Jump to content
Siggypony

Integration between Kafka and Kazoo via Kafka connector for Couchdb

Recommended Posts

Hello,

I was wondering, would using a Kafka connect source to stream changes to account documents in Couchdb be likely to cause issues for Kazoo?  I intend on try this soon with a basic install soon but was wondering if bypassing things like crossbar, amqp etc and monitoring changes in Couchdb directly in a readonly manner would be likely to cause issues?

Looking to use, https://github.com/xebia/kafka-connect-couchdb but will only be sourcing data out of couchdb, not trying to sink into it.  I think any updates to Kazoo should be made though the appropriate methods such as over crossbar etc. 

I know there is notifications for changes to account docs over AMQP but they lack the details of what specifically has changed requiring a call to crossbar to get that account doc, but with uncertainty over what has actually changed.  So the idea would be to bypass this process and stream changes from couchdb directly to a Topic in Kafka for a stream processor to determine whats changed and then make updates to topic for sink connectors to effect other systems appropriately.

Basically am wondering if there is anything about Kazoo which result in this being a no go? Or is there a better approach for determining changes to docs without putting an unnecessarily high load into Kazoo / Crossbar etc

  • Upvote 1

Share this post


Link to post
Share on other sites

Hi.

If it's done with replication, you may experience high load on CouchDB itself. Very likely you're going to have a big number of databases and lots of running replication tasks.

And clearly, if a DB struggles - all other components are going to have problems too.

This question is not that simple, actually.  But you can try ))

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...