Jump to content
KAZOOcon: hackathon signup and details here! ×

Integration between Kafka and Kazoo via Kafka connector for Couchdb


Siggypony

Recommended Posts

Hello,

I was wondering, would using a Kafka connect source to stream changes to account documents in Couchdb be likely to cause issues for Kazoo?  I intend on try this soon with a basic install soon but was wondering if bypassing things like crossbar, amqp etc and monitoring changes in Couchdb directly in a readonly manner would be likely to cause issues?

Looking to use, https://github.com/xebia/kafka-connect-couchdb but will only be sourcing data out of couchdb, not trying to sink into it.  I think any updates to Kazoo should be made though the appropriate methods such as over crossbar etc. 

I know there is notifications for changes to account docs over AMQP but they lack the details of what specifically has changed requiring a call to crossbar to get that account doc, but with uncertainty over what has actually changed.  So the idea would be to bypass this process and stream changes from couchdb directly to a Topic in Kafka for a stream processor to determine whats changed and then make updates to topic for sink connectors to effect other systems appropriately.

Basically am wondering if there is anything about Kazoo which result in this being a no go? Or is there a better approach for determining changes to docs without putting an unnecessarily high load into Kazoo / Crossbar etc

Link to comment
Share on other sites

  • 4 weeks later...

Hi.

If it's done with replication, you may experience high load on CouchDB itself. Very likely you're going to have a big number of databases and lots of running replication tasks.

And clearly, if a DB struggles - all other components are going to have problems too.

This question is not that simple, actually.  But you can try ))

Link to comment
Share on other sites

×
×
  • Create New...