copy of couchdb every first of the month

[Riccardo]

I saw that every first of the month,  kazzo tells couchdb to create a new databases for all the accounts and rename the old one like YEARMONTH (like account_XXXXXXXXXXXXXX_202202).

This starts at midnight and lasts for over 8 hours...in mean time the kazoo UI is not available and the sip calls neither.

I need to restar the dbs machines for getting back available.

What I have to do to disable this function?

Thanks

[mc_]

These are MODBs: https://docs.2600hz.com/dev/core/kazoo_modb/doc/

They are critical to KAZOO usage, so there's no disabling of them. If the MODB doesn't exist when a document (like a CDR) needs to be saved, the MODB will be created. If you have a lot of activity, they'll all stack up waiting for the MODB to be created.

What you can do is configure the MODB task (https://docs.2600hz.com/dev/applications/tasks/doc/modb_creation/)

Set that to pre-create the next month's MODB for all accounts and give it enough lead time (say 3-5 days depending on how many accounts you have); the task will automatically spread out the MODB creations across that time period to hopefully minimize impact on the DB cluster.

[Riccardo]

thanks for the answer

the routines key is empty. I think I have to create one to fit my problem

Edited March 28, 2022 by Riccardo (see edit history)

[Riccardo]

Hi, I cannot resolve the problem.

I try to configure the  MODBs to pre-create the next month MODB 5 days before. But this is not the problem, the creation process does not hang the couchdb...the problem is at the midnight of the new month. The beam.smp  process stay to 100% cpu and 1.0 load and all the platform did not respond (in and out telephone calls does not work). The only way to restore is to restart the couchdb service.

I have the same infrastructure in test and I saw that this whole process least 16 hours or so.

In the log I have plenty of this

[error] 2022-06-01T00:00:44.974482Z couchdb@db01.viavoip.it <0.27791.1982> b0d91beda3 rexi_server: from: couchdb@db02.viavoip.it(<14389.1393.2123>) mfa: fabric_rpc:reduce_view/4 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,reduce_view,5,[{file,"src/fabric_rpc.erl"},{line,162}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:00:44.974657Z couchdb@db01.viavoip.it <0.30663.1983> b0d91beda3 rexi_server: from: couchdb@db02.viavoip.it(<14389.1393.2123>) mfa: fabric_rpc:reduce_view/4 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,reduce_view,5,[{file,"src/fabric_rpc.erl"},{line,162}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:00:44.974689Z couchdb@db01.viavoip.it <0.28616.2008> b0d91beda3 rexi_server: from: couchdb@db02.viavoip.it(<14389.1393.2123>) mfa: fabric_rpc:reduce_view/4 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,reduce_view,5,[{file,"src/fabric_rpc.erl"},{line,162}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]

[error] 2022-06-01T00:00:59.544608Z couchdb@db01.viavoip.it <0.7070.1991> -------- rexi_server: from: couchdb@db01.viavoip.it(<0.29884.1995>) mfa: fabric_rpc:open_revs/4 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,with_db,3,[{file,"src/fabric_rpc.erl"},{line,331}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:00:59.545507Z couchdb@db01.viavoip.it <0.2624.2003> -------- rexi_server: from: couchdb@db02.viavoip.it(<14389.23319.2071>) mfa: fabric_rpc:open_revs/4 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,with_db,3,[{file,"src/fabric_rpc.erl"},{line,331}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:00:59.556963Z couchdb@db01.viavoip.it <0.7615.2023> -------- rexi_server: from: couchdb@db02.viavoip.it(<14389.31395.1807>) mfa: fabric_rpc:update_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,read_repair_filter,4,[{file,"src/fabric_rpc.erl"},{line,349}]},{fabric_rpc,update_docs,3,[{file,"src/fabric_rpc.erl"},{line,274}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:00:59.557601Z couchdb@db01.viavoip.it <0.23392.1992> -------- rexi_server: from: couchdb@db01.viavoip.it(<0.22397.2019>) mfa: fabric_rpc:update_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,read_repair_filter,4,[{file,"src/fabric_rpc.erl"},{line,349}]},{fabric_rpc,update_docs,3,[{file,"src/fabric_rpc.erl"},{line,274}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]

[error] 2022-06-01T00:01:19.553779Z couchdb@db01.viavoip.it <0.22227.2025> 5d0dde7442 rexi_server: from: couchdb@db02.viavoip.it(<14389.15892.2116>) mfa: fabric_rpc:reduce_view/4 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,reduce_view,5,[{file,"src/fabric_rpc.erl"},{line,162}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:19.554185Z couchdb@db01.viavoip.it <0.16384.1978> 5d0dde7442 rexi_server: from: couchdb@db02.viavoip.it(<14389.15892.2116>) mfa: fabric_rpc:reduce_view/4 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,reduce_view,5,[{file,"src/fabric_rpc.erl"},{line,162}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:19.554212Z couchdb@db01.viavoip.it <0.31432.2016> 5d0dde7442 rexi_server: from: couchdb@db02.viavoip.it(<14389.15892.2116>) mfa: fabric_rpc:reduce_view/4 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,reduce_view,5,[{file,"src/fabric_rpc.erl"},{line,162}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:19.554603Z couchdb@db01.viavoip.it <0.26844.2029> 5d0dde7442 rexi_server: from: couchdb@db02.viavoip.it(<14389.15892.2116>) mfa: fabric_rpc:reduce_view/4 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,reduce_view,5,[{file,"src/fabric_rpc.erl"},{line,162}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:19.554683Z couchdb@db01.viavoip.it <0.15894.2034> 5d0dde7442 rexi_server: from: couchdb@db02.viavoip.it(<14389.15892.2116>) mfa: fabric_rpc:reduce_view/4 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,reduce_view,5,[{file,"src/fabric_rpc.erl"},{line,162}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]

[error] 2022-06-01T00:01:26.404812Z couchdb@db01.viavoip.it <0.526.2025> 1ffb8f975c rexi_server: from: couchdb@db03.viavoip.it(<14390.24218.2109>) mfa: fabric_rpc:reduce_view/4 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,reduce_view,5,[{file,"src/fabric_rpc.erl"},{line,162}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]

[error] 2022-06-01T00:01:29.758155Z couchdb@db01.viavoip.it <0.11584.2007> -------- rexi_server: from: couchdb@db02.viavoip.it(<14389.30415.2124>) mfa: fabric_rpc:all_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,124}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:29.758375Z couchdb@db01.viavoip.it <0.29088.1989> -------- rexi_server: from: couchdb@db02.viavoip.it(<14389.30415.2124>) mfa: fabric_rpc:all_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,124}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:29.758505Z couchdb@db01.viavoip.it <0.20117.2006> -------- rexi_server: from: couchdb@db02.viavoip.it(<14389.30415.2124>) mfa: fabric_rpc:all_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,124}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:29.758566Z couchdb@db01.viavoip.it <0.16052.1976> -------- rexi_server: from: couchdb@db02.viavoip.it(<14389.30415.2124>) mfa: fabric_rpc:all_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,124}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[info] 2022-06-01T00:01:29.767272Z couchdb@db01.viavoip.it <0.16323.2006> -------- Index update finished for db: shards/20000000-3fffffff/account/c7/b0/5a6b6f3f0310f58a2be6c07a506b-202206.1653973252 idx: _design/ledgers
[error] 2022-06-01T00:01:29.769811Z couchdb@db01.viavoip.it <0.22301.2013> -------- rexi_server: from: couchdb@db01.viavoip.it(<0.30092.2043>) mfa: fabric_rpc:all_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,124}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:29.769827Z couchdb@db01.viavoip.it <0.10687.1997> -------- rexi_server: from: couchdb@db01.viavoip.it(<0.30092.2043>) mfa: fabric_rpc:all_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,124}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:29.769969Z couchdb@db01.viavoip.it <0.31001.1997> -------- rexi_server: from: couchdb@db01.viavoip.it(<0.30092.2043>) mfa: fabric_rpc:all_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,124}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:29.770063Z couchdb@db01.viavoip.it <0.25544.2021> -------- rexi_server: from: couchdb@db01.viavoip.it(<0.30092.2043>) mfa: fabric_rpc:all_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,124}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:29.770163Z couchdb@db01.viavoip.it <0.4392.1992> -------- rexi_server: from: couchdb@db01.viavoip.it(<0.30092.2043>) mfa: fabric_rpc:all_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,124}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:29.770238Z couchdb@db01.viavoip.it <0.27073.1922> -------- rexi_server: from: couchdb@db01.viavoip.it(<0.30092.2043>) mfa: fabric_rpc:all_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,124}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2022-06-01T00:01:29.770329Z couchdb@db01.viavoip.it <0.8736.2015> -------- rexi_server: from: couchdb@db01.viavoip.it(<0.30092.2043>) mfa: fabric_rpc:all_docs/3 error:function_clause [{couch_db,incref,[undefined],[{file,"src/couch_db.erl"},{line,185}]},{couch_server,open,2,[{file,"src/couch_server.erl"},{line,85}]},{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,124}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]

[error] 2022-06-01T00:02:51.542456Z couchdb@db01.viavoip.it <0.7426.1942> -------- fabric_worker_timeout open_revs,'couchdb@db02.viavoip.it',<<"shards/20000000-3fffffff/account/6d/14/3ff92037072529ce0f4113dbd43d.1615819907">>
[error] 2022-06-01T00:02:51.542589Z couchdb@db01.viavoip.it <0.7426.1942> -------- fabric_worker_timeout open_revs,'couchdb@db03.viavoip.it',<<"shards/20000000-3fffffff/account/6d/14/3ff92037072529ce0f4113dbd43d.1615819907">>
[error] 2022-06-01T00:02:51.543264Z couchdb@db01.viavoip.it <0.31416.2000> -------- fabric_worker_timeout open_doc,'couchdb@db02.viavoip.it',<<"shards/20000000-3fffffff/account/6d/14/3ff92037072529ce0f4113dbd43d.1615819907">>
[error] 2022-06-01T00:02:51.543366Z couchdb@db01.viavoip.it <0.31416.2000> -------- fabric_worker_timeout open_doc,'couchdb@db03.viavoip.it',<<"shards/20000000-3fffffff/account/6d/14/3ff92037072529ce0f4113dbd43d.1615819907">>
[error] 2022-06-01T00:02:51.673274Z couchdb@db01.viavoip.it <0.5017.1971> -------- fabric_worker_timeout open_doc,'couchdb@db02.viavoip.it',<<"shards/20000000-3fffffff/account/6d/48/34fdb9f708eec564417dff81f04b-202205.1651222714">>

 

[mc_]

What is KAZOO doing at this time? There are routines that run monthly for updating account balances and things like that; could also be that your server is underpowered