Forums

Could be a ulimit issue? Check those settings, make sure you have enough file descriptors...

@mc_ I got issue again, and here is my bigcouch log entry [Mon, 11 Nov 2019 14:10:43 GMT] [info] [<0.2254.0>] [ecf5d584] undefined - - 'GET' /faxes/_design/faxes/_view/schedule_accounts?group_level=1&group=true&reduce=true 500 [Mon, 11 Nov 2019 14:10:43 GMT] [info] [<0.2254.0>] [ecf5d584] 45.76.23.242 127.0.0.1:15984 GET /faxes/_design/faxes/_view/schedule_accounts?group_level=1&group=true&reduce=true 500 ok 1 [Mon, 11 Nov 2019 14:10:43 GMT] [info] [<0.285.0>] [c98bdde3] 52.40.138.157 144.202.79.50:5984 GET /_utils/database.html?accounts 304 ok 1 [Mon, 11 Nov 2019 14:10:43 GMT] [info] [<0.2263.0>] [7fb8cfb1] undefined - - 'GET' / 200 [Mon, 11 Nov 2019 14:10:43 GMT] [info] [<0.2263.0>] [7fb8cfb1] 45.76.23.242 undefined GET / 200 ok 1 [Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.280.0>] [331f2827] undefined - - 'GET' /_session 200 [Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.280.0>] [331f2827] 52.40.138.157 144.202.79.50:5984 GET /_session 200 ok 1 [Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.281.0>] [4eef4492] undefined - - 'GET' /_config/query_servers/ 200 [Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.281.0>] [4eef4492] 52.40.138.157 144.202.79.50:5984 GET /_config/query_servers/ 200 ok 0 [Mon, 11 Nov 2019 14:10:44 GMT] [error] [<0.282.0>] [c675e7a5] Uncaught error in HTTP request: {error,                                                                 {internal_server_error,                                                                  "No DB shards could be opened."}} [Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.282.0>] [c675e7a5] Stacktrace: [{fabric_util,get_shard,3,                                             [{file,"src/fabric_util.erl"},                                              {line,67}]},                                            {fabric,get_security,2,                                             [{file,"src/fabric.erl"},                                              {line,138}]},                                            {chttpd_db,do_db_req,2,                                             [{file,"src/chttpd_db.erl"},                                              {line,198}]},                                            {chttpd,handle_request,1,                                             [{file,"src/chttpd.erl"},                                              {line,198}]},                                            {mochiweb_http,headers,5,                                             [{file,"src/mochiweb_http.erl"},                                              {line,126}]},                                            {proc_lib,init_p_do_apply,3,                                             [{file,"proc_lib.erl"},                                              {line,227}]}] [Mon, 11 Nov 2019 14:10:44 GMT] [error] [<0.285.0>] [1d214b0c] Uncaught error in HTTP request: {error,                                                                 {internal_server_error,                                                                  "No DB shards could be opened."}} [Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.282.0>] [c675e7a5] undefined - - 'GET' /accounts/_all_docs?limit=11 500 [Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.285.0>] [1d214b0c] Stacktrace: [{fabric_util,get_shard,3,                                             [{file,"src/fabric_util.erl"},                                              {line,67}]},                                            {fabric,get_security,2,                                             [{file,"src/fabric.erl"},                                              {line,138}]},                                            {chttpd_db,do_db_req,2,                                             [{file,"src/chttpd_db.erl"},                                              {line,198}]},                                            {chttpd,handle_request,1,                                             [{file,"src/chttpd.erl"},                                              {line,198}]},                                            {mochiweb_http,headers,5,                                             [{file,"src/mochiweb_http.erl"},                                              {line,126}]},                                            {proc_lib,init_p_do_apply,3,                                             [{file,"proc_lib.erl"},                                              {line,227}]}]     Regards Abbasi

@mc_ Thanks lot for your time, i have checked that there were no permission issues with the dbs files(directories and subdirectories), I can see that issue does not happen since last Friday(1st-Nov-19). If that issue gets in again, i will surely look into bigcouch logs and get back to you.   Regards Abbasi

Possibly permissions on the database files themselves (I think they're under /srv/dbs ?). Try accessing a single bigcouch node and see what its logging. Also, n=4 (or any even number) is problematic in a network split. It is generally good to have n be odd to make it more likely to have a majority on one side of the split.

Hi Team, I have successfully setup kazoo-bigcouch cluster, and everything is working good except that after some time i get some shard error, and all of the cluster stops working. example: sup crossbar_maintenance create_account admin sip.mogility.cloud admin Lmkt@ptcl1234
Command failed: {'EXIT',{{badmatch,{error,<<"No DB shards could be opened.">>}},[{kz_json_schema,default_object,1,[{file,"src/kz_json_schema.erl"},{line,997}]},{kzd_accounts,new,0,[{file,"src/kzd_accounts.erl"},{line,120}]},{crossbar_maintenance,create_account,4,[{file,"src/crossbar_maintenance.erl"},{line,394}]},{sup,in_kazoo,4,[{file,"src/sup.erl"},{line,98}]},{rpc,'-handle_call_call/6-fun-0-',5,[{file,"rpc.erl"},{line,197}]}]}} It goes away when i restart every database nodes, here are my database members(4 members of couch database). [root@db1zone1 ~]# curl 127.0.0.1:5984/_membership   {"all_nodes":["bigcouch@db1zone1.xxxx.cloud","bigcouch@db1zone2.xxxx.cloud","bigcouch@db2zone1.xxxx.cloud","bigcouch@db2zone2.xxxx.cloud"],"cluster_nodes":["bigcouch@db1zone1.xxxx.cloud","bigcouch@db1zone2.xxxx.cloud","bigcouch@db2zone1.xxxx.cloud","bigcouch@db2zone2.xxxxx.cloud"]} And here are my cluster configurations in local.ini file   [cluster] q=1 r=3 w=3 n=4   Can you please guid me whats the issue here.   Regards Abbasi