Jump to content
  • 0
BilalAbbasi

[Bigcouch Shard Error]

Question

Hi Team,

I have successfully setup kazoo-bigcouch cluster, and everything is working good except that after some time i get some shard error, and all of the cluster stops working.

example:

sup crossbar_maintenance create_account admin sip.mogility.cloud admin Lmkt@ptcl1234
Command failed: {'EXIT',{{badmatch,{error,<<"No DB shards could be opened.">>}},[{kz_json_schema,default_object,1,[{file,"src/kz_json_schema.erl"},{line,997}]},{kzd_accounts,new,0,[{file,"src/kzd_accounts.erl"},{line,120}]},{crossbar_maintenance,create_account,4,[{file,"src/crossbar_maintenance.erl"},{line,394}]},{sup,in_kazoo,4,[{file,"src/sup.erl"},{line,98}]},{rpc,'-handle_call_call/6-fun-0-',5,[{file,"rpc.erl"},{line,197}]}]}}

It goes away when i restart every database nodes, here are my database members(4 members of couch database).

[root@db1zone1 ~]# curl 127.0.0.1:5984/_membership  

{"all_nodes":["bigcouch@db1zone1.xxxx.cloud","bigcouch@db1zone2.xxxx.cloud","bigcouch@db2zone1.xxxx.cloud","bigcouch@db2zone2.xxxx.cloud"],"cluster_nodes":["bigcouch@db1zone1.xxxx.cloud","bigcouch@db1zone2.xxxx.cloud","bigcouch@db2zone1.xxxx.cloud","bigcouch@db2zone2.xxxxx.cloud"]}

And here are my cluster configurations in local.ini file

 

[cluster]

q=1

r=3

w=3

n=4

 

Can you please guid me whats the issue here.

 

Regards

Abbasi

 

Share this post


Link to post
Share on other sites

4 answers to this question

Recommended Posts

  • 0

Possibly permissions on the database files themselves (I think they're under /srv/dbs ?). Try accessing a single bigcouch node and see what its logging.

Also, n=4 (or any even number) is problematic in a network split. It is generally good to have n be odd to make it more likely to have a majority on one side of the split.

Share this post


Link to post
Share on other sites
  • 0

@mc_

Thanks lot for your time, i have checked that there were no permission issues with the dbs files(directories and subdirectories), I can see that issue does not happen since last Friday(1st-Nov-19).

If that issue gets in again, i will surely look into bigcouch logs and get back to you.

 

Regards

Abbasi

Share this post


Link to post
Share on other sites
  • 0

@mc_

I got issue again, and here is my bigcouch log entry

[Mon, 11 Nov 2019 14:10:43 GMT] [info] [<0.2254.0>] [ecf5d584] undefined - - 'GET' /faxes/_design/faxes/_view/schedule_accounts?group_level=1&group=true&reduce=true 500

[Mon, 11 Nov 2019 14:10:43 GMT] [info] [<0.2254.0>] [ecf5d584] 45.76.23.242 127.0.0.1:15984 GET /faxes/_design/faxes/_view/schedule_accounts?group_level=1&group=true&reduce=true 500 ok 1

[Mon, 11 Nov 2019 14:10:43 GMT] [info] [<0.285.0>] [c98bdde3] 52.40.138.157 144.202.79.50:5984 GET /_utils/database.html?accounts 304 ok 1

[Mon, 11 Nov 2019 14:10:43 GMT] [info] [<0.2263.0>] [7fb8cfb1] undefined - - 'GET' / 200

[Mon, 11 Nov 2019 14:10:43 GMT] [info] [<0.2263.0>] [7fb8cfb1] 45.76.23.242 undefined GET / 200 ok 1

[Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.280.0>] [331f2827] undefined - - 'GET' /_session 200

[Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.280.0>] [331f2827] 52.40.138.157 144.202.79.50:5984 GET /_session 200 ok 1

[Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.281.0>] [4eef4492] undefined - - 'GET' /_config/query_servers/ 200

[Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.281.0>] [4eef4492] 52.40.138.157 144.202.79.50:5984 GET /_config/query_servers/ 200 ok 0

[Mon, 11 Nov 2019 14:10:44 GMT] [error] [<0.282.0>] [c675e7a5] Uncaught error in HTTP request: {error,

                                                                {internal_server_error,

                                                                 "No DB shards could be opened."}}

[Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.282.0>] [c675e7a5] Stacktrace: [{fabric_util,get_shard,3,

                                            [{file,"src/fabric_util.erl"},

                                             {line,67}]},

                                           {fabric,get_security,2,

                                            [{file,"src/fabric.erl"},

                                             {line,138}]},

                                           {chttpd_db,do_db_req,2,

                                            [{file,"src/chttpd_db.erl"},

                                             {line,198}]},

                                           {chttpd,handle_request,1,

                                            [{file,"src/chttpd.erl"},

                                             {line,198}]},

                                           {mochiweb_http,headers,5,

                                            [{file,"src/mochiweb_http.erl"},

                                             {line,126}]},

                                           {proc_lib,init_p_do_apply,3,

                                            [{file,"proc_lib.erl"},

                                             {line,227}]}]

[Mon, 11 Nov 2019 14:10:44 GMT] [error] [<0.285.0>] [1d214b0c] Uncaught error in HTTP request: {error,

                                                                {internal_server_error,

                                                                 "No DB shards could be opened."}}

[Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.282.0>] [c675e7a5] undefined - - 'GET' /accounts/_all_docs?limit=11 500

[Mon, 11 Nov 2019 14:10:44 GMT] [info] [<0.285.0>] [1d214b0c] Stacktrace: [{fabric_util,get_shard,3,

                                            [{file,"src/fabric_util.erl"},

                                             {line,67}]},

                                           {fabric,get_security,2,

                                            [{file,"src/fabric.erl"},

                                             {line,138}]},

                                           {chttpd_db,do_db_req,2,

                                            [{file,"src/chttpd_db.erl"},

                                             {line,198}]},

                                           {chttpd,handle_request,1,

                                            [{file,"src/chttpd.erl"},

                                             {line,198}]},

                                           {mochiweb_http,headers,5,

                                            [{file,"src/mochiweb_http.erl"},

                                             {line,126}]},

                                           {proc_lib,init_p_do_apply,3,

                                            [{file,"proc_lib.erl"},

                                             {line,227}]}]

 

 

Regards

Abbasi

Share this post


Link to post
Share on other sites
  • 0

Could be a ulimit issue? Check those settings, make sure you have enough file descriptors...

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×