FrankP Posted January 14, 2022 Report Posted January 14, 2022 Despite my effort to follow the direction, config.ini on cluster setup is not being applied when kazoo-applications is starting... The console log shows that it's setting the zone to local despite the matching host exists under kazoo_apps section... and "no local zone configured, adding default AMQP" happens.. Any thoughts on why none of the values are being applied from config.ini even though it is being loaded? 16:42:39.990 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application lager started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:39.990 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application inet_cidr started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:39.990 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application erlang_localtime started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:39.990 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application jiffy started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:39.990 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application kazoo_stdlib started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:39.990 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application rfc3339 started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:39.990 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application jsx started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:39.990 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application jesse started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:39.991 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application kazoo_schemas started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:39.991 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application recon started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:39.991 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application kazoo started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:39.991 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application proper started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:40.001 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application gun started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:40.011 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application ranch started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:40.021 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application cowboy started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:40.021 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application kazoo_proper started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:40.021 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application zucchini started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:40.035 [warning] |0000000000|kazoo_config_init:48(<0.180.0>) file /etc/kazoo/config.ini does not exist or is not accessible 16:42:40.039 [info] |0000000000|kazoo_config_init:44(<0.180.0>) loaded configs from file /etc/kazoo/core/config.ini 16:42:40.040 [notice] |0000000000|kazoo_config_init:59(<0.180.0>) loaded settings : [{zone,[{name,nyc3},{amqp_uri,"amqp://guest:guest@rabbit1.nyc3.domain.test"}]},{bigcouch,[{compact_automatically,true},{cookie,change_me},{ip,"127.0.0.1"},{port,15984},{admin_port,15986}]},{kazoo_apps,[{cookie,change_me},{zone,nyc3},{host,"kz1.nyc3.domain.test"}]},{ecallmgr,[{cookie,change_me},{zone,nyc3},{host,"kz1.nyc3.domain.test"}]},{log,[{syslog,info},{console,notice},{file,error}]}] 16:42:40.053 [notice] |0000000000|kazoo_config_init:66(<0.180.0>) setting zone to local 16:42:40.053 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application kazoo_config started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:40.053 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application kazoo_etsmgr started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:41.090 [info] |0000000000|kazoo_bindings_init:18(<0.187.0>) kazoo bindings is ready 16:42:41.090 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application kazoo_bindings started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:41.091 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application ranch_proxy_protocol started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:41.091 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application rabbit_common started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:41.117 [info] |0000000000|Undefined:Undefined(<0.31.0>) Application amqp_client started on node 'kazoo_apps@kz1.nyc3.domain.test' 16:42:41.161 [info] |00000000000|kz_amqp_bootstrap:148(<0.205.0>) no local zone configured, adding default AMQP 16:42:41.192 [info] |00000000000|kz_amqp_bootstrap:54(<0.205.0>) waiting for first amqp connection... 16:42:41.320 [warning] |00000000000|kz_amqp_connection:385(<0.206.0>) connection refused to amqp://guest:guest@localhost:5672 (check that the broker is running) 16:42:41.482 [info] |00000000000|kz_amqp_connection:278(<0.207.0>) successfully connected to 'amqp://guest:guest@rabbit1.nyc3.domain.test' 16:42:41.522 [warning] |00000000000|kz_amqp_connection:385(<0.206.0>) connection refused to amqp://guest:guest@localhost:5672 (check that the broker is running) 16:42:41.924 [warning] |00000000000|kz_amqp_connection:385(<0.206.0>) connection refused to amqp://guest:guest@localhost:5672 (check that the broker is running) 16:42:42.726 [warning] |00000000000|kz_amqp_connection:385(<0.206.0>) connection refused to amqp://guest:guest@localhost:5672 (check that the broker is running) 16:42:44.350 [warning] |00000000000|kz_amqp_connection:385(<0.206.0>) connection refused to amqp://guest:guest@localhost:5672 (check that the broker is running) config.ini below for reference... domains are FQDN, it's simply replaced with domain.test. ; section are between [] = [section] ; key = value ; to comment add ";" in front of the line ;[amqp] ;uri = "amqp://guest:guest@127.0.0.1:5672" [zone] name = "nyc3" amqp_uri = "amqp://guest:guest@rabbit1.nyc3.domain.test" [bigcouch] compact_automatically = true cookie = change_me ip = "127.0.0.1" port = 15984 admin_port = 15986 [kazoo_apps] cookie = change_me zone = "nyc3" host = "kz1.nyc3.domain.test" [ecallmgr] cookie = change_me zone = "nyc3" host = "kz1.nyc3.domain.test" [log] syslog = info console = notice file = error
fmateo05 Posted January 14, 2022 Report Posted January 14, 2022 That is related to dns resolution on the node. Check /etc/hosts and exec hostname (also hostname -f)
FrankP Posted January 14, 2022 Author Report Posted January 14, 2022 (edited) /etc/hosts and hostname has a matching host value in config.ini, and the hostname (i.e. kz1.nyc3.domain.test) resolves to itself. Anything else I can troubleshoot and/or correct? Thanks in advance. Edited January 14, 2022 by FrankP (see edit history)
FrankP Posted January 14, 2022 Author Report Posted January 14, 2022 I don't know if this is a bug or expected behavior, but I think I found it. config.ini host value has to match the hostname -s (short hostname) and not hostname -f (FQDN). When I changed the host=kz1 instead of host=kz1.nyc3.domain.test, it recognized it. Is this the right behavior? Upon troubleshooting further... This is what I found.. Perhaps this is not a bug... Looks like the host value in config has to resolve to 127.0.0.1 in /etc/hosts instead of its public IP address. I know I had this in my build script, but at some point manually removed this line from /etc/hosts since it is resolving to its public IP address.. I guess this was the crucial mistake on my part. Hopefully someone else finds this useful in the future.
Recommended Posts