Raise log level of a specific module

[Eduardo Almeida]

Hello all,

 

Introduction

First, I just started working with the Kazoo/Kamailio/FreeSWITCH stack and recently started learning some Erlang for the greater good; please be patient.
 

The company I work at uses an old version of Kazoo (3.18), and they have created a few custom modules (e.g., conference) in the past that are working to this day. Unfortunately, all developers with experience with this stack - and programmed these modules - are no longer around. Due to the growth the company has experienced, the number of clients rose considerably, and now our system is having troubles during peak hours (i.e., conferencing, among other things, does not work).

 

Question

I saw that the log library used by Kazoo in this version is lager; since in production the default log level is ERROR, I would like to raise the log level to DEBUG for the specific modules used in conferences. The default log backend used in our application is the console.
 

What was tried

Lager provides a way to trace a specific module with the desired log level, but every time I try it, everything else seems to stop. This is how I tried:

 

echo '{ok, Trace} = lager:trace_file("log/conference.log", [{module, conference}], debug), Trace.' | ./erl_call -e -c XXX -n whistle_apps -h command_node

 

I followed the documentation of erl_call of the exact version present in the system. Once I run this, all the apps stop responding, and the only way to get things working again is to restart the whole process. Using lager:clear_all_traces and lager:stop_trace do not help.
 

Finally

I just started, so I am full of questions. Am I going in the wrong direction? Can I do what I am trying to do? Does Kazoo provide another way of doing this? If anyone can offer a little help, it would be fantastic.

 

Best regards,

Eduardo Almeida.

[mc_]

@Eduardo Almeida The "easier" option might be to set the syslog logging level to debug for a time, capturing the logs you want, then using tools like grep to extract the log lines you're interested in.

In general log lines will contain the call-id (or API request-id) and the erlang module / line number in the source file where the log line was called, followed by the message.

You should be able to "attach" to the running Erlang VM to run your trace command - my guess off hand is the trace is tied to the PID you start the trace with and, when your erl_call is done, that PID (and trace) die. Why the node is unresponsive after that I don't know.

Aside - 3.18 last got attention in 2015. You might want to encourage time/money to be spent upgrading (or investigating hosting with us) as the scalability and feature set of latest KAZOO versions (4.3 currently for community members, 5.0 for paid members) are leaps and bounds better than the 3.x series. If you are comfortable talking about the custom code and its goals, we could also see if you can use stock KAZOO now as well.

Welcome to the land of Erlang; its the best koolaid of any language out there! (probably)

[Eduardo Almeida]

Thank you for the comment @mc_,

Unfortunately, it is unfeasible to raise the log level for the entire Kazoo. When the problem happens - at busy hours - there are just to many calls going through. So I went further with the idea of tracing individual modules.

Kazoo Update

We will update our Kazoo version in the near future, but we first need to solve these pressing issues so we won't lose clients. Once we stabilize our environment we are going to prepare a migration plan.

Tracing individual modules

Fortunately, I was able to enable tracing for specific modules. I know that the Kazoo version we use is old, and newer ones probably don't even use the same log library (Lager), but I will explain here how I did it for future reference.

I created a Erlang module to run as a service. This server awaits for messages containing the module names, and upon receiving them, start traces for them. Follows the simplified version:

run(State)->
  receive
    {status} ->
      io:fwrite("Lager status: ~p~n", [lager:status()]),
      run(State);

    {start_trace, File, Module} ->
      {ok, Trace} = lager:trace_file(File, [{module, M}], debug),
      run([{module, Module, trace, Trace} | State]);

    {stop_all_traces} ->
      lager:clear_all_traces(),
      run([]);
  end.

start() ->
  register(tracemodulectl, spawn(?MODULE, run, [[]])).

After loading this module in the Erlang VM I can use the 'erl_call' to start and stop traces:

#!/bin/bash

echo 'tracemodulectl:start().' | ./erl_call -e -c <cookie> -n <node_name> -h command_node

echo 'tracemodulectl ! {start_trace, "/var/log/conference.log", conference}.' | ./erl_call -e -c <cookie> -n <node_name> -h command_node
echo 'tracemodulectl ! {status}.' | ./erl_call -e -c <cookie> -n <node_name> -h command_node

sleep 60

echo 'tracemodulectl ! {stop_all_traces}.' | ./erl_call -e -c <cookie> -n <node_name> -h command_node
echo 'tracemodulectl ! {status}.' | ./erl_call -e -c <cookie> -n <node_name> -h command_node

And this how I did it.
 

P.s.: Is there a way to close this topic?

Best regards,
Eduardo Almeida.

[krzykat]

On 9/13/2022 at 6:05 PM, mc_ said:

@Eduardo Almeida

 latest KAZOO versions (4.3 currently for community members, 5.0 for paid members) 

I thought 5.0 was going to be release on github ??  Is that no longer going to happen?  I was told this almost 2 years ago.  It's hard to contribute when you aren't given access.

[mc_]

@Eduardo Almeidacircling back, great that you got into some Erlang code!

Future versions of KAZOO have this tracing functionality added too.

[krzykat]

@mc_  Do you have ANY timeline on when this project will go to github so that we can work on some modules that we have planned for it?

Thanks.

 

[mc_]

There are other threads where this is discussed; let's not hijack this one.