Rick Guyton

Hello all! So, I thought I'd share some progress I've made on an app that connects to 2600hz websockets, listens for and saves events that come across. Then stores those files on backblaze for later processing. Today, I can only give you the 10,000 foot view on how it works and basic instructions on getting it going. I'll be adding another agent later that'll commit the events to a MySQL DB and do some processing on them to make them user readable. If you want to join in the fun though, you might want to get this agent up and running now so you can start gathering the data. Repo: https://github.com/wildernesstechie/socketsink 10,000 foot view: 2 VPS machines run sink.py - These agents connect to your 2600hz cluster and record the events coming off of it to memory. Then, after 30 minutes the agent terminates, saving the events to a gzipped file and uploads them to backblaze 1 VPS running dbupload.py - This agent collects files from the unprocessed folder on backblaze, and commits them to a MySql database. A series of SQL statements are run to transform the raw events into more friendly data like hold time, park time, ringing time, missed, answered, ect. 1 VPS running a WebGUI - You will be on your own for this one. I'm using Databik for this. But, it's not free or open source, so I can't give it out to you all. Basic Install Insructions: Install python3.6+ clone the repo setup a venv pip the required packages (websocket-client, kazoo-sdk, b2sdk, mysql-connector-python) Get yourself a backblaze b2 account and setup API keys for it go into sink_settings.py and put your backblaze API key and id in name your unprocessed folder (it can be anything) put your account ID where it says 'KazooAccountIdHere' and your API key where it days 'ApiKeyHere' (sorry, only API keys work ATM) If you aren't on hosted, you'll need to update your api base and webhook base URLs. change to another ca bundle if you'd like. The setup cron to run the agent every 0 and 30 minutes Rinse, repeat on a second VPS except cron every 15 and 45 minutes. Sit back and watch the data flow into your backblaze! I'll be back soon with an agent that imports to a DB. (It's already done, just needs some cleanup) Merry Gift Giving Holiday Of Your Choosing and New Year!!!

So, the device naming is very strange. Is there any way to control that? Everything else is AMAZING. So easy to onboard now. BTW, for anyone else using this, you can avoid having to confirm account rate increases by including the following JSON in your device/user. Obviously KNOW what your charges are and be ok with them auto-adding... You can also add caller ID info this way. Or add custom hold music on a per device level. Really anything. So nice... { "accept_charges": "true" }

Few things, FYI Macs and local IPs for phone are available in the debugging app. No need to broadcast scan and ARP. I use this all the time to access phone WebUI while remoting into a client PC. BTW simple-help.com is a great cost effective self hosted remote access service. Also, provisioner can reboot phones easily. The real issue with shipping routers is you have to be 100% right or you are screwed. Also, sometimes you need to setup PPPoE bridging to avoid double NAT. It’s a mess, if you are going to offer a router you need someone onsite.

A lot of what @Karl Stallknecht is saying is very true. At the end of the day you need three things. A good LAN, solid connectivity and sufficient bandwidth. A Good LAN: Well, this is 50% of the battle TBH. At the end of the day if the LAN sucks, so will the phones. Grab a VLAN if you can get it. (remarkably hard with many IT people for some reason) But, more than anything you need good IT. If you are targeting SMBs that means finding local IT folks or doing the IT for them. (I think Karl does his customer's IT) Good IT folks will usually be comfrotable with Cisco (not Meraki, but Cisco IOS), mikrotik, palo alto or juniper equipment. All they REALLY need to know is basics like jitter, packet loss, latency, and VLANing. But, most people who are proficient in this stuff aren't using netgear. Solid Connectivity: We use PingPlotter extensivly for this. I like to run continuous pings to a couple of the phones, the router, cable modem (192.168.100.1 typically), ISP gateway, 1.0.0.1, 8.8.8.8 and 4.2.2.2. This will give you a VERY accurate picture of jitter, PL, latency OVER TIME. And, tell you where it's starting. For instance if you are getting packet loss to the phones, there's LAN issues. To the router, LAN issues or bad router. To the modem bad modem. To the GW, bad circuit. GW ok, but others bad, peering issues with ISP. I'll roll this in advance if I'm able and get these issues resolved before deploying if I can. Bandwidth: Nope, 1.5Mbps ADSL isn't going to cut it... There's two ways to handle this. Either have them buy way more than they'll ever use (most people do this) or setup bandwidth management. I use Mikrotik routers personally wherever I can to do the latter. So, to your question on how to prevent these issues, figure out a way to cover the above bases. Top of the list should be partnering with good IT folks. To your question about how does everyone else do it? Well... Most go Karl's route and do the IT too. Or they don't mess with it. They sell to everyone and take the 80%. The 20% will get tech support that basically just points the finger at IT (I mean, they aren't wrong really). The vast majority or SMBs, especially with the rise of cloud services, don't even have any IT support AT ALL so they will deal with the occasional bad quality or move on to a land line provider. The remainder will blame their IT folks until they figure it out or do the same.

Ok, last update and I'm leaving this be. The timestamp in the msg_id is down to the microsecond. So, I really don't need the seperate timestamp field on top of it. Messages can be de-duped with routing key + msg_id

Ugh, just in case anyone is following along... from_tag and to_tag eventually stay the same. It differs between hold/unhold #1 and hold/unhold #2. But, not thereafter. I'll be using both msg_id and timestamp to de-dup with. This leaves open the possibility that if someone managed to hold/unhold/hold again within 1 second that it'd throw off my stats. But, I feel like that'd be getting pretty edge case. I think AQMP had internal IDs it'd be nice to get at least a hash of AQMP's internal ID to de-dup with. But, this is the best there is for the moment as far as I can see.

Awesome as always @mc_ thanks! I'm going to concat all the things!!!

Hey @mc_, do you know what "from_tag" is? This also seems to be unique across different hold events. Also, seems like you are right, the msg_id does appear to be a timestamp

I've got an agent in python that connects, subscribes to one or more accounts (and sub-accounts), records events and dumps them out to a file. It then uploads to backblaze storage for processing by another agent. I'll be open sourcing that part for sure most likely before the end of the year. If you could use that, I'd hate for you to duplicate your efforts.

NP, if you haven't started on the websocket code code yet and python is acceptable to you, you might want to wait a bit before diving in. 😉

So, you are listening on the AQMP. So regardless of the origin, it should come up. Are you looking to verify that this is the case? Or are you trying to differentiate between hold on Op Console vs physical phone holds? EDIT: my example was done on a physical phone

I'll concat msg_id and the routing key just to be safe. That'll do it I think. Thanks a bunch @mc_!

@FASTDEVICE Neither? AFAIK connecting to WS is equivalent to listening to the AQMP. So effectively you are seeing ecallmgr internally reporting to kazoo apps that a hold event has occurred after freeswitch told it so. I'm not 100% sure about that TBH. But, here's an example of a hold straight off the wire if it helps (censored like crazy obviously) {"action": "event", "subscribed_key": "call.*.*", "subscription_key": "call.MY_ACCOUNT_ID_HERE.CHANNEL_HOLD.*", "name": "CHANNEL_HOLD", "routing_key": "call.MY_ACCOUNT_ID_HERE.CHANNEL_HOLD.CALL_ID_HERE", "data": {"to_tag": "NOT_SURE_IF_THIS_IS_SENSITIVE", "timestamp": 63743922729, "switch_url": "sip:mod_sofia@SOME_IP:11000", "switch_uri": "sip:SOME_IP:11000", "switch_nodename": "freeswitch@fs003.ord.p.zswitch.net", "switch_hostname": "fs003.ord.p.zswitch.net", "presence_id": "MY_EXT@MY_REALM", "other_leg_direction": "inbound", "other_leg_destination_number": "+MY_PHONE_NUM", "other_leg_caller_id_number": "+MY_CELL_PHONE_NUM", "other_leg_caller_id_name": "Rich Guyton Iii", "other_leg_call_id": "OTHER_CALL_LED_ID", "media_server": "fs003.ord.p.zswitch.net", "from_tag": "NOT_SURE_IF_THIS_IS_SENSITIVE", "disposition": "ANSWER", "custom_sip_headers": {"x_kazoo_invite_format": "contact", "x_kazoo_aor": "sip:MY_USER_NAME@MY_REALM"}, "custom_channel_vars": {"account_id": "MY_ACCOUNT_ID_HERE", "authorizing_id": "NOT_SURE_IF_THIS_IS_SENSITIVE", "authorizing_type": "device", "bridge_id": "OTHER_CALL_LED_ID", "call_interaction_id": "INTERACTION_ID", "channel_authorized": "true", "ecallmgr_node": "ecallmgr@apps002.ord.p.zswitch.net", "global_resource": "false", "inception": "+MY_PHONE_NUM@SOME_IP", "owner_id": "MY_OWNER_ID", "realm": "MY_REALM", "username": "MY_USER_NAME"}, "custom_application_vars": {}, "channel_state": "EXCHANGE_MEDIA", "channel_name": "sofia/sipinterface_1/MY_USER_NAME@MY_REALM", "channel_created_time": 1576703524752278, "channel_call_state": "HELD", "caller_id_number": "+MY_CELL_PHONE_NUM", "caller_id_name": "CleaRing - Rich Guyton Iii", "callee_id_number": "+MY_PHONE_NUM", "callee_id_name": "CleaRing", "call_direction": "outbound", "call_id": "CALL_ID_HERE", "msg_id": "NOT_SURE_IF_THIS_IS_SENSITIVE", "event_name": "CHANNEL_HOLD", "event_category": "call_event", "app_version": "4.0.0", "app_name": "ecallmgr"}}

I'm developing a reporting app and as part of it, I have two "agent" servers connecting to my Kazoo API using websockets and listening to events. They each listen for 30 minutes before closing and relaunching. "agent 1" starts on every 0 and 30 minute and "agent 2" starts on every 15 and 45 minute of the hour. This way, I have some redundancy. This obviously leaves me with duplicate messages though and I need a way to de-dup them. I have been using the routing key and this worked really well until I tried to do reporting on call hold time when the calls were held/unheld multiple times in a session. I've looked further into it and it seems like the key I should have been pulling is msg_id. But, before I invest the time to re-write my code to use this key to de-dup my messages, I want to be sure this would work as expected. Where you at @mc_? 😀

Hey Karl! Glad I could help out! I had a feeling a few others probably just gave root creds...