Rick Guyton Posted October 5, 2016 Report Posted October 5, 2016 Hello all, as you may have seen from my previous posts, we've been having a heck of a time with BLF for a long time now. When I went to a training event in SF @ 2600hz, Darren mentioned something about a previous issue that they had resolved a while back with BLF. He was telling me that there's a BLF dialog message ID that's incrementally updated by +1 every time a new BLF message is sent. This ID is tracked by each phone separately and if the tracking ID the phone receives in a BLF message doesn't match what it has on file, it's discarded. Now the issue they had and resolved was that it would transmit the same tracking ID to all phones and it would match on some, but not others. They've since fixed that(a long time ago).But, it got me thinking... I can think of a bunch of ways a single message might be lost over time especially when you are talking over a period of weeks or months. And in that scenario, from what I understand, the phone will stop responding to BLF updates correctly until it reboots or completely looses/regains registration. And then I remembered an obscure setting I read in the Yealink provisioning guide... "out_dialog_blf_enable" Yealink says "It enables or disables the phone to handle NOTIFY messages out of the BLF dialog for account X." Erika? Well, I've deployed it for a few clients that were complaining about having to reboot their phones every couple days to make sure their lights work correctly. And since then, they haven't had any problems. But, I have a pretty small sample. So, I'm hoping some of you might be willing to try this for some of your clients to see if it helps you all before I roll it out to everyone. So, long story short, if you use Yealink and are having BLF issues, try this and PLEASE let me know if it helps them out or if you see any unexpected behavior. Via GUI:1) Account Tab2) Advanced3) Set "Out Dialog BLF" to enabled.4) Rinse/Repeat for any other accounts you might have subscribing to BLfs.Screenshot: http://prnt.sc/cq9j7iVia Provisioning:1) Add this line to your provisioning profile: account.1.out_dialog_blf_enable=1Again, if you try this please let me know either way if it does or does not work for you. Thanks!!!
esoare Posted October 7, 2016 Report Posted October 7, 2016 I gave it a shot at a site that is having issues... Did this yesterday at 2:30pm not sure when they went down, but i am rebooting again..
esoare Posted October 7, 2016 Report Posted October 7, 2016 Too bad the BLF's can't be TCP, so the message always gets through...if that is the problem.
Logicwrath Posted October 7, 2016 Report Posted October 7, 2016 I assume if you switch the SIP signaling to TCP it would also include the BLF. Although, TCP requires more overhead and is certainly not as good as UDP for real time audio. I advise against using TCP.Getting a packet capture and trying to identify the problem is the best start. I think the option you found about allowing out of order packets is a great start.You might also look into firewall, MTU, and datagram fragmentation type stuff.
Mads Mortensen Posted October 9, 2016 Report Posted October 9, 2016 We have had major problems with BLF at our own site, however, after we replaced our Zywall 35 (old one, i know) with Pfsense- we have not had any BLF issues.We have had few customer whom have reported BLF issues, in all cases it turned out to be firewall issues.
esoare Posted October 10, 2016 Report Posted October 10, 2016 Rick Guyton: I had to revert back to Disabled last night. It seemed to get worse for the site (BLF issues were happening every 24 hour period) I am going to see how it goes over the next 48 hours, and let you know. I am wondering, does anyone open up their firewall to allow any/any from the 2600hz data center IP's? I don't think it's necessary, but thought I would ask. esoare
Rick Guyton Posted October 10, 2016 Author Report Posted October 10, 2016 Thanks for letting me know Eugine. I don't, but I heard of people doing that.
Rick Guyton Posted October 10, 2016 Author Report Posted October 10, 2016 Yes, firewall can defiantly cause this. These clients are all behind known PfSense or Mikrotik routers though. With configs we use very widely.
Rick Guyton Posted October 31, 2016 Author Report Posted October 31, 2016 Well, guys, no dice here. I just had 2 different clients call me to tell me that their lights were going crazy today. apparently they are having issues where all of the lights will start flashing simultaneously and not stop. This problem is infuriating... Anyone have any more ideas???
Rick Guyton Posted October 31, 2016 Author Report Posted October 31, 2016 Also, parking seems to be involved a lot. Anyone seeing BLF issues with parking?
esoare Posted October 31, 2016 Report Posted October 31, 2016 I think this was a system wide issue. My clients called me about it also. it was Park BLF's as well as Extension BLF's on the consoles... One of the Park BLF's kept the flashing up and I had to "Flush" that from the Debug Portal under Presence... It is very frustrating!
Rick Guyton Posted October 31, 2016 Author Report Posted October 31, 2016 Thanks for the update eugine! At least I don't feel like I'm going crazy over here.
esoare Posted October 31, 2016 Report Posted October 31, 2016 Yeah, It's not the greatest thing when you have confirmed that a system issue was the cause, versus some reliability/configuration issue was the problem, but in this case, I actually think we were happy that it was an outage problem! lol, at least it came up quickly!
Karl Stallknecht Posted October 31, 2016 Report Posted October 31, 2016 This infuriates us/our customers to no end. Last week we had multiple customers complaining and all we were told from 2600hz support was to flush the BLF data and let them know if it happens again (which it did). We haven't received any complaints today, but honestly I think a lot of our customers get sick of it and just stop reporting it to us since it's such a frequent occurrence.
Rick Guyton Posted October 31, 2016 Author Report Posted October 31, 2016 Yea, I'm really worried we are getting that report fatigue too. You can hear it in their voice when they call. And the wort part it I really don't know what to tell them. I'm starting to feel like a cat in a Skinner box swatting at every lever I can to make this issue stop...I think I'm going to try inputting IP addresses instead of host names for the SIP servers. Maybe that'll help? Ugh.
esoare Posted October 31, 2016 Report Posted October 31, 2016 Karl, my only customer that uses this, has had increased stability, since I removed a console at my office. (the router I have is some kinda Zyxel PK501 router) they have had increase reliability! (other than this glitch). I believe that there was an issue with a: the blf's going to two different IP's b: my crummy zyxel dsl modem c: the dsl service itself do you guys know what type of router's your customers are using?
esoare Posted October 31, 2016 Report Posted October 31, 2016 Rick, Do you use Advanced Provisioner yet? you could make that change there...but be careful if you have Combo Key's programmed...
Rick Guyton Posted October 31, 2016 Author Report Posted October 31, 2016 This morning, I ran into this with two clients at three locations.1) Buffalo running DD-WRT2) Cox comunications all in one router3) Netgear ProsafeI've been working with Jeff on some MikroTik configs. I'm getting pretty close to a fully functional config. When I have that I'll see if replacing the router works.
Karl Stallknecht Posted October 31, 2016 Report Posted October 31, 2016 Rick: I doubt that will help - we actually are still using IPs instead of hostnames since we're still 100% on Kazoo and the old provisioner.Esoare: Hmm, interesting. Our customers are on a very wide variety of equipment, but most of them are using their ISP's default modem/router. The BLF problems seem to come and go and aren't constant which made me think it wasn't likely to be an issue with their equipment. Also, last week as an example we had one customer on FiOS and one customer on Cox (one of the local cable companies) both simultaneously complaining that they were experiencing BLF issues in the same exact behavior.
Rick Guyton Posted October 31, 2016 Author Report Posted October 31, 2016 Yea, I do use the advanced provisioner and that's where I'll make my changes to try it out.
Rick Guyton Posted October 31, 2016 Author Report Posted October 31, 2016 Hey Karl: Thanks! I didn't know you were using IPs still and having this issue. There goes that idea. I mean on the plus side, you probably didn't even notice the DYN outage. :)
Karl Stallknecht Posted October 31, 2016 Report Posted October 31, 2016 Regarding DYN: yep! It was actually kind of nice considering we a) don't use them for anything and b) are using IPs. Not a single complaint from a single customer that day :-)That being said, we always recommend using multiple DNS providers which apparently now 2600hz is going to be implementing. We run our own DNS servers as well as use DNS Made Easy. Having both us and them go down would pretty much be impossible and I was shocked that 2600hz didn't have a similar setup to be honest considering it's cheap and easy to have multiple DNS providers.
Logicwrath Posted October 31, 2016 Report Posted October 31, 2016 I have had BLF issues in the past, however, the most recent reports I have received deal with BLF on the Call Park keys.I had the same user report the problem (light doesnt turn red) when a call is parked for two weeks in a row.When I go to the debugging tool I can see that there is no presence listed at all for any extensions or parking spaces. If I reboot the users phone, then after reboot, I can see all the presence values showing up. At that time I can flush the parking lot presence subscriptions.The interesting thing is that both times I had to look at this, all presence listing for this account were blank. Rebooting the phone will get them to re-appear.I wish i had a better idea on how this works, so I could infer what the missing presence data means in the debugging tool 2600 give us. Is it empty because of out of order packets and all the phones stop updating the cluster or subscribing?
Rick Guyton Posted October 31, 2016 Author Report Posted October 31, 2016 @Karl abut DYN: Yea, I like DNSMadeEasy, but they don't support NAPTR rescords last I checked. We were using them because they have that ANAME feature and we were using that thinking they cached the IP. Turns out not, if you use an ANAME on DNSMadeEasy, it'll stop resolving if the name you were pointing to stops resolving. Opps... So as soon as I'm done with this MikroTik golden config, I'l going to deploy brand new DNS with EasyDNS + GoogleDNS.
Recommended Posts