Tuly Posted September 20, 2017 Report Posted September 20, 2017 Hi, Is anyone who has Verizon FIOS having issues lately with the US-east? we’re having for the last I would say 2 weeks or maybe even more, high pings and a lot of packet loos to us-east, and this is only happening on FIOS customers Interesting part is that pinging .70.1 or 70.10 is good ( I guess it’s in the same datacenter) only the 2600HZ IPs are bad, and its happening only during working hours, about 10-4, we are not getting anywhere with Verizon, they keep saying that up till the IP 70.3 everything looks good, (and it does, the last hop to that IP by level 3 is also very good) How could we start to troubleshoot to see where the issue is? Thanks,
Karl Stallknecht Posted September 20, 2017 Report Posted September 20, 2017 Funny you mention this. At my house I FiOS and also have one of our office phones and I've noticed poor call quality this week. I thought maybe FiOS was just doing something funky in my area since no clients have complained, and it's fine at our office...
Tuly Posted September 20, 2017 Author Report Posted September 20, 2017 It's very interesting I've already been logged into about 20 Clients and all of them have the same results high pings and packet loss , And it's only in the middle of the day......
Karl Stallknecht Posted September 21, 2017 Report Posted September 21, 2017 So we had two clients report to us today that they've been having issues over the last two weeks. And multiple calls I was on today had terrible quality with the call cutting in and out. I'll open a ticket about this.
Administrators Darren Schreiber Posted September 21, 2017 Administrators Report Posted September 21, 2017 We now have three FiOS folks complaining about this. I have opened a ticket with Level3 to see if we can get some traces and figure out what is going on here.
Administrators Darren Schreiber Posted September 23, 2017 Administrators Report Posted September 23, 2017 We'll see for sure on Monday but we have just completed some routing changes that should put this issue to rest. Sorry for any headaches this has caused.
esoare Posted September 23, 2017 Report Posted September 23, 2017 Don't want to high jack this. Any us-west.p.zswitch.net West Coast Data Center customers with "Comcast" having issues? Brought 3 customers on-board over the past Month 8-10 forward. And they seem to have issues. Spent about 1-3 cumulative man days on one customer in troubleshooting. Pulling hair out...oh wait, hair is just jumping ship.
Tuly Posted September 24, 2017 Author Report Posted September 24, 2017 @esoare Can you take a good long ping in the middle of the day to us-west.p.zswitch.net and to the 173.1 IP? Also a trace route to both,
amn Posted September 25, 2017 Report Posted September 25, 2017 (edited) On 9/23/2017 at 9:25 AM, esoare said: Don't want to high jack this. Any us-west.p.zswitch.net West Coast Data Center customers with "Comcast" having issues? Brought 3 customers on-board over the past Month 8-10 forward. And they seem to have issues. Spent about 1-3 cumulative man days on one customer in troubleshooting. Pulling hair out...oh wait, hair is just jumping ship. It is rumored that Comcast throttles SIP. I know I have seen some issues with Comcast on residential plans. Could just be that the Comcast network is getting congested in some areas at some times of the day too. Edited September 25, 2017 by amn (see edit history)
Logicwrath Posted September 25, 2017 Report Posted September 25, 2017 1 minute ago, amn said: It is widely rumored that Comcast throttles SIP. I know I have seen some issues with Comcast on residential plans. Assuming this is true. What is the easiest way to stop giving them the option to throttle it? Does encrypting the audio mask it enough or is something like a VPN required? I have read plenty of posts over the years where users setup a VPN to their hosted PBX and all audio issues cease to exist. I would absolutely hate to be in this situation. It would be great if just encrypting the audio would hide this enough to stop Comcast from filtering the traffic but I suspect the unencrypted signaling traffic would give them the ability to identify which traffic to shape or sabotoge. I believe as Net Neutrality laws get weaker we will see more specific anti-competitive actions being taken by providers like Comcast.
amn Posted September 25, 2017 Report Posted September 25, 2017 (edited) 2 minutes ago, Logicwrath said: Assuming this is true. What is the easiest way to stop giving them the option to throttle it? Does encrypting the audio mask it enough or is something like a VPN required? I have read plenty of posts over the years where users setup a VPN to their hosted PBX and all audio issues cease to exist. I would absolutely hate to be in this situation. It would be great if just encrypting the audio would hide this enough to stop Comcast from filtering the traffic but I suspect the unencrypted signaling traffic would give them the ability to identify which traffic to shape or sabotoge. I believe as Net Neutrality laws get weaker we will see more specific anti-competitive actions being taken by providers like Comcast. You could certainly test the theory by adding a VPN. Edited September 25, 2017 by amn (see edit history)
Karl Stallknecht Posted September 25, 2017 Report Posted September 25, 2017 @Logicwrath I've read tons of posts from people who had to do this exact thing...we had one customer that we did it for and it worked, but luckily the root issue went away so we could remove the VPN and just go back to the normal setup. I've always been afraid it might start back up at any of our clients though.
amn Posted September 25, 2017 Report Posted September 25, 2017 (edited) 36 minutes ago, Karl Stallknecht said: @Logicwrath I've read tons of posts from people who had to do this exact thing...we had one customer that we did it for and it worked, but luckily the root issue went away so we could remove the VPN and just go back to the normal setup. I've always been afraid it might start back up at any of our clients though. I don't know what else would cause problems to disappear by adding VPN other than throttling. Edited September 25, 2017 by amn (see edit history)
Administrators Darren Schreiber Posted September 25, 2017 Administrators Report Posted September 25, 2017 I don't think this is throttling. BlueTel has been working with me on this with some traces. What is super weird is Level3 (or Verizon?) is sending traffic to our gateway IP (.1) via one peering point, but traffic to our other nodes is going through an alternate peering point. I think something is still wrong, and am working on it. We did confirm that if you're impacted by this, you can reroute to us-central.p.zswitch.net for now and the problem is resolved. I may even do that for the remainder of the day for all traffic until I understand this issue more.
Administrators Darren Schreiber Posted September 25, 2017 Administrators Report Posted September 25, 2017 Also, I've re-confirmed that this appears to be only traffic to NY so far that I've been able to simulate this with. Very strange. I think a peer somewhere is saturated. I wonder if Level3 is having peering disputes again with Verizon and/or Comcast or their backhaul providers.
Administrators Darren Schreiber Posted September 25, 2017 Administrators Report Posted September 25, 2017 OK, for those on-lookers. I've just gone through about 15 of Tuly's accounts and now believe that the information he is posting here is not accurate. For the sake of avoiding any misleading assumptions by other customers, we CAN NOT produce any failing packet traces except from ONE of Tuly's locations thus far. So, until I have something more conclusive that shows this is more widespread, please note that I do not think there is any issue on our equipment or with our network provider in EWR at this time. Happy to investigate more reports, and waiting for Tuly to provide a more comprehensive list of where he's seeing failures, but at this point, I can't reproduce this, including to his own customers, so I want to prevent any FUD on this thread until I have more data. I'll circle back when I've collected more data.
Recommended Posts