Jump to content
KAZOOcon: hackathon signup and details here! ×

us-east on Verizon FIOS issues


Tuly

Recommended Posts

Hi,

Is anyone who has Verizon FIOS having issues lately with the US-east? we’re having for the last I would say 2 weeks or maybe even more,  high pings and a lot of packet loos to us-east,  and this is only happening on FIOS customers

 

Interesting part is that pinging .70.1 or 70.10 is good ( I guess it’s in the same datacenter) only the 2600HZ IPs are bad, and its happening only during working hours, about 10-4, we are not getting anywhere with Verizon, they keep saying that up till the IP 70.3 everything looks good, (and it does, the last hop to that IP by level 3 is also very good)

 

How could we start to troubleshoot to see where the issue is?

 

 

Thanks,

Link to comment
Share on other sites

Don't want to high jack this. 

Any us-west.p.zswitch.net West Coast Data Center customers with "Comcast" having issues? 

Brought 3 customers on-board over the past Month 8-10 forward. And they seem to have issues. Spent about 1-3 cumulative man days on one customer in troubleshooting.

Pulling hair out...oh wait, hair is just jumping ship. ;) 

Link to comment
Share on other sites

 

On 9/23/2017 at 9:25 AM, esoare said:

Don't want to high jack this. 

Any us-west.p.zswitch.net West Coast Data Center customers with "Comcast" having issues? 

Brought 3 customers on-board over the past Month 8-10 forward. And they seem to have issues. Spent about 1-3 cumulative man days on one customer in troubleshooting.

Pulling hair out...oh wait, hair is just jumping ship. ;) 

It is rumored that Comcast throttles SIP.  I know I have seen some issues with Comcast on residential plans.  Could just be that the Comcast network is getting congested in some areas at some times of the day too.

Edited by amn (see edit history)
Link to comment
Share on other sites

1 minute ago, amn said:

 

It is widely rumored that Comcast throttles SIP.  I know I have seen some issues with Comcast on residential plans.

Assuming this is true.  What is the easiest way to stop giving them the option to throttle it?  Does encrypting the audio mask it enough or is something like a VPN required?

I have read plenty of posts over the years where users setup a VPN to their hosted PBX and all audio issues cease to exist.  I would absolutely hate to be in this situation.  It would be great if just encrypting the audio would hide this enough to stop Comcast from filtering the traffic but I suspect the unencrypted signaling traffic would give them the ability to identify which traffic to shape or sabotoge.  I believe as Net Neutrality laws get weaker we will see more specific anti-competitive actions being taken by providers like Comcast.

Link to comment
Share on other sites

2 minutes ago, Logicwrath said:

Assuming this is true.  What is the easiest way to stop giving them the option to throttle it?  Does encrypting the audio mask it enough or is something like a VPN required?

I have read plenty of posts over the years where users setup a VPN to their hosted PBX and all audio issues cease to exist.  I would absolutely hate to be in this situation.  It would be great if just encrypting the audio would hide this enough to stop Comcast from filtering the traffic but I suspect the unencrypted signaling traffic would give them the ability to identify which traffic to shape or sabotoge.  I believe as Net Neutrality laws get weaker we will see more specific anti-competitive actions being taken by providers like Comcast.

You could certainly test the theory by adding a VPN.

Edited by amn (see edit history)
Link to comment
Share on other sites

36 minutes ago, Karl Stallknecht said:

@Logicwrath I've read tons of posts from people who had to do this exact thing...we had one customer that we did it for and it worked, but luckily the root issue went away so we could remove the VPN and just go back to the normal setup. I've always been afraid it might start back up at any of our clients though.

I don't know what else would cause problems to disappear by adding VPN other than throttling.

Edited by amn (see edit history)
Link to comment
Share on other sites

  • Administrators

I don't think this is throttling. BlueTel has been working with me on this with some traces. What is super weird is Level3 (or Verizon?) is sending traffic to our gateway IP (.1) via one peering point, but traffic to our other nodes is going through an alternate peering point. I think something is still wrong, and am working on it.

We did confirm that if you're impacted by this, you can reroute to us-central.p.zswitch.net for now and the problem is resolved. I may even do that for the remainder of the day for all traffic until I understand this issue more.

 

Link to comment
Share on other sites

  • Administrators

OK, for those on-lookers. I've just gone through about 15 of Tuly's accounts and now believe that the information he is posting here is not accurate. For the sake of avoiding any misleading assumptions by other customers, we CAN NOT produce any failing packet traces except from ONE of Tuly's locations thus far. 

So, until I have something more conclusive that shows this is more widespread, please note that I do not think there is any issue on our equipment or with our network provider in EWR at this time.

Happy to investigate more reports, and waiting for Tuly to provide a more comprehensive list of where he's seeing failures, but at this point, I can't reproduce this, including to his own customers, so I want to prevent any FUD on this thread until I have more data.

I'll circle back when I've collected more data.

Link to comment
Share on other sites

×
×
  • Create New...