2600Hz Employees Isaac Posted May 9, 2016 2600Hz Employees Report Share Posted May 9, 2016 We are currently experiencing an issue with the switches in our new datacenter. Specifically, you will notice degradation in audio and problems with BLF. This is only impacting calls on our east coast customers. We are experiencing some choppy audio, dropped calls and a few other symptoms for calls passing through our EWR datacenter (east coast). Customers in the central states or west coasts are not impacted. The issue has been narrowed down a faulty switch card or cable (we're still identifying which). In the meantime, we are in the process of mitigating the impact of the issue by shifting traffic to alternate servers so that we can replace the switch which is causing the issue. BLF lights may be inaccurate while we work to resolve this. We apologize for the trouble this is causing. Link to comment Share on other sites More sharing options...
Administrators Darren Schreiber Posted May 9, 2016 Administrators Report Share Posted May 9, 2016 FYI, realtime updates are at http://status.2600hz.com/ Link to comment Share on other sites More sharing options...
Michael D Posted May 9, 2016 Report Share Posted May 9, 2016 Appreciate the status updates, i subscribed as well Is this a 2600hz issue specifically? or the datacenter/ Telco building Link to comment Share on other sites More sharing options...
Administrators Darren Schreiber Posted May 9, 2016 Administrators Report Share Posted May 9, 2016 This is a 2600hz issue. We have nice new shiny GigE switches in the new DC. They're redundant and stacked.All servers are racked with bonded cabling and we have multiple ISPs connecting into the core.There's literally only one component that could cause disruptions to the redundancy, and that's the cable that makes the switches redundant (plugs one switch into the other). And, of course, that's what failed...Murphy's law Link to comment Share on other sites More sharing options...
Karl Stallknecht Posted May 9, 2016 Report Share Posted May 9, 2016 To clarify, are you just adjusting DNS records or what exactly is going on? i.e. if we have a customer with the east coast proxy IP manually programmed in will they be fine with your fixes, or were your "re-routing traffic" fixes just adjusting DNS records? Link to comment Share on other sites More sharing options...
Administrators Darren Schreiber Posted May 9, 2016 Administrators Report Share Posted May 9, 2016 No need to adjust anything. We're able to re-route traffic after hitting the external switch and route the calls themselves to alternate media servers. Link to comment Share on other sites More sharing options...
Karl Stallknecht Posted May 9, 2016 Report Share Posted May 9, 2016 Fantastic.Thanks for being proactive with posting this guys. Saved both you and us a support ticket! Link to comment Share on other sites More sharing options...
Administrators Darren Schreiber Posted May 9, 2016 Administrators Report Share Posted May 9, 2016 Please do check status.2600hz.com - it has more updates. We'll be using that for a while until our own status tool is online and fully automated.Thanks for your patience! Link to comment Share on other sites More sharing options...
Karl Stallknecht Posted May 9, 2016 Report Share Posted May 9, 2016 Sorry, I wasn't clear. I meant thanks for being proactive with posting it on StatusPage! :-) I understand the community isn't the expected place for quick updates. Link to comment Share on other sites More sharing options...
Karl Stallknecht Posted May 9, 2016 Report Share Posted May 9, 2016 We subscribed our staff email to it so we will all receive notifications! Link to comment Share on other sites More sharing options...
Administrators Darren Schreiber Posted May 9, 2016 Administrators Report Share Posted May 9, 2016 This is resolved. Link to comment Share on other sites More sharing options...
Recommended Posts