sporkman Posted December 15, 2017 Report Posted December 15, 2017 We use DHCP option 66 to provision new phones to the zswitch server. This is obviously way faster than manually keying the provisioning server into each phone and it works great with new phones. For used phones, we similarly point all the phones at Polycom's boot server which clears and updates them if necessary. Today we were on site with 70 phones to provision. After a round of 30 or so phones, we found no further phones could be provisioned. After much troubleshooting, we realized that "p3.zswitch.net" was not reachable from our public IP at this site. We flipped to another public IP and were able to do another batch of phones. Then same thing, "p3.zswitch.net" becomes unreachable. My spidey sense says that there's some kind of DDoS protection in place and turning up an office full of phones at once is triggering it. This is worrisome in that a) we should be able to bulk provision phones and b) would this same thing happen if, say a large office lost power and then all the phones hit the server at once when the power returned we'd be in a similar situation Thoughts?
Administrators Darren Schreiber Posted December 16, 2017 Administrators Report Posted December 16, 2017 Yup, that's what you hit. Basically don't plug in the phones until they're configured, as a rule, and it should be fine. And don't try to load configs from a browser. I can unblock you manually if you msg me your IP.
sporkman Posted December 16, 2017 Author Report Posted December 16, 2017 3 minutes ago, Darren Schreiber said: Yup, that's what you hit. Basically don't plug in the phones until they're configured, as a rule, and it should be fine. And don't try to load configs from a browser. I can unblock you manually if you msg me your IP. Thanks! So my remaining question is what happens in a more normal situation like a power outage where you similarly have an inrush of phones looking to hit the server? Will they be blocked?
Administrators Darren Schreiber Posted December 16, 2017 Administrators Report Posted December 16, 2017 Nope, not if they load known files perfectly.
sporkman Posted December 16, 2017 Author Report Posted December 16, 2017 1 minute ago, Darren Schreiber said: Nope, not if they load known files perfectly. I think I see what you're looking at then - an unprovisioned phone is going to be generating many more 404s than a provisioned phone? When we bring up a large number 40-100, is there a batch size we can work with that won't trigger this? The method of having the DHCP server set the boot server saves literally hours of work compared to entering stuff by hand on every phone, especially if you're going into an office and transferring phones from another provider. DDoS generator attached:
Administrators Darren Schreiber Posted December 16, 2017 Administrators Report Posted December 16, 2017 Basically you have to program the phones first unfortunately. It's the only way (for now). We are working on a whitelist IP for 24 hours mechanism.
Karl Stallknecht Posted December 18, 2017 Report Posted December 18, 2017 @sporkman I love your optimism with the dollar symbols lol.
sporkman Posted January 9, 2018 Author Report Posted January 9, 2018 On 12/17/2017 at 7:05 PM, Karl Stallknecht said: @sporkman I love your optimism with the dollar symbols lol. Yeah, maybe misplaced. It's a huge ob/gyn office and the call center app is hugely not functioning. You want to up the stress level in an office like that? Get your phone system dumping calls from ladies in labor. Huge black eye for us and a bit of a "is this ready for prime time" sort of moment.
sporkman Posted January 16, 2018 Author Report Posted January 16, 2018 And... that call center app failure lost us the customer. Misplaced optimism. Think twice about "mission critical" customers on the platform.
Administrators Darren Schreiber Posted January 16, 2018 Administrators Report Posted January 16, 2018 2 minutes ago, sporkman said: And... that call center app failure lost us the customer. Misplaced optimism. Think twice about "mission critical" customers on the platform. That's unfortunate. While there was an issue last Monday/Tuesday, it was resolved. I would not agree that this should stop you from mission critical clients in the future, but it sounds like the timing on this did kill it for this customer. Obviously nobody wants that to be the end result. Sorry to hear this lost you your deal. Hopefully the next one will work out better.
sporkman Posted January 16, 2018 Author Report Posted January 16, 2018 2 minutes ago, Darren Schreiber said: That's unfortunate. While there was an issue last Monday/Tuesday, it was resolved. I would not agree that this should stop you from mission critical clients in the future, but it sounds like the timing on this did kill it for this customer. Yes, the call center issue. It's a very large ob/gyn office. I mean, not having ladies in labor being able to reach their nurse or doctor is a really big deal - in many ways this client is probably more demanding than an office full of traders. Total disaster. And the hours spent convincing our reseller there was a problem, just painful. Even after the fix, there are still ongoing issues that I suspect don't show up in lower call-volume queues, and definitely some browser-specific issues with the web app that we never really nailed down.
Tuly Posted January 16, 2018 Report Posted January 16, 2018 Maybe it's just me, but I would have not accepted such a client in the 1st place, the liability and the headache that comes with it is not worth,
Administrators Darren Schreiber Posted January 16, 2018 Administrators Report Posted January 16, 2018 3 minutes ago, Tuly said: Maybe it's just me, but I would have not accepted such a client in the 1st place, the liability and the headache that comes with it is not worth, It's reasonable to accept it should work. We have to make that the target.
Recommended Posts