Jump to content
KAZOOcon: hackathon signup and details here! ×

Rate-limiting on hosted server?


sporkman

Recommended Posts

We use DHCP option 66 to provision new phones to the zswitch server.  This is obviously way faster than manually keying the provisioning server into each phone and it works great with new phones.  For used phones, we similarly point all the phones at Polycom's boot server which clears and updates them if necessary.

Today we were on site with 70 phones to provision.  After a round of 30 or so phones, we found no further phones could be provisioned.  After much troubleshooting, we realized that "p3.zswitch.net" was not reachable from our public IP at this site.  We flipped to another public IP and were able to do another batch of phones.  Then same thing, "p3.zswitch.net" becomes unreachable.

My spidey sense says that there's some kind of DDoS protection in place and turning up an office full of phones at once is triggering it.  This is worrisome in that a) we should be able to bulk provision phones and b) would this same thing happen if, say a large office lost power and then all the phones hit the server at once when the power returned we'd be in a similar situation

Thoughts?

Link to comment
Share on other sites

3 minutes ago, Darren Schreiber said:

Yup, that's what you hit. Basically don't plug in the phones until they're configured, as a rule, and it should be fine. And don't try to load configs from a browser.

I can unblock you manually if you msg me your IP.

 

Thanks!

So my remaining question is what happens in a more normal situation like a power outage where you similarly have an inrush of phones looking to hit the server?  Will they be blocked?

Link to comment
Share on other sites

1 minute ago, Darren Schreiber said:

Nope, not if they load known files perfectly.

 

I think I see what you're looking at then - an unprovisioned phone is going to be generating many more 404s than a provisioned phone?

When we bring up a large number 40-100, is there a batch size we can work with that won't trigger this?  The method of having the DHCP server set the boot server saves literally hours of work compared to entering stuff by hand on every phone, especially if you're going into an office and transferring phones from another provider.

DDoS generator attached:

IMG_20171215_155737.jpg

Link to comment
Share on other sites

  • 4 weeks later...
On 12/17/2017 at 7:05 PM, Karl Stallknecht said:

@sporkman I love your optimism with the dollar symbols lol.

Yeah, maybe misplaced.  It's a huge ob/gyn office and the call center app is hugely not functioning.  You want to up the stress level in an office like that?  Get your phone system dumping calls from ladies in labor.  Huge black eye for us and a bit of a "is this ready for prime time" sort of moment.

Link to comment
Share on other sites

  • Administrators
2 minutes ago, sporkman said:

And... that call center app failure lost us the customer.

Misplaced optimism.  Think twice about "mission critical" customers on the platform.

That's unfortunate. While there was an issue last Monday/Tuesday, it was resolved. I would not agree that this should stop you from mission critical clients in the future, but it sounds like the timing on this did kill it for this customer. Obviously nobody wants that to be the end result.

Sorry to hear this lost you your deal. Hopefully the next one will work out better.

Link to comment
Share on other sites

2 minutes ago, Darren Schreiber said:

That's unfortunate. While there was an issue last Monday/Tuesday, it was resolved. I would not agree that this should stop you from mission critical clients in the future, but it sounds like the timing on this did kill it for this customer.

Yes, the call center issue.  It's a very large ob/gyn office.  I mean, not having ladies in labor being able to reach their nurse or doctor is a really big deal - in many ways this client is probably more demanding than an office full of traders.  Total disaster.  And the hours spent convincing our reseller there was a problem, just painful.  Even after the fix, there are still ongoing issues that I suspect don't show up in lower call-volume queues, and definitely some browser-specific issues with the web app that we never really nailed down.

Link to comment
Share on other sites

×
×
  • Create New...