Q&A re: upcoming 4.0 upgrade

[Darren Schreiber]

Questions about the 4.0 upgrade? This thread will be an ongoing Q&A. I will edit this top level post as people ask questions so this becomes a collection of questions asked and answers.

Q: Why am I required to sign new contracts?
A: We have a number of varying contracts that have gone out to clients since 2012. Many of them have not been renewed and contain outdated language, outdated pricing, etc. In some cases we have incidents where, for example, calling 911 without provisioning 911 is not something we can charge a client for, even though we get charged $150 each time you do it, so there is no incentive for such clients to "fix" their 911 provisioning behavior. In other cases, we have issues where contracts include services we don't even offer anymore. This all needs to be cleaned up.

More important, we've heard loud and clear from resellers that they want faster responses to our tickets. One way to do that is to roll 4.0 (because we will no longer be in a major release cycle). The other way, however, is to make sure that we can more quickly update terms and conditions electronically when they change. Right now, many contracts require snail mail updates to be sent. We are sunsetting this, which is more inline with most other cloud-based service providers where T&C, Privacy Policy and Legal Updates can be pushed via the portal, email or e-sign.

This is more a house-keeping routine than anything else.


Q: Are you raising my rates?
A: If you've signed with us recently (last 6-9 months), very unlikely. This is more to get your contract's terms inline with the electronic items listed above.

If you have a contract that is a year older or more, than there may be some changes (mostly decreases) because we now bill for support separately (and optionally). Many customers are still on "unlimited' support contracts, which we are officially phasing out across the board. Those contracts are likely the ones with the most changes.

Each contract varies, as everyone has slightly different ways they sell our product. Thus, we're crafting individual rate sheet updates for everyone. The most important thing is that all rates are actually represented, as many items in older contracts are just wrong at this point.



Q: Why was I not given more notice?
A: We are allowing two weeks notice for the upgrade, in addition to announcing the upgrade at KazooCon and sending multiple emails prior to KazooCon warning of the upcoming upgrade. In addition, we provided a sandbox login for people to test the update before we rolled it nearly 3 months ago. We would love to provide even more notice, but as we get too close to Christmas and staff start going on holiday, it becomes harder to patch things quickly. This is the best compromise we could come up with.

As for contracts, we are targeting 90 days to get everyone up to date so that we can utilize a unified legal system and app roll-out process. We think this is the fairest thing for everyone involved, including us. 90 days allows a lot of time to negotiate if you don't like something, while also giving us a reasonable window for when all these contract updates will be "done" and we can switch to a single notification system for T&C and Privacy Policy updates.


Q: Why does this upgrade have anything to do with pricing?
A: For customers with contracts signed in 2015 or earlier, some of the pricing and names of products are very outdated. In most cases the prices for most items have gone down, but we have split out some services (like support) that cost us a lot of money to offer. Since we're fixing the terminology in contracts, we figured we should update the pricing at the same time.


Q: I would have preferred more than a 90 day window for pricing, why are you giving us this window?
See above


Q: Why are you performing a disruptive upgrade that I get no immediate benefit from?
A: The upgrade to get all the new features requires us to "fork lift" the plumbing to a new operating system (CentOS 7). In addition, we really don't want to introduce new features the same day we are burdened with making sure all the old features still work. It's too much. So, actually, the new features are being rolled out but just not turned on yet. It's a two-phased approach.

Once support is satisfied that 4.0 at least covers all existing features without new bugs or issues, we'll continue rolling the new features.


Q: When will I benefit from the new features?
A: As soon as we're sure 4.0 is stable with existing features, we'll start rolling out the new ones. We'll announce each one as we go and you'll always be able to test them first on Sandbox from now on.


Q: What is the start and end time of the maintenance?
A: The SaaS / Hosted Platform maintenance window starts at 8pm Eastern (no disruption expected at that time, but we will be shifting call traffic to Chicago or San Jose. Normally this isn't noticeable to most clients but it's POSSIBLE some call quality issues such as pops, hisses or jitter may occur). At 11pm Eastern / 8pm Pacific we will begin what is considered more disruptive maintenance (still hoping no disruption, but there will likely be some). The disruption will likely be some calls failing to complete and/or phones failing to register. Calls should ultimately roll to voicemail if we get everything perfect, but there's a chance we won't, so notifying clients in advance is a good bet.


Q: Why are the 209.X.X.X and 184.X.X.X IPs changing?
A: Those IPs represent our original Rackspace servers. Rackspace themselves no longer supports this version of their servers, they are considered "v1" and they are trying to get rid of them. As such the pricing is higher, we can't upgrade them anymore (they've hit their max) and they're basically dying. As with the general "housekeeping" theme above, we're going to use this opportunity to get our house in order and retire these servers.

We aren't going to just turn off servers without checking the logs to make sure nobody's actually using them. If we see usage still we'll hunt down whoever is using them, notify them and help them move off. This note in our announcement is more of an FYI - if you can help us out and stop using those IPs, that'd be awesome - one less person to contact manually.

The 209. and 184. address are still used in a few places, I'm aware of that and having that corrected. If you do see it, please point it out (as you've done here) and I'll get it updated! Thank you as always for the help.


Q: Is BLF going to be fixed in this upgrade?
A: BLF has three possible issues. One we have been tracking and, thanks to extensive help and debugging reports from Rick Guyton, we've finally figured out last week. Basically, a particular queue that processes BLFs was getting "too full" before the messages were processed (this all happens in miliseconds and is hard to catch without examples). We've basically fixed that issue, with a few recurrences still popping up but I think we've figured out how to track them. Note that this issue is VERY SPECIFIC - if rebooting your phone clears the light, it's NOT this issue.. This issue ALWAYS had the symptom that a light would get stuck in the GUI and rebooting your phone would not work - the light would still be there. You'd have to go into the GUI to "flush" it. Many of you have said "oh yeah, that's the issue I'm having" but in testing, we found this was not true! That a reboot of the phone WOULD fix the issue (which means it's NOT this issue). That said, we are still getting some ancillary reports of lights stuck AFTER a phone is rebooted, but we are researching each of them and they all look to be the same root cause. 4.0 itself has some patches that should help with this, but also the maintenance we're doing along 4.0's rollout should fix this as well. So, yes, this should be resolved in 4.0.

Again, to see if the issue you're having with BLF is the one we believe is resolved, you should debug by:
1) Restarting the phone. If the phone comes back up and all lights are working, THIS IS NOT the issue we have resolved.
2) If a restart doesn't work, go into the GUI and clear the light. If this works, then the issue IS resolved with the patches we'll be doing.

Please note carefully that for the above to be true you MUST do it in the order of operations I listed - reboot the phone first, and if no-go, then clear it via the GUI. If that COMBINATION works in that ORDER, then the bug we are trying to squash is infact squashed in the future version! (Or so we think!)

BUT, onto issue number two... BLF is tough to debug. If I'm wrong on the above (and even if I'm not), the real reason BLF issues have dragged on so long is that they're impossible to catch. We must find the call which caused the light to get stuck. Right now, there's a binary data file we can look in, and it's painfully slow to do, and only employees with access to those servers can do it. So to debug BLF today, we have to have a reseller partner (you) willing to debug with us, the exact call that caused the issue, and a lot of time and luck. This is painful. In 4.0, this is moved to a GUI where we can actually see (and so can you) the last BLF packet we sent to a phone. This should make this problem really fast to debug.

The third issue is occasional reports of BLF just stopping working and a reboot of the phone itself fixes the issue. We have traced this one to no end and in every case, it's come back as an internet or firewall issue. Every single case. We still want to find a way to resolve this without having to restart the phone, but we consider this a lower issue since we don't believe our software is actually causing this. Nonetheless, it remains a priority after 4.0 rolls out because people are tired of it. We have some ideas how we can work around crummy internet connections and firewall issues.

[esoare]

Does the Dec. 2nd stuff have anything to do with BLF?
Please say yes! and that it will fix everything once and for all!...Please!

[Karl Stallknecht]

Are the IPs going out of service that you listed in your email all proxy IPs? I noticed we had 209.x.x.233 and 184.x.x.174 listed in some DNS records of ours related to SIP proxies/what our SIP realms are pointed to, so we replaced them with 166.x.x.67 and 8.x.x.3, respectively. Just wanted to make sure this was okay to do.

**FYI 209.x.x.233 is still listed in Monster under DNS Helper as the "general proxy fallback" but you said in your email that this IP is going bye bye...is this a mistake on the email or does it need to be removed from Monster?

I know you don't want us sharing the IPs publicly, so I intentionally was vague. Let me know if you want us to directly email you with the exact IPs.

[Darren Schreiber]

Hi there,
     Yes, some of those servers are old Rackspace servers and they have EOL'd the type of server we are on. We can no longer upgrade their speed or memory :-/

     So we are being forced to migrate them. And we're taking the opportunity to just move them out of Rackspace.

     We aren't going to turn off any servers without checking the logs to make sure nobody's actually using them. If they are, we'll hunt them down and notify them and help them move off. This is more of an FYI - if you can help us out and stop using those IPs, that'd be awesome - one less person to contact manually.

     The 209. and 184. address are still used in a few places, I'm aware of that and having that corrected. If you do see it, please point it out (as you've done here) and I'll get it updated! Thank you as always for the help.

[Karl Stallknecht]

Thanks!

[Darren Schreiber]

I've been trying to make time to answer all the BLF complaints and questions posted here, but have not been able. I know this isn't super helpful but:

1) We think there are two different issues occurring
2) We've identified and fixed one of the issues
3) We've added a debugger tool in 4.0 that lets us VERY QUICKLY find any remaining issues in case I'm wrong on the above (including the remaining issue we don't yet understand). All of you will be able to see this and the BLF packets directly from the GUI and we can show you how to use it. It's pretty cool and easy.

The main problem with BLF is examples. We have to catch the BLF lights go wonky at the time they go wonky OR know exactly which call it is that triggered the BLF to be wrong.

4.0 lets us do that. Which will literally save all of us time and headaches until everything is 100% perfect.

I'll write more on this next week. But yes, this is still our highest priority issue.

[Darren Schreiber]

I've updated the White-Label setting in Monster UI on the Hosted Platform.

There may be one more round of changes sometime in January when we upgrade our circuits in Chicago but should be good for now for the 4.0 upgrade.

[Darren Schreiber]

Posted more about BLF above.

[Logicwrath]

Will 4.0 allow us to copy and paste device configurations in advanced provisioner or tier account BLF configurations?

[Darren Schreiber]

No. We did add this to the feature request log. However, considering how many complaints are still coming in with BLF, and considering we're coming out with operator console (which I'd much prefer people use instead of this archaic SIP standard), we're going to go in that order.

[Karl Stallknecht]

Whoops, I guess we made a slight oversight. It actually looks like some of our phones that were provisioned a long time ago using Kazoo are still pointing to 184.x.x.174 manually (non-DNS). Is there a way to push an update out? We really don't have time between now and December 2nd to finish switching everyone over to Monster/advanced provisioner.

[Darren Schreiber]

We won't be turning off any old servers until we've given a reasonable amount of time for people to move. Don't freak out about the December 2nd date in regards to this particular issue.

Let me clarify the dates some more:

Phase 1: Upgrade of Hosted Platform to 4.0: December 2nd, 2016


Rollout of new features: Once 4.0 is rolled out and we believe things are working well. Likely ETA is December 2016 or January 2017 (some features may still be considered beta, but will be usable / available)


Phase 2: Upgrade of Hardware for Hosted Platform: January 2017


Finalization of all contracts moved to 4.0 version / pricing: February 17th, 2017.


Shutdown of ancient servers: January/February 2017, depending on speed of decomission and customer cooperation.

[Karl Stallknecht]

Thanks

[Rick Guyton]

Will there be any additional add-on costs in 4.0?For instance, at kazoo-con I thought I heard someone say that call center light and web sockets would require an additional cost. I wasn't sure if that was just for the opens source people or if there would be an additional cost for the SAAS people too. ::fingers crossed for no::

[Darren Schreiber]

If you've signed a contract recently (last 6-9 months), your costs will stay the same for what you have now and even for a few new apps which will be introduced.

Not everything will be free that is introduced in 4.0, though. The biggest item is likely call center lite.

Most companies charge something for Call Center / Call Queues so we are debating how we should position that right now. Obviously everyone wants free, but it cost us about $250k to develop all the queue stuff so far, so we have to recoup that somehow. So likely we'll charge something. We're actively learning what other WHOLESALE vendors charge now and how that's marketed retail today. Most of the time it's some base fee for their product plus $ per agent, and/or $ per queue. We'll likely do the same. We will not be open-sourcing call queues for the time being.

WebSockets is included and is open to everyone at no cost.

We are moving to an app-exchange based model, where each app costs something (either an annual flat license or a usage based model, such as per-user or per-queue). That reflects that we can't possibly maintain a zillion features for a few bucks a month per product offering. It also allows others to put stuff into the same app exchange and charge for their work, too. The goal being you have the most options, and even a few choices for the same feature.

The pricing and purchasing will become automated. You go into the app exchange, you look at the apps, you buy (or if they're free, you just turn them on).

Simple. Just like every other application exchange / store out there. Check out Atlassian's market place as an example. We're probably closest to that model at this point - open-source core, open-source a bunch of the core/main apps, some apps can be added for a fee as an add-on, pay for hosting if you need that.

[Rick Guyton]

Ok, yea I kind of expected an up charge on call center. But I'm happy that web sockets will be open. Thanks!

[esoare]

How about costs for Operator Console?

[Rick Guyton]

Oh, and webphone?

[Darren Schreiber]

This one is in debate, mostly because Webphone still needs some work done to it. I'll defer to the product team for announcing Webphone / Operator Console once they're ready to go.

Google has made some changes to WebRTC in Chrome which are mucking everything up.

[Logicwrath]

Will 4.0 allow us to specify a fail-over landline numbers so we can properly configure an account to automatically fail-over to a cell phone or landline?  I know we talked about this on one of the re-seller calls and some ideas were discussed.  I would like to have all of my accounts pre-configured for fail-over so the customer does not have to tell us things aren't working and rely on us to do manual work to enable forwarding.

[Logicwrath]

Will the new carrier and number features in 4.0 allow us to update the CNAM name on phone number ourselves?

[Darren Schreiber]

No, 4.0 has nothing to do with this. There is already a failover feature in the backend that if a phone is de-registered it will trigger use of the call forwarding number programmed onto the line instead. We can probably just add that to the GUI if you want that.

[Darren Schreiber]

No, but our next big project is redoing all phone number management. We are tired of people sending in tickets for buying phone numbers (as I'm sure you are). The issue is that we used to rely on bandwidth.com for their API, and then they changed their API with only 30 days notice, so we had to reprogram for the new API quickly and a lot of it still didn't work right. We're circling back to fix that.

The goal will be that you can order from any of our providers who have API support, including number porting, and complete the process yourself. It should be "just as good" as if you had access to the carrier's own upstream portal by the time we're done, with all features exposed except directory assistance.

But that is probably 4.1 or 4.2 at this point.

Do note that the issue with 4.0 is it was such a massive overhaul, it took over a year to do. But once it's rolled I expect much more frequent rollouts and feature updates because they will be less risky and change less portions of the code.

[Logicwrath]

This is something we all want.  We look bad when we can't configure this to automatically happen.  Especially, as we port numbers away from competitors who are doing this for them already.  I believe we had some discussion on this during one of the re-seller calls.

[Darren Schreiber]

Hi there,
     It is a popular request, I agree. I will see what we can do here.