Status

Production upgrade triggered session leak

Emergency maintenance is required, which will result in some live, active calls likely being dropped.  If you encounter a call failure, please email support@voxeolabs.com with information about the call and we can confirm whether if failed as a result of the server maintenance and restart.

Sincere apologies for the inconvenience.

Unplanned provisioning server maintenance

From 12:00 to 12:30 a.m. Eastern tonight (5/8/2013 into 5/9/2013), unplanned emergency server maintenance will be implemented; this will block all access to the Application Manager and Provisioning API, so app changes like URL modification, phone number addition/deletion will be unavailable during the 30 minute window.

Tropo production issue (resolved)

This morning Voxeo Labs Operations noticed a session leak which started consuming all application session in production.   This did cause several of our production browsers to start rejecting calls during this time period.   We quickly removed and restarted these browsers however some calls where likely impacted during this time period.  At this time all traffic levels are back to normal and all alerts have cleared.   

     As for background, we have already identified the cause of this issue, and of course consider it a top priority to address.  Our engineering team is already testing a fix internally which we plan to push out in the very near future.  We have also taken internal steps to mitage this risk wherever possible until then.  If there are any questions please don’t hesitate to contact support@tropo.com. 

Regards

Voxeo Labs Operations

Tropo.com and api.tropo.com down (FIXED)

A currently unknown datacenter problem is impacting the ability for both sites to load, which is impacting outbound calling and SMS.  We are working diligently to determine the root and resolve quickly.

RESOLUTION UPDATE:

Our datacenter encountered connectivity issues related to an upstream provider (Internet connectivity problems, basically) and the NOC failed over to a backup in order to prevent customer impact.  The failover load balancer failed to run correctly, causing our sites to appear down to the outside world (including api.tropo.com, which caused outbound call/SMS issues) while appear functioning internally.  

Connectivity issues resolved at the datacenter, so shifted back to the primary load balancer and the problem with tropo.com and api.tropo.com resolved as well.  We are planning a controlled failover test this weekend to ensure things are working moving forward, and to ensure the issue does not reoccur.

Billing maintenance Monday 8 Eastern

We’ll be performing maintenance on our billing system on Monday for approximately one hour starting at 8pm Eastern. During this time, your account balance may be inaccurate or unavailable and you will be unable to add or change your credit card information.

New tokens failing

Newly added tokens (typically tokens added when a new app is created) are currently failing to launch; existing applications are not impacted.  We are investigating now with our network operations team to identify what’s causing the problem and will update again as soon as we have a resolution.

RESOLVED: 12:15 EST on 2/27/2013 - issue with a transfer agent caused a back up, once resolved all tokens began working as expected again.

ISP related maintenance with potential impact

We have received notice from one of our Internet providers, that they are implementing emergency maintenance updates.  We have multiple providers and customers should not see significant impact, however we have noticed intermittent connectivity issues while new routes are established.  We continue to monitor and will alert if necessary.

Alerts cleared ( 4:45pm EST )

Tropos Network Operations team have confirmed that the connectivity issues specific to our hosted network in Orlando have been remediated, and services in Orlando have returned to operational status.

(resolved) Datacenter issues

Tropo operations has been alerted to connectivity issues in our Orlando data center which may be impacting call traffic  We are currently investigating the issue and will post updates as soon as they are made available.  ( 4:30pm EST )

Issue detected on one of the development servers

…and already resolved.  This will have impacted a portion of sessions launched in the development environment, but production was not impacted.