February 2017

Change to Recording Functionality
The behavior of the record method in Tropo has changed slightly with the current build, and the change was not published in the changelog erroneously. Previously, when a recording was sent to an HTTPS destination URL via a REST POST, Tropo would send two POSTs; the first improved our ability to validate SSL certificates in specific circumstances, and then the second would send the actual recording file. This behavior changed to be a single POST now, as enhanced methods to validate the certificate are present that eliminate the need for the preceding POST. This could cause file processing issues if the recipient application was built to intentionally discard the content of the first POST and process only the content of the second. Apologies for the late notice of this change; while an overall improvement, the potential customer impact was not properly evaluated. If you notice you are encountering recording failures, please contact support@tropo.com for further help
Feb 24, 14:01 EST
[Scheduled] Transcription Provider Maintenance
The scheduled maintenance has been completed.
Feb 24, 01:00-05:00 EST
[Scheduled] Shared Production env upgrade (15.6.1) for Tropo.com
The scheduled maintenance has been completed.
Feb 20, 21:00 - Feb 21, 00:03 EST
Brief connectivity issues
It appears the issue with related to a single monitoring node in our environment. This node failed and seems to have caused several false alarms. We will go ahead and close this issue out at this time. If there are any questions please let us know - support@tropo.com
Feb 19, 17:59-20:00 EST
Toll Free Outbound Messaging Delay
This impairment has been resolved.
Feb 17, 13:38-14:58 EST
Power issue on single shared production runtime
During routine hardware maintenance this morning a runtime was inadvertently restarted while taking active calls. Our support team will be reaching out to the limited number of users who were impacted. If there are any questions please let us know at support@tropo.com
Feb 17, 07:51 EST
Short Code Provider Datacenter Outage
The issue has been resolved and service is fully restored.
Feb 15, 15:09 - Feb 16, 17:00 EST
[Scheduled] Global forbidden rule revision, and expanded internal whitelists
The scheduled maintenance has been completed.
Feb 14, 21:00-23:32 EST
Transcription service unavailable
Our vendor reports service has been restored.
Feb 7, 15:08-16:28 EST
Rollback of update due to media issue.
This incident has been resolved.
Feb 1, 09:28 - Feb 2, 07:46 EST
[Scheduled] Transcription maintenance
The scheduled maintenance has been completed.
Feb 2, 01:00-05:00 EST
[Scheduled] Shared Production env upgrade (15.6.1) for Tropo.com and Tropo.eu deployments
The scheduled maintenance has been completed.
Jan 31, 21:00 - Feb 1, 04:28 EST

January 2017

Media Server Crash
January 31st at 00:20 GMT (midnight), we encountered a media server crash on one runtime in the production environment, which may have caused a small percentage of calls to encounter media related issues, such as missing recordings. The media server recovered and calls are currently stable.
Jan 31, 14:52 EST
Reduced number availability in some Canadian regions
Due to a vendor transition, we have lower than typical stock of Canadian SMS and Voice enabled phone numbers. This is a temporary situation, and will be resolved shortly, but if a region you need does not show as available in the drop down when adding a new number, please contact support@tropo.com for help. We deeply apologize for any inconvenience this may cause.
Jan 27, 10:00 EST
Inbound SMS failures in the development environment.
This incident has been resolved.
Jan 26, 13:48-17:10 EST
Tropo.eu text message delivery outage
Service has been fully restored. We apologize for any inconvenience.
Jan 25, 10:11-10:55 EST
[Scheduled] Enable concatenation, new runtime TLS certs, update forbidden destinations rule
We're all set, just took a tad longer than expected. Have a great evening!
Jan 24, 21:00 - Jan 25, 00:56 EST
[Scheduled] Provisioning API upgrade, and installing new Gateway certs
Work is complete, have a great evening!
Jan 17, 21:00-22:06 EST
[Scheduled] Dedicated Environments: runtime upgrade (15.5.1)
This is complete. All tests are passing, and monitoring is GREEN. Have a great evening!
Jan 10, 21:00-23:19 EST
Potential instability for international text message delivery
This has been resolved.
Jan 10, 11:14-17:05 EST
Routing issue in production environment
This incident has been resolved.
Jan 9, 12:50-21:41 EST
Application update delays
This incident has been resolved.
Jan 9, 17:52-21:41 EST
[Scheduled] Tropo.eu firewall upgrade (Master)
The scheduled maintenance has been completed.
Jan 4, 21:00-23:29 EST
[Scheduled] Global Provisioning API upgrade
The scheduled maintenance has been completed.
Jan 4, 21:00-23:28 EST
[Scheduled] Tropo.eu firewall upgrade
This task was completed for the Slave firewall, but we decided to postpone upgrading the Master for 24hours. We will be posting a new bulletin for the remaining tasks that will be completed on the Master this evening.
Jan 3, 21:00 - Jan 4, 09:19 EST
[Scheduled] Emergency Backend Router Maintenance on Secondary (DR) Datacenter
The scheduled maintenance has been completed.
Jan 4, 00:00-01:00 EST

December 2016

Possible audio quality degradation on some domestic voice calls
Our team has identified the root cause and contacted any users that were impacted. Please let us know if you have any questions. (support@tropo.com).
Dec 19, 15:06 - Dec 23, 17:08 EST
Investigating alerts indicating possible instability at the Provisioning API
This issue has not reoccurred, and we feel confident in the applied correction.
Dec 12, 21:43 - Dec 19, 13:59 EST
Tropo.com Docs are currently down
Incident mistakenly left open beyond actual duration; this was resolved by 12:30pm -est Friday (12/16/2016)
Dec 16, 12:19 - Dec 19, 13:57 EST
Transcription is currently down
This incident has been resolved.
Dec 9, 15:40 - Dec 16, 12:17 EST
[Scheduled] Tropo.com portal upgrade, and deleting legacy environment records
The scheduled maintenance has been completed.
Dec 12, 21:00-23:07 EST
[Scheduled] Shared Production env upgrade, API env cleanup (Shared Prod env), Number formatter correction
The scheduled maintenance has been completed.
Dec 6, 20:00 - Dec 7, 00:59 EST

November 2016

[Scheduled] Provisioning API upgrade, and legacy DNS record removal
Provisioning API upgrade is complete (15.5.1), and all tests are passing.
Nov 29, 21:00-22:24 EST
Degraded service in some runtime nodes
After review of the runtime nodes, a defective customer script was found running in a such a way as to make that app non-responsive. All affected runtime nodes have been inspected, tested and placed back in routing.
Nov 29, 14:49-21:09 EST
Inbound SMS unavailable for staging
This incident has been resolved.
Nov 23, 08:43-09:33 EST
[Scheduled] Development env upgrade, security and cipher tuning, and legacy DNS record removal
The scheduled maintenance has been completed.
Nov 22, 21:00 - Nov 23, 01:04 EST
[Scheduled] API Environment cleanup (Development env), legacy mapping removal
The scheduled maintenance has been completed.
Nov 21, 21:00-22:41 EST
Instability in TTS and ASR overnight
It's been well over 24 hours since the correction was implemented, and we have not seen another occurrence of this failure.
Nov 17, 10:32 - Nov 18, 12:08 EST
[Scheduled] API Environment cleanup, and legacy Dev DNS record removal
The DNS record was removed successfully, but we've postponed the Environment cleanup and may revisit this tomorrow evening (Development Environment only). If we do, a new bulletin will be opened. Please let us know if you have any questions (support@tropo.com).
Nov 16, 21:00-23:26 EST
[Scheduled] Shared Production upgrade, patch speech mgmnt hosts, add new Dedicated Prod api/sip endpoints
The scheduled maintenance has been completed.
Nov 15, 21:01 - Nov 16, 01:13 EST
[Scheduled] Production Call Detail Record engine upgrade
The scheduled maintenance has been completed.
Nov 14, 21:01-22:48 EST
[Scheduled] Dedicated Production environment; runtime upgrade
The scheduled maintenance has been completed.
Nov 8, 21:00 - Nov 9, 08:56 EST
[Scheduled] Shared Production upgrade, and transcription logging enhancement
The Transcription logging upgrade has completed successfully, but we have postponed the Shared Production 15.5.0 upgrade due to the potential impact of an isolated issue. We are grateful for your feedback (we listen :-) )! A new update will follow with details of the next deployment date.
Nov 8, 21:01-22:14 EST
Seeing high failure rates in Eastern datacenter.
This incident has been resolved.
Nov 2, 17:01 - Nov 4, 10:05 EST
Inbound US carrier outage
This incident has been resolved.
Nov 3, 09:12 - Nov 4, 09:32 EST

October 2016

[Scheduled] Shared Production traffic routed to failover datacenter
The scheduled maintenance has been completed.
Oct 25, 21:00 - Oct 26, 03:06 EST
[Scheduled] Development environment upgrade
All set; all Tropo Development environments are now running version 15.5
Oct 24, 17:00 - Oct 25, 14:29 EST
Instability in ProvisioningAPI
We have identified and resolved the underlying issue resulting in this temporary instability. There was a condition where logging queues to be come overloaded, so we've decreased the logging context for the problematic appender, and have not seen this occur since.
Oct 24, 11:47-15:21 EST
[Scheduled] Provisioning API upgrade
The scheduled maintenance has been completed.
Oct 18, 21:01 - Oct 19, 01:19 EST
[Scheduled] Speech recognition upgrade (en-us); Tropo.com Development env
The scheduled maintenance has been completed.
Oct 18, 21:01 - Oct 19, 01:18 EST
[Scheduled] Production logging upgrade & legacy environment deprecation
The scheduled maintenance has been completed.
Oct 4, 21:01 - Oct 5, 02:38 EST
[Scheduled] Speech recognition upgrade; Tropo.eu
The scheduled maintenance has been completed.
Oct 4, 21:01-23:45 EST
Inbound Toll-free carrier outage
Service has been fully restored. Please let us know if you have any questions or concerns (support@tropo.com).
Oct 4, 11:11-13:10 EST

September 2016

[Scheduled] Urgent maintenance; Provider certificate update
The scheduled maintenance has been completed.
Sep 27, 20:01-20:30 EST
[Scheduled] Routing Tropo staging envrionment to secondary data center
Due to an unforeseen vendor issue we been forced to postpone this maintenance for 24 hours. We apologize for any inconvienance.
Sep 21, 20:30-21:16 EST
[Scheduled] Dev environment test runtime security hardening
This is complete
Sep 20, 21:01 - Sep 21, 00:46 EST
[Scheduled] Prod environment Provisioning API patch and gateway security hardening
Sorry for the delay... We are complete here.
Sep 20, 21:01 - Sep 21, 00:46 EST
[Scheduled] Prod environment runtime upgrade
This maintenance is complete, all tests are returning positive results, and all systems are green. Please let us know if you happen to encounter any failures (support@tropo.com).
Sep 14, 21:00 - Sep 15, 01:05 EST
[Scheduled] Provisioning API patch
We have postponed this maintenance for now, and will revisit this item for next weeks window. Apologies for the notification flood; please let us know if you have any questions (support@tropo.com).
Sep 13, 21:01 - Sep 14, 09:06 EST
[Scheduled] Prod environment runtime upgrade
Apologies, the previous update was saved to the wrong bulletin; this maintenance is still on-schedule for Wednesday evening beginning at 9pm -et.
Sep 13, 16:57 EST
[Scheduled] Dev environment ASR upgrade en-us
The scheduled maintenance has been completed.
Sep 12, 11:00-11:15 EST
[Scheduled] Development environment; TTS & ASR upgrade (en-us)...
* We decided to postpose the ASR/TTS upgrade to a later date. Because this change is limited to the Dev env, we may try again this week. Please stay tuned for updates. * The new Dev hostname endpoints have been successfully deployed to the global proxies.
Sep 6, 21:00-23:40 EST
[Scheduled] Production and global environment changes...
This is complete, and all tests are passing.
Sep 6, 22:01-23:36 EST
[Scheduled] Internal tools and monitoring
The scheduled maintenance has been completed.
Sep 6, 22:00-23:30 EST
[Scheduled] Priority maintenance; Implement brand specific sync policy for hosted resources
The scheduled maintenance has been completed.
Sep 1, 22:00-22:08 EST

August 2016

Instability on a single Production runtime
This incident has been resolved.
Aug 25, 15:18 - Aug 26, 17:55 EST
[Scheduled] Urgent test-host configuration change
This change is now complete. The Provisioning API is running at 100% capacity, and no issues have been introduced. Have a great evening!
Aug 23, 23:01-23:25 EST
[Scheduled] Dev environment upgrade, new endpoint hostnames, and portal backend hostname config
This is complete. We did hold-off on deploying the new SIP and API endpoints (Dev env) to allow for additional testing; we will revisit that task during the next maintenance window.
Aug 23, 21:02-22:31 EST
Website issues
This incident has been resolved.
Aug 22, 15:40 - Aug 23, 17:33 EST
Provisioning server issues
This incident has been resolved.
Aug 20, 10:11 - Aug 21, 12:26 EST
[Scheduled] Provisioning API Upgrade, Security patching, transcription API update
The scheduled maintenance has been completed.
Aug 16, 21:00-23:54 EST
[Scheduled] Proxy hardware upgrade & Runtime port configuration
Took just a few minutes longer than anticipated, but we are now complete. Tropo.com is running at 100% capacity, with no present alerts.
Aug 2, 21:00-23:40 EST

July 2016

Application Update Delays
This incident has been resolved.
Jul 28, 08:03-10:48 EST
[Scheduled] Maintenance: US vendor billing system upgrade
The scheduled maintenance has been completed.
Jul 26, 23:45 - Jul 27, 00:30 EST
Transcription issues
This incident has been resolved.
Jul 25, 19:15 - Jul 26, 23:17 EST
[Scheduled] Emergency maintenance on speech servers
The scheduled maintenance has been completed.
Jul 22, 23:00 - Jul 23, 07:00 EST
[Scheduled] Transcription proxy logging upgrade
The scheduled maintenance has been completed.
Jul 19, 21:00 - Jul 20, 08:49 EST
[Scheduled] Billing provider system upgrade
The scheduled maintenance has been completed.
Jul 18, 22:30 - Jul 19, 01:19 EST
Issue affecting the proper routing of calls to newly created apps
Since applying the correction there have been no further occurrences of this issue.
Jul 13, 14:44 - Jul 14, 08:55 EST
[Scheduled] Scheduled Maintenance: U.S number provider
The scheduled maintenance has been completed.
Jul 12, 22:00 - Jul 13, 02:01 EST
Slow asset fetches from webhosting
The root cause has been identified and corrected, in addition to the upgraded hardware.
Jun 15, 17:49 - Jul 8, 18:32 EST
Single runtime failure in hosted
We've identified the root cause, and are adding triggers to alert in the event this occurs again.
Jun 29, 12:41 - Jul 8, 18:31 EST
[Scheduled] Gateway memory allocation and additions to global routing proxy
The scheduled maintenance has been completed.
Jul 5, 21:00 - Jul 8, 18:25 EST
[Scheduled] Maintenance: Israel
The scheduled maintenance has been completed.
Jul 5, 02:00-02:00 EST

June 2016

Single runtime failure in hosted
This incident has been resolved.
Jun 22, 11:29-17:42 EST
Investigating reports of recording upload failures
This issue was resolve by rolling back the build. We have fixed the issue in build 15.3.3 which is in our development environment. If you have any questions please don't hesitate to let us know.
Jun 8, 09:36 - Jun 22, 11:30 EST
Issue mapping addresses to applications
This issue was caused by a the addition of new address capabilities for a soon to be announced expansion. This new option was enabled by default and our integration testing unfortunately did not catch the problem. Our engineering team quickly identified the issue and was deployed a patch to resolve it. We have seen no further indications or reports of trouble since that patch was deployed. We are also performing an internal review to see how this issue was missed in order to ensure it doesn't happen again. If there are any questions please don't hesitate to let us know ( support@tropo.com ).
Jun 7, 15:09 - Jun 9, 15:42 EST
[Scheduled] Production runtime upgrade
The scheduled maintenance has been completed.
Jun 7, 21:01 - Jun 8, 02:20 EST
[Scheduled] Maintenance: International SMS
The scheduled maintenance has been completed.
Jun 7, 01:00-14:00 EST
SMS callerID formatting issues identified
This incident has been resolved.
Jun 1, 11:14 - Jun 2, 05:26 EST
[Scheduled] Provisioning API upgrade
The scheduled maintenance has been completed.
Jun 1, 22:01 - Jun 2, 01:01 EST

May 2016

[Scheduled] Transcription proxy config change
The scheduled maintenance has been completed.
May 26, 21:00-21:12 EST
[Scheduled] Provisioning API update & logging correction to Production
The scheduled maintenance has been completed.
May 24, 21:00 - May 25, 00:27 EST
SMS CallerID validation issues
This incident has been resolved.
May 19, 10:39 - May 20, 10:56 EST
[Scheduled] Production transcription upgrade & callerId validation fix
The scheduled maintenance has been completed.
May 17, 21:00 - May 18, 00:44 EST
[Scheduled] Emergency maintenance, Development env
The scheduled maintenance has been completed.
May 17, 21:00-22:14 EST
Investigating reports of ftp instability
This incident has been resolved.
May 16, 16:24-21:56 EST
[Scheduled] Emergency provider config update
The scheduled maintenance has been completed.
May 11, 22:00 - May 12, 00:37 EST
[Scheduled] Provider portal migration
The scheduled maintenance has been completed.
May 10, 22:00 - May 11, 02:00 EST
[Scheduled] Maintenance for production runtimes' webhosting
The scheduled maintenance has been completed.
May 10, 21:00 - May 11, 01:00 EST
[Scheduled] Tropo.com portal upgrade
The scheduled maintenance has been completed.
May 10, 22:00-22:35 EST
Transcription Proxy issues
This incident has been resolved.
May 10, 11:53-11:55 EST
Provisioning API reported errors
This has been resolved following a drive replacement. Please let us know if you have any questions (support@tropo.com).
May 9, 17:32 - May 10, 10:11 EST
Provisioning API reported errors
This issue is now resolved. A block store volume failed, and accordingly the Provisioning systems failed over to a readonly mode. We have recovered the volume, and validated that all services have been restored. If you have any questions please feel free to reach out to our team (support@tropo.com).
May 9, 15:37-16:23 EST
Potential outbound delivery failures with short code text message service in U.S.
This incident has been resolved.
May 5, 10:24 - May 9, 10:31 EST
Provisioing API Issues
This incident has been resolved.
May 3, 13:10 - May 9, 10:27 EST
Media Server instabilites
This incident has been resolved.
May 5, 12:34 - May 9, 10:27 EST
Media server instability
This has been resolved, and have not seen any instability since this bulletin was originally posted. Please let us know if you have any questions (support@tropo.com).
May 4, 16:31 - May 5, 10:19 EST
[Scheduled] Production Runtime Upgrade
The scheduled maintenance has been completed.
May 3, 21:30 - May 4, 02:00 EST

April 2016

Tropo portal application view limitation
We have deployed the correction, and confirmed the issue has been resolved. Please let us know if you have any questions or concerns (support@tropo.com).
Apr 29, 10:46-11:26 EST
Media server restart
This incident has been resolved.
Apr 20, 14:09 - Apr 28, 13:06 EST
[Scheduled] Dev env patch, Global proxy patch and service discovery upgrade
Updates have completed, and all tests are yielding successful results. This maintenance is complete. Please let us know if you have any questions (support@tropo.com).
Apr 26, 21:00-23:24 EST
[Scheduled] Maintenance: Belgium
The scheduled maintenance has been completed.
Apr 26, 15:00-19:00 EST
Inbound SMS outage: UK
This issue is now resolved and inbound traffic on UK geographic DIDs are operational again. The problem started on 25 April 2016, at 18:00 GMT and resolved today 26 April 2016. If you are experiencing any trouble with UK numbers please open a ticket with Tropo support at (support@tropo.com) and a member of our team will assess if the issue is related. Thank you
Apr 25, 14:56 - Apr 26, 15:34 EST
[Scheduled] Maintenance: Sweden Inbound Voice
The scheduled maintenance has been completed.
Apr 24, 18:00-23:00 EST
[Scheduled] Emergency vendor patch for text-to-speech software
The patch has been applied successfully to the text-to-speech servers and no downtime has been recorded. This maintenance window is now complete. As usual, if you have any questions, feel free to reach out to support@tropo.com. Thanks! –Tropo Operations
Apr 20, 22:00-23:40 EST
[Scheduled] Emergency Maintenance: Israel
The scheduled maintenance has been completed.
Apr 20, 21:00-22:00 EST
Single Media Server Restart
This server was removed from routing and inspected for errors. It has since been passing all tests and has been placed back into routing.
Apr 19, 15:42 - Apr 20, 14:12 EST
[Scheduled] Platform Maintenance
The scheduled maintenance has been completed.
Apr 19, 22:01-22:57 EST
[Scheduled] Emergency addressing change for teir 3 outbound carrier
The scheduled maintenance has been completed.
Apr 13, 08:30-13:58 EST
[Scheduled] Upgrading api.tropo.com
The scheduled maintenance has been completed.
Apr 12, 21:00-23:07 EST
[Scheduled] Maintenance: Switzerland Inbound Voice
The scheduled maintenance has been completed.
Apr 11, 20:00 - Apr 12, 11:00 EST
Development environment: Media server instability
This has been resolved, and no further issues have been seen for over 5-hours.
Apr 11, 11:54-18:09 EST
[Scheduled] Critical maintenance: Production runtime environment patch
This maintenance is complete. All tests have passed, and the Production environment is running at 100% capacity.
Apr 6, 21:00-23:09 EST
[Scheduled] Website patch this evening
The scheduled maintenance has been completed.
Apr 6, 21:45-22:15 EST
Issues editing applications via portal
This incident has been resolved.
Apr 2, 14:26-17:31 EST
[Scheduled] Billing provider migration
We apologize for the delay on this maintenance. At this time all work has been completed. If you have any questions please don't hesitate to let us know - support@tropo.com
Apr 1, 19:14 - Apr 2, 02:22 EST

March 2016

[Scheduled] Scheduled Maintenance: US SMS
The scheduled maintenance has been completed.
Mar 31, 01:00-04:00 EST
[Scheduled] Proxy configuration update
The scheduled maintenance has been completed.
Mar 29, 21:00-23:00 EST
[Scheduled] Maintenance: Belgium
The scheduled maintenance has been completed.
Mar 24, 16:00-19:00 EST
[Scheduled] Routine database maintenance
The scheduled maintenance has been completed.
Mar 23, 22:00-22:53 EST
[Scheduled] Scheduled Maintenance: International SMS
The scheduled maintenance has been completed.
Mar 23, 03:00-14:00 EST
Issues connecting to the support portal
Our provider has corrected the issue within their network, and service has been restored to the support.tropo.com portal. Please let us know if you have any questions (support@tropo.com).
Mar 21, 10:13-11:14 EST
[Scheduled] Vendor Routing Upgrade Maintenance
The scheduled maintenance has been completed.
Mar 18, 22:00 - Mar 19, 08:01 EST
Investigating reports of failures when adding a phone number to a Prod app or moving a Dev app to Prod
We've seen no evidence of this issue having occurred since the correction was applied yesterday, and are considering the incident resolved. Please feel free to contact our team if you have any questions (support@tropo.com).
Mar 17, 09:29 - Mar 18, 09:49 EST
[Scheduled] Provider migration: Production environment, U.S. phone numbers
The scheduled maintenance has been completed.
Mar 17, 21:00 - Mar 18, 03:00 EST
[Scheduled] Provisioning API update
The scheduled maintenance has been completed.
Mar 16, 21:00-23:00 EST
[Scheduled] Provider migration: Development environment, U.S. phone numbers
The scheduled maintenance has been completed.
Mar 15, 21:00 - Mar 16, 00:00 EST
[Scheduled] Production environment config change
The scheduled maintenance has been completed.
Mar 15, 21:01-23:00 EST
[Scheduled] Emergency Maintenance: United Kingdom
The scheduled maintenance has been completed.
Mar 15, 18:30-20:30 EST
[Scheduled] Portal update and Development environment config change
The scheduled maintenance has been completed.
Mar 14, 09:00-10:00 EST
[Scheduled] Production cluster maintenance
The scheduled maintenance has been postponed. The work was canceled and will be rescheduled.
Mar 8, 21:30 - Mar 9, 01:00 EST
Speech recognition failures in Development env
This has been resolved. We identified a license key that was not applied correctly, and have patched the environment. Internal testing is returning 100% success, but please let us know if you encounter any issues (support@tropo.com).
Mar 8, 16:07-17:23 EST
[Scheduled] Development environment upgrade
The scheduled maintenance has been completed.
Mar 7, 15:00-16:00 EST
[Scheduled] Emergency Maintenance: International SMS
The scheduled maintenance has been completed.
Mar 4, 03:00-17:00 EST
[Scheduled] Maintenance: Switzerland Inbound Voice
The scheduled maintenance has been completed.
Mar 1, 21:00 - Mar 2, 00:00 EST

February 2016

[Scheduled] Development environment patch to address recording failures in a specific scenario.
The scheduled maintenance has been completed.
Feb 29, 11:30-12:00 EST
[Scheduled] Security Maintenance (day 4)
Security maintenance completed.
Feb 23, 00:21 EST
[Scheduled] Security maintenance ( day 3 )
The scheduled maintenance has been completed.
Feb 19, 21:30 - Feb 20, 01:30 EST
[Scheduled] Security maintenance ( day 2 )
The scheduled maintenance has been completed.
Feb 18, 21:30 - Feb 19, 01:30 EST
Tropo API experiencing inconsistent service
A credential change for one of our vendor connections occured and during this change, an issue arose with our gateways which ended up disruption token sessions for a portion of our users. The issue has been identified and is currently resolved.
Feb 18, 16:45-16:56 EST
[Scheduled] Security maintenance ( day 1 )
The scheduled maintenance has been completed.
Feb 17, 21:30 - Feb 18, 01:30 EST
[Scheduled] Production gateway upgrades
The scheduled maintenance has been completed.
Feb 16, 21:30 - Feb 17, 01:00 EST
Webhosting issue in Virginia data center ( resolved )
This incident has been resolved.
Feb 15, 11:49-17:32 EST
[Scheduled] Drive replacement on secondary file hosting server in Orlando data center
The scheduled maintenance has been completed.
Feb 12, 21:00 - Feb 13, 09:00 EST
[Scheduled] Production runtime upgrade
The update completed without any issues. Please let us know if you have questions or run into any troubles. (support@tropo.com)
Feb 10, 21:30-23:06 EST
[Scheduled] Development runtime hosting maintenance
The scheduled maintenance was completed this afternoon without issue. If there are any questions please let us know (support@tropo.com).
Feb 9, 14:57-21:48 EST
[Scheduled] Portal update: security and international rates
This scheduled maintenance was completed successfully.
Feb 5, 15:04 EST
[Scheduled] Emergency Maintenance: Netherlands
The scheduled maintenance has been completed.
Feb 3, 18:00-22:00 EST
[Scheduled] Emergency Maintenance: International SMS
The scheduled maintenance has been completed.
Feb 3, 03:00-07:00 EST
[Scheduled] Hosted scheduled maintenance ( developer cluster only )
The scheduled maintenance has been completed.
Feb 2, 08:30-12:00 EST

January 2016

[Scheduled] Failover Route Change
The scheduled maintenance has been completed.
Jan 28, 21:30-23:30 EST
Developement Platform Instability Issues
This incident has been resolved.
Jan 28, 17:47-21:25 EST
[Scheduled] Hosted scheduled maintenance
The proxy upgrade has been completed and all tests are passing. The SMS vendor peering change was postponed until a later date. If there are any questions please let us know - support@tropo.com
Jan 27, 21:30-22:53 EST
[Scheduled] Hosted scheduled maintenance
This maintenance work has been completed. Unfortunately the following two items were post postponed. * Tropo Operations will be deploying a new version of Tropo's global routing proxy to address edge cases with call forwarded numbers. * Tropo Operations will be changing the default destination for all traffic from Orlando to Virginia data center.
Jan 26, 21:30-23:39 EST
Production data centers issues
Our data center vendor has patched all switches in the data center and we have no seen any additional occurrences of this issue. We will no close this issue as as resolved.
Jan 15, 22:38 - Jan 24, 12:41 EST
Outbound token launches reporting failues
This incident has been resolved.
Jan 23, 14:27 - Jan 24, 12:40 EST
[Scheduled] Emergency Maintenance: Panama Inbound Voice
The scheduled maintenance has been completed.
Jan 22, 23:30 - Jan 23, 01:30 EST
[Scheduled] Emergency maintenance on file replication service in Orlando data center
The scheduled maintenance has been completed.
Jan 20, 22:00 - Jan 21, 03:00 EST
[Scheduled] Moving inbound international SMS to new data center
The scheduled maintenance has been completed.
Jan 19, 21:00-23:02 EST
[Scheduled] Emergency switch maintenance tonight
The scheduled maintenance has been completed.
Jan 19, 00:00-02:00 EST
Potential memory failure
We have replaced the memory in the failing server and at it has been put back into routing.
Jan 15, 18:22-21:39 EST
File replication delays in Orlando
Primary file replication agent was restarted and all files in Orlando are now up to date.
Jan 15, 08:47-08:53 EST
Brief Network Backend Instability
This incident has been resolved.
Jan 14, 21:52 - Jan 15, 08:35 EST
[Scheduled] Production data center migration
The scheduled maintenance has been completed.
Jan 12, 22:31 - Jan 13, 04:30 EST
[Scheduled] Network change in development environment
The scheduled maintenance has been completed.
Jan 7, 22:00-23:00 EST
TTS license issue
Earlier today we experienced TTS license issue with one of our runtime in development environment. Server was immediately taken out which operations will review later. This issue only impacted the development environment, production was not affected.
Jan 3, 12:15 EST

December 2015

Phono product public access and refactoring
In order to better align Phono with our overall offering, we are removing public access to the project as of today, 12/22/2015. It will be back, and with more functionality integrated into both Tropo and other Cisco products. Please watch both the Tropo and DevNet blogs (https://blog.tropo.com and https://communities.cisco.com/community/developer/blog for more information on the release of the new and improved Phono, and email support@tropo.com if you have any questions. Note that existing applications utilizing Phono will remain active and live for the time being, as the gateways are not going to be turned down yet, but new applications will not be possible until the product is relaunched. If you have a production application using Phono now and want to discuss options moving forward, please email support@tropo.com as soon as possible. If you want to learn more about our Products & APIs, please check out Tropo.com (https://tropo.com) and Cisco Spark for Developers (https://developer.ciscospark.com).
Dec 22, 16:24 EST
External Vendor Issue
We have seen no further issues with this carrier for over 24 hours. We will be closing this issue as resolved. If there are any questions please let us know ( support@tropo.com ).
Dec 15, 15:00 - Dec 16, 21:37 EST
Detected issues with SMS traffic
This incident has been resolved.
Dec 3, 12:18 - Dec 16, 21:36 EST
[Scheduled] Maintenance: Germany Inbound Voice
The scheduled maintenance has been completed.
Dec 14, 18:00 - Dec 15, 00:00 EST
[Scheduled] Emergency Maintenance: Switzerland Inbound Voice
The scheduled maintenance has been completed.
Dec 9, 18:00 - Dec 10, 01:00 EST
New Account Balance Alerts activation
The account balance alerts referenced on 11/10 will be activated tonight at 11 p.m. EST - deployment was intentionally delayed a few weeks to resolve a couple account specific issues. Please let us know if you have any questions prior by emailing support@tropo.com.
Dec 1, 16:18 EST

November 2015

[Scheduled] Firewall maintenance in developer cluster.
The scheduled maintenance has been completed.
Nov 25, 21:30 - Nov 26, 01:00 EST
Intermittent packet loss in Orlando data center
After investigating in tandem with our datacenter provider, a latency issue with a carrier was found in the Orlando location, upstream from the datacenter itself. The issue looks to be resolved at this time.
Nov 24, 09:37-17:53 EST
Vendor issues for Canadian SMS numbers
We have confirmed that outbound messages during this period were unfortunately dropped. Our Engineering team is investigating ways to be better queue outbound SMS messages so we can be more tolerant of these types of issues in the future. If there are any questions please don't hesitate to let us know - support@tropo.com
Nov 21, 09:59 - Nov 22, 21:42 EST
[Scheduled] Firewall maintenance in developer cluster.
The scheduled maintenance has been completed.
Nov 21, 21:00 - Nov 22, 01:00 EST
Vendor made changes that affects some inbound voices numbers
The vendor identified root cause as an incorrect "calls per second" setting at the trunk level, which caused a large portion of inbound U.S. calls to fail due to hard limiting call concurrency. This setting is fixed moving forward.
Nov 18, 07:03 - Nov 21, 12:39 EST
[Scheduled] Emergency VPN maintenance in staging environment
The scheduled maintenance has been completed.
Nov 19, 11:30-12:30 EST
[Scheduled] Staging firewall maintenance
Tonights maintenance has been rescheduled. We will post an update shortly with the new time.
Nov 18, 21:30-23:16 EST
Tropo.com Developer Portal Updated
We just updated the Tropo portal to add several long standing requested enhancements, with more to come in the near future: * Previously if a developer attempted to save a file without an extension, they would receive an error, but could not then re-save with an extension - had to back out and start all over. This is fixed, you can now just add the file extension and the error will resolve. * If a number pool is out of numbers, previously it would just cycle and display no information as to why the number could not be added. A message will now display indicating the number pool is depleted. If you need a number in that region, please contact support to request one. * You can now search for a number in the number pool by region or area code, instead of scrolling through the entire list. * Sorting of application numbers by country or region is now available. * Icons for files in the File Manager are now unique per file extension. Note that to reflect the new changes, you'll need to clear your cache so any historical records of Tropo.com are removed; we will be fixing that as well in an upcoming version, so cache will clear on it's own.
Nov 18, 12:53 EST
[Scheduled] Provisioning server and staging upgrade
The scheduled maintenance has been completed.
Nov 17, 22:00 - Nov 18, 01:00 EST
File replication delays
This incident has been resolved.
Nov 17, 13:02-22:03 EST
Staging data center issues
All staging services are now recovered. The problem appears to have been a failure of the active router and the secondary not correctly assuming control of routing. Tropo Operations manually forced the failover and services immediately recovered. We will be reaching out to the firewall vendor to better understand why the service did not correctly failover and manual intervention was required. If there are any questions please let us know ( support@tropo.com ).
Nov 15, 05:22-05:45 EST
[Scheduled] New Account Balance Alerts
The scheduled maintenance has been completed.
Nov 10, 23:00 - Nov 11, 02:00 EST
Billing provider delays
Issue is now resolved; Oracle database issue that is fully recovered.
Nov 10, 14:04-15:33 EST
Service Impairment: Inbound Toll Free
Issue Resolved- Our toll free provider identified an issue with call origination routing and was able to resolve it. As of today 11-10-2015 09:42 EST, the problem is now confirmed as fixed. We will continue to monitor to ensure stability. If you are still experiencing any trouble with inbound toll free calls, please open a ticket with our support team (support@tropo.com).
Nov 9, 17:21 - Nov 10, 14:01 EST
[Scheduled] Updating support portal security certificate
The scheduled maintenance has been completed.
Nov 4, 21:00-23:00 EST

October 2015

[Scheduled] Website maintenance
The scheduled maintenance has been completed.
Oct 23, 23:15 - Oct 24, 06:00 EST
[Scheduled] Billing system maintenance
This maintenance has been completed and all post deployment tests have passed. Thank you very much for your patience during this upgrade.
Oct 21, 20:00-20:44 EST
SMS delays in both production and development
This incident has been resolved.
Oct 20, 10:44-15:21 EST
Sip proxy issues for inbound calling
The proxy has been rebuilt,tested, and returned to routing. We are closing this incident as resolved. If there are any questions please let us know - support@tropo.com
Oct 20, 11:46-15:02 EST
Outbound SMS delivery issues from developer environment
Tropo Operations uncovered a configuration issue on the SMS gateways which was introduced during this mornings upgrade. This issue likely caused problems with outbound SMS delivery from the free development environment and was fixed at 22:00 UTC. This issue did NOT impact the production environment at all. If there are any questions please let us know - support@tropo.com
Oct 19, 17:26 EST
[Scheduled] Developer platform upgrade
The scheduled maintenance has been completed.
Oct 19, 08:00-15:00 EST
[Scheduled] Support Portal Update
The scheduled maintenance has been completed.
Oct 18, 12:01-14:00 EST
Inbound Service Impairment- WV, VA & PA Only
Thank you for your patience. Our underlying carrier rebuilt their databases that lead to resolution. As of this morning (4:30am EST) inbound services for WV, VA & PA have been restored. Please contact Tropo support if you are in need of assistance. (support@tropo.com)
Oct 14, 20:08 - Oct 15, 10:54 EST
[Scheduled] Emergency Maintenance: Norway
The scheduled maintenance has been completed.
Oct 11, 20:00-21:00 EST
[Scheduled] Orlando data center route away.
The scheduled maintenance has been completed.
Oct 9, 23:00 - Oct 10, 07:00 EST
[Scheduled] Scheduled Maintenance: Ireland
The scheduled maintenance has been completed.
Oct 9, 19:00 - Oct 10, 01:00 EST
[Scheduled] Route away from Orlando data center
The scheduled maintenance has been completed.
Oct 9, 11:04-11:04 EST
Inbound failures on Staging
This issue has been resolved. It was caused by a platform misconfiguration and only impacted inbound voice traffic to the free developer platform. Production was not effected by this issue at all. The issue has been resolved and please let us know if there are any questions - support@tropo.com
Oct 7, 08:52-10:27 EST
[Scheduled] Weekly Maintenance
The scheduled maintenance has been completed.
Oct 6, 21:00 - Oct 7, 01:00 EST
Conference creation issues
This incident has been resolved.
Oct 5, 16:13-19:58 EST
ASR issues
The ASR servers are back up and running normally.
Oct 4, 15:08-15:15 EST
Users unable to make application changes
This incident has been resolved.
Oct 1, 09:41-22:59 EST

September 2015

[Scheduled] Scheduled Maintenance: Norway Numbers
The scheduled maintenance has been completed.
Sep 27, 18:00 - Sep 28, 00:00 EST
[Scheduled] Emergency Maintenance: SMS Canada Only
The scheduled maintenance has been completed.
Sep 16, 04:00-06:00 EST
Errors moving applications to production
This incident has been resolved.
Sep 11, 15:48 - Sep 12, 09:41 EST
Inbound call setup delays
This incident has been resolved.
Sep 4, 20:03-21:46 EST
Transcription issues
This incident has been resolved.
Sep 3, 13:39-16:55 EST
[Scheduled] Maintenance in staging network
The scheduled maintenance has been completed.
Sep 2, 22:00 - Sep 3, 01:00 EST

August 2015

Webhosting connectivity issues causing file replication delays
This incident has been resolved.
Aug 28, 11:51-13:48 EST
Local Market Impairment - 720 - Reno, NV only
Carrier reports fiber cut is fixed and services are restored.
Aug 18, 11:52 - Aug 19, 10:50 EST
[Scheduled] Route away from Orlando data center
The scheduled maintenance has been completed.
Aug 14, 22:45 - Aug 15, 05:00 EST
Transcription API issues
The vendor issue has been resolved and all messages sent during their outage should have been delivered at this time. We apologize for the delay in transcription delivery and if there are any questions please don't hesitate to let us know - support@tropo.com
Aug 10, 11:51-16:36 EST

July 2015

[Scheduled] Emergency Maintenance: Israel
The scheduled maintenance has been completed.
Jul 29, 15:01-18:00 EST
[Scheduled] Production Maintenance
The scheduled maintenance has been completed.
Jul 28, 21:30-22:14 EST
[Scheduled] Hosted Maintenance for staging network
The scheduled maintenance has been completed.
Jul 28, 11:00-16:00 EST
[Scheduled] Maintenance for Canadian SMS traffic
The scheduled maintenance has been completed.
Jul 16, 08:00-17:00 EST
Inbound Voice Impairment: Select Markets Only
Update: Our provider has repaired the fiber. Inbound services have remained stable. We are closing this incident at this time, however please contact Tropo support (support@tropo.com) if you are in need of assistance.
Jul 13, 19:47 - Jul 14, 13:43 EST
[Scheduled] Vendor change for inbound Canadian voice numbers
The scheduled maintenance has been completed.
Jul 13, 06:30-08:00 EST
[Scheduled] Emergency Maintenance: Sweden
The scheduled maintenance has been completed.
Jul 11, 03:00-08:00 EST
SMS delivery delays
This incident has been resolved.
Jul 7, 08:47 - Jul 8, 16:09 EST
ASR server issues
This incident has been resolved.
Jul 7, 20:17-21:46 EST
[Scheduled] Carrier routing change
The scheduled maintenance has been completed.
Jul 7, 06:00-08:30 EST
[Scheduled] Billing maintenance
The scheduled maintenance has been completed.
Jul 6, 22:00 - Jul 7, 01:00 EST

June 2015

[Scheduled] Weekly Maintenance
Due to an unforeseen issue with our vendor we have been forced to cancel this planned maintenance item. We hope to have a new date very soon, and will update status.tropo.com as soon as it's available. If there are any questions please let us know - support@tropo.com
Jun 30, 23:39 EST
[Scheduled] Additional carrier migrations
Due to an scheduling issue with one of our vendors we have had to push this maintenance item back at week. Currently we have a tentative date of July 1st, and as soon as this is confirmed we will post a new maintenance notification to http://status.tropo.com. If there are any questions please let us know - support@tropo.com
Jun 24, 22:17 EST
[Scheduled] Weekly maintenance
The scheduled maintenance has been completed.
Jun 23, 22:00 - Jun 24, 01:01 EST
Webhosting issues
Tropo Operations isolated the high load on the web hosting servers to a corruption of the file replication sync database. This corruption was causing the replication process to go into a tight loop which triggered the high load. This issue was limited to the state databases for Tropo's secondary data centers and once our Operations team forced a re-sync load quickly subsided. The re-sync has now finished and load is now running within expected norms. If there are any questions please let us know - support@tropo.com
Jun 22, 12:58-23:13 EST
Web hosting replication delay
This incident has been resolved.
Jun 22, 07:10-09:49 EST
Receiving reports of voice and SMS failure
This incident has been resolved.
Jun 20, 02:19 - Jun 21, 20:49 EST
[Scheduled] Scheduled Maintenance: Hong Kong
The scheduled maintenance has been completed.
Jun 17, 14:00-16:00 EST
[Scheduled] Hosted maintenance
The scheduled maintenance has been completed.
Jun 16, 22:00 - Jun 17, 01:00 EST
[Scheduled] Route away from Orlando data center
The scheduled maintenance has been completed.
Jun 12, 23:00 - Jun 13, 06:00 EST
[Scheduled] Emergency Maintenance: Netherlands
The scheduled maintenance has been completed.
Jun 10, 21:00 - Jun 11, 00:01 EST
conference replication issues
This incident has been resolved.
Jun 9, 12:57-16:04 EST
[Scheduled] Emergency Maintenance: Netherlands
The scheduled maintenance has been completed.
Jun 5, 21:00 - Jun 6, 00:00 EST
[Scheduled] Scheduled maintenance
The scheduled maintenance has been completed.
Jun 2, 21:30 - Jun 3, 03:00 EST
[Scheduled] Carrier maintenace German DIDs and UK Inbound SMS
The scheduled maintenance has been completed.
May 29, 14:00 - Jun 1, 02:00 EST

May 2015

[Scheduled] Carrier maintenace German DIDs and UK Inbound SMS
The scheduled maintenance has been completed.
May 26, 14:00 - May 29, 14:00 EST
[Scheduled] Vendor changes for inbound international termination.
The scheduled maintenance has been completed.
May 27, 06:15-08:15 EST
[Scheduled] Provisioning API upgrade
The scheduled maintenance has been completed.
May 26, 21:30 - May 27, 01:00 EST
[Scheduled] Maintenance: Argentina
The scheduled maintenance has been completed.
May 21, 23:00 - May 22, 02:00 EST
[Scheduled] Scheduled Maintenance: Argentina
The scheduled maintenance has been completed.
May 20, 23:00 - May 21, 00:00 EST
[Scheduled] Hosted maintenance
The scheduled maintenance has been completed.
May 19, 22:30 - May 20, 01:00 EST
Provisioning API issues
The server has been restarted and all services and tests are green and passing at this time. We have opened a ticket with product engineering to investigate for a root cause but in the meantime Operations has put the server back into routing.
May 19, 09:31-09:39 EST
Support portal down
This incident has been resolved.
May 14, 23:08 - May 15, 10:25 EST
[Scheduled] Hosted maintenance next week
The scheduled maintenance has been completed.
May 12, 21:30 - May 13, 01:00 EST

April 2015

[Scheduled] Hosted maintenance next week
The scheduled maintenance has been completed.
Apr 28, 21:00-22:57 EST
Issues with foreign language speech recognition servers
We have identified an issue with a subset of our speech recognition servers. The problematic servers have been removed from routing and at this point tests are green. We will continue to monitor the situation and if there are any questions please let us know ( support@tropo.com ).
Apr 22, 19:37-20:51 EST
Orlando Datacenter issue ( resolved )
This incident has been resolved.
Apr 20, 02:38 - Apr 22, 19:38 EST
Possible conference issues in shared production
This incident has been resolved.
Apr 14, 12:47-15:23 EST
[Scheduled] Upgrade for HAProxy and OpenSSL
This update has been completed
Apr 7, 08:27 EST
Phono registration down
The issue is related to the user registration part. If the users are not meeting the password complexity rules, the registration page doesn't act properly. We'll fix the bug.
Apr 3, 20:03-21:37 EST
Tropo.com has an internal issue
We have gotten the data we need for our investigation and have already rolled back the update for both load balancers. We will continue to monitor them but we expect no further issues. If there are any questions please don't hesitate to let us know ( support@tropo.com ).
Apr 2, 04:46-11:33 EST

March 2015

[Scheduled] Updates to API servers
The scheduled maintenance has been completed.
Mar 31, 22:00-22:17 EST
[Scheduled] Update SSL certificate chain for ftp.tropo.com
The scheduled maintenance has been completed.
Mar 26, 23:00-23:02 EST
[Scheduled] Portal and Provisioning upgrade Wednesday evening
The scheduled maintenance has been completed.
Mar 25, 21:30-21:48 EST
SMS Billing fix
We identified an issue where large SMS messages (those over 160 characters, typically, with variance for multi-byte characters and the like) were being billed as one SMS instead of multiple SMS as intended. This issue will be fixed as of midnight GMT 3/26, so SMS will be billed properly moving forward, with no retroactive billing.
Mar 25, 12:44 EST
[Scheduled] Service Notification: Planned Maintenance in the United Kingdom
The scheduled maintenance has been completed.
Mar 21, 18:00 - Mar 22, 02:01 EST
[Scheduled] Service Notification: Planned Maintenance in the United Kingdom
The scheduled maintenance has been completed.
Mar 17, 17:00 - Mar 19, 02:00 EST
Portal issues
This incident has been resolved.
Mar 11, 22:26-23:30 EST
[Scheduled] Production maintenance
The scheduled maintenance has been completed.
Mar 10, 22:00 - Mar 11, 01:00 EST
[Scheduled] Update to https://api.tropo.com to enable CORS
The scheduled maintenance has been completed.
Mar 3, 22:00-22:37 EST

February 2015

Media server on a single runtime crashed, already recovered
This incident has been resolved.
Feb 27, 11:38-14:45 EST
Provisioning Outage
Our database server VM crashed earlier this afternoon at around 5:30pm EST. The outage lasted until around 5:50pm EST.
Feb 21, 18:30 EST
New build in development environment
Please test all applications, production and development, against the new build to verify no issues are present. There have been a few build push attempts recently, so the list of changes to validate are split up among multiple listings on changes.tropo.com - you will want to validate against all of them: http://changes.tropo.com/post/111316904195/build-14-2-16-bugs-fixed-when-connecting-two http://changes.tropo.com/post/110075894780/build-14-2-15-bugs-fixed-if-the-same-signal-name http://changes.tropo.com/post/105354773190/build-14-2-12-bugs-fixed-python-applications http://changes.tropo.com/post/103641957300/build-14-2-10-bugs-fixed-route-header-missing Please email support@tropo.com with any questions or issues.
Feb 18, 12:04 EST
Scheduled Maintenance: United States
Update: There should be no impact to calls or call quality during this maintenance. Please contact us if there are any questions.
Feb 17, 11:26-15:53 EST
Provisioning issues
This incident has been resolved.
Feb 12, 15:03-18:55 EST
[Scheduled] Transfer agent update
The scheduled maintenance has been completed.
Feb 10, 22:00 - Feb 11, 01:00 EST
Scheduled Maintenance: Argentina
This incident has been resolved.
Feb 9, 15:40-19:56 EST
Scheduled Maintenance: Estonia
This incident has been resolved.
Feb 9, 15:47-19:56 EST
[Scheduled] Security patches of production servers
The scheduled maintenance has been completed.
Feb 4, 19:00 - Feb 5, 01:00 EST
[Scheduled] Portal and Provisioning API updates
The scheduled maintenance has been completed.
Feb 3, 22:01 - Feb 4, 01:00 EST
[Scheduled] Carrier change for staging outbound traffic
The scheduled maintenance has been completed.
Feb 3, 10:00-12:00 EST
Transcription issues
Issues with transcription have now been resolved and transcription is back online.
Feb 2, 12:23-13:26 EST

January 2015

[Scheduled] Orlando data center maintenance
The scheduled maintenance has been completed.
Jan 24, 01:06-05:00 EST
Transcription delays
Transcription provider indicates back to normal, please email support@tropo.com if you notice any further trouble.
Jan 22, 12:53-13:34 EST
[Scheduled] Tropo.com website update
Encountered some issues during the upgrade of Tropo.com. The upgrade has been rolled back and Tropo.com is back and fully operational. Maintenance for this evening (Wed, Jan 21 EST) has been completed.
Jan 21, 21:00-22:07 EST
[Scheduled] Token router maintenance release
All maintenance as been completed and post deployment tests are green. If there are any questions please don't hesitate to let us know ( support@tropo.com ).
Jan 20, 22:00-22:42 EST
SMS Issues
After an exhaustive investigation we have confirmed the reported problem was limited to a batch of a few dozen sms addresses. These numbers were incorrectly mapped in our provisioning database which caused them to fail when mapped to users applications. We have corrected the issue and have taken steps to ensure the issue does not happen again.
Jan 15, 14:47-21:25 EST

December 2014

Detected API sluggishness
This incident has been resolved.
Dec 26, 12:25 - Dec 27, 14:31 EST
Media server issues
This incident has been resolved.
Dec 22, 15:51-16:53 EST
New build in development environment
Please test all applications, production and development, against the new build to verify no issues are present. Changes are listed here: http://changes.tropo.com/post/105354773190/build-14-2-12-bugs-fixed-python-applications If you did not test recently, the changes shown here may also apply: http://changes.tropo.com/post/103641957300/build-14-2-10-bugs-fixed-route-header-missing Please email support@tropo.com with any questions or issues.
Dec 22, 13:39 EST
Media server issues
This incident has been resolved.
Dec 19, 10:13-16:32 EST
Emergency Maintenance: Brazil
Emergency maintenance has been completed for Brazil. If you are still experiencing issues and are in need of assistance, please open a support ticket (support@tropo.com) and we will assess if your issue is related. Thank you.
Dec 12, 17:15 - Dec 15, 10:30 EST
Media server issue on single runtime
This incident has been resolved.
Dec 11, 13:52-20:08 EST
[Scheduled] Update to Tropo.com web portal
The scheduled maintenance has been completed.
Dec 9, 20:00-21:00 EST

November 2014

Development environment updated to new build
This incident has been resolved.
Nov 26, 11:36-11:36 EST
[Scheduled] Tuesday mainteance
The scheduled maintenance has been completed.
Nov 25, 22:30 - Nov 26, 01:00 EST
Transcriptions Delayed
The transcription provider made a change to their firewall that broke after a max number of connections was hit - they fixed the issue and all is well again.
Nov 21, 12:48-16:17 EST
Application updates delayed due to identified network trouble
All is resolved, and has been since midafternoon.
Nov 20, 12:52-19:13 EST
[Scheduled] Tuesday maintenance
This maintenance as been completed successfully.
Nov 18, 22:30-22:51 EST
Application update delays
All application changes have been fully processed and any further updates should be processed in under a minute from this point forward.
Nov 17, 17:34-21:28 EST
[Scheduled] Migration to new name servers and security patches.
All maintenance has been completed and all post migration tests are passing. If there are any questions please don't hesitate to contact us ( support@tropo.com ).
Nov 13, 22:45-23:25 EST
High webserver load
Migration to new webhosting and dns servers has been fully completed.
Nov 1, 12:02 - Nov 13, 23:19 EST
Local Market Impairment - Burlington, VT
All services to the network have been restored. Start time: 11 November 2014 - 10:35AM ET Stop time: 11 November 2014 - 01:39PM ET The following markets were impacted: Burlington, VT - Down Hard If you are still experiencing issues for the Burlington, VT area, please open a ticket with Tropo support so that we can assess and determine whether or not your issue was related to the initial outage. Thank you
Nov 11, 10:51-14:07 EST
[Scheduled] Web Hosting maintenance ( 2 of 2 )
The scheduled maintenance has been completed.
Nov 7, 23:00 - Nov 8, 03:00 EST
[Scheduled] Web Hosting maintenance 1 of 2
The scheduled maintenance has been completed.
Nov 4, 22:30 - Nov 5, 01:00 EST
High webserver load
Services fully restored
Oct 31, 17:58 - Nov 1, 08:39 EST
[Scheduled] DNS zone change
The scheduled maintenance has been completed.
Oct 31, 23:00 - Nov 1, 02:00 EST

October 2014

Issues with Jruby script engines
This incident has been resolved.
Oct 29, 11:27-14:30 EST
[Scheduled] Production Environment Build Update
This change was completed at approximately 5am UTC but unfortunately the status page was not updated. We apologize for any confusion by the task being left as "in progress" over night. If there are any questions please let us know ( support@tropo.com )
Oct 28, 20:00 - Oct 29, 10:41 EST
[Scheduled] Data center maintenance
The scheduled maintenance has been completed.
Oct 24, 23:16 - Oct 25, 05:00 EST
Canada: Inbound/Outbound SMS
This incident has been resolved.
Oct 20, 16:46 - Oct 24, 10:58 EST
Scheduled Maintenance: Chicago, IL
Chicago maintenance is now complete. If you have any questions or concerns about this notification, please contact Tropo support (support@tropo.com). Thank you
Oct 20, 16:50 - Oct 24, 10:43 EST
[Scheduled] DNS Change for tropo.com domain
The scheduled maintenance has been completed.
Oct 17, 22:00 - Oct 18, 01:00 EST
[Scheduled] Security Update
This change has been completed and all systems green.
Oct 16, 22:00-22:15 EST
[Scheduled] Scheduled maintenance - 10/14/2014
COMPLETED: Proxy update for api.tropo.com - This is an update from the version deployed last Tuesday and will correct a change to the default content type returned by the proxy.
Oct 14, 22:00-23:38 EST
Emergency Maintenance: Peru
This incident has been resolved.
Oct 13, 18:42-23:19 EST
[Scheduled] Emergency maintenance this evening
The scheduled maintenance has been completed.
Oct 8, 21:00 - Oct 9, 01:00 EST
[Scheduled] Maintenance Tuesday
maintenance complete.
Oct 7, 10:50 - Oct 8, 00:31 EST
Website issues ( resolved )
This morning our Operations teams monitored some connectivity issues between our web servers and our database servers. This issue may have caused intermittent timeouts for visitors of the Tropo.com website site. This issue has been tracked down to a networking issue with our vendor and has since been resolved. If you have any questions please don't hesitate to let us know ( support@tropo.com ).
Oct 4, 09:06 EST
[Scheduled] Emergency DNS maintenance
The scheduled maintenance has been completed.
Oct 3, 07:00-09:00 EST
[Scheduled] Transcription Service Maintenance
The scheduled maintenance has been completed.
Oct 1, 09:07 EST
Media server crash
This runtime is testing positive and is being returned to active routing.
Oct 1, 07:51-09:06 EST
[Scheduled] Regularly schedule network maintenance
The scheduled maintenance has been completed.
Sep 30, 22:00 - Oct 1, 01:00 EST

September 2014

[Scheduled] Regularly schedule network maintenance
All work has been completed.
Sep 30, 22:00-23:33 EST
[Scheduled] Transcription Service Maintenance
The scheduled maintenance has been completed.
Sep 30, 02:00-04:00 EST
[Scheduled] FTP Server maintenance
The scheduled maintenance has been completed.
Sep 29, 02:00-06:00 EST
[Scheduled] Orlando data center maintenance ( route away )
Our data center provider has canceled their planned maintenance this weekend. Once this is rescheduled we'll post an update to the status page so our users can plan accordingly. If there any questions please don't hesitate to contact us ( support@tropo.com ).
Sep 26, 22:55 EST
[Scheduled] Emergency maintenance
The scheduled maintenance has been completed.
Sep 25, 15:00-16:00 EST
[Scheduled] Increase memory on Production runtimes
Phase 1 of the production runtimes have been updated. At this time all runtimes are back in routing and all tests are passing.
Sep 23, 22:00-23:09 EST
Provisioning load
We have seen no additional alerts since increasing capacity of the provisioning cluster.
Sep 17, 19:25-21:54 EST
[Scheduled] Hosted network maintenance
Maintenance completed.
Sep 16, 22:00-23:09 EST
Internet Service Provider Outage
The ISP is back in routing after fixing their difficulties.
Sep 9, 11:06 - Sep 15, 17:30 EST
[Scheduled] Logging format change
This has been completed
Sep 11, 22:00-23:13 EST
Inbound Outage: Abilene, TX
All services to the network have been restored for Abilene, TX . Start time: 10 September 2014 - 11:30AM ET Stop time: 11 September 2014 - 12:50AM ET Responsible Party: LEC / Carrier Fix / Cause of Outage: Wire / Cable
Sep 10, 11:59 - Sep 11, 12:36 EST
[Scheduled] Upcoming maintenance
This was completed and all systems are functioning as expected.
Sep 9, 22:00 - Sep 10, 00:12 EST
Service Outage Notification
Issue was *not* related to the previous infrastructure issue we encountered Tuesday & Wednesday - this was isolated to a carrier with an intermittently failing inbound voice route (outbound was not impacted, nor was SMS). The carrier indicates the problem was fully resolved by 3:00 a.m EST ( 7:00 a.m. GMT) - start time listed previously is incorrect; issue began near 10:15 P.M. PST / 2:15 A.M. EST / 6:15 A.M. GMT. All calls are and have been connecting without issue since the resolution was reported back to us. Please email support@tropo.com with any further questions. (Small update to reflect correct times - missed an hour in the conversion)
Sep 5, 09:52-13:01 EST
api.tropo.com issues
This incident has been resolved.
Sep 2, 12:14 - Sep 4, 00:34 EST
Provisioning API Issues
The issues with the provisioning API are resolved as of 6:23 EST; issue was a persistent deadlock. if any issues persist, please email support@tropo.com for further help.
Sep 3, 14:50-19:24 EST

August 2014

[Scheduled] Webhosting Stability Improvements
This was completed as of 11:00 p.m. EST yesterday, 8/19.
Aug 19, 22:00 - Aug 20, 12:37 EST
Emergency Maintenance
Update: Emergency maintenance is complete. If you have any questions or concerns about this notification, please open a ticket with Tropo Support (support@tropo.com).
Aug 16, 22:47 - Aug 18, 14:13 EST
***IMPAIRMENT NOTICE***
Update: The majority of our customers are back in service. For those still impacted, we are working with customers directly to explore the best options for resolution. If you suspect your service has been impacted, please open a ticket with Tropo support (support@tropo.com).
Aug 12, 12:26 - Aug 14, 15:21 EST
Inbound Outage: Akron Ohio
Final: We have confirmed that splicing has been completed and that all service has been restored.
Aug 11, 10:18 - Aug 12, 23:42 EST
Inbound Outage: Mansfield, Ohio
Splicing has been completed and services restored.
Aug 11, 12:26-20:07 EST
Development environment updated to new build
Release notes are located here: http://changes.tropo.com/ Please test all existing production and development applications against the new build to confirm whether any issues are present, and contact support@tropo.com immediately with any difficulties - or even just if you have any questions.
Aug 7, 16:24 EST

July 2014

Website issues ( resolved )
We are seeing issues with the primary database server which handles the tropo.com website. We are investigating the issue now and expect to have it resolved in short order. This issue is only impacting the public tropo.com website and would not impact the api or any call traffic. Update: This issue has been resolved 14:04 UTC, 7/21
Jul 21, 09:58 EST
FTP server issues ( resolved )
We are seeing very high load on one of our file hosting servers which is effecting users ability to view their web hosting files on the website. This is not the same server that actually serves the files to the runtimes so no calls are impacted. We are continuing to investigate and should have an update shortly. UPDATE: As of 13:22 UTC 7/16, this issue is resolved.
Jul 16, 09:15 EST
Maintenance Notification: Hong Kong
Our provider is planning to carry out maintenance work in Hong Kong during the following time window: Start time: 2014-07-30, 16:00 GMT
 End time: 2014-07-30, 18:00 GMT
 Estimated impact: Possible disconnections during the above time window. If you have any questions or feel you have been affected by this maintenance, please open a ticket with Tropo Support and we will determine if your issue is related. (support@tropo.com)
Jul 15, 12:13 EST
Outbound/Inbound SMS delays and failures
We have identified an SMS issue today, which looks to be originating from our vendor’s gateway. This is causing SMS messages to fail completely or be delayed for an inordinate amount of time. This issue is actively being worked on and another update will be forthcoming. UPDATE: SMS services have been restored and we’re no longer seeing errors from the gateway. Our vendor is still investigating the matter, to provide a formal explanation of the issue and to implement a permanent fix.
Jul 10, 10:35 EST
SMS & Inbound Voice Delays
We have identified two issues - outbound and inbound SMS delays / failures, and inbound calls taking numerous rings before connecting. The SMS issue was identified as a vendor gateway failure, which was restarted and the issue appears to be cleared - we’re isolating the start time now so we can better determine the scope of the problem. Inbound voice, we continue to work on. It also appears at the moment to be vendor related, but we have not made much progress as of yet on the root cause or the resolution. We continue to focus on the problem and will update once we have more information. UPDATE: Inbound voice issue is now resolved - to summarize, we were forced to take a DNS server out of routing for a repair. Our SBCs (session border controls, essentially a network edge device which routes voice calls to and from the carriers) were not properly configured to use our load balancer and instead were trying the DNS servers via round robin. This meant that 50% of calls used the downed DNS server and had to timeout before they tried the other server. This lead to the call delays and timeouts. The server came out of routing at 01:55 a.m. UTC time 7/8, which would officially be when call delays could have started, and the issue was fully resolved as of 14:56 p.m. UTC. We are still pending the specific start date for the SMS issue, however that was resolved fully as of 12:57 p.m. UTC.
Jul 8, 09:44 EST
Webhosting Sync issue
We are currently experiencing an issue with our webhosting sync, which is preventing changes in new accounts from syncing between our data centers. This problem is only effecting new users, existing users should see no impact. We will update as soon as the sync issue is resolved. As of 15:00 p.m. UTC 7/7, this issue is resolved.
Jul 7, 10:46 EST
Transcription service outage
Our transcription provider is experiencing a persistent outage throughout a good part of today - we are working with them to resolve and will update again once we’re back up and live. As of 12:00 p.m. EST, 7/2/2014, all services were restored.
Jul 1, 14:09 EST

June 2014

No incidents reported for this month.

May 2014

No incidents reported for this month.

April 2014

No incidents reported for this month.

March 2014

No incidents reported for this month.

February 2014

No incidents reported for this month.

January 2014

No incidents reported for this month.

December 2013

No incidents reported for this month.

November 2013

No incidents reported for this month.

October 2013

No incidents reported for this month.

September 2013

No incidents reported for this month.

August 2013

No incidents reported for this month.

July 2013

No incidents reported for this month.