Change to Recording Functionality
The behavior of the record method in Tropo has changed slightly with the current build, and the change was not published in the changelog erroneously. Previously, when a recording was sent to an HTTPS destination URL via a REST POST, Tropo would send two POSTs; the first improved our ability to validate SSL certificates in specific circumstances, and then the second would send the actual recording file. This behavior changed to be a single POST now, as enhanced methods to validate the certificate are present that eliminate the need for the preceding POST. This could cause file processing issues if the recipient application was built to intentionally discard the content of the first POST and process only the content of the second. Apologies for the late notice of this change; while an overall improvement, the potential customer impact was not properly evaluated. If you notice you are encountering recording failures, please contact email@example.com for further help
Feb 24, 14:01 EST
Brief connectivity issues
It appears the issue with related to a single monitoring node in our environment. This node failed and seems to have caused several false alarms. We will go ahead and close this issue out at this time. If there are any questions please let us know - firstname.lastname@example.org
Feb 19, 17:59-20:00 EST
Power issue on single shared production runtime
During routine hardware maintenance this morning a runtime was inadvertently restarted while taking active calls. Our support team will be reaching out to the limited number of users who were impacted. If there are any questions please let us know at email@example.com
Feb 17, 07:51 EST
Media Server Crash
January 31st at 00:20 GMT (midnight), we encountered a media server crash on one runtime in the production environment, which may have caused a small percentage of calls to encounter media related issues, such as missing recordings. The media server recovered and calls are currently stable.
Jan 31, 14:52 EST
Reduced number availability in some Canadian regions
Due to a vendor transition, we have lower than typical stock of Canadian SMS and Voice enabled phone numbers. This is a temporary situation, and will be resolved shortly, but if a region you need does not show as available in the drop down when adding a new number, please contact firstname.lastname@example.org for help.
We deeply apologize for any inconvenience this may cause.
Jan 27, 10:00 EST
[Scheduled] Tropo.eu firewall upgrade
This task was completed for the Slave firewall, but we decided to postpone upgrading the Master for 24hours. We will be posting a new bulletin for the remaining tasks that will be completed on the Master this evening.
Jan 3, 21:00 - Jan 4, 09:19 EST
Tropo.com Docs are currently down
Incident mistakenly left open beyond actual duration; this was resolved by 12:30pm -est Friday (12/16/2016)
Dec 16, 12:19 - Dec 19, 13:57 EST
Degraded service in some runtime nodes
After review of the runtime nodes, a defective customer script was found running in a such a way as to make that app non-responsive. All affected runtime nodes have been inspected, tested and placed back in routing.
Nov 29, 14:49-21:09 EST
Instability in TTS and ASR overnight
It's been well over 24 hours since the correction was implemented, and we have not seen another occurrence of this failure.
Nov 17, 10:32 - Nov 18, 12:08 EST
[Scheduled] API Environment cleanup, and legacy Dev DNS record removal
The DNS record was removed successfully, but we've postponed the Environment cleanup and may revisit this tomorrow evening (Development Environment only). If we do, a new bulletin will be opened. Please let us know if you have any questions (email@example.com).
Nov 16, 21:00-23:26 EST
[Scheduled] Shared Production upgrade, and transcription logging enhancement
The Transcription logging upgrade has completed successfully, but we have postponed the Shared Production 15.5.0 upgrade due to the potential impact of an isolated issue. We are grateful for your feedback (we listen :-) )! A new update will follow with details of the next deployment date.
Nov 8, 21:01-22:14 EST
Instability in ProvisioningAPI
We have identified and resolved the underlying issue resulting in this temporary instability. There was a condition where logging queues to be come overloaded, so we've decreased the logging context for the problematic appender, and have not seen this occur since.
Oct 24, 11:47-15:21 EST
Inbound Toll-free carrier outage
Service has been fully restored. Please let us know if you have any questions or concerns (firstname.lastname@example.org).
Oct 4, 11:11-13:10 EST
[Scheduled] Prod environment runtime upgrade
This maintenance is complete, all tests are returning positive results, and all systems are green. Please let us know if you happen to encounter any failures (email@example.com).
Sep 14, 21:00 - Sep 15, 01:05 EST
[Scheduled] Provisioning API patch
We have postponed this maintenance for now, and will revisit this item for next weeks window. Apologies for the notification flood; please let us know if you have any questions (firstname.lastname@example.org).
Sep 13, 21:01 - Sep 14, 09:06 EST
[Scheduled] Development environment; TTS & ASR upgrade (en-us)...
* We decided to postpose the ASR/TTS upgrade to a later date. Because this change is limited to the Dev env, we may try again this week. Please stay tuned for updates.
* The new Dev hostname endpoints have been successfully deployed to the global proxies.
Sep 6, 21:00-23:40 EST
This incident has been resolved.
Aug 22, 15:40 - Aug 23, 17:33 EST
Single runtime failure in hosted
We've identified the root cause, and are adding triggers to alert in the event this occurs again.
Jun 29, 12:41 - Jul 8, 18:31 EST
Investigating reports of recording upload failures
This issue was resolve by rolling back the build. We have fixed the issue in build 15.3.3 which is in our development environment. If you have any questions please don't hesitate to let us know.
Jun 8, 09:36 - Jun 22, 11:30 EST
Issue mapping addresses to applications
This issue was caused by a the addition of new address capabilities for a soon to be announced expansion. This new option was enabled by default and our integration testing unfortunately did not catch the problem. Our engineering team quickly identified the issue and was deployed a patch to resolve it. We have seen no further indications or reports of trouble since that patch was deployed. We are also performing an internal review to see how this issue was missed in order to ensure it doesn't happen again. If there are any questions please don't hesitate to let us know ( email@example.com ).
Jun 7, 15:09 - Jun 9, 15:42 EST
Provisioning API reported errors
This has been resolved following a drive replacement. Please let us know if you have any questions (firstname.lastname@example.org).
May 9, 17:32 - May 10, 10:11 EST
Provisioning API reported errors
This issue is now resolved. A block store volume failed, and accordingly the Provisioning systems failed over to a readonly mode. We have recovered the volume, and validated that all services have been restored. If you have any questions please feel free to reach out to our team (email@example.com).
May 9, 15:37-16:23 EST
Media server instability
This has been resolved, and have not seen any instability since this bulletin was originally posted. Please let us know if you have any questions (firstname.lastname@example.org).
May 4, 16:31 - May 5, 10:19 EST
Tropo portal application view limitation
We have deployed the correction, and confirmed the issue has been resolved. Please let us know if you have any questions or concerns (email@example.com).
Apr 29, 10:46-11:26 EST
Inbound SMS outage: UK
This issue is now resolved and inbound traffic on UK geographic DIDs are operational again. The problem started on 25 April 2016, at 18:00 GMT and resolved today 26 April 2016.
If you are experiencing any trouble with UK numbers please open a ticket with Tropo support at (firstname.lastname@example.org) and a member of our team will assess if the issue is related. Thank you
Apr 25, 14:56 - Apr 26, 15:34 EST
[Scheduled] Emergency vendor patch for text-to-speech software
The patch has been applied successfully to the text-to-speech servers and no downtime has been recorded. This maintenance window is now complete.
As usual, if you have any questions, feel free to reach out to email@example.com.
Apr 20, 22:00-23:40 EST
Single Media Server Restart
This server was removed from routing and inspected for errors. It has since been passing all tests and has been placed back into routing.
Apr 19, 15:42 - Apr 20, 14:12 EST
[Scheduled] Billing provider migration
We apologize for the delay on this maintenance. At this time all work has been completed. If you have any questions please don't hesitate to let us know - firstname.lastname@example.org
Apr 1, 19:14 - Apr 2, 02:22 EST
Issues connecting to the support portal
Our provider has corrected the issue within their network, and service has been restored to the support.tropo.com portal. Please let us know if you have any questions (email@example.com).
Mar 21, 10:13-11:14 EST
Speech recognition failures in Development env
This has been resolved. We identified a license key that was not applied correctly, and have patched the environment. Internal testing is returning 100% success, but please let us know if you encounter any issues (firstname.lastname@example.org).
Mar 8, 16:07-17:23 EST
Tropo API experiencing inconsistent service
A credential change for one of our vendor connections occured and during this change, an issue arose with our gateways which ended up disruption token sessions for a portion of our users. The issue has been identified and is currently resolved.
Feb 18, 16:45-16:56 EST
[Scheduled] Production runtime upgrade
The update completed without any issues. Please let us know if you have questions or run into any troubles. (email@example.com)
Feb 10, 21:30-23:06 EST
[Scheduled] Hosted scheduled maintenance
The proxy upgrade has been completed and all tests are passing. The SMS vendor peering change was postponed until a later date. If there are any questions please let us know - firstname.lastname@example.org
Jan 27, 21:30-22:53 EST
[Scheduled] Hosted scheduled maintenance
This maintenance work has been completed. Unfortunately the following two items were post postponed.
* Tropo Operations will be deploying a new version of Tropo's global routing proxy to address edge cases with call forwarded numbers.
* Tropo Operations will be changing the default destination for all traffic from Orlando to Virginia data center.
Jan 26, 21:30-23:39 EST
Production data centers issues
Our data center vendor has patched all switches in the data center and we have no seen any additional occurrences of this issue. We will no close this issue as as resolved.
Jan 15, 22:38 - Jan 24, 12:41 EST
Potential memory failure
We have replaced the memory in the failing server and at it has been put back into routing.
Jan 15, 18:22-21:39 EST
TTS license issue
Earlier today we experienced TTS license issue with one of our runtime in development environment. Server was immediately taken out which operations will review later. This issue only impacted the development environment, production was not affected.
Jan 3, 12:15 EST
Phono product public access and refactoring
In order to better align Phono with our overall offering, we are removing public access to the project as of today, 12/22/2015. It will be back, and with more functionality integrated into both Tropo and other Cisco products. Please watch both the Tropo and DevNet blogs (https://blog.tropo.com and https://communities.cisco.com/community/developer/blog for more information on the release of the new and improved Phono, and email email@example.com if you have any questions.
Note that existing applications utilizing Phono will remain active and live for the time being, as the gateways are not going to be turned down yet, but new applications will not be possible until the product is relaunched. If you have a production application using Phono now and want to discuss options moving forward, please email firstname.lastname@example.org as soon as possible.
If you want to learn more about our Products & APIs, please check out Tropo.com (https://tropo.com) and Cisco Spark for Developers (https://developer.ciscospark.com).
Dec 22, 16:24 EST
External Vendor Issue
We have seen no further issues with this carrier for over 24 hours. We will be closing this issue as resolved. If there are any questions please let us know ( email@example.com ).
Dec 15, 15:00 - Dec 16, 21:37 EST
New Account Balance Alerts activation
The account balance alerts referenced on 11/10 will be activated tonight at 11 p.m. EST - deployment was intentionally delayed a few weeks to resolve a couple account specific issues. Please let us know if you have any questions prior by emailing firstname.lastname@example.org.
Dec 1, 16:18 EST
Intermittent packet loss in Orlando data center
After investigating in tandem with our datacenter provider, a latency issue with a carrier was found in the Orlando location, upstream from the datacenter itself. The issue looks to be resolved at this time.
Nov 24, 09:37-17:53 EST
Vendor issues for Canadian SMS numbers
We have confirmed that outbound messages during this period were unfortunately dropped. Our Engineering team is investigating ways to be better queue outbound SMS messages so we can be more tolerant of these types of issues in the future. If there are any questions please don't hesitate to let us know - email@example.com
Nov 21, 09:59 - Nov 22, 21:42 EST
Vendor made changes that affects some inbound voices numbers
The vendor identified root cause as an incorrect "calls per second" setting at the trunk level, which caused a large portion of inbound U.S. calls to fail due to hard limiting call concurrency. This setting is fixed moving forward.
Nov 18, 07:03 - Nov 21, 12:39 EST
Tropo.com Developer Portal Updated
We just updated the Tropo portal to add several long standing requested enhancements, with more to come in the near future:
* Previously if a developer attempted to save a file without an extension, they would receive an error, but could not then re-save with an extension - had to back out and start all over. This is fixed, you can now just add the file extension and the error will resolve.
* If a number pool is out of numbers, previously it would just cycle and display no information as to why the number could not be added. A message will now display indicating the number pool is depleted. If you need a number in that region, please contact support to request one.
* You can now search for a number in the number pool by region or area code, instead of scrolling through the entire list.
* Sorting of application numbers by country or region is now available.
* Icons for files in the File Manager are now unique per file extension.
Note that to reflect the new changes, you'll need to clear your cache so any historical records of Tropo.com are removed; we will be fixing that as well in an upcoming version, so cache will clear on it's own.
Nov 18, 12:53 EST
Staging data center issues
All staging services are now recovered. The problem appears to have been a failure of the active router and the secondary not correctly assuming control of routing. Tropo Operations manually forced the failover and services immediately recovered. We will be reaching out to the firewall vendor to better understand why the service did not correctly failover and manual intervention was required. If there are any questions please let us know ( firstname.lastname@example.org ).
Nov 15, 05:22-05:45 EST
Billing provider delays
Issue is now resolved; Oracle database issue that is fully recovered.
Nov 10, 14:04-15:33 EST
Service Impairment: Inbound Toll Free
Our toll free provider identified an issue with call origination routing and was able to resolve it. As of today 11-10-2015 09:42 EST, the problem is now confirmed as fixed. We will continue to monitor to ensure stability. If you are still experiencing any trouble with inbound toll free calls, please open a ticket with our support team (email@example.com).
Nov 9, 17:21 - Nov 10, 14:01 EST
[Scheduled] Billing system maintenance
This maintenance has been completed and all post deployment tests have passed. Thank you very much for your patience during this upgrade.
Oct 21, 20:00-20:44 EST
Sip proxy issues for inbound calling
The proxy has been rebuilt,tested, and returned to routing. We are closing this incident as resolved. If there are any questions please let us know - firstname.lastname@example.org
Oct 20, 11:46-15:02 EST
Outbound SMS delivery issues from developer environment
Tropo Operations uncovered a configuration issue on the SMS gateways which was introduced during this mornings upgrade. This issue likely caused problems with outbound SMS delivery from the free development environment and was fixed at 22:00 UTC. This issue did NOT impact the production environment at all. If there are any questions please let us know - email@example.com
Oct 19, 17:26 EST
Inbound Service Impairment- WV, VA & PA Only
Thank you for your patience. Our underlying carrier rebuilt their databases that lead to resolution. As of this morning (4:30am EST) inbound services for WV, VA & PA have been restored. Please contact Tropo support if you are in need of assistance. (firstname.lastname@example.org)
Oct 14, 20:08 - Oct 15, 10:54 EST
Inbound failures on Staging
This issue has been resolved. It was caused by a platform misconfiguration and only impacted inbound voice traffic to the free developer platform. Production was not effected by this issue at all. The issue has been resolved and please let us know if there are any questions - email@example.com
Oct 7, 08:52-10:27 EST
The ASR servers are back up and running normally.
Oct 4, 15:08-15:15 EST
Transcription API issues
The vendor issue has been resolved and all messages sent during their outage should have been delivered at this time. We apologize for the delay in transcription delivery and if there are any questions please don't hesitate to let us know - firstname.lastname@example.org
Aug 10, 11:51-16:36 EST
Inbound Voice Impairment: Select Markets Only
Our provider has repaired the fiber. Inbound services have remained stable. We are closing this incident at this time, however please contact Tropo support (email@example.com) if you are in need of assistance.
Jul 13, 19:47 - Jul 14, 13:43 EST
[Scheduled] Weekly Maintenance
Due to an unforeseen issue with our vendor we have been forced to cancel this planned maintenance item. We hope to have a new date very soon, and will update status.tropo.com as soon as it's available. If there are any questions please let us know - firstname.lastname@example.org
Jun 30, 23:39 EST
[Scheduled] Additional carrier migrations
Due to an scheduling issue with one of our vendors we have had to push this maintenance item back at week. Currently we have a tentative date of July 1st, and as soon as this is confirmed we will post a new maintenance notification to http://status.tropo.com. If there are any questions please let us know - email@example.com
Jun 24, 22:17 EST
Tropo Operations isolated the high load on the web hosting servers to a corruption of the file replication sync database. This corruption was causing the replication process to go into a tight loop which triggered the high load. This issue was limited to the state databases for Tropo's secondary data centers and once our Operations team forced a re-sync load quickly subsided. The re-sync has now finished and load is now running within expected norms. If there are any questions please let us know - firstname.lastname@example.org
Jun 22, 12:58-23:13 EST
Provisioning API issues
The server has been restarted and all services and tests are green and passing at this time. We have opened a ticket with product engineering to investigate for a root cause but in the meantime Operations has put the server back into routing.
May 19, 09:31-09:39 EST
Issues with foreign language speech recognition servers
We have identified an issue with a subset of our speech recognition servers. The problematic servers have been removed from routing and at this point tests are green. We will continue to monitor the situation and if there are any questions please let us know ( email@example.com ).
Apr 22, 19:37-20:51 EST
Phono registration down
The issue is related to the user registration part. If the users are not meeting the password complexity rules, the registration page doesn't act properly. We'll fix the bug.
Apr 3, 20:03-21:37 EST
Tropo.com has an internal issue
We have gotten the data we need for our investigation and have already rolled back the update for both load balancers. We will continue to monitor them but we expect no further issues. If there are any questions please don't hesitate to let us know ( firstname.lastname@example.org ).
Apr 2, 04:46-11:33 EST
SMS Billing fix
We identified an issue where large SMS messages (those over 160 characters, typically, with variance for multi-byte characters and the like) were being billed as one SMS instead of multiple SMS as intended. This issue will be fixed as of midnight GMT 3/26, so SMS will be billed properly moving forward, with no retroactive billing.
Mar 25, 12:44 EST
This incident has been resolved.
Mar 11, 22:26-23:30 EST
Our database server VM crashed earlier this afternoon at around 5:30pm EST. The outage lasted until around 5:50pm EST.
Feb 21, 18:30 EST
New build in development environment
Please test all applications, production and development, against the new build to verify no issues are present. There have been a few build push attempts recently, so the list of changes to validate are split up among multiple listings on changes.tropo.com - you will want to validate against all of them:
Please email email@example.com with any questions or issues.
Feb 18, 12:04 EST
Scheduled Maintenance: United States
There should be no impact to calls or call quality during this maintenance. Please contact us if there are any questions.
Feb 17, 11:26-15:53 EST
Issues with transcription have now been resolved and transcription is back online.
Feb 2, 12:23-13:26 EST
Transcription provider indicates back to normal, please email firstname.lastname@example.org if you notice any further trouble.
Jan 22, 12:53-13:34 EST
[Scheduled] Tropo.com website update
Encountered some issues during the upgrade of Tropo.com. The upgrade has been rolled back and Tropo.com is back and fully operational. Maintenance for this evening (Wed, Jan 21 EST) has been completed.
Jan 21, 21:00-22:07 EST
[Scheduled] Token router maintenance release
All maintenance as been completed and post deployment tests are green. If there are any questions please don't hesitate to let us know ( email@example.com ).
Jan 20, 22:00-22:42 EST
After an exhaustive investigation we have confirmed the reported problem was limited to a batch of a few dozen sms addresses. These numbers were incorrectly mapped in our provisioning database which caused them to fail when mapped to users applications. We have corrected the issue and have taken steps to ensure the issue does not happen again.
Jan 15, 14:47-21:25 EST
New build in development environment
Please test all applications, production and development, against the new build to verify no issues are present. Changes are listed here:
If you did not test recently, the changes shown here may also apply:
Please email firstname.lastname@example.org with any questions or issues.
Dec 22, 13:39 EST
Emergency Maintenance: Brazil
Emergency maintenance has been completed for Brazil. If you are still experiencing issues and are in need of assistance, please open a support ticket (email@example.com) and we will assess if your issue is related. Thank you.
Dec 12, 17:15 - Dec 15, 10:30 EST
The transcription provider made a change to their firewall that broke after a max number of connections was hit - they fixed the issue and all is well again.
Nov 21, 12:48-16:17 EST
Application update delays
All application changes have been fully processed and any further updates should be processed in under a minute from this point forward.
Nov 17, 17:34-21:28 EST
High webserver load
Migration to new webhosting and dns servers has been fully completed.
Nov 1, 12:02 - Nov 13, 23:19 EST
Local Market Impairment - Burlington, VT
All services to the network have been restored.
Start time: 11 November 2014 - 10:35AM ET
Stop time: 11 November 2014 - 01:39PM ET
The following markets were impacted:
Burlington, VT - Down Hard
If you are still experiencing issues for the Burlington, VT area, please open a ticket with Tropo support so that we can assess and determine whether or not your issue was related to the initial outage. Thank you
Nov 11, 10:51-14:07 EST
[Scheduled] Production Environment Build Update
This change was completed at approximately 5am UTC but unfortunately the status page was not updated. We apologize for any confusion by the task being left as "in progress" over night. If there are any questions please let us know ( firstname.lastname@example.org )
Oct 28, 20:00 - Oct 29, 10:41 EST
Scheduled Maintenance: Chicago, IL
Chicago maintenance is now complete. If you have any questions or concerns about this notification, please contact Tropo support (email@example.com). Thank you
Oct 20, 16:50 - Oct 24, 10:43 EST
[Scheduled] Scheduled maintenance - 10/14/2014
COMPLETED: Proxy update for api.tropo.com - This is an update from the version deployed last Tuesday and will correct a change to the default content type returned by the proxy.
Oct 14, 22:00-23:38 EST
Website issues ( resolved )
This morning our Operations teams monitored some connectivity issues between our web servers and our database servers. This issue may have caused intermittent timeouts for visitors of the Tropo.com website site. This issue has been tracked down to a networking issue with our vendor and has since been resolved. If you have any questions please don't hesitate to let us know ( firstname.lastname@example.org ).
Oct 4, 09:06 EST
Media server crash
This runtime is testing positive and is being returned to active routing.
Oct 1, 07:51-09:06 EST
[Scheduled] Orlando data center maintenance ( route away )
Our data center provider has canceled their planned maintenance this weekend. Once this is rescheduled we'll post an update to the status page so our users can plan accordingly. If there any questions please don't hesitate to contact us ( email@example.com ).
Sep 26, 22:55 EST
We have seen no additional alerts since increasing capacity of the provisioning cluster.
Sep 17, 19:25-21:54 EST
Inbound Outage: Abilene, TX
All services to the network have been restored for Abilene, TX .
Start time: 10 September 2014 - 11:30AM ET
Stop time: 11 September 2014 - 12:50AM ET
Responsible Party: LEC / Carrier
Fix / Cause of Outage: Wire / Cable
Sep 10, 11:59 - Sep 11, 12:36 EST
Service Outage Notification
Issue was *not* related to the previous infrastructure issue we encountered Tuesday & Wednesday - this was isolated to a carrier with an intermittently failing inbound voice route (outbound was not impacted, nor was SMS).
The carrier indicates the problem was fully resolved by 3:00 a.m EST ( 7:00 a.m. GMT) - start time listed previously is incorrect; issue began near 10:15 P.M. PST / 2:15 A.M. EST / 6:15 A.M. GMT. All calls are and have been connecting without issue since the resolution was reported back to us.
Please email firstname.lastname@example.org with any further questions.
(Small update to reflect correct times - missed an hour in the conversion)
Sep 5, 09:52-13:01 EST
Provisioning API Issues
The issues with the provisioning API are resolved as of 6:23 EST; issue was a persistent deadlock. if any issues persist, please email email@example.com for further help.
Sep 3, 14:50-19:24 EST
Update: Emergency maintenance is complete.
If you have any questions or concerns about this notification, please open a ticket with Tropo Support (firstname.lastname@example.org).
Aug 16, 22:47 - Aug 18, 14:13 EST
Update: The majority of our customers are back in service. For those still impacted, we are working with customers directly to explore the best options for resolution. If you suspect your service has been impacted, please open a ticket with Tropo support (email@example.com).
Aug 12, 12:26 - Aug 14, 15:21 EST
Inbound Outage: Akron Ohio
We have confirmed that splicing has been completed and that all service has been restored.
Aug 11, 10:18 - Aug 12, 23:42 EST
Development environment updated to new build
Release notes are located here:
Please test all existing production and development applications against the new build to confirm whether any issues are present, and contact firstname.lastname@example.org immediately with any difficulties - or even just if you have any questions.
Aug 7, 16:24 EST
Website issues ( resolved )
We are seeing issues with the primary database server which handles the tropo.com website. We are investigating the issue now and expect to have it resolved in short order. This issue is only impacting the public tropo.com website and would not impact the api or any call traffic.
Update: This issue has been resolved 14:04 UTC, 7/21
Jul 21, 09:58 EST
FTP server issues ( resolved )
We are seeing very high load on one of our file hosting servers which is effecting users ability to view their web hosting files on the website. This is not the same server that actually serves the files to the runtimes so no calls are impacted. We are continuing to investigate and should have an update shortly.
UPDATE: As of 13:22 UTC 7/16, this issue is resolved.
Jul 16, 09:15 EST
Maintenance Notification: Hong Kong
Our provider is planning to carry out maintenance work in Hong Kong during the following time window:
Start time: 2014-07-30, 16:00 GMT
End time: 2014-07-30, 18:00 GMT
Estimated impact: Possible disconnections during the above time window.
If you have any questions or feel you have been affected by this maintenance, please open a ticket with Tropo Support and we will determine if your issue is related.
Jul 15, 12:13 EST
Outbound/Inbound SMS delays and failures
We have identified an SMS issue today, which looks to be originating from our vendor’s gateway. This is causing SMS messages to fail completely or be delayed for an inordinate amount of time. This issue is actively being worked on and another update will be forthcoming.
UPDATE: SMS services have been restored and we’re no longer seeing errors from the gateway. Our vendor is still investigating the matter, to provide a formal explanation of the issue and to implement a permanent fix.
Jul 10, 10:35 EST
SMS & Inbound Voice Delays
We have identified two issues - outbound and inbound SMS delays / failures, and inbound calls taking numerous rings before connecting. The SMS issue was identified as a vendor gateway failure, which was restarted and the issue appears to be cleared - we’re isolating the start time now so we can better determine the scope of the problem.
Inbound voice, we continue to work on. It also appears at the moment to be vendor related, but we have not made much progress as of yet on the root cause or the resolution. We continue to focus on the problem and will update once we have more information.
Inbound voice issue is now resolved - to summarize, we were forced to take a DNS server out of routing for a repair. Our SBCs (session border controls, essentially a network edge device which routes voice calls to and from the carriers) were not properly configured to use our load balancer and instead were trying the DNS servers via round robin. This meant that 50% of calls used the downed DNS server and had to timeout before they tried the other server. This lead to the call delays and timeouts.
The server came out of routing at 01:55 a.m. UTC time 7/8, which would officially be when call delays could have started, and the issue was fully resolved as of 14:56 p.m. UTC.
We are still pending the specific start date for the SMS issue, however that was resolved fully as of 12:57 p.m. UTC.
Jul 8, 09:44 EST
Webhosting Sync issue
We are currently experiencing an issue with our webhosting sync, which is preventing changes in new accounts from syncing between our data centers. This problem is only effecting new users, existing users should see no impact.
We will update as soon as the sync issue is resolved.
As of 15:00 p.m. UTC 7/7, this issue is resolved.
Jul 7, 10:46 EST
Transcription service outage
Our transcription provider is experiencing a persistent outage throughout a good part of today - we are working with them to resolve and will update again once we’re back up and live.
As of 12:00 p.m. EST, 7/2/2014, all services were restored.
Jul 1, 14:09 EST