VirtualPBX Status

Inbound call connection delays and dead air.

Update - The VirtualPBX Operations team is monitoring the efforts of the underlying carrier, Bandwidth.com, to resolve this incident and investigating potential solutions for our customers.

We do not have an ETA from Bandwidth.com on a resolution at this time.

Outbound calling remains unaffected, as do all VirtualPBX services.

We will provide updates as they become available.
Sep 27, 13:41 PDT

Update - This incident continues to affect VirtualPBX customers, as well as customers of VoIP providers across the US and Canada today.

We are actively investigating options to workaround this incident until the underlying carrier mitigates the attack.

Please note, outbound calling is operating normally. However, calls to phone numbers that also belong to Bandwidth.com will experience the same call delivery issues.

We understand the impact this is causing our customers and apologize for the inconvenience.
Sep 27, 11:55 PDT

Update - We have confirmed the cause of this incident. An underlying carrier for inbound calls is experiencing a DDoS attack on their infrastructure.

This is affecting inbound calls across the industry today, including VirtualPBX customers.

Please note that VirtualPBX systems are fully functional and operating normally at this time.

The VirtualPBX Operations team is working with the underlying carrier to resolve this issue and will provide updates as they become available.
Sep 27, 10:24 PDT

Identified - We have identified the issue with phone keypad selections not working. VirtualPBX has not had any issues with our own service as this issue has been solely with our upstream provider, Bandwidth.com. Bandwidth is working hard to resolve the issue and we are monitoring their progress closely.
Sep 27, 10:05 PDT

Investigating - Investigating -
It seems our upstream provider,Bandwidth, is having more troubles today. Details available at https://status.bandwidth.com/. We are also receiving numerous reports of DTMF issues.

VirtualPBX selects providers that we believe are good quality. Bandwidth.com always has been good quality. They're having a bad day today, though, and it's impacting us, and you as well. We trust their engineers are all over this and we'll be waiting it out, along with you, for resolution.

Please note that we can NOT move your numbers to another provider for inbound - that's not a thing that can be done in VoIP or telecom in the US (phone numbers belong to one carrier only and the only way to change that is a port out, which takes days).
Sep 27, 09:21 PDT

VirtualPBX Operational

Phones Operational

Carriers Partial Outage

Web Applications Operational

Web Site Operational

API Operational

Mobile Apps Operational

Advanced Reporting Operational

ACD Queues Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Past Incidents

Sep 27, 2021

Unresolved incident: Inbound call connection delays and dead air..

Sep 26, 2021

Tracking upstream carrier incident

Resolved - The upstream carrier has resolved the incident.

All VirtualPBX services continue to operate normally.
Sep 26, 18:05 PDT

Update - Please note that VirtualPBX systems are fully functional and operating normally at this time.

The incident affecting one of our upstream providers, Bandwidth.com, is ongoing. We are have not identified any impact on VirtualPBX customers.

We will post updates if we confirm any impact on customers moving forward.
Sep 26, 15:59 PDT

Update - VirtualPBX systems are fully functional and operating normally at this time.

We are continuing to monitor our upstream provider, Bandwidth.com's, issue that is ongoing today. This incident affected SMS and E911 calls intermittently this morning.

We will post updates if we confirm any impact on customers moving forward.
Sep 26, 14:04 PDT

Monitoring - Please note that VirtualPBX systems are fully functional and operating normally at this time.

The incident affecting one of our upstream providers, Bandwidth.com, is ongoing today. This incident affected SMS and E911 calls intermittently this morning.

We will post updates if we confirm any impact on customers moving forward.
Sep 26, 11:11 PDT

Sep 25, 2021

Upstream Carrier Issue

Resolved - This issue is resolved
Sep 25, 14:30 PDT

Monitoring - We see inbound call routing recovering.

The incident is not resolved at this time by the upstream carrier.

Here is some background on this issue.

All providers who do not own physical wires in the ground interconnect with the telecom network providers to deliver services. There are only a few companies that provide these services - Bandwidth, Level3, Inteliquent, Peerless to name a few.

Bandwidth.com always has been dependable and delivered excellent call quality. They're having a bad day today, though, and it's impacting you and us too.

Please note that we can NOT move your numbers to another provider for inbound - that's not a thing that can be done in VoIP or telecom in the US (phone numbers belong to one carrier only, and the only way to change that is a port out, which takes days).

We hope that helps you understand the current situation. We apologize for the inconvenience this may have caused.
Sep 25, 13:59 PDT

Update - We have updated outbound call routing to minimize the impact on outbound calls from VirtualPBX. Your outbound calls should connect, unless the number dialed is managed by the upstream carrier.

We continue to monitor the carrier's efforts to resolve their issue. Please note that VirtualPBX systems are fully functional and operating normally at this time.
Sep 25, 13:31 PDT

Identified - Please note that VirtualPBX systems are fully functional and operating normally at this time.

That said, Bandwidth.com has reported an outage. Bandwidth.com provided numbers and routing for our platform, and a significant number of providers OFF our system, so while we may be up, calls TO destinations who use Bandwidth.com may fail also.

Bandwidth.com has reported that they are aware of the outage and working on it. From their status site is providing updates:

"As of 3:31pm EDT Bandwidth has acknowledged the issue:

Investigating - Bandwidth is investigating an incident impacting Voice
and Messaging Services. Calls and Messages may experience unexpected
failures. All teams are actively engaged.

Sep 25, 15:31 EDT"

We're posting this as mostly informational. We don't control the Bandwidth.com network. Outages with Bandwidth.com are rare. We expect it will resolve it ASAP. But since they are one of the largest providers in the US, this will impact a lot of customers across the US and Canada.

As we learn more we'll post here, and you can follow updates and https://status.bandwidth.com
Sep 25, 12:55 PDT

Investigating - We are investigating issues with one our our upstream carriers, Bandwidth, that is affect inbound call routing intermittently. We are also rerouting outbound call routing to minimize impact. No VirtualPBX services are affected.
Sep 25, 12:43 PDT

Sep 24, 2021

Network Provider Service Interruption

Resolved - All affected services are operating correctly.

Our team will continue to closely monitor the source of the incident and work with the upstream providers to prepare a postmortem.
Sep 24, 12:29 PDT

Update - All systems continue to work correctly.

Our upstream providers continue to work on the cause of this incident.

We are monitoring all systems and will provide updates as needed.
Sep 23, 19:45 PDT

Update - All systems are working correctly at this time, and we will be monitoring them until this incident is fully resolved.

We will continue working with the upstream providers and provide updates as we receive them.
Sep 23, 16:11 PDT

Monitoring - All known issues are resolved, and our team is monitoring all systems.

Our upstream providers continue mitigation efforts on the network incident and provide updates as they have more information.
Sep 23, 14:46 PDT

Update - One-way call audio is resolved at this time. Please let our team know if you experience the issue again.
Sep 23, 14:34 PDT

Update - We have identified the cause for one-way call audio and deployed a resolution.
Sep 23, 14:09 PDT

Update - We are working on isolated call quality issues that are affecting some customers at this time. We expect that to stabilize as the upstream providers mitigate their network incident.

All services are working properly at this time.
Sep 23, 13:48 PDT

Update - At this time all services are working properly, and we are working with our upstream providers to resolve this incident.

We do not have an ETA on full resolution and will continue working until it is fully resolved.
Sep 23, 13:41 PDT

Update - ACD Queue operations have been restored.
Sep 23, 13:36 PDT

Update - All core features, phone registrations, and call handling are working correctly at this time.

We are still working to resolve issues with ACD Queues and those are also coming back online.

The upstream incident is not fully resolved and we are monitoring it closely.
Sep 23, 12:55 PDT

Update - We are continuing to work on a fix for this issue.
Sep 23, 12:30 PDT

Update - We have resolved the registration and call handling issue. You may need to restart your phone to reregister with our platform.
Sep 23, 12:25 PDT

Update - We have identified the issue impacting phone registration and call delivery and are working to resolve the issue.
Sep 23, 12:20 PDT

Update - We understand this incident is impacting your business and apologize.

Our entire team working with upstream providers to mitigate this incident.
Sep 23, 12:06 PDT

Update - We are working on the backend issue affecting registration of phone and call handling completely.
Sep 23, 11:57 PDT

Update - We are continuing to work with our upstream providers to mitigate the incident. We do not have an ETA on a full resolution at this time.
Sep 23, 11:27 PDT

Update - Our team is working to resolve the issue with ACD Queues. We are investigating some issues of one-way audio affecting some customers.
Sep 23, 11:14 PDT

Update - We continue to work with the upstream provider on their mitigation efforts.

We have made routing adjustments and continue to work with affected customers.
Sep 23, 10:37 PDT

Update - Our upstream providers alerted us that mitigation efforts are ongoing to address their network incidents.

We are responding to issues as they arise and working to minimize the impact on customers.

We expect to have intermittent connectivity issues until the provider's incidents are fully resolved.
Sep 23, 10:11 PDT

Update - We are working to reroute traffic for Webphone and ACD Queues away from the datacenters impacted by the upstream provider's incidents.

This does not affect normal call routing and handling.
Sep 23, 09:56 PDT

Update - We have identified an issue with the Dashboard web interface that displays an error message due to the network incident.

We have deployed a fix to resolve the issue. It may take up to 5 minutes to fully deploy to all customers.
Sep 23, 09:48 PDT

Update - Call handling and routing are working as expected and our team is continuing to monitor systems affected this morning.

Customers experiencing issues with ACD Queues may try to log out and back into their ACD Queues to resolve their issues.

We are monitoring the affected upstream providers and will post updates until this incident is fully resolved.
Sep 23, 09:31 PDT

Identified - Our team is investigating reports that some customers may be experiencing issues with ACD Queues.

This may be related to the upstream provider's network incident that is ongoing.
Sep 23, 09:22 PDT

Update - Our upstream providers have identified the cause and working to mitigate the issue.

Our team is monitoring the affected systems and working with customers impacted by this incident.
Sep 23, 08:57 PDT

Investigating - We are currently experiencing intermittent connectivity issues with upstream network providers affecting customers.

This affects call handling and routing intermittently for some customers.

Our team is working with our upstream providers to fully resolve this incident and will be posting updates on our status site as work progresses.

We deeply apologize for any inconvenience you may experience and assure you it will be resolved as quickly as possible.
Sep 23, 08:09 PDT

Sep 23, 2021

Sep 22, 2021

Webphone Login Issue

Resolved - This incident is resolved.
Sep 22, 16:19 PDT

Monitoring - A fix has been implemented and We are monitoring the results.
Sep 22, 15:59 PDT

Update - We are still actively working to stabilize services. We will keep you informed as we progress and will be back with another update in 45 minutes.
Sep 22, 15:33 PDT

Update - We are still actively working to stabilize services. We will keep you informed as we progress and will be back with another update in 45 minutes.
Sep 22, 14:48 PDT

Update - We are still actively working to stabilize services. We will keep you informed as we progress and will be back with another update in 45 minutes.
Sep 22, 14:03 PDT

Identified - We have identified the issue affecting Webphone logins. Engineers are actively working to stabilize services. We will post another update in 30 minutes.
Sep 22, 13:33 PDT

Update - We are still actively investigating this issue. We will keep you informed as we progress and will be back with another update in 30 minutes.
Sep 22, 13:20 PDT

Update - We are still actively investigating this issue. We will keep you informed as we progress and will be back with another update in 15 minutes.
Sep 22, 13:05 PDT

Investigating - We are currently investigating an issue where some customers are reporting they are unable to log into the webphone page.
Sep 22, 12:50 PDT

Sep 21, 2021

No incidents reported.

Sep 20, 2021

Partner Firewall Maintenance

Completed - The schedule maintenance is complete.
Sep 20, 01:43 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 19, 23:01 PDT

Scheduled - Our partner will be conducting a Firewall update on September 19th at 11:00 PM PST for 3 hours.

The Dash portal will be accessible during this time and we do not expect any impact on service.

We will share live status updates and the outcome of these performance improvements on our real-time status site, status.virtualpbx.com, where you may also subscribe to updates via email, SMS, or RSS feed.

If you have any questions, please feel free to reach out to us by phone, email, or online chat. Thank you for your understanding.
Sep 15, 12:31 PDT

Sep 19, 2021

Sep 18, 2021

No incidents reported.

Sep 17, 2021

No incidents reported.

Sep 16, 2021

Dash Maintenance

Completed - The scheduled maintenance has been completed.
Sep 16, 05:25 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 15, 23:55 PDT

Scheduled - We will be conducting system maintenance for Dash on September 15th at 11:55 PM PST for 3 hours.

This maintenance will deploy performance improvements across our entire platform and lay the groundwork for a major platform upgrade that will be scheduled in the near future.

Customers using ACD Queues during maintenance will have their active Agents logged out of ACD Queues when updates are applied to the service. When this occurs Agents will need to log back into their assigned ACD Queues.

All other platform features and functionalities will be unaffected.

We will share live status updates and the outcome of our performance improvements on our real-time status site, status.virtualpbx.com, where you may also subscribe to updates via email, SMS, or RSS feed.

Thank you for your patience as we make these updates. If you have any questions, please feel free to reach out to us by phone, email, or online chat.
Sep 8, 16:28 PDT

Sep 15, 2021

Sep 14, 2021

No incidents reported.

Sep 13, 2021

SMS Message Issue

Resolved - This issue is resolved.
Sep 13, 11:58 PDT

Monitoring - The SMS issue has been fixed. We're closely monitoring systems to ensure stability. We will post another update in 60 minutes.
Sep 13, 10:37 PDT

Identified - We are resolving the issue. Affected customers will see the queued SMS messages begin to be delivered.
Sep 13, 10:12 PDT

Update - We are still investigating this issue.
Sep 13, 09:06 PDT

Update - We are continuing to investigate this issue.
Sep 13, 06:59 PDT

Update - We are still working on this issue. We will provide a new update tomorrow morning.
Sep 12, 21:08 PDT

Update - We are still actively investigating this issue.
Sep 12, 18:58 PDT

Update - We are continuing to investigate this issue.
Sep 12, 17:09 PDT

Update - We are still investigating the SMS routing delay issue.
Sep 12, 15:00 PDT

Update - We are continuing to investigate the SMS routing delay.
Sep 12, 13:03 PDT

Update - We are still actively investigating this issue.
Sep 12, 11:00 PDT

Update - We are still investigating this issue. We will provide additional updates as they become available.
Sep 12, 08:53 PDT

Update - We are continuing to investigate this issue, We will provide updates as they become available.
Sep 12, 07:00 PDT

Update - We are still working on investigating the SMS routing delay issue. We will be updating the status again tomorrow morning.
Sep 11, 21:40 PDT

Update - We are continuing to investigate this issue.
Sep 11, 19:05 PDT

Investigating - We are currently investigating an issue with SMS routing delays that is affecting some customers.
Sep 11, 18:30 PDT

Subscribe to Incident