← Homepage

Orderbox Status Page

Issue with Shared Hosting & Email Hosting services

Opened on April 18th, 2022 8:18 am GMT, last updated April 21st, 2022 2:45 pm GMT
Resolved

Issue: Network issues impacting Shared Hosting and Email Hosting Servers hosted on our Austin Data Center

Impact:

  • Shared Linux servers were down restricting ability to perform any action
  • Unable to log in to webmail due to intermittent packet loss
  • Delay in email delivery and email client configuration due to problems with POP/IMAP as a result of issues with one of the proxy servers
  • Renewal actions were impacted for these orders

Sequence of events:

  • Engaged the Netops and Data Center (DC) ops Team who identified a network issue.
  • They restored the grid power first followed by A and B feeds. Servers were powered on and network connectivity was enabled.
  • VXLAN was recovered post which Netops worked to establish access to the VXLAN routers and restore the services.
  • Internet access to Austin DC via VXLAN was down and a case opened with the vendor - Juniper.
  • VXLAN issue was resolved and Netops continued to restore network connectivity.
  • DC ops identified a faulty switch and swapped cables. They identified that a few IPs were not reachable.
  • Juniper Engineers helped Netops to restore configuration lost on the virtual machines after the power outage at Austin DC on 13-April.
  • Issue confirmed as resolved on reviewing server alerts. An additional inbound server was added to alleviate the load.
  • As a preventive measure, the team increased the connection limits on the servers to max capacity and allow emails to flow in. A second additional inbound servers was added to alleviate the load.

**Root Cause: **Network issue and hardware impacted in Austin DC as a result of the power outage on 13-April-2022.

Posted April 21st, 2022 2:45 pm GMT

Monitoring

The issue has been identified and fixed by our System Administration team and the services are now up and running. An increase in load on the outbound server caused the outage which is now rectified.

Our Server Administration team is actively monitoring the server. We will post further updates on this thread regarding the issue.

Posted April 19th, 2022 9:00 am GMT

Identified

Our network team has identified intermittent issues with our Email Hosting Servers. You might face issues while sending emails and also experience slowness while accessing Webmail.

The team is working on restoring the services. An update will be provided to this thread as soon as additional information is available.

Posted April 19th, 2022 7:16 am GMT

Monitoring

We have successfully restored the impacted services. You should now be able to receive emails without any delay.

You should be able to access the website, panels and email services seamlessly on all the platforms again.

We shall continue to monitor the servers for another 12 hours and shall update further on this thread.

Posted April 18th, 2022 5:34 pm GMT

Monitoring

We have successfully fixed the network router in Austin DC which has restored all the services.

You should be able to access the website, panels and email services seamlessly on all the platforms again. You will observe a slight delay in the email delivery while the queues are stabilizing which will also be fixed in the next 1 hour.

A detailed RCA will be shared on this thread once we have all the details.

We will continue to monitor the servers for another 6 hours and shall update the post accordingly.

Posted April 18th, 2022 10:56 am GMT

Identified

On further investigation, we've identified that the network router in Austin DC is currently down and we'll have to change it physically. Once that has been changed, our network administrators will check the setup to initiate the recovery process for server connectivity.

We will update this thread as and when we get more information about this.

Posted April 18th, 2022 9:37 am GMT

Investigating

We have received reports of network issues impacting Shared Hosting and Email Hosting Servers, hosted on our Austin DC.

You might face issues while:

  • Accessing the Website on a browser
  • Accessing the cPanel/WHM Panel
  • Accessing webmail
  • Sending and receiving emails
  • Searching respective orders in Control Panel

Our network team is further working on restoring the server connectivity. An update will be provided to this thread as soon as additional information is available.

Posted April 18th, 2022 8:18 am GMT

Affected Services

  • Linux Web Hosting
  • Windows Web Hosting
  • Email Hosting
  • Network