Intermittent issues with Orderbox

Here is the RCA for the OrderBox Downtime we faced on the 23rd of March, 2020.

Issue:

At around 05:40 PM IST / 12:10 PM GMT, we received a few down alerts from our Monitoring System.

Our System Administration Team along with the Development Team immediately began troubleshooting the issue. They noticed that all database connections were being used up as a result of which requests were queuing up on our Load Balancer.

Impact:

API, Control Panels & Supersite were not functioning at the optimum level.

Root Cause:

The database connection pool was getting exhausted as database connections were not being released by a particular process once a transaction was completed.

We identified this and modified the code such that every database connection is closed once the transaction is completed.

Once this was done the load stabilized and everything started functioning normally.

We were observing load on our OrderBox Servers due to which you were facing an issue while accessing the Control Panel and API. Our System Administration team has worked towards stabilizing the load and the services are now working fine.

We are currently monitoring our Servers closely and will update this post with a detailed RCA soon.

We are currently facing issues with OrderBox causing intermittent issues while accessing the control Panel, making API calls & managing orders.

Our System Administration Team is already working on this issue and are trying to fix this as soon as possible.

We sincerely regret the inconvenience caused. Please watch this thread for further updates.