Intermittent issue with Orderbox Login

Here is the RCA for the Control Panel login issue we faced earlier today.

Issue:

At around 11:50 AM IST, the System Admin team observed alerts for high response time for API calls. Connectivity from GPX agent nodes to HAProxy indicated everything to be normal.

Upon further investigation, the team found out that the login related issue was being caused due to rate-limiting on our new OKD cluster to which the Control Panel was migrated recently. The rate-limiting rules that were in force on the new infrastructure were determined to be low compared to the incoming traffic.

Since we received comparatively high traffic today to our control panel endpoints, it triggered the rate-limiting rules for API calls.

Impact:

Customers were unable to login to the Control Panel.

Root Cause:

The rate-limiting rules on the new OKD cluster were triggered as a result of the comparatively high traffic earlier today, which led to login requests being dropped and thereby returning an error message.

The System Admin team identified this and updated the rate-limiting threshold to compensate for the increased traffic on the new infrastructure.

Once this was done the load stabilized and everything started functioning normally.

The issues with OrderBox Control Panel login have now been fixed and services are operational.

We are continuously monitoring our Servers to ensure that services are stable.

We are currently experiencing intermittent login issues while accessing the OrderBox control panel. Our System Administration team has been alerted of this issue and is already working towards resolving it at the earliest.

We regret the inconvenience caused to you and shall continue to post updates regarding the issue here at regular intervals.

Your patience and cooperation are highly appreciated.