Domain Renewals - Queued

Recently, two new vulnerabilities known as 'Spectre' and 'Meltdown' were discovered in a number of processors from Intel, ARM and AMD that could allow hackers to access passwords, encryption keys and other private information from open applications.

To fix this vulnerability, a new kernel security patch by KernelCare was applied. However, due to a bug in this patch, it caused performance degradation of application/database servers. We have rolled back and disabled Auto Update post which servers are now stable and running smoothly.

At approximately 07:30 AM IST(02:00 AM GMT) we noticed an issue with domain renewals. The renewals actions were being placed successfully, however they were not being processed and were getting stuck at ‘ExecutionStarted’.

We immediately escalated this issue to the Product Development Team and all hands were on deck. Initial investigations revealed that there was an issue with one of our Archive Databases, as a result of which the renewal actions were stuck at the status “ExecutionStarted”.

In order to be compliant with the Registry Business Rules, the renewal CRON checks if a domain was restored in the last 5 days. This check is made on our Archive Database since the System archives all actions within 48 hours. Now, since the renewal CRON was unable to connect to the Archive Database server, the system could not process the renewal actions.

The Product Development Team worked closely and restarted the Archive Database Server which fixed the issue. All stuck renewal actions were processed immediately.

A detailed RCA will follow soon however listed below are the possible causes for the Database Server issue:

1. There could be a segmentation fault with the memory used on our Archive Database Servers
2. A firmware update may be required
   

As mentioned earlier all renewal actions have been processed and there is currently no issue with our CRON servers, stay tuned for further updates on this thread.