RCA - Runtime JS Issue of August 3rd, 2023 

Executive Summary  

On August 3rd 2023, Discover runtime JS services were unreachable for around five hours.  During this time, runtime JS based implementations of Preview Search, Search Results Page, Recommendations and Hosted Product Listing Pages were not displayable on customer sites. 

Sitecore Discover team worked on a recovery strategy and applied the fix for each customer after this issue was investigated and identified  

Root Cause 

One of the Sitecore’s Content Delivery Network (CDN) resources residing within our AWS infrastructure was inadvertently removed due to a misinterpretation of the resource’s labelling. The removal of the CDN distribution hosts reflektion.js and customer CSS files, critical to the runtime JS components implemented on customer sites, directly caused customer’s site experiences to not load. 

Timeline (all time in PDT) 

6:00AM - Internal alert fired for CDN monitoring 

6:30AM - Customer started to file support case regarding Discover services not working 

7:00AM - Discover Engineering team received escalation and moved to further investigation 

8:30AM - Discover Engineering team worked out a recovery plan and updated one customer  

11:00AM - Discover engineering team applied fixes for all affected customers 

14:15PM - All affected accounts are updated and verified to be back to normal 

Impacts 

All and only customers implementing Sitecore Discover’s services via runtime JS experienced an outage whereby the runtime JS components did not load on their corresponding site.  

Planned changes / follow-ups 

1. Internal alert escalation policy has been revisited and enforced for faster reaction times of any future like-issue.   
2. All AWS resources’ labels will be revisited to improve comprehension and avoid future confusion. 
3. Enhancements for faster recovery in such cases where runtime JS-related updates across our customer base are required.