Platform stopped processing events after new release deployment
Incident Report for Simplr
Postmortem

Incident:

Platform unusable on 11/21/2021 between 11:00am - 12:15pm PDT

Timeline:

  • Operations team reported platform sluggishness around 11:00 AM PDT
  • Problem resolved at 12:15 pm PDT after rolling back the release and killing long running DB queries

RCA:

A new slow query caused database to back up and platform stopped processing new events. Release was reverted back.

Posted Sep 21, 2021 - 12:27 PDT

Resolved
Release reverted back
Posted Sep 21, 2021 - 12:15 PDT