Starting at approximately 12:30 Eastern on 2017-06-30, we started noticing backlogs with writing data to our Elasticsearch cluster. The problem was isolated to today's data. After many attempts to resolve the issue, we decided to remove the problem index and restore from a backup. We attempted to take a snapshot of the current data, but the cluster problem prevented a new snapshot from succeeding. Our last successful snapshot was from 10:30AM Eastern. At approximately 2:00PM Eastern we removed the problem data and recreated the index to allow current events to write successfully.
Unfortunately this means that we were unable to recover activity events from approximately 10:30AM to 2:00PM Eastern.
What this means for customers
- Activity events between approximately 10:30AM to 2:00PM are gone. This includes Sent, Opened, Delivered, and Clicked events.
- Statistics and web hooks for unique opens and unique clicks might not be accurate for today.
- Actual email sending was not affected.
- Inbound processing and activity was not affected.