← Incidents

Starting at approximately 12:30 Eastern on 2017-06-30, we started noticing backlogs with writing data to our Elasticsearch cluster. The problem was isolated to today's data. After many attempts to resolve the issue, we decided to remove the problem index and restore from a backup. We attempted to take a snapshot of the current data, but the cluster problem prevented a new snapshot from succeeding. Our last successful snapshot was from 10:30AM Eastern. At approximately 2:00PM Eastern we removed the problem data and recreated the index to allow current events to write successfully.

Unfortunately this means that we were unable to recover activity events from approximately 10:30AM to 2:00PM Eastern.

What this means for customers

  • Activity events between approximately 10:30AM to 2:00PM are gone. This includes Sent, Opened, Delivered, and Clicked events.
  • Statistics and web hooks for unique opens and unique clicks might not be accurate for today.
  • Actual email sending was not affected.
  • Inbound processing and activity was not affected.



Incident’s History

UP

We were to restore the activity data for today from before 10:30AM Eastern. Data from 10:30AM to 2:00PM is still gone.

We're still working on restoring outbound activity from before 10:30AM Eastern.

Messages sent moving forward will update in Activity as normal. We are still working on message activity for emails sent before 10:30 AM ET.

DOWN

Activity messages are currently delayed, we are working on a fix.