Reduce data loss on deploy

[devinmatte]

When we push a new deploy, a few things happen or can happen:

1. Service restarts, missing a few data points

In a rare case, the EC2 instance needs to be recreated. In this case we're going to miss a lot of data points. Things we can do to reduce that:

1. Force data to be pushed to s3 during a deploy so we lose the minimal amount
2. When new EC2 is started up, pull down all of todays data from s3 so the service doesn't overwrite all earlier data on first upload
3. Only apply these updates at 3am 🙃

[ankoure]

We may be able to use the Last-Event-ID header to reduce the number of data points lost on re-connection. From reading about the SSE specification, it sounds like if this id is passed as a header to the server, it will resend all events from the last successfully received event. Not sure how far back the server would re-send events but worth giving it a try.

https://stackoverflow.com/questions/61541750/server-sent-events-sse-automatic-reconnect

https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events