Update to Emergency Maintenance
Incident Report for Figure Eight
Resolved
Production is stable and we have resolved the emergency maintenance. Certain functionality will be significantly delayed including reports. Ok to resume normal activity. Thank you for your understanding. We are continuing to monitor the situation for any abnormalities.
Posted Feb 21, 2019 - 13:31 PST
Update
Production continues to be up in a limited capacity. Report downloads are still impacted and intermittent errors are being reported by contributors, however completed judgements are accessible via API. We will have another planned outage at 1pm PST for approximately 30min for emergency maintenance to increase capacity to resolve worker issues. Reports will continue to be delayed while a database index continues to be build. Thank you for your continued patience as we work around the clock to get back up to 100% operational.
Posted Feb 21, 2019 - 11:32 PST
Update
Thank you for your patience. We have restored partial operations. Some key functionality including reports is still not operational. Please refrain from launching high volume jobs as we continue to bring back to full functionality.
Posted Feb 21, 2019 - 09:38 PST
Update
We continue to work on restoring production - we are smoke testing and bringing various systems back online slowly but surely. Our sincerest apologies for the inconvenience.
Posted Feb 20, 2019 - 23:56 PST
Update
We are continuing to rebuild database indexes. Thank you for your patience. We want this up and running as much as you do and are working with some of the leading experts in the world to restore production as soon as possible.

Apologies for the inconvenience.
Posted Feb 20, 2019 - 20:35 PST
Update
We are rebuilding database indexes now and hope to have service restored soon after completion. Engineers are working diligently to optimize recovery time.

We apologize for the inconvenience caused by this interruption in service availability.
Posted Feb 20, 2019 - 15:30 PST
Update
Thank you for your continued patience. We are all hands on deck working on issues with our database. We will be back as soon as possible
Posted Feb 20, 2019 - 11:02 PST
Update
We are continuing to work on resolving systems issues and will provide updates and we near resolution.

Please accept our apologies for this interruption in service. Thank you for your patience.
Posted Feb 20, 2019 - 06:10 PST
Update
We are still working on restoring service to our main system. We will continue to update as we proceed.

Thank you for your continued patience. We sincerely apologize for any inconvenience this may be causing.
Posted Feb 20, 2019 - 04:36 PST
Monitoring
We are still working on bringing the systems back up. The operation is taking longer than expected and we sincerely apologize.
Posted Feb 20, 2019 - 01:02 PST
Investigating
Unfortunately our maintenance is taking longer than expected. Our team is very much hard at work at resolving the database related issues. We are looking to restore service as soon as possible.

Our sincere apologies for the inconvenience this may have caused.
Posted Feb 19, 2019 - 21:26 PST
This incident affected: Customer Portal and API.