On September 6th, 2017, we experienced two separate incidents affecting API services. Both were related to underlying database issues. Layer utilizes a database and access library to more easily map database objects to more usable collections in our software. We have identified several instances where queries generated by this automated mapping tool resulted in sub-par performance. When coupled with overall elevated growth of our customer base and usage patterns, previously un-identified issues with query optimization, indexing, and connection pools manifested as extreme latency and API call failures.
To address this, we have begun immediately performing a holistic look at all database queries and data models related to our API services. The first phase of this, involving queries and model issues that came to light today, has already been completed. We plan to identify and remedy any additional optimizations over the course of the next 72 hours. Additionally, we are assessing and implementing options to gracefully degrade non-critical messaging features during failure scenarios such as those that occurred.
Lastly, we are immediately implementing a moratorium on feature deployments and a less aggressive cadence on deployments overall, until such a time that we have completed the assessment outlined above.