About Integrates failing to load on 2019-12-11

What happened

As part of the implementation of a series of new functionalities in which we are working, yesterday morning (9:57-10:50) and afternoon (17:13-18:50) the users of Integrates experimented errors when they tried to access any tab of a project.

Both errors were caused by changes in the structure of directories and files in our repository. With new deployments, a cache was stored from previous deployments and the API endpoint was unable to respond to new requests.

What we have done

Both errors were caused by the same problem but were fixed by different means. The first one was solved with a fresh deployment of the application. The other one was solved with a purge of the server cache.

What was the impact

During approximately two and a half hours, all projects, with their indicators, findings, events, etc, were inaccessible to all Integrates users.

What we are doing to help

  • We made some changes to our CI/CD in order to improve the identification of critical files that requiere cache invalidation during new builds.