About Integrates data fetch error on 2020-04-23

What happened

  • We are currently updating our API using a completely new backend. As part of this work, and seeking for creating a faster API, a commit was introduced with the goal of improving some technical aspects of data fetching and concurrency handling. Although the change accomplished the desired effect, the current in-memory cache data was not updated to reflect the changes performed.
  • This led to unavailability of parts of the application retrieving information using the method introduced by the change.

What we’ve done

  • After identifying the problem, the cache was purged to allow filling it with the new data structure

What’s the impact

  • The change was deployed to production at 10:28 am of April 23. 12 minutes was the time between the change was deployed to production and the cache was purged.
  • During that time, 13 users were affected by the unavailability of retrieving parts of the projects information.

What we are doing to help

  • Those kind of changes are rare. However, we identified the pattern of code changes that may affect the in-memory cache structure and defined means to purge the cache when this happens.