About Integrates outage on 2020-10-09

What happened

  • We are currently working on standardizing our infrastructure through nix .
  • At 14:33 (COT) our team deployed a change that made use of docker experimental syntax. The change was expected to affect only base infrastructure, but due to how docker works internally, caches for Integrates containers were also lost.
  • At the same time, one of our sub-dependencies was updated and broke our source code compatibility.

What we’ve done

  • After an intensive debug process, our team reproduced the issue at 10:33 (COT) on 2020/10/10.
  • A temporary solution was implemented and service was restored at 2020/10/10 10:40 (COT)
  • Definitive fix was committed at 11 (COT).
  • A complementary solution was committed at 11:32 (COT).

What’s the impact

  • Users were unable to login to Integrates or use API since 2020/10/09 14:33 until 2020/10/10 10:40.

What we are doing to help

3 Likes