This was caused by a recent upgrade to an application responsible for load-balancing requests to the AoC API. Upgrading this application is normally called non-disruptive, since it historically handles rolling restarts seamlessly. Our standard operating procedures requires that all changes go through QA and are verified to be non-disruptive before going through to production, which was the case here. However, there was one configuration setting in QA which did match production. The setting was for enabling/disabling gzip compression. It was disabled in QA, but enabled in prod. There was an issue with gzip compression in the new version of the application that went undetected as a result. Performing a standard roll back of the application version change remediated the issue.

This scenario highlighted the need for additional scrutiny of configuration differences between our Prod and QA environments, of which, many are legitimate. Fortunately, we are big proponents of GitOps here at Aspera, so looking back and doing a thorough check of our key configuration differences is a straight-forward task.

We apologize for the inconvenience caused.