Microsoft has finally broken Linux –
The outage prevented Linux from installing or upgrading any Microsoft software.
In 2017, Tux was sad to have a Microsoft logo on his chest. In 2021, he was mostly upset that Microsoft’s software library was down for most of the day.
Jim Salter
Yesterday, Packages.Microsoft.com, Microsoft’s repository of software installers for Linux distributions, including CentOS, Debian, Fedora, OpenSUSE and others, went down severely for about 18 hours. This failure affected the attempted installation. NET Core, Microsoft Teams, Microsoft SQL Server for Linux (yes, that’s a thing), and Azure’s own development pipeline.
We first became aware of the problem Wednesday night when we saw a 404 error in the apt Update output on our Ubuntu workstation with Microsoft Teams installed. This one on Github. The NET Core Issue Report has a better record of this outage, with many users from around the world sharing their experiences and theories.
In short, the entire repository cluster that supplies all of Microsoft’s Linux packages went completely down — sending out a series of HTTP 404 (content not found) and 500 (internal server error) messages to any URL — for about 18 hours. Microsoft engineer Rahul Bhandari confirmed the glitch about five hours after the initial report, with a crypto comment that the infrastructure team was “experiencing some space issues.”
Eighteen hours after the problem was detailed, Bhandari said, mirroring was available again — though performance was temporarily reduced, possibly due to cold caching. In the update, Bhandari said that the initial cause of the outage was “a backsliding of the [APT repository] during some feature migration work that made these packages unavailable on mirrors.”
We are still waiting for a full report of the incident as Bhandari’s status update provides clues but no real explanation. The good news is. We can confirm that Packages.Microsoft.com is indeed up and running again, and that it is providing the packages it should.