Yesterday, we had a 6-hour long tree-closure. From 11:48 am PST to 16:43 pm PST. Philor closed the trees when he noticed the issue. By the time I noticed it, it was much later and luckily, I was in a room with the A-team and Releng. I walked over to catlee and nthomas. They both took a look and we were trying to figure out the problem. Wes and Ryan were helping me later and we just called rail and [mrrrgn][morgan] over to help us sort it out. This is much better than IRC!
Couple of confounding things at that point – It was happening across trees, and it didn’t feel like a code issue. However, Releng are all here and haven’t landed any changes. Morgan and rail identified the problem pretty soon. We were deleting temp files on AWS machines as well instead of only on SCL3 machines. Among the temp files we deleted were pulseaudio files. Deleting these files will break pulseaudio.
Now the funny thing about this bug. We ran into it only now because we had so few commits. Let me say that again, we triggered a bug in the automation because we had so few commits. That is hilarious, probably in a gallows humor kind of way.
Leave a Reply