I got paged a bit ago because one of our sites was down. This site does go down occasionally because of an issue with Solr. Usually, it just needs a restart of Solr and a restart of background queues. Obviously, that didn’t work this time around. I had a sneaky suspicion this was not the case since we didn’t have any Solr related alerts.
This looked more like a problem with the frontend server and it was indeed. The logs said it was out of space, and df -h reported plenty of free space. Having been in this situation before, I knew it was likely to be inodes. df -i is handy to debug in these situations.
I narrowed down the problem thanks to this great answer on Stack Overflow by simon. Essentially, go to your root folder and run this command:
sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
You may need to edit it to the following form to get it to actually work (thanks to Frederick Nord’s comment):
sudo find . -xdev -type f | cut -d "/" -f 2 | sort --buffer=1G | uniq -c | sort -n --buffer=1G
Now find the folder with the largest number of inodes and start digging deeper. In my case it was a cache folder that created tmp files for sessions, which wasn’t ever cleared.
Image Credit: Crowd by James Cridland on Flickr.
Leave a Reply