The other day at work, I was working on setting up a new database server. This is the first time in a while we’re doing this. Almost no-one remembers who or how it was done the last time. Our data is kinda big, so we tend to put the mysql data files into an EBS volume by itself so that we always have the data separate from the machine and because we get as much space as want. We created the new machine, new disk, changed the path of the data folders, and started mysql. BAM! It threw a whole bunch of errors about permissions.
I went in and checked the ownership, which seemed to be correct, but re-ownershipped everything anyway. Tried again. Nope, didn’t work. Out of frustration, tried again after doing a chmod -R 777. Still failed. For a while, we googled extensively for the error, leading us to nothing much to go on. Before this, we had some backup stuff to do, so I think it was close to 1 am when we actually got down to troubleshooting this. After sometime, we had the sense to google for what we wanted to accomplish, leading me to apparmor.
Then, my memory kicked in about Apparmor and what it did. I figured out that mysql probably didn’t have permission to use other directories. We gave it permissions and it worked! But, we ended up not having enough time to restore data on this new server in and rotate out the old server. Overall, we were working on this from 12 am to 4 am. The next day, my QOTD was from my friend, who shall not be named – ‘Oops. That said, it’s happened to me, too. The irony bit is that I’m one of the primary upstream apparmor devs.’
Leave a Reply