Softserve is a service that lets our community loan machines to debug test failures. They create cloud VMs based on the image that we use for our test machines. We originally used pyrax to create and delete VMs. I spent some time trying to re-do that with libcloud.
Part 1: Writing libcloud code
I started by writing a simple script that created a VM with libcloud. Then, I modified it to do an SSHKeyDeployment, and further re-wrote that code to work with MultiStepDeployment with two keys. Once I got that working, all I had left was to delete the server. All went well, I plugged in the code and pushed it. Have you seen the bug yet?
Part 2: Deepshikha tries to deploy it
Because I’m an idiot, I didn’t test my code with our application. Our tests don’t actually go and create a cloud server. We ran into bugs. We rolled back and I went about fixing them. The first one we ran into was installing dependencies. Turns out that the process of installing dependencies for libcloud was slightly more complicated for some reason (more on that later!). We needed to pull in a few new devel packages. I sat down and actually fixed all bugs I could trigger. Turns out, there were plenty.
Part 3: I find bugs
Now I ran into subtle installation bugs. Pip would throw up some weird error. The default Python on Centos 7 is pretty old. I upgraded pip and setuptools inside my virtualenv to see if that would solve the pip errors and it did. I suspect some newer packages depend on newer setuptools and pip features and it fails quite badly when those are older.
After that, I ran into an bug that was incredibly satisfying to debug. The logs had a traceback that said I wasn’t passing a string as pubkey. I couldn’t reproduce that bug locally. On my local setup, the type was str, so I had to debug on that server with a few print statements. It turns out that the variable had type unicode. Well, that’s weird. I don’t know why that’s happening – unicode sounds right, something is broken on my local setup. A check for “strings” in python2 should check for str and unicode. The code does the following check which returns False when pubkey is a unicode:
isinstance(pubkey, basestring) On a first glance, that looked right. On python2, str and unicode are instances of basestring. A bit of sleuthing later, I discovered that libcloud has their own overridden basestring. This does not consider unicode to be an instance of basestring. I found this definition of basestring for python2:
basestring = unicode = str That doesn’t work as I expect on Python 2. How did this ever work? Is it that almost everyone passes strings and never runs into this bug? I have a bug filed. When I figure out how to fix this correctly, I’ll send a patch. My first instinct is to replace it with a line that just says
basestring In python2, that should Just Work™. Link to code in case anyone is curious. One part of me is screaming to replace all of this with a library like six that will handle edge cases way better.
Leave a Reply