Softserve is a service that lets our community loan machines to debug test failures. They create cloud VMs based on the image that we use for our test machines. We originally used
pyrax to create and delete VMs. I spent some time trying to re-do that with
Part 1: Writing libcloud code
I started by writing a simple script that created a VM with libcloud. Then, I modified it to do an SSHKeyDeployment, and further re-wrote that code to work with MultiStepDeployment with two keys. Once I got that working, all I had left was to delete the server. All went well, I plugged in the code and pushed it. Have you seen the bug yet?
Part 2: Deepshikha tries to deploy it
Because I’m an idiot, I didn’t test my code with our application. Our tests don’t actually go and create a cloud server. We ran into bugs. We rolled back and I went about fixing them. The first one we ran into was installing dependencies. Turns out that the process of installing dependencies for libcloud was slightly more complicated for some reason (more on that later!). We needed to pull in a few new devel packages. I sat down and actually fixed all bugs I could trigger. Turns out, there were plenty.
Part 3: I find bugs
Now I ran into subtle installation bugs. Pip would throw up some weird error. The default Python on Centos 7 is pretty old. I upgraded pip and setuptools inside my virtualenv to see if that would solve the pip errors and it did. I suspect some newer packages depend on newer setuptools and pip features and it fails quite badly when those are older.
After that, I ran into an bug that was incredibly satisfying to debug. The logs had a traceback that said I wasn’t passing a string as pubkey. I couldn’t reproduce that bug locally. On my local setup, the type was
str, so I had to debug on that server with a few print statements. It turns out that the variable had type
unicode. Well, that’s weird. I don’t know why that’s happening –
unicode sounds right, something is broken on my local setup. A check for “strings” in python2 should check for
unicode. The code does the following check which returns
pubkey is a
On a first glance, that looked right. On python2,
unicode are instances of
basestring. A bit of sleuthing later, I discovered that libcloud has their own overridden
basestring. This does not consider
unicode to be an instance of
basestring. I found this definition of
basestring for python2:
basestring = unicode = str
That doesn’t work as I expect on Python 2. How did this ever work? Is it that almost everyone passes strings and never runs into this bug? I have a bug filed. When I figure out how to fix this correctly, I’ll send a patch. My first instinct is to replace it with a line that just says
In python2, that should Just Work™. Link to code in case anyone is curious. One part of me is screaming to replace all of this with a library like six that will handle edge cases way better.