Softserve is a service that lets our community loan machines to debug test
failures. They create cloud VMs based on the image that we use for our test
machines. We originally used
pyrax to create and delete VMs. I spent some
time trying to re-do that with
I started by writing a simple script that created a VM with libcloud. Then, I modified it to do an SSHKeyDeployment, and further re-wrote that code to work with MultiStepDeployment with two keys. Once I got that working, all I had left was to delete the server. All went well, I plugged in the code and pushed it. Have you seen the bug yet?
Because I’m an idiot, I didn’t test my code with our application. Our tests don’t actually go and create a cloud server. We ran into bugs. We rolled back and I went about fixing them. The first one we ran into was installing dependencies. Turns out that the process of installing dependencies for libcloud was slightly more complicated for some reason (more on that later!). We needed to pull in a few new devel packages. I sat down and actually fixed all bugs I could trigger. Turns out, there were plenty.
Now I ran into subtle installation bugs. Pip would throw up some weird error. The default Python on Centos 7 is pretty old. I upgraded pip and setuptools inside my virtualenv to see if that would solve the pip errors and it did. I suspect some newer packages depend on newer setuptools and pip features and it fails quite badly when those are older.
After that, I ran into an bug that was incredibly satisfying to debug. The logs
had a traceback that said I wasn’t passing a string as pubkey. I couldn’t
reproduce that bug locally. On my local setup, the type was
str, so I had to
debug on that server with a few print statements. It turns out that the
variable had type
unicode. Well, that’s weird. I don’t know why that’s
unicode sounds right, something is broken on my local setup.
A check for “strings” in python2 should check for
unicode. The code
does the following check which returns
pubkey is a
On a first glance, that looked right. On python2,
basestring. A bit of sleuthing later, I discovered that libcloud
has their own overridden
basestring. This does not consider
unicode to be an
basestring. I found this definition of
basestring for python2:
basestring = unicode = str
That doesn’t work as I expect on Python 2. How did this ever work? Is it that almost everyone passes strings and never runs into this bug? I have a bug filed. When I figure out how to fix this correctly, I’ll send a patch. My first instinct is to replace it with a line that just says
In python2, that should Just Work™. Link to code in case anyone is curious. One part of me is screaming to replace all of this with a library like six that will handle edge cases way better.