Nigel Babu

Moving from pyrax to libcloud: A story in 3 parts

Softserve is a service that lets our community loan machines to debug test failures. They create cloud VMs based on the image that we use for our test machines. We originally…

Softserve is a service that lets our community loan machines to debug test failures. They create cloud VMs based on the image that we use for our test machines. We originally used pyrax to create and delete VMs. I spent some time trying to re-do that with libcloud.

Part 1: Writing libcloud code

I started by writing a simple script that created a VM with libcloud. Then, I modified it to do an SSHKeyDeployment, and further re-wrote that code to work with MultiStepDeployment with two keys. Once I got that working, all I had left was to delete the server. All went well, I plugged in the code and pushed it. Have you seen the bug yet?

Part 2: Deepshikha tries to deploy it

Because I’m an idiot, I didn’t test my code with our application. Our tests don’t actually go and create a cloud server. We ran into bugs. We rolled back and I went about fixing them. The first one we ran into was installing dependencies. Turns out that the process of installing dependencies for libcloud was slightly more complicated for some reason (more on that later!). We needed to pull in a few new devel packages. I sat down and actually fixed all bugs I could trigger. Turns out, there were plenty.

Part 3: I find bugs

Now I ran into subtle installation bugs. Pip would throw up some weird error. The default Python on Centos 7 is pretty old. I upgraded pip and setuptools inside my virtualenv to see if that would solve the pip errors and it did. I suspect some newer packages depend on newer setuptools and pip features and it fails quite badly when those are older.

After that, I ran into an bug that was incredibly satisfying to debug. The logs had a traceback that said I wasn’t passing a string as pubkey. I couldn’t reproduce that bug locally. On my local setup, the type was str, so I had to debug on that server with a few print statements. It turns out that the variable had type unicode. Well, that’s weird. I don’t know why that’s happening – unicode sounds right, something is broken on my local setup. A check for “strings” in python2 should check for str and unicode. The code does the following check which returns False when pubkey is a unicode:

isinstance(pubkey, basestring)

On a first glance, that looked right. On python2, str and unicode are instances of basestring. A bit of sleuthing later, I discovered that libcloud has their own overridden basestring. This does not consider unicode to be an instance of basestring. I found this definition of basestring for python2:

basestring = unicode = str

That doesn’t work as I expect on Python 2. How did this ever work? Is it that almost everyone passes strings and never runs into this bug? I have a bug filed. When I figure out how to fix this correctly, I’ll send a patch. My first instinct is to replace it with a line that just says

basestring

In python2, that should Just Work™. Link to code in case anyone is curious. One part of me is screaming to replace all of this with a library like six that will handle edge cases way better.

Dublin Adventures: Howth Cliff Walk

In Dublin last month, I did the Howth cliff walk. I originally intended to do the walk, but when I reached Howth, I wasn’t sure anymore. I was tired and it looked like it might rain. Turns out, I was just hungry. I stopped at a pub for fish and chips, and cider. Then, I decided I was going to do the walk anyway. I picked up a map of the route along the way. There are four different routes: Lower Cliff Loop (easy), Tramline Loop (easy), Masts Loop (moderate), and Bog of Frogs Loop (hard). I was pretty sure I didn’t want to do just the easiest loop. I was contemplating doing the Bog of Frogs Loop as I started. The route starts off with a paved road that takes you along a sharp uphill. There are some views out to the beach along the way. Once you reach a reasonable height, there’s a parking area. This is where the hiking trail actually starts. The map in the parking lot said that the Tramline Loop would take me a good 1.5h to 2h to complete and Bog of Frogs Loop would take about 3.5h. I decided I’d take a mix of both. I wanted to go as far as the Baily lighthouse. Then, I’d trek back to the Summit and head to the railway station.

The weather was near perfect. To Dubliners, I suspect, it was just hot. The trail was crowded as well. I saw people of all ages along the way. On the trail, I watched the Dublin Bay Cruise ship making its way out of Howth. I even saw a couple of flights on their arrival path in Dublin. On my own flight into Dublin, I had seen the cliffs. The cliffs were windy, green, and the air was filled with the sound of seagulls and the waves crashing along the cliffs. I stopped to take pictures along the way, but was among the more faster walkers.

I’d picked up a local SIM earlier in the day, so I was also sending pictures of the cliff walk to friends and we had a banter about it going on. Every turn brought with it new sights. I finally reached the turn, from where I could see the lighthouse. That’s when I noticed that I’m at the highest point of the path I was currently taking. The walk down to the cliffs would be a sharp downhill. I have pictures of this, but the pictures don’t do justice. Climbing down was scarier than climbing up. If I had to do it all over again, I’d tell myself to travel with a good pair of hiking shoes rather than running shoes 🙂

After I reached the lighthouse, I went back up to the sort of “peak”. The path to the actual Summit was a little more harder than I expected. I managed to brave it through. Sadly, there are no pictures of this feat. I was busy trying not to fall. From the top, I expected to see Dublin city. It was a gorgeous day, but not clear enough for me to get a clear picture of the view. With most of the hike done, all I had to do was walk back to the railway station and finally head back to my hotel. This stretch was much easier. I should have done this hike in reverse. That would have been a much smoother experience. When I was crossing a row of houses, I met someone who lived nearby. He walked in step with me and asked me about how the hike went 🙂 He’s done the Bog of Frogs Loop, the path I did not fully take. The next time I travel, I’ll plan to allocate more time to hiking.

Welp, I ran out of passport pages

In 2007, when my first passport expired, I didn’t bother applying for a new one. I don’t really travel all that much, what’s the point of doing a whole bunch of

In 2007, when my first passport expired, I didn’t bother applying for a new one. I don’t really travel all that much, what’s the point of doing a whole bunch of paperwork for nothing. Fast-forward a few years, I couldn’t apply to travel to a UDS because I didn’t have a passport. That’s when I applied for a new passport. When I got the passport, I really just wanted to go to UDS. I believe the first visa I applied with this passport got denied as well 🙂 It’s been 8 years or so since I got this passport. I’m now out of pages. All of my empty pages have been taken up by visas and passport stamps. I have a few pages left but not enough continuous pages to get a new visa.

Flipping through the pages, I’ve traveled to Hungary, Malaysia, Singapore, UK, Kenya, Tanzania, Germany, USA, and Czech Republic, and Ireland. There are stories about a few of the visas and the trouble I went through to get them. In fact, every single visa involves, on average, at least 16 hours of paperwork that I’ve done per visa. Some countries require more and some countries less. The visa that I’ve often been most nervous about getting on time has been UK. The visa that I found easiest to get was Kenya. Kenya is among the few countries that has visa on arrival for Indian citizens.

It’s been a ride for the last few years. I’m hoping that the next passport doesn’t fill up before 10 years 🙂

Resetting My Running

2017 was admittedly not a good year for running. Mumbai is a great city, but I couldn’t find the time or the space to get my running in the rhythm it used…

2017 was admittedly not a good year for running. Mumbai is a great city, but I couldn’t find the time or the space to get my running in the rhythm it used to be in the previous season. I loved the long runs in Mumbai. It was amazing to run on Marine drive or to run with the Mumbai Road Runners. But the daily easy runs never happened and I fell off the wagon. This year, the first book I finished reading is Daniels’ Running Formula. If you’re starting out running or somewhere in the mid-level where you feel stuck, I fully recommend you read this book. It’s oriented towards fast runners as well as beginners. It gives you a great amount of foundational knowledge of running and workouts. For instance, it explains what the different kinds of workouts do and how they help you become a better runner. This means you know exactly why you’re doing a specific speed workout.

I’m resetting my running training from this month. I’ve gained weight almost to the exact point when I first finished the C25K plan. Instead of restarting C25K, I’m starting with Jack Daniels’ White Plan. It’s meant for people who’re absolute beginners. I’ve checked my VDOT and I’m starting with a VDOT of 25.6.My goal this year is to build this up. I’m going to be running ADHM, but I plan on racing very little otherwise. I’ll be doing 10Ks if I’m racing. I plan to work through a few of the beginner plans to build up my mileage and VDOT. This year, I’m focusing my running on actually being fitter and less to do with running running races or pace. I want to build a habit of waking up every day with a run.

The White Plan has me running easy for the most part with walking breaks in between at a pace that’s getting more and more comfortable as I keep repeating the workout. This year, I have a modest goal, set a habit of running every week 🙂

Building Gluster with Address Sanitizer

We occasionally find leaks in Gluster via bugs filed by users and customers. We definitely have benefits from checking for memory leaks and address…

We occasionally find leaks in Gluster via bugs filed by users and customers. We definitely have benefits from checking for memory leaks and address corruption ourselves. The usual way has been to run it under valgrind. With ASAN, the difference is we can compile the binary with ASAN and then anyone can run their tests on top of this binary and it should crash in case it comes across a memory leak or memory corruption. We’ve fixed at least one bug with the traceback from ASAN.

Here’s how you run Gluster under ASAN.

./autogen.sh ./configure --enable-gnfs --enable-debug --silent make install CFLAGS="-g -O0 -fsanitize-recover=address -fsanitize=address" -j 4

You need to make sure you have libasan installed or else this might error out. Once this is done, compile and install like you would normally. Now run tests and see how it works. There are problems with this approach though. If there’s a leak in cli, it’s going to complain about it all the time. The noise doesn’t imply that fixing that is important. The Gluster CLI is going away soon. Additionally, the CLI isn’t a long running daemon. It’s started, does it’s job, and dies immediately.

The tricky part though is catches memory you’ve forgotten to free. It does not catch memory that you’ve allocated unnecessarily. In the near future, I want to create downloadable RPMs which you can download and run tests against.

The configuration I’ve setup lets you continue to run the program after the first memory corruption by setting the environment variable ASAN_OPTIONS=halt_on_error=0. If you find an existing leak you are not interested in fixing, you can suppress it. More information on the wiki page