Screwups are important

it’s been quite a while since i blogged, more than 6 months or so. between a family emergency in december,

At my day job, one of the tasks that Kiran and I have to do frequently is to watch the job posting on the HasGeek Job Board and reject the ones that don’t conform to our Terms of Service. Last night, Kiran pinged me on IRC with a link to a job that he wanted me to knock off. Usually, I use phpmyadmin to do this, but I thought I’d turn it into a bash script and started writing a mysql update query.

Head in Hands by Alex E Proimos on Flickr

In retrospect, that’s probably my first mistake right there. Production server is really not the right place to have done this and I don’t even know why I thought it was a good idea. I wrote the query and executed it. Suddenly, I realized that I screwed up. I had that sinking feeling where you know exactly what went wrong and that it’s entirely your fault. I forgot the WHERE clause in the query and managed to reject every job in the job board. Thankfully, there was a backup handy, from about 10 minutes earlier too. Because of the power outages in North India and the fact that our servers are hosted by E2E Networks in Delhi, I had set up hourly backups earlier in the day. Quickly, I brought down apache and started restoring from the backup. In about 10 minutes from executing the wrong query, we were back and running.

We had two things to take away from this mistake – modifying the database directly should stop, and hourly backups are a good. We don’t have a lot of data yet, and hourly backups don’t take a lot of time. I spent all day today writing code so that we don’t touch the database manually ever for this. There’s been a plan to write this code for months, but true motivation came in the form of this embarrassing mistake. Most of us hate admitting our mistakes, but when working on servers, it is essential we move to a culture of blameless post-mortems to fix broken systems and ensure the same mistakes aren’t repeated or at least occur less frequently.

Switched to Clang

So, yeah, I’ve switched to clang on Ubuntu. A quick rundown of what I did for any Ubuntu or other Linux users.

A recent conversation in #devtools

<nigelb> My current build segfaults when I touch the webconsole. <robcee> that's no good <nigelb> That's counter productive when I'm patching the webconsole. <robcee> are you on linux? <nigelb> Ya. <robcee> are you building with clang? <nigelb> No I'm not. <nigelb> gcc forever <3 . a little while later . <nigelb> Wait, that's a gcc bug? <msucan> yes <nigelb> Sadness. <nigelb> gcc </3 <nigelb> its time to move on. <robcee> it was a good run 

So, yeah, I’ve switched to clang on Ubuntu. A quick rundown of what I did for any Ubuntu or other Linux users. The instructions are pretty much the same as what ehsan posted for Mac, except I didn’t install clang system-wide.

So, here’s what I did.

mkdir clang cd clang svn co llvm cd llvm/tools svn co clang cd ../../ mkdir build cd build ../llvm/configure --enable-optimized --disable-assertions make 

This got the clang binary in Release/bin. I use ZSH, so I added that folder to my $PATH variable like so

export PATH=$PATH:$HOME/clang/build/Release/bin 

The last bit is to add the following to your .mozconf

export CC=clang export CXX=clang++ 

Note – If you’re working on Ubuntu 10.04 with gcc version 4:4.4.3–1ubuntu1, it might be essential to use clang and not gcc. As of now the segfaults are attributed to gcc. For more details, see bug 694594

Mozcamp Asia

The long pending post about Asia Camp

Finally, I’m getting around the write my MozCamp Asia post. This has been long pending and got delayed mostly because I’ve been writing patches for Firefox and Thunderbird instead of writing a blog post 😛


The night before I was to due to travel, I fell ill and almost called the whole thing off. After taking a break for a day, I decided that I was good to go, started packing, and left for the airport. At the airport, I was sitting down and finishing the slides for my talk since I never got a chance thanks to work and other commitments. We landed in Kuala Lumpur at about 12 am local time and I got into my room at 2:30 am. I ended up waking up my roommate, Siddharth, who I’ve known for a while but never met. With all the excitement of the event, and also because it was just 12 am in India, I was wide awake (Proud member of Weird Hours) . At some point, I even thought of not sleeping, but thankfully, I caught about 3 hours of sleep, which at least saved the day to some extent.

Day 1

Day 1, right before we started

At Breakfast, I met the ‘IT crowd’ – fox2mike, zandr, phong, and bkero. Fun gang, we were hanging out later on Sunday. I then hurried to the Welcome session by Mitchell Baker. I don’t remember most of the session now, but I remember tweeting like crazy all through the morning. The update from Mozilla China was particular interesting because, as the representative from China said, ‘We don’t have Facebook or Youtube on our internet’. I also remember Mozilla Japan had an interseting video of their work and Chibi was on stage for the first time giving a talk in English. We had extra applause for her courage 🙂 During the coffee break, I caught up with Tim Watts, who works on the SUMO team. We realized that we were planning on talking about the same thing and decided we’d merge our talks into one (all the work I did in the airport had to be re-done. YAY!). Mary then talked about Engagement. There was a lot of learning from that session, like the fact that 2.2 million people download firefox every day, and about the huge poster in the JFK arrival area. Everyone loved the ‘Don’t work for the man, work for the mankind’ poster. Then, we had the most exciting bit ever, the community quilt. It was great to listen to all the communities talk about their efforts and also about their challenges. Again, it was great to hear people give a talk for the first time, especially in a langauge they’re not used to speaking all the time.

At lunch, I met Khairul (ejat) who I also know from the Ubuntu community and Stormy Peters who I’ve heard about from friends. After lunch I took one look at the schedule and wished I had a time machine. I wanted to attend all the sessions! After lunch I attended the session about building community websites. It was interesting to hear Laura talk about the Mozilla websites from the product owner’s point of view. After this, I went to the talk about Developer Evangelism, MDN, and Docs. We all agreed to blog about MDN and push it up in search rankings instead of ‘other’ usual sources (I’m looking at you w3schools). Next was probably one of the most interesting sessions for me, David Dahl’s ‘From Web Developer to Firefox Hacker’ session. I’ve been shying away from hacking on Firefox or Thunderbird, because I’m not exactly a C++ person. ddahl’s session kind of got me intersted in Firefox hacking and I’ve been doing a fair bit of Firefox hacking since I got back (Thanks again ddhalimage. The last session for the day was the user engagement session by Chelsea and William. It was nice to hear about “Affiliates”: and about how campaigns could use a lot of localized help to connect to the right audience.

Most of the things after this is a blur thanks to the lack of sleep from previous night. We went had dinner at a resturant which also had a cultural show, but I was too tired to watch any of it and just headed back to the hotel to crash.

Day 2

On the Day 1 open session, Mitchell announced that we’d all have wake up calls at 8 am and they actually did it) Thankfully, I’d set a 7 am wake up call, so we didn’t have the 8 am wake up call, but I did see other people comment about it on twitter. Despite all this, we did start about 15 minutes late because we waited for everyone to finish breakfast. I was much more alert thanks to a good night’s sleep. We opened the day with the ‘State of the Product’ talk. There was a lot of applause during the talk, especially for the B2G demo (philikon and qDot recently managed to get the B2G phone to dial). It was mind blowing to see the View Source. The devtools demo, especially, the Tilt demo also generated a fair bit of applause. After this, I attended Siddharth’s session about building a Thunderbird Extension. Sid0 demoed writing a simple Hello World extension to using Gloda. It was a fun session with a lot of code to digest through. The WebFWD talk was something I was looking forward to and Diane explained about what WebFWD does and how it can help potential startups with the same ideologies as Mozilla, especially openess.

The leader Q & A was interesting for its format and for the questions that were asked. Some of the answers, like the ones about Mozilla Spaces, and Thunderbird were particularly interesting. Right after lunch, was the session about contributing to Mozilla webdev led by Tim Watts and I. We talked about contributing to Mozilla’s Webdev projects, some of the blockers, and the respective solutions. Tim talked about everything except trouble with setting up the environment, while I focused on getting an environment setup with Vagrant. The audience already had people using Vagrant so it was fun to talk about it. We had most of Mozilla IT at our talk as well, so we had some good feeback from the about getting our environment as close as possible to production environment. I summarized most of these thoughts into a blog post on the Mozilla WebDev blog a few weeks back.

I then went to Dan’s session about BrowserID and apps. I had a lot of questions about BrowserID and few of us heckled Dan throughout and after the talk 😛 Nevertheless, it cleared up a lot of questions I had about BrowserID. The last session of the day before the closing was Community IT. Shyam, Phong, and Ben talked about the scale of IT at Mozilla and how they’re trying to open it up for the community to contribute. Memorable phrase from the session was ‘We’ll all be one big happy family and we’ll all have root’ by fox2mike 🙂 (and yeah, he’s probably going to kill me for that). Another issue raised was how bouncer wasn’t working and needed some debugging; as time permits over the next few months I intend to look into this and figure out how to improve it. Finally, we had the closing remarks from Mitchell. It dawned on me that Mozcamp was over and most of us would be leaving that night or the next morning, and it was indeed saddening.

Always stylish ddahl in Jalan Aloor

I joined Arky, bkero, cedricv, Dave, ddhal, fox2mike, Harinder, John, Mihca, phong, and zandr in exploring Jalan Aloor. Harinder suggested a chineese place there for dinner. It was indeed a great dinner. We did some experimentation with food – the biggest of which was Durian. When I entered the street I did smell some sort of sticky sweet smell which seemed pleasant. When we had the Durian, I matched the smell to Durian and it wasn’t so pleasant after that 😉 We also tasted some bakkwa (bacon candyimage and “Rambutan”: It was a long street filled with the most eatable things ever) We had a 10 pm drink up planned at the hotel, so we got back to hotel to be there on time, only to walk into an empty pub. Later everyone else joined us and we pretty much took over the pub. We were all chatting for the next few hours. I think it was eventually 4 am when I headed to bed.

tl;dr: I went to Mozcamp Asia, and started writing patches for Firefox 🙂

Durian adventure
The drinkup

Please nitpick

Please don’t rubber-stamp my reviews. Nitpick. Nitpick a lot.

When working with Open Source, some of the best learnings come from code reviews. When I first started coding on projects with strict code reviews Launchpad), I was uncomfortable getting nitpicked. I (wrongly) thought the goal was to have a code review that has absolutely no nitpicks (crazy right?). Whenever someone had a nitpick with my code, I felt like I couldn’t code properly. That’s when I read this excellent mail from Jeroen Vermeulen to the launchpad-dev team:

A rubber-stamp approval can save you minutes or more in the short term, but it does nothing for your longer-term development. A rubber-stamp approval leaves you without proof that the reviewer has read and understood the branch. A rubber-stamp approval sets no performance bar, no communicable standard for reviewers to live up to. A rubber-stamp approval does not tell you whether your branch was excellent, so-so, or too hard to read. A rubber-stamp approval buys you time that you could use towards the self-improvement you’re missing out on, but fails to tell you where you need it most. A rubber-stamp approval deprives you of a chance to harmonize your part of the codebase with our best practices. So I don’t like getting rubber-stamped any more than I like to rubber-stamp others. Good reviews take time, and that’s not time I put in for the sheer fun of it.

It changed my world. I’ve had epic nitpicky code reviews after that. Its even changing how I do my code reviews; I’m warming up to be more nitpicky. Lesson learnt – code reviews are one of the strongest takeaways from open source. So, if you’re reviewing code, please nitpick. Don’t rubber stamp. The professional development of the person whose code you’re reviewing depends on it!

PS: For acceptable levels of nitpick 😉

How I Got Involved With Mozilla

There have been a fair number of responses to David Boswell’s post about how he got involved with Mozilla. Most of the responses have been from fairly established contributors and contributing for a while. I thought I’d share my experience since I’ve been around for hardly 4 months now and moderately active.

There have been a fair number of responses to David Boswell’s post about how he got involved with Mozilla. Most of the responses have been from fairly established contributors and contributing for a while. I thought I’d share my experience since I’ve been around for hardly 4 months now and moderately active.

My first attempt at contributing to Mozilla was a disaster. It was around January 2011 and I tried to contribute to (AMO) before its cleaned database was publically available. It took me a while to set the whole thing up and eventually when I did have things setup, I couldn’t properly reproduce bugs since I had no data and I had trouble adding new data. Eventually, I gave up out of frustration and I had lesser time available and my interest died out. Things have definitely changed since, the database for AMO is now downloadable and I’m pretty sure my setup troubles were my lack of experience; thank you Wil Clouser for being patient with me back then.

Around June 2011, I applied to a web developer opening at Mozilla. The email which confirms my application said this, ‘If you’ve gotten this far, we’re certain you care about the future of the web. It’s easy to get involved and further that mission.’ I think this struck a note in me (whoever thought to add it, great job!). I wanted to give it another shot. This time, I just joined #webdev and asked if any projects needed help. Dave quickly responded saying that Input could use some help and I started checking out the code and setting it up. By July, I had cleaned up most of the easy bugs on Input and I continue to help fix bugs. I ended up not getting picked for the position I applied for, but I’m really glad to be contributing.

Things have changed since my attempt in January. There is now a significant interest in making virtual machine images available via either vagrant or some other means so setting up a dev environment is much easier (See posts by tofumatt, lochard, and groovecoder). This reduces the barrier of entry and frustration levels significantly. A frontend person need not be a linux sysadmin anymore to be able to fix a small CSS bug. Another good thing about this is, since everyone will be on the same environment, we’re reducing the chances of a weird environment problem.

Is the route I took still possible? I’d definitely say yes. If you know Python and Django or PHP or CSS or JavaScript, there are a bunch of webdev projects, and just asking in #webdev channel could guide you to projects that need help. It’s probably a good idea to ask during UTC–8 working hours since working hours of most of Mozilla WebDev overlap significantly around that time.