err &errhttps://www.erranderr.com/blog/2016-07-12T00:00:00-04:00Untangling WebDriver and the Browser Automation Landscape I Live In2016-07-12T00:00:00-04:002016-07-12T00:00:00-04:00Maja Frydrychowicztag:www.erranderr.com,2016-07-12:/blog/webdriver-ontology.html<p>I define WebDriver, <span class="caps">W3C</span> WebDriver, Selenium, geckodriver, FirefoxDriver, Marionette harness/runner/client/driver and more.</p><p>This piece is about too few names for too many things, as well as a kind of origin story for a web standard. For the past year or so, I’ve been contributing to a Mozilla project broadly named Marionette — <a href="http://vakila.github.io/blog/marionette-act-i-automation/">a set of tools for automating and testing Gecko-based browsers like Firefox</a>. Marionette is part of a larger browser automation universe that I’ve managed to mostly ignore so far, but the time has finally come to make sense of it.</p>
<p>The main challenge for me has been nailing down imprecise terms that have changed over time. From my perspective, “Marionette” may refer to any combination of two to four things, and it’s related to equally vague names like “Selenium” and “WebDriver”… and then there are things like “FirefoxDriver” and “geckodriver”. Blargh. Untangling needed.</p>
<p><em>Aside: integrating a new team member (like, say, a volunteer contributor or an intern) is the best! They ask big questions and you get to teach them things, which leads to filling in your own knowledge. Everyone wins.</em></p>
<h2>The <span class="caps">W3C</span> WebDriver Specification</h2>
<p>Okay, so let’s work our way backwards, starting from the future. (“The future is now.”) We want to remotely control browsers so that we can do things like write <a href="https://github.com/davehunt/bedrock/blob/e1580816fcedbd3e6fc7d7b95a06270d6cd4f08e/tests/functional/test_navigation.py#L12-L23">automated tests for the content they run</a> or <a href="http://www.hskupin.info/2016/06/02/firefox-ui-tests-platform-operations-project-of-the-month/">tests for the browser <span class="caps">UI</span> itself</a>. It sucks to have to write the same test in a different way for each browser or each platform, so let’s have a common interface for testing all browsers on all platforms. (Yay, open web standards!) To this end, a group of people from several organizations is working on the <a href="https://w3c.github.io/webdriver/webdriver-spec.html">WebDriver Specification</a>.</p>
<p>The main idea in this specification is the <strong>WebDriver Protocol</strong>, which provides a platform- and browser- agnostic way to send <a href="https://w3c.github.io/webdriver/webdriver-spec.html#list-of-endpoints">commands</a> to the browser you want to control, commands like “open a new window” or “execute some JavaScript.” It’s a communication protocol<sup id="fnref-1"><a class="footnote-ref" href="#fn-1">1</a></sup> where the payload is some <span class="caps">JSON</span> data that is sent over <span class="caps">HTTP</span>. For example, to tell the browser to navigate to a url, a client sends a <span class="caps">POST</span> request to the endpoint <code>/session/{session id of the browser instance you're talking to}/url</code> with body <code>{"url": "http://example.com/"}</code>.</p>
<p>The server side of the protocol, which might be implemented as a browser add-on or might be built into the browser itself, listens for commands and sends responses. The client side, such as a Python library for automating browsers, send commands and processes the responses.</p>
<p>This broad idea is already implemented and in use: an open source project for browser automation, Selenium WebDriver, became widely adopted and is now the basis for an open web standard. Awesome! (<em>On the other hand, oh no! The overlapping names begin!</em>) </p>
<h2>Selenium WebDriver</h2>
<p>Where does this WebDriver concept come from? You may have noticed that lots of web apps are tested across different browsers with <a href="http://www.seleniumhq.org/">Selenium</a> — that’s precisely what it was built for back in 2004-2009<sup id="fnref-2"><a class="footnote-ref" href="#fn-2">2</a></sup>. One of its components today is <strong>Selenium WebDriver</strong>. </p>
<p>(<em>Confusingly<sup id="fnref-3"><a class="footnote-ref" href="#fn-3">3</a></sup>, the terms “Selenium Webdriver, “Webdriver”, “Selenium 2” and “Selenium” are often used interchangeably, as a consequence of the project’s <a href="http://www.aosabook.org/en/selenium.html">history</a>.</em>)</p>
<p>Selenium WebDriver provides APIs so that you can write code in your favourite language to simulate user actions like this:</p>
<div class="highlight"><pre><span></span>client.get("https://www.mozilla.org/")
link = client.find_element_by_id("participate")
link.click()
</pre></div>
<p>Underneath that <span class="caps">API</span>, commands are transmitted via <span class="caps">JSON</span> over <span class="caps">HTTP</span>, as described in the previous section. A fair name for the protocol currently implemented in Selenium is <strong>Selenium <span class="caps">JSON</span> Wire Protocol</strong>. We’ll come back to this distinction later.</p>
<p>As mentioned before, we need a server side that understands incoming commands and makes the browser do the right thing in response. The Selenium project provides this part too. For example, they wrote <strong>FirefoxDriver</strong> which is a Firefox add-on that takes care of interpreting WebDriver commands. There’s also InternetExplorerDriver, AndroidDriver and more. I imagine it takes a lot of effort to keep these browser-specific “drivers” up-to-date.</p>
<h3>Then something cool happened</h3>
<p>A while after Selenium 2 was released, browser vendors started implementing the Selenium <span class="caps">JSON</span> Wire Protocol themselves! Yay! This makes a lot of sense: they’re in the best position to maintain the server side and they can build the necessary behaviour directly into the browser.</p>
<p>It started with <a href="https://seleniumhq.wordpress.com/2011/02/09/operadriver_released/">OperaDriver</a> in <a href="https://dev.opera.com/blog/operadriver-now-a-part-of-selenium-and-experimental-android-support-2/">2009-2011</a>, and then others followed such as <a href="https://sites.google.com/a/chromium.org/chromedriver/">ChromeDriver</a> and Mozilla’s <a href="https://github.com/mozilla/geckodriver">geckodriver</a> with <a href="https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette">Marionette</a>.<sup id="fnref-4"><a class="footnote-ref" href="#fn-4">4</a></sup> This is where the motivation for a WebDriver standard comes from.</p>
<h3>Let’s Review</h3>
<p>Selenium Webdriver (a.k.a. Selenium 2, WebDriver) provides a common <span class="caps">API</span>, protocol and browser-specific “drivers” to enable browser automation. Browser vendors started implementing the Selenium <span class="caps">JSON</span> Wire Protocol themselves, thus gradually replacing some of Selenium’s browser-specific drivers. Since WebDriver is already being implemented by <a href="http://www.theautomatedtester.co.uk/blog/2016/the-final-major-player-is-set-to-ship-webdriver.html">all</a> major browser vendors to some degree, it’s being turned into a rigorous web standard, and some day all browsers will implement it in a perfectly compatible way and we’ll all live happily ever after.</p>
<p>Is the Selenium <span class="caps">JSON</span> Wire Protocol the same as the <span class="caps">W3C</span> WebDriver protocol? Technically, no. The <span class="caps">W3C</span> spec is describing the future of WebDriver<sup id="fnref-5"><a class="footnote-ref" href="#fn-5">5</a></sup>, but it’s based on what Selenium WebDriver and browser vendors are already doing. The goal of the spec is to coordinate the browser automation effort and make sure we’re all implementing the same interface; each command in the protocol should mean the same thing across all browsers.</p>
<h2>A Fresh Look at the Marionette Family</h2>
<p>Now that I understand the context, my view of Marionette’s components is much clearer.</p>
<ul>
<li>Marionette Server together with <a href="https://github.com/mozilla/geckodriver">geckodriver</a> make up Mozilla’s implementation of the <span class="caps">W3C</span> WebDriver protocol.</li>
<li>Marionette Server is built directly into Firefox (into the Gecko rendering engine) and it speaks a slightly different protocol. To make Marionette truly WebDriver-compatible, we need to translate between Marionette’s custom protocol and the WebDriver protocol, which is exactly what geckodriver does. The Selenium client can talk to geckodriver, which in turn talks to Marionette Server.</li>
<li>As I mentioned earlier, the plan for Selenium 3 is to have geckodriver replace Selenium’s FirefoxDriver. This is an important change: since FirefoxDriver is a Firefox add-on, it has limitations and is <a href="https://wiki.mozilla.org/Add-ons/Extension_Signing">going to stop working altogether</a> with future releases.</li>
<li><a href="http://marionette-client.readthedocs.io/en/latest/">Marionette Client</a> is Mozilla’s official Python library for remote control of Gecko, but it’s not covered by the <span class="caps">W3C</span> WebDriver spec and it’s not compatible with WebDriver in general. Think of it as an alternative to Selenium’s Python client with Gecko-specific features. Selenium + geckodriver should eventually replace Marionette Client, including the Gecko-specific features.</li>
<li>The Marionette project also includes tools for integrating with Mozilla’s intricate test infrastructure: <a href="https://developer.mozilla.org/en-US/docs/Marionette_Test_Runner">Marionette Test Runner</a>, a.k.a. the Marionette test harness. This part of the project has nothing to do with WebDriver, really, except that it knows how to run tests that depend on Marionette Client. The runner collects the tests you ask for, takes care of starting a Marionette session with the right browser instance, runs the tests and reports the results.<sup id="fnref-6"><a class="footnote-ref" href="#fn-6">6</a></sup></li>
</ul>
<p>As you can see, “Marionette” may refer to many different things. I think this ambiguity will always make me a little nervous… Words are hard, especially as a loose collection of projects evolves and becomes unified. In a few years, the terms will firm up. For now, let’s be extra careful and specify which piece we’re talking about. </p>
<h2>Acknowledgements</h2>
<p>Thanks to <a href="https://twitter.com/AutomatedTester">David Burns</a> for patiently answering my half-baked questions last week, and to James Graham and <a href="https://sny.no">Andreas Tolfsen</a> for providing detailed and delightful feedback on a draft of this article. Bonus high-five to <a href="http://vakila.github.io/">Anjana Vakil</a> for contributions to Marionette Test Runner this year and for inspiring me to write this post in the first place. </p>
<div class="footnote">
<hr>
<ol>
<li id="fn-1">
<p>Terminology lesson: the WebDriver protocol is a <a href="https://en.wikipedia.org/wiki/Wire_protocol">wire protocol</a> because it’s at the application level and requires several applications working together. <a class="footnote-backref" href="#fnref-1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-2">
<p>I give a range of years because Selenium WebDriver is a merger of two projects that started at different times. <a class="footnote-backref" href="#fnref-2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-3">
<p>Abbreviated Selenium history and roadmap: Selenium 1 used an old <span class="caps">API</span> and mechanism called SeleniumRC, Selenium 2 favours the WebDriver <span class="caps">API</span> and <span class="caps">JSON</span> Wire Protocol, Selenium 3 will officially designate SeleniumRC as deprecated (“LegRC”, harhar), and Selenium 4 will implement the authoritative <span class="caps">W3C</span> WebDriver spec. <a class="footnote-backref" href="#fnref-3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn-4">
<p>Many of my claims about Marionette are confirmed by this <a href="http://www.theautomatedtester.co.uk/blog/2012/marionette-the-future-of-firefoxdriver-in-selenium.html">historical artifact from 2012</a>, which I came across shortly before publishing this post. <a class="footnote-backref" href="#fnref-4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn-5">
<p>For example, until recently Selenium WebDriver only included commands that are common to all browsers, with no way to use features that are specific to one. In contrast, the <span class="caps">W3C</span> WebDriver spec allows the possibility of <a href="https://w3c.github.io/webdriver/webdriver-spec.html#dfn-extension-commands">extension commands</a>. Extension commands are being implemented in Selenium clients right now! The future is now! <a class="footnote-backref" href="#fnref-5" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn-6">
<p>Fun fact: Marionette is not only used for “Marionette Tests” at Mozilla. The client/server are also used to instrument Firefox for other test automation like mochitests and Web Platform Tests.  <a class="footnote-backref" href="#fnref-6" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
</ol>
</div>A Week of Reflection and Programming for Fun2016-05-17T00:00:00-04:002016-05-18T00:00:00-04:00Maja Frydrychowicztag:www.erranderr.com,2016-05-17:/blog/recurse-center-review.html<p>Returning to Recurse Center for a week inspires me to reflect on my experience at their retreat 2 years go.</p><p>I recently got to spend a week back at the heart of an <del>excellent</del> <del>delightful</del> inspiring technical community: <a href="http://www.recurse.com">Recurse Center</a> or <span class="caps">RC</span>. This friendly group consists mostly of programmers from around the world who have, at some point, participated in <span class="caps">RC</span>’s three-month “retreat” in New York City to work on whatever <a href="https://www.recurse.com/blog/82-what-people-do-at-the-recurse-center-apr-15">projects</a> happen to interest them. The retreat’s motto is “never graduate”, and so participants continue to support each other’s technical growth and curiosity forever and ever. </p>
<p>I’m an <span class="caps">RC</span> alum from 2014! <span class="caps">RC</span>’s retreat is how I ended up contributing to open source software and eventually gathering the courage to join Mozilla. Before <span class="caps">RC</span>, despite already having thousands of hours of programming and fancy math under my belt, I held myself back with doubts about whether I’m a “real programmer”, whatever that stereotype means. That subconscious negativity hasn’t magically disappeared, but I’ve had a lot of good experiences in the past few years to help me manage it. Today, <span class="caps">RC</span> helps me stay excited about learning all the things for the sake of learning all the things.</p>
<p>A retreat at <span class="caps">RC</span> looks something like this: you put your life more-or-less on hold, move to <span class="caps">NYC</span>, and spend three months tinkering in a big, open office with around fifty fellow (thoughtful, kind, enthusiastic) programmers. During my 2014 retreat, I worked mostly on lowish-level networking things in Python, pair programmed on whatever else people happened to be working on, gave and received code review, chatted with wise <a href="http://marijnhaverbeke.nl/blog/recurse-center.html">“residents”</a>, attended spontaneous workshops, presentations and so on. </p>
<p>Every May, alumni are invited to return to the <span class="caps">RC</span> space for a week, and this year I got to go! (Thanks, Mozilla!) It was awesome! Exclamation points! This past week felt like a tiny version of the 3-month retreat. After two years away, I felt right at home — that says a lot about the warm atmosphere <span class="caps">RC</span> manages to cultivate. My personal goal for the week was just to work in a language that’s relatively new to me - JavaScript - but I also happened to have really interesting conversations about things like:</p>
<ul>
<li>How to implement a basic debugger?</li>
<li>How to improve the technical interview process?</li>
<li>What holds developers back or slows them down? What unnecessary assumptions do we have about our tools and their limitations?</li>
</ul>
<p><span class="caps">RC</span>’s retreat is a great environment for growing as a developer, but I don’t want to make it sound like it’s all effortless whimsy. Both the hardest and most wonderful part of <span class="caps">RC</span> (and many other groups) is being surrounded by extremely impressive, positive people who never seem to struggle with anything. It’s easy to slip into showing off our knowledge or to get distracted by measuring ourselves against our peers. Sometimes this is impostor syndrome. Sometimes it’s the myth of the 10x developer. <span class="caps">RC</span> puts a lot of effort into being a <a href="https://www.recurse.com/diversity">safe space</a> where you can reveal your ignorance and ask questions, but insecurity can always be a challenge.</p>
<p>Similarly, the main benefit of <span class="caps">RC</span> is learning from your peers, but the usual ways of doing this seem to be geared toward people who are outgoing and think out loud. These are valuable skills, but when we focus on them exclusively we don’t hear from people who have different defaults. There is also little structure provided by <span class="caps">RC</span> so you are free to self-organize and exchange ideas as you deem appropriate. The risk is that quiet people are allowed to hide in their quiet corners, and then <a href="https://hbr.org/2016/04/run-meetings-that-are-fair-to-introverts-women-and-remote-workers">everyone misses out on their contributions</a>. I think <span class="caps">RC</span> makes efforts to balance this out, but the overall lack of structure means you really have to take charge of how you learn from others. I’m definitely better at this than I used to be.</p>
<p><span class="caps">RC</span> is an experiment and it’s always changing. Although at this point my involvement is mostly passive, I’m glad to be a part of it. I love that I’ve been able to work closely with vastly different people, getting an inside look at their work habits and ways of thinking. Now, long after my “never-graduation”, the <span class="caps">RC</span> community continues to expose me to a variety of ideas about technology and learning in a way that makes us all get better. Continuous improvement, yeah! </p>Not Testing a Firefox Build (Generic Tasks in TaskCluster)2016-05-02T00:00:00-04:002016-05-02T00:00:00-04:00Maja Frydrychowicztag:www.erranderr.com,2016-05-02:/blog/taskcluster-generic-tasks.html<p>I take advantage of generic tasks to run a mozharness script in a gecko source checkout.</p><p>A few months ago I wrote about my <a href="https://www.erranderr.com/blog/taskcluster-learning.html">tentative setup</a> of a TaskCluster task that was neither a build nor a test. Since then, gps has implemented <a href="https://groups.google.com/forum/#!searchin/mozilla.dev.platform/generic$20task/mozilla.dev.platform/bNYp2HDyeqU/tg4mnGHEAwAJ">“generic” in-tree tasks</a> so I <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1227367#c116">adapted my initial work</a> to take advantage of that.</p>
<h1>Triggered by file changes</h1>
<p>All along I wanted to run some in-tree tests without having them wait around for a Firefox build or any other dependencies they don’t need. So I originally implemented this task as a <a href="https://www.erranderr.com/blog/taskcluster-learning.html#scheduling_summary">“build”</a> so that it would get scheduled for every incoming changeset in Mozilla’s repositories. </p>
<p>But forget “builds”, forget “tests” — now there’s a third category of tasks that we’ll call “generic” and it’s exactly what I need. </p>
<p>In <a href="https://hg.mozilla.org/mozilla-central/diff/e4ea9261d5bb/testing/taskcluster/tasks/branches/base_jobs.yml">base_jobs.yml</a> I say, “hey, here’s a new task called <code>marionette-harness</code> — run it whenever there’s a change under (branch)/testing/marionette/harness”. Of course, I can also just trigger the task with try syntax like <code>try: -p linux64_tc -j marionette-harness -u none -t none</code>.</p>
<p>When the task is triggered, a chain of events follows: </p>
<ul>
<li><code>marionette-harness</code> is defined by <a href="https://hg.mozilla.org/mozilla-central/file/e4ea9261d5bb/testing/taskcluster/tasks/tests/harness_marionette.yml">harness_marionette.yml</a>, which depends on <a href="https://hg.mozilla.org/mozilla-central/file/e4ea9261d5bb/testing/taskcluster/tasks/harness_test.yml">harness_test.yml</a></li>
<li><a href="https://hg.mozilla.org/mozilla-central/file/e4ea9261d5bb/testing/taskcluster/tasks/harness_test.yml">harness_test.yml</a> says to run <a href="https://hg.mozilla.org/mozilla-central/diff/e4ea9261d5bb/testing/docker/desktop-build/bin/build.sh">build.sh</a> with the appropriate mozilla branch and revision.</li>
<li><a href="https://hg.mozilla.org/mozilla-central/file/e4ea9261d5bb/testing/taskcluster/tasks/tests/harness_marionette.yml">harness_marionette.yml</a> sets more environment variables and parameters for build.sh to use (<code>JOB_SCRIPT</code>, <code>MOZHARNESS_SCRIPT</code>, etc.)</li>
<li>So build.sh checks out the source tree and executes <a href="https://hg.mozilla.org/mozilla-central/diff/e4ea9261d5bb/testing/taskcluster/scripts/tester/harness-test-linux.sh">harness-test-linux.sh</a> (<code>JOB_SCRIPT</code>)…</li>
<li>…which in turn executes <a href="https://hg.mozilla.org/mozilla-central/file/1e0b4e27bd51/testing/mozharness/scripts/marionette_harness_tests.py">marionette_harness_tests.py</a> (<code>MOZHARNESS_SCRIPT</code>) with the parameters passed on by build.sh</li>
</ul>
<h1>For Tasks that Make Sense in a gecko Source Checkout</h1>
<p>As you can see, I made the <code>build.sh</code> script in the <code>desktop-build</code> docker image execute an arbitrary in-tree <code>JOB_SCRIPT</code>, and I created <code>harness-test-linux.sh</code> to run mozharness within a gecko source checkout. </p>
<h2>Why not the desktop-test image?</h2>
<p>But we can also run arbitrary mozharness scripts thanks to the configuration in the desktop-test docker image! Yes, and all of that configuration is geared toward testing a Firefox binary, which implies downloading tools that my task either doesn’t need or already has access to in the source tree. Now we have a lighter-weight option for executing tests that don’t exercise Firefox.</p>
<h2>Why not mach?</h2>
<p>In my lazy work-in-progress, I had originally executed the Marionette harness tests via a simple call to mach, yet now I have this crazy chain of shell scripts that leads all the way mozharness. The mach command didn’t disappear — you can run Marionette harness tests with <code>./mach python-test ...</code>. However, mozharness provides clearer control of Python dependencies, appropriate handling of return codes to report test results to Treeherder, and I can write a job-specific script and configuration.</p>First Experiment with TaskCluster2016-02-09T00:00:00-05:002016-02-16T00:00:00-05:00Maja Frydrychowicztag:www.erranderr.com,2016-02-09:/blog/taskcluster-learning.html<p>Adding a new task to TaskCluster continuous integration system.</p><p><a href="https://docs.taskcluster.net/">TaskCluster</a> is a new-ish continuous integration system made at Mozilla. It manages the scheduling and execution of tasks based on a graph of their dependencies. It’s a general <span class="caps">CI</span> tool, and could be used for any kind of job, not just Mozilla things. </p>
<p>However, the example I describe here refers to a Mozilla-centric use case of TaskCluster<sup id="fnref-1"><a class="footnote-ref" href="#fn-1">1</a></sup>: tasks are run per check-in on the branches of Mozilla’s Mercurial repository and then results are posted to <a href="https://github.com/mozilla/treeherder">Treeherder</a>. For now, the tasks can be configured to run in Docker images (Linux), but other platforms are in the works<sup id="fnref-2"><a class="footnote-ref" href="#fn-2">2</a></sup>. </p>
<p>So, I want to schedule a task! I need to add a new task to the task graph that’s created for each revision submitted to hg.mozilla.org. (This is part of my work on deploying a suite of <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1227367">tests for the Marionette Python test runner</a>, i.e. testing the test harness itself.) </p>
<p>The rest of this post describes what I learned while making <a href="https://hg.mozilla.org/try/rev/6b7479c4aa30">this work-in-progress</a>.</p>
<h1>There are builds and there are tests</h1>
<p>mozilla-taskcluster operates based on the info under <a href="https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks"><code>testing/taskcluster/tasks</code></a> in Mozilla’s source tree, where there are yaml files that describe tasks. Specific tasks can inherit common configuration options from base yaml files. </p>
<p>The yaml files are organized into two main categories of tasks: builds and tests. This is just a convention in mozilla-taskcluster about how to group task configurations; <span class="caps">TC</span> itself doesn’t actually know or care whether a task is a build or a test.</p>
<p>The task I’m creating doesn’t quite fit into either category: it runs harness tests that just exercise the Python runner code in <a href="https://marionette-client.readthedocs.org">marionette_client</a>, so I only need a source checkout, not a Firefox build. I’d like these tests to run quickly without having to wait around for a build. Another example of such a task is the recently-created <a href="https://hg.mozilla.org/mozilla-central/rev/4b34c9d1a31a">ESLint task</a>.</p>
<h1>Scheduling a task</h1>
<p>Just adding a yaml file that describes your new task under <code>testing/taskcluster/tasks</code> isn’t enough to get it scheduled: you must also add it to the list of tasks in <a href="https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/branches/base_jobs.yml"><code>base_jobs.yml</code></a>, and define an identifier for your task in <a href="https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/branches/base_job_flags.yml"><code>base_job_flags.yml</code></a>. This identifier is used in <code>base_jobs.yml</code>, and also by people who want to run your task when pushing to <a href="https://wiki.mozilla.org/ReleaseEngineering/TryServer">try</a>.</p>
<p>How does scheduling work? First a <a href="https://docs.taskcluster.net/introduction/getting-started/#decision-tasks-and-task-graphs">decision task</a> generates a <em>task graph</em>, which describes all the tasks and their relationships. More precisely, it looks at <code>base_jobs.yml</code> and other yaml files in <code>testing/taskcluster/tasks</code> and spits out a json artifact, <code>graph.json</code><sup id="fnref-3"><a class="footnote-ref" href="#fn-3">3</a></sup>. Then, <code>graph.json</code> gets sent to <span class="caps">TC</span>’s <a href="https://docs.taskcluster.net/queue/api-docs/#createTask"><code>createTask</code></a> endpoint, which takes care of the actual scheduling. </p>
<p>In the excerpt below, you can see a task definition with a <code>requires</code> field and you can recognize a lot of fields that are in common with the ‘task’ section of the yaml files under <code>testing/taskcluster/tasks/</code>.</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"tasks"</span><span class="o">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">"requires"</span><span class="o">:</span> <span class="p">[</span>
<span class="c1">// id of a build task that this task depends on</span>
<span class="s2">"fZ42HVdDQ-KFFycr9PxptA"</span>
<span class="p">],</span>
<span class="s2">"task"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"taskId"</span><span class="o">:</span> <span class="s2">"c2VD_eCgQyeUDVOjsmQZSg"</span>
<span class="s2">"extra"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"treeherder"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"groupName"</span><span class="o">:</span> <span class="s2">"Reftest"</span><span class="p">,</span>
<span class="s2">"groupSymbol"</span><span class="o">:</span> <span class="s2">"tc-R"</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="s2">"metadata"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"description"</span><span class="o">:</span> <span class="s2">"Reftest test run 1"</span><span class="p">,</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"[TC] Reftest"</span><span class="p">,</span>
<span class="c1">//...</span>
<span class="p">]</span>
<span class="p">}</span>
</pre></div>
<p>For now at least, a major assumption in the task-graph creation process seems to be that <em>test</em> tasks can depend on <em>build</em> tasks and <em>build</em> tasks don’t really<sup id="fnref-4"><a class="footnote-ref" href="#fn-4">4</a></sup> depend on anything. So: <a name="scheduling_summary"></a></p>
<ul>
<li>If you want your tasks to run for every push to a Mozilla hg branch, add it to the list of <strong>builds</strong> in <code>base_jobs.yml</code>. </li>
<li>If you want your task to run after certain build tasks succeed, add it to the list of <strong>tests</strong> in <code>base_jobs.yml</code> and specify which build tasks it depends on.</li>
<li>Other than the above, I don’t see any way to specify a dependency between task A and task B in <code>testing/taskcluster/tasks</code>.</li>
</ul>
<p>So, I added <code>marionette-harness</code> under <code>builds</code>. Recall, my task isn’t a build task, but it doesn’t depend on a build, so it’s not a test, so I’ll treat it like a build.</p>
<div class="highlight"><pre><span></span><span class="c1"># in base_job_flags.yml</span>
<span class="l l-Scalar l-Scalar-Plain">builds</span><span class="p p-Indicator">:</span>
<span class="c1"># ...</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">marionette-harness</span>
<span class="c1"># in base_jobs.yml</span>
<span class="l l-Scalar l-Scalar-Plain">builds</span><span class="p p-Indicator">:</span>
<span class="c1"># ...</span>
<span class="l l-Scalar l-Scalar-Plain">marionette-harness</span><span class="p p-Indicator">:</span>
<span class="l l-Scalar l-Scalar-Plain">platforms</span><span class="p p-Indicator">:</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">Linux64</span>
<span class="l l-Scalar l-Scalar-Plain">types</span><span class="p p-Indicator">:</span>
<span class="l l-Scalar l-Scalar-Plain">opt</span><span class="p p-Indicator">:</span>
<span class="l l-Scalar l-Scalar-Plain">task</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">tasks/tests/harness_marionette.yml</span>
</pre></div>
<p>This will allow me to trigger my task with the following try syntax: <code>try: -b o -p marionette-harness</code>. Cool.</p>
<h1>Make your task do stuff</h1>
<p>Now I have to add some stuff to <code>tasks/tests/harness_marionette.yml</code>. Many of my choices here are based on the work done for the <a href="https://hg.mozilla.org/mozilla-central/rev/4b34c9d1a31a">ESLint task</a>. I created a base task called <code>harness_test.yml</code> by mostly copying bits and pieces from the basic build task, <code>build.yml</code> and making a few small changes. The actual task, <code>harness_marionette.yml</code> inherits from <code>harness_test.yml</code> and defines specifics like Treeherder symbols and the command to run.</p>
<h2>The command</h2>
<p>The heart of the task is in <code>task.payload.command</code>. You could chain a bunch of shell commands together directly in this field of the yaml file, but it’s better not to. Instead, it’s common to call a TaskCluster-friendly shell script that’s available in your task’s environment. For example, the <a href="https://dxr.mozilla.org/mozilla-central/source/testing/docker/desktop-test"><code>desktop-test</code></a> docker image has a script called <code>test.sh</code> through which you can call the <a href="https://wiki.mozilla.org/ReleaseEngineering/Mozharness">mozharness</a> script for your tests. There’s a similar <code>build.sh</code> script on <code>desktop-build</code>. Both of these scripts depend on environment variables set elsewhere in your task definition, or in the Docker image used by your task. The environment might also provide utilities like <a href="https://tc-vcs.readthedocs.org/en/latest/">tc-vcs</a>, which is used for checking out source code.</p>
<div class="highlight"><pre><span></span><span class="c1"># in harness_marionette.yml</span>
<span class="l l-Scalar l-Scalar-Plain">payload</span><span class="p p-Indicator">:</span>
<span class="l l-Scalar l-Scalar-Plain">command</span><span class="p p-Indicator">:</span>
<span class="l l-Scalar l-Scalar-Plain">+ bash</span>
<span class="l l-Scalar l-Scalar-Plain">+ -cx</span>
<span class="l l-Scalar l-Scalar-Plain">+ ></span>
<span class="l l-Scalar l-Scalar-Plain">tc-vcs checkout ./gecko {{base_repository}} {{head_repository}} {{head_rev}} {{head_ref}} &&</span>
<span class="l l-Scalar l-Scalar-Plain">cd gecko &&</span>
<span class="l l-Scalar l-Scalar-Plain">./mach marionette-harness-test</span>
</pre></div>
<p>My task’s <code>payload.command</code> should be moved into a custom shell script, but for now it just chains together the source checkout and a call to <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/mach">mach</a>. It’s not terrible of me to use mach in this case because I expect my task to work in a build environment, but most tests would likely call mozharness.</p>
<h1>Configuring the task’s environment</h1>
<p>Where should the task run? What resources should it have access to? This was probably the hardest piece for me to figure out.</p>
<h2>docker-worker</h2>
<p>My task will run in a docker image using a <a href="https://docs.taskcluster.net/workers/docker-worker/">docker-worker</a><sup id="fnref-5"><a class="footnote-ref" href="#fn-5">5</a></sup>. The image, called <code>desktop-build</code>, is defined in-tree under <a href="https://dxr.mozilla.org/mozilla-central/source/testing/docker/desktop-build"><code>testing/docker</code></a>. There are many other images defined there, but I only considered <code>desktop-build</code> versus <code>desktop-test</code>. I opted for <code>desktop-build</code> because <code>desktop-test</code> seems to contain mozharness-related stuff that I don’t need for now.</p>
<div class="highlight"><pre><span></span><span class="c1"># harness_test.yml</span>
<span class="l l-Scalar l-Scalar-Plain">image</span><span class="p p-Indicator">:</span>
<span class="l l-Scalar l-Scalar-Plain">type</span><span class="p p-Indicator">:</span> <span class="s">'task-image'</span>
<span class="l l-Scalar l-Scalar-Plain">path</span><span class="p p-Indicator">:</span> <span class="s">'public/image.tar'</span>
<span class="l l-Scalar l-Scalar-Plain">taskId</span><span class="p p-Indicator">:</span> <span class="s">'{{#task_id_for_image}}desktop-build{{/task_id_for_image}}'</span>
</pre></div>
<p>The image is stored as an artifact of another <span class="caps">TC</span> task, which makes it a ‘task-image’. Which artifact? The default is <code>public/image.tar</code>. Which task do I find the image in? The magic incantation <code>'{{#task_id_for_image}}desktop-build{{/task_id_for_image}}'</code> somehow<sup id="fnref-6"><a class="footnote-ref" href="#fn-6">6</a></sup> obtains the correct <span class="caps">ID</span>, and if I look at a particular run of my task, the above snippet does indeed get populated with an actual <code>taskId</code>. </p>
<div class="highlight"><pre><span></span><span class="s2">"image"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"path"</span><span class="o">:</span> <span class="s2">"public/image.tar"</span><span class="p">,</span>
<span class="c1">// Mystery task that makes a desktop-build image for us. Thanks, mystery task!</span>
<span class="s2">"taskId"</span><span class="o">:</span> <span class="s2">"aqt_YdmkTvugYB5b-OvvJw"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="o">:</span> <span class="s2">"task-image"</span>
<span class="p">}</span>
</pre></div>
<p>Snooping around in the handy <a href="https://tools.taskcluster.net/task-inspector/">Task Inspector</a>, I found that the magical mystery task is defined in <a href="https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/image.yml">image.yml</a> and runs <a href="https://dxr.mozilla.org/mozilla-central/source/testing/docker/image_builder/bin/build_image.sh"><code>build_image.sh</code></a>. Fun. It’s also quite convenient to <a href="https://docs.taskcluster.net/presentations/TC-102/#/images-00">define and test your own custom image</a>.</p>
<h2>Other details that I mostly ignored</h2>
<div class="highlight"><pre><span></span><span class="c1"># in harness_test.yml</span>
<span class="l l-Scalar l-Scalar-Plain">scopes</span><span class="p p-Indicator">:</span>
<span class="c1"># Nearly all of our build tasks use tc-vcs</span>
<span class="p p-Indicator">-</span> <span class="s">'docker-worker:cache:level-{{level}}-{{project}}-tc-vcs'</span>
<span class="l l-Scalar l-Scalar-Plain">cache</span><span class="p p-Indicator">:</span>
<span class="c1"># The taskcluster-vcs tooling stores the large clone caches in this</span>
<span class="c1"># directory and will reuse them for new requests this saves about 20s~</span>
<span class="c1"># and is the most generic cache possible.</span>
<span class="l l-Scalar l-Scalar-Plain">level-{{level}}-{{project}}-tc-vcs</span><span class="p p-Indicator">:</span> <span class="s">'/home/worker/.tc-vcs'</span>
</pre></div>
<ul>
<li><em>Routes</em> allow your task to be looked up in the task index. This isn’t necessary in my case so I just omitted routes altogether.</li>
<li><em>Scopes</em> are permissions for your tasks, and I just copied the scope that is used for checking out source code.</li>
<li><em>workerType</em> is a configuration for managing the workers that run tasks. To me, this was a choice between <code>b2gtest</code> and <code>b2gbuild</code>, which aren’t specific to b2g anyway. <code>b2gtest</code> is more lightweight, I hear, which suits my harness-test task fine.</li>
<li>I had to include a few dummy values under <code>extra</code> in <code>harness_test.yml</code>, like <code>build_name</code>, just because they are expected in <em>build</em> tasks. I don’t use these values for anything, but my task fails to run if I don’t include them.</li>
</ul>
<h1>Yay for trial and error</h1>
<ul>
<li>If you have syntax errors in your yaml, the Decision task will fail. If this happens during a try push, look under Job Details > Inspect Task to fine useful error messages.</li>
<li>Iterating on your task is pretty easy. Aside from pushing to try, you can <a href="https://docs.taskcluster.net/presentations/TC-101/#/run-locally-environment">run tasks locally using vagrant</a> and you can build a task graph locally as well with <code>mach taskcluster-graph</code>. </li>
</ul>
<h1>Resources</h1>
<p>Blog posts from other TaskCluster users at Mozilla:</p>
<ul>
<li><a href="https://ehsanakhgari.org/blog/2015-09-29/my-experience-adding-new-build-type-taskcluster">https://ehsanakhgari.org/blog/2015-09-29/my-experience-adding-new-build-type-taskcluster</a></li>
<li><a href="https://elvis314.wordpress.com/2015/11/09/adventures-in-task-cluster-running-tests-locally/">https://elvis314.wordpress.com/2015/11/09/adventures-in-task-cluster-running-tests-locally/</a></li>
<li><a href="https://elvis314.wordpress.com/2015/11/11/adventures-in-task-cluster-running-a-custom-docker-image/">https://elvis314.wordpress.com/2015/11/11/adventures-in-task-cluster-running-a-custom-docker-image/</a></li>
</ul>
<p>There is lots of great documentation at <a href="https://docs.taskcluster.net">docs.taskcluster.net</a>, but these sections were especially useful to me:</p>
<ul>
<li><a href="https://docs.taskcluster.net/queue/api-docs/#createTask">createTask <span class="caps">API</span></a></li>
<li><a href="https://docs.taskcluster.net/workers/">Workers</a></li>
</ul>
<h1>Acknowledgements</h1>
<p>Thanks to <a href="http://code.v.igoro.us/">dustin</a>, pmoore and others for corrections and feedback.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-1">
<p>This is accomplished in part thanks to <a href="http://blog.gregarndt.com/taskcluster/2015/08/05/demystifying-in-tree-scheduling/">mozilla-taskcluster</a>, a service that links Mozilla’s hg repo to TaskCluster and creates each decision task. More at <a href="https://docs.taskcluster.net/introduction/getting-started/#taskcluster-at-mozilla">TaskCluster at Mozilla</a> <a class="footnote-backref" href="#fnref-1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-2">
<p>Run tasks on any platform thanks to <a href="https://docs.taskcluster.net/workers/generic-worker/">generic worker</a> <a class="footnote-backref" href="#fnref-2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-3">
<p>To look at a <code>graph.json</code> artifact, go to <a href="https://treeherder.mozilla.org/">Treeherder</a>, click a green ‘D’ job, then Job details > Inspect Task, where you should find a list of artifacts. <a class="footnote-backref" href="#fnref-3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn-4">
<p>It’s not <em>really</em> true that build tasks don’t depend on anything. Any task that uses a task-image depends on the task that creates the image. I’m sorry for saying ‘task’ five times in every sentence, by the way. <a class="footnote-backref" href="#fnref-4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn-5">
<p>…as opposed to a <a href="https://docs.taskcluster.net/workers/generic-worker/">generic worker</a>. <a class="footnote-backref" href="#fnref-5" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn-6">
<p><code>{{#task_id_for_image}}</code> is an example of a predefined variable that we can use in our <span class="caps">TC</span> yaml files. Where do they come from? How do they get populated? I don’t know. <a class="footnote-backref" href="#fnref-6" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
</ol>
</div>Snapshots from my OPW Internship with Mozilla QA2014-08-07T00:00:00-04:002014-08-23T00:00:00-04:00Maja Frydrychowicztag:www.erranderr.com,2014-08-07:/blog/opw-snapshot.html<p>A description of a “typical” two weeks during my <span class="caps">OPW</span> internship with Mozilla.</p><p>Throughout my <span class="caps">OPW</span><sup id="fnref-1"><a class="footnote-ref" href="#fn-1">1</a></sup> internship with Mozilla <span class="caps">QA</span><sup id="fnref-2"><a class="footnote-ref" href="#fn-2">2</a></sup> I’ve been keeping an informal log in outline form<sup id="fnref-3"><a class="footnote-ref" href="#fn-3">3</a></sup>. In it, I briefly describe what I accomplish (or fail to accomplish) each day, problems I encounter, who I talk to about them, which meetings I attend, what I read, useful tricks I learn, etc. So far, I have about 60-days worth of these tiny log entries about Mozilla. Here’s what they look like:</p>
<div style="text-align:center">
<p><img alt="Checkvist Mozilla Log Screenshot" src="https://www.erranderr.com/blog/images/checkvist-screenshot.png"></p>
</div>
<p>Day-to-day, the log helps me answer questions like “How did I solve this weird configuration problem three weeks ago?” or “What should I ask about at the next team meeting?” Writing also generally helps me think through a task, and the log is a quick and effective outlet for that. The other major benefit is that I can take a step back and see the overall progress of my projects.</p>
<h1>So, what’s it like being an intern with Mozilla <span class="caps">QA</span>?</h1>
<p>I’m so glad you asked! First, some context. </p>
<ul>
<li><span class="caps">OPW</span> interns <a href="https://www.erranderr.com/blog/remote-work.html">work remotely</a>.</li>
<li>The internship position I applied for is called “<a href="https://wiki.mozilla.org/GNOME_Outreach_Summer2014#Mozilla_Bug_Wrangler_.28Desktop_QA.29">Bug Wrangler</a>“, which refers to tasks like reproducing and triaging incoming Firefox bugs, but I’ve actually (mostly) been doing Django web development.</li>
</ul>
<p><em><strong>To future interns</strong>: as in my case, there can be some flexibility about your internship activities, and during your application process, you’ll narrow down what you will work on. The mentor I applied to offered a Django project as an option under the Bug Wrangler umbrella, and that was more in line with my interests and experience than bug triage, so that’s what I chose to focus on.</em></p>
<p>Based on my handy log, I’ll answer a slightly more specific question:</p>
<blockquote>
<blockquote>
<p><span class="dquo">“</span>What did Maja do during a typical week while working on a Django project for Mozilla <span class="caps">QA</span>?”</p>
</blockquote>
</blockquote>
<h2>Routines</h2>
<p>Often, I start my day by skimming my latest “bug mail” (updates from Bugzilla) and checking my Bugzilla dashboard to see if I need to follow up on anything immediately. </p>
<p>The other regular occurrence is about 2 hours of video meetings per week. I meet with my mentor once a week to discuss my general progress and my post-internship plans. I lurk at one <span class="caps">QA</span> team meeting almost every week, where I mostly don’t have enough context to understand much. My mentor filled me in on some things and my understanding gradually improved. There are also two regular meetings for <a href="https://github.com/mozilla/oneanddone">One and Done</a>, the project I’m contributing to: a weekly technical meeting to discuss the design of new features, and a biweekly check-in meeting with project managers, developers and a few key users.</p>
<h2>Week 3</h2>
<p>The early weeks of the internship involved a lot of reading and trying things out, of course. At this point, I was finishing up the official Django tutorial as well as responding to some administrative requests about the internship. </p>
<p>Just for fun, I used vim throughout my Django learnings to rediscover some handy vim commands. I also applied the tutorial concepts to the One and Done source code as much as I could, and thus discovered what other parts of Django I need to become familiar with, like generic class-based views. </p>
<p>I gradually became more familiar with how the One and Done code is structured by looking at how its models are used, poking at its URLconf, and populating my local database with example data.</p>
<h2>Week 5</h2>
<p>At this point, I was just about finished with my first substantial <a href="https://github.com/mozilla/oneanddone/pull/124">pull request to One and Done</a>. My changes broke some unit tests, which caused me to discover that some of our test data was using the wrong data type: a regular Python dictionary instead of a Django <a href="https://docs.djangoproject.com/en/1.4/ref/request-response/#django.http.QueryDict">QueryDict</a>. Cool.</p>
<p>I actually spent a bunch of time getting the unit tests to run in my dev environment, which is on a Linux virtual machine. My local copy of the project is stored in a directory that is shared between my Linux guest <span class="caps">OS</span> and Windows host <span class="caps">OS</span>, which happens to rely on file permissions that the <a href="https://nose.readthedocs.org/en/latest/">nose</a> testing library doesn’t like. In the end, I chose to have a clone of the project in a non-shared directory that I used just for running unit tests.</p>
<p>My work log also describes in detail how unintended changes to my development branch in git turned my Github pull request into a giant, unreadable mess. Aaah! (Be careful what you branch from and what you merge with, friends.) I had to close my original pull request and make a new, clean one, which was fairly embarrassing. Now I remember that on that day my friend and I were co-working in my apartment to battle <a href="https://www.erranderr.com/blog/remote-work.html">the loneliness of remote work</a>, and she generously listened to me venting my misery about the incident. :) In retrospect, I learned a lot about git.</p>
<p>Later that week, that same pull request got merged and I started investigating a bug I ran into in one of the libraries our project relies on, which involved asking some questions on <span class="caps">IRC</span>. </p>
<p>All around, a good week.</p>
<h2>Week 9</h2>
<p>First I finished up a couple of things I had started earlier:</p>
<ul>
<li>updating the <a href="https://github.com/mozilla/oneanddone/pull/139">One and Done readme</a> to make it easier for new contributors to get started, </li>
<li>adding tests to a <a href="https://github.com/mozilla/oneanddone/pull/151">pull request about form validation</a> I had made the week before,</li>
<li>cleaning up an article about my <a href="https://www.erranderr.com/blog/borked-migrations.html">database migration blunders</a>.</li>
</ul>
<p>I also contributed my first few code reviews: the week before I missed an issue that someone else caught (doh!), but this week I found something that needed to be fixed (yay!). This was cool because I found the problem by simply taking the time to understand code that was <a href="https://github.com/mozilla/oneanddone/pull/154#discussion_r14786905">mostly mysterious to me</a>. Bonus: I learned a bit about <a href="http://www.voidspace.org.uk/python/mock/">Mock and patch</a>.</p>
<p>By the end of the week, I was focused on sketching out the functionality and implementation of a <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1030972">new One and Done feature</a>. I enjoyed working with the project managers to define the feature requirements. Figuring out how to implement them required a few more weeks of research and learning on my part, but it all worked out in the end.</p>
<h1>This is why I like work logs!</h1>
<p>Reviewing my work log to write this article was eye-opening for me, especially due to the perspective it offers of the ups and downs I experienced during my internship. On some days, I felt quite frustrated, stuck, discouraged, and all that bad stuff. So, I like how the log shows that feeling crappy for a few days here and there totally doesn’t matter overall. I learned a lot in the past couple of months and it’s incredibly satisfying to see that itemized in one big list.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-1">
<p>Outreach Program for Women <a class="footnote-backref" href="#fnref-1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-2">
<p>Quality Assurance <a class="footnote-backref" href="#fnref-2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-3">
<p>I write the log using <a href="https://checkvist.com/">Checkvist</a>. It’s fantastic. I did the same while at Hacker School. <a class="footnote-backref" href="#fnref-3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>Database Migrations (“You know nothing, Version Control.”)2014-07-04T18:00:00-04:002014-07-25T00:00:00-04:00Maja Frydrychowicztag:www.erranderr.com,2014-07-04:/blog/borked-migrations.html<p>How to rewrite database migration history in a Django project with South and git.</p><p>This is the story of how I rediscovered what version control doesn’t do for you. Sure, I understand that git doesn’t track what’s in my project’s local database, but to understand is one thing and to <em>feel in your heart forever</em> is another. In short, learning from mistakes and accidents is the greatest!</p>
<p>So, I’ve been working on a Django project and as the project acquires new features, the database schema changes here and there. Changing the database from one schema to another and possibly moving data between tables is called a <em>migration</em>. To manage database migrations, we use <a href="http://south.aeracode.org/">South</a>, which is sort of integrated into the project’s <code>manage.py</code> script. (This is because we’re really using <a href="https://github.com/mozilla/playdoh">playdoh</a>, Mozilla’s augmented, specially-configured flavour of Django.)</p>
<p>South is lovely. Whenever you change the model definitions in your Django project, you ask South to generate Python code that defines the corresponding schema migration, which you can customize as needed. We’ll call this Python code a <em>migration file</em>. To actually update your database with the schema migration, you feed the migration file to <code>manage.py migrate</code>.</p>
<p>These migration files are safely stored in your git repository, so your project has a history of database changes <em>that you can replay backward and forward</em>. For example, let’s say you’re working in a different repository branch on a new feature for which you’ve changed the database schema a bit. Whenever you switch to the feature branch <strong>you must remember</strong> to apply your new database migration (migrate forward). Whenever you switch back to master <strong>you must remember</strong> to migrate backward to the database schema expected by the code in master. <strong>Git doesn’t know which migration your database should be at.</strong> Sometimes I’m distracted and I forget. :(</p>
<p>As always, it gets more interesting when you have project collaborators because they might push changes to migration files and <strong>you must pay attention and remember</strong> to actually apply these migrations in the right order. We will examine one such scenario in detail.</p>
<h1>Adventures with Overlooked Database Migrations</h1>
<p>Let’s call the actors Sparkles and Rainbows. Sparkles and Rainbows are both contributing to the same project and so they each regularly push or pull from the same “upstream” git repository. However, they each use their own local database for development. As far as the database goes, git is only tracking South migration files. Here is our scenario.</p>
<ol>
<li>Sparkles pushes Migration Files 1, 2, 3 to upstream and applies these migrations to their local db in that order. </li>
<li>Rainbows pulls Migration Files 1, 2, 3 from upstream and applies them to their local db in that order.<blockquote>
<p>All is well so far. The trouble is about to start.</p>
</blockquote>
</li>
<li>Sparkles reverses Migration 3 in their local database (backward migration to Migration 2) and pushes a delete of the Migration 3 file to upstream.</li>
<li>Rainbows pulls from upstream: the Migration 3 file no longer exists at <code>HEAD</code> <strong>but it must also be reversed in the local db</strong>! Alas, Rainbows does not perform the backward migration. :(</li>
<li>Life goes on and Sparkles now adds Migration Files 4 and 5, applies the migrations locally and pushes the files to upstream. </li>
<li>Rainbows happily pulls Migrations Files 4 and 5 and applies them to their local db. <blockquote>
<p>Notice that Sparkles’ migration history is now 1-2-4-5 but Rainbows’ migration history is 1-2-3-4-5, but 3 is nolonger part of the up-to-date project! </p>
</blockquote>
</li>
</ol>
<p>At some point Rainbows will encounter Django or South errors, depending on the nature of the migrations, because the database doesn’t match the expected schema. No worries, though, it’s git, it’s South: you can go back in time and fix things.</p>
<p>I was recently in Rainbows’ position. I finally noticed that something was wrong with my database when South started refusing to apply the latest migration from upstream, telling me “Sorry! I can’t drop table TaskArea, it doesn’t exist!” </p>
<div class="highlight"><pre><span></span><span class="n">tasks</span><span class="o">:</span><span class="mi">0011</span><span class="n">_auto__del_taskarea__del_field_task_area__add_field_taskkeyword_name</span>
<span class="n">FATAL</span> <span class="n">ERROR</span> <span class="o">-</span> <span class="n">The</span> <span class="n">following</span> <span class="n">SQL</span> <span class="n">query</span> <span class="n">failed</span><span class="o">:</span> <span class="n">DROP</span> <span class="n">TABLE</span> <span class="n">tasks_taskarea</span> <span class="n">CASCADE</span><span class="o">;</span>
<span class="n">The</span> <span class="n">error</span> <span class="n">was</span><span class="o">:</span> <span class="o">(</span><span class="mi">1051</span><span class="o">,</span> <span class="s2">"Unknown table 'tasks_taskarea'"</span><span class="o">)</span>
<span class="o">></span><span class="n">snip</span>
<span class="n">KeyError</span><span class="o">:</span> <span class="s2">"The model 'taskarea' from the app 'tasks' is not available in</span>
<span class="s2">this migration."</span>
</pre></div>
<p>In my instance of the Sparkles-Rainbows story, Migration 3 and Migration 5 both drop the TaskArea table; I’m trying to apply Migration 5, and South grumbles in response because I had never reversed Migration 3. As far as South knows, there’s no such thing as a TaskArea table. </p>
<p>Let’s take a look at my migration history, which is conveniently stored in the database itself:</p>
<div class="highlight"><pre><span></span><span class="k">select</span> <span class="n">migration</span> <span class="k">from</span> <span class="n">south_migrationhistory</span> <span class="k">where</span> <span class="n">app_name</span><span class="o">=</span><span class="s2">"tasks"</span><span class="p">;</span>
</pre></div>
<p>The output is shown below. The lines of interest are <code>0010_auth__del</code> and <code>0010_auto__chg</code>; I’m trying to apply migration <code>0011</code> but I can’t, because it’s the same migration as <code>0010_auto__del</code>, which should have been reversed a few commits ago. </p>
<div class="highlight"><pre><span></span>+------------------------------------------------------------------------------+
| migration |
+------------------------------------------------------------------------------+
| 0001_initial |
| 0002_auto__add_feedback |
| 0003_auto__del_field_task_allow_multiple_finishes |
| 0004_auto__add_field_task_is_draft |
| 0005_auto__del_field_feedback_task__del_field_feedback_user__add_field_feed |
| 0006_auto__add_field_task_creator__add_field_taskarea_creator |
| 0007_auto__add_taskkeyword__add_tasktype__add_taskteam__add_taskproject__ad |
| 0008_task_data |
| 0009_auto__chg_field_task_team |
| 0010_auto__del_taskarea__del_field_task_area__add_field_taskkeyword_name |
| 0010_auto__chg_field_taskattempt_user__chg_field_task_creator__chg_field_ta |
+------------------------------------------------------------------------------+
</pre></div>
<p>I want to migrate backwards until <code>0009</code>, but I can’t do that directly because the migration file for <code>0010_auto__del</code> is not part of <code>HEAD</code> anymore, just like Migration 3 in the story of Sparkles and Rainbows, so South doesn’t know what to do. However, that migration does exist in a previous commit, so let’s go back in time.</p>
<p>I figure out which commit added the migration I need to reverse:</p>
<div class="highlight"><pre><span></span><span class="c1"># Display commit log along with names of files affected by each commit. </span>
<span class="c1"># Once in less, I searched for '0010_auto__del' to get to the right commit.</span>
git log --name-status <span class="p">|</span> less
</pre></div>
<p>With that key information, the following sequence of commands tidies everything up:</p>
<div class="highlight"><pre><span></span><span class="c1"># Switch to the commit that added migration 0010_auto__del</span>
git checkout e67fe32c
<span class="c1"># Migrate backward to a happy migration; I chose 0008 to be safe. </span>
<span class="c1"># ./manage.py migrate [appname] [migration]</span>
./manage.py migrate oneanddone.tasks <span class="m">0008</span>
git checkout master
<span class="c1"># Sync the database and migrate all the way forward using the most </span>
<span class="c1"># up-to-date migrations.</span>
./manage.py syncdb <span class="o">&&</span> ./manage.py migrate
</pre></div>Open Source Bridge 20142014-06-30T09:18:00-04:002014-07-01T00:00:00-04:00Maja Frydrychowicztag:www.erranderr.com,2014-06-30:/blog/osb14.html<p>Open Source Bridge 2014 was my first conference experience in about 5 years and my first non-academic conference. It had a great mix of talks and was thoughtfully organized and super friendly. I learned about Firefox quality assurance, communication strategies, remote work, discrimination in the tech industry and being a …</p><p>Open Source Bridge 2014 was my first conference experience in about 5 years and my first non-academic conference. It had a great mix of talks and was thoughtfully organized and super friendly. I learned about Firefox quality assurance, communication strategies, remote work, discrimination in the tech industry and being a better human. That mix could have been totally different though: during almost every session, there were 3-4 interesting talks competing for my attention, which is a good thing.</p>
<p>Attendees were welcome to help out with conference logistics and encouraged to organize spontaneous, open meetings. For example, I spent a couple of hours volunteering as session chair, which amounts to helping each speaker set up for their talk, introducing them and giving them timing cues. </p>
<p>Liz and I also set up a table at the conference Hacker Lounge to <a href="https://www.erranderr.com/blog/one-and-done-osbrige.html">introduce potential contributors to Firefox quality assurance and One and Done</a>. We ended up supporting four people as they each set up a development environment for our project. As expected, each participant ran into snags along the way with things like database configuration so it was good to work on this in a group. </p>
<p>It felt awesome to share my recent experience and speed up their setup process. I think the fact that One and Done is my first Django project helped a lot: when someone is new to a project, having expert mentors is great but in many cases it’s much easier to pose questions and relate to someone who is just a bit less of a beginner than you are. </p>
<p>That particular Hacker Lounge session got me thinking about how easy it is for potential <span class="caps">FOSS</span> contributors to get discouraged by setup alone. A good readme is priceless! The <a href="https://github.com/mjzffr/oneanddone/blob/master/README.md">One and Done readme</a> is quite detailed and helpful, but we will likely add a setup guide or dev <span class="caps">FAQ</span> to the projet wiki for additional support. It’s silly for someone to miss out on contributing just because they get mysterious errors while installing mysql! </p>
<h1>Firefox Quality Assurance</h1>
<p>This conference happened to be an opportunity for me to work and talk in person with Liz for the first time during my <span class="caps">OPW</span> internship. In addition to discussing additions to One and Done, Liz gave me a nice overview of how Firefox features and bug fixes get tested and gradually make their way through a series of release channels: from Nightly, to Aurora, then Beta, then a real release. We looked through this <a href="https://wiki.mozilla.org/Releases/Firefox_31/Test_Plan">Firefox 31 Test Plan</a> and talked about the supporting tools used by the <span class="caps">QA</span> team.</p>
<p>Many of these tools are actually developed by Mozilla and there was a really nice talk at this year’s Open Source Bridge about one such tool, Socorro, which is used for <a href="http://www.opensourcebridge.org/sessions/1319">Firefox crash reporting and analysis</a>. Fun facts:</p>
<ul>
<li>If you go to <code>about:crashes</code> in Firefox, you can easily access the <a href="https://crash-stats.mozilla.com/">crash reports</a> you’ve submitted and see whether anyone is working on the associated bug(s).</li>
<li>Socorro stores about 3 months worth of crash reports (3-5 million crashes per day!) and those crashes are initially sent to file system storage rather than a database. Why? A file system has better up-time than a database.</li>
<li>To access any particular crash quickly on that file system, a <a href="http://www.twobraids.com/2012/12/socorro-file-system-storage.html">radix storage scheme</a> is implemented with <strong>symbolic links</strong>. Cool. (You can’t just store thousands of files in one directory — file access would be ridiculously slow.)</li>
</ul>
<p>Those points refer to just one component of Socorro — there’s all kinds of other neat stuff about it.</p>
<h1>Selected Talks</h1>
<p>Some of the talks I attended, like the one about Socorro, related quite closely to my current internship with Mozilla.</p>
<ul>
<li>Roan Kattouw showed us surprising and bizarre browser <a href="http://www.opensourcebridge.org/sessions/1247">rendering bugs</a> that he discovered while working on the <a href="http://www.mediawiki.org/wiki/VisualEditor">Visual Editor</a> for Wikimedia.</li>
<li>As it turns out, lots of people are as <a href="https://www.erranderr.com/blog/remote-work.html">ambivalent about remote work</a> as I am, and we all flocked to two talks about how great it is to work on a distributed team. Handy tips for remote work: <a href="http://www.opensourcebridge.org/sessions/1244">overcommunicate</a> and <a href="http://www.opensourcebridge.org/sessions/1278">always assume good faith in text communication</a>. I feel hopeful.</li>
</ul>
<p>Other talks helped me think about my future career path in the tech industry and the open source community. Unexpected bonus: many of these talks caused me to reflect about diversity and bias in our field.</p>
<ul>
<li>The way Julie Pagano described <a href="http://www.opensourcebridge.org/sessions/1185">impostor syndrome</a> really hit home: “People with impostor syndrome do not share knowledge, collaborate, help with <span class="caps">OSS</span>, apply for jobs. [They] start small, remain small, end up small.” </li>
<li>In a talk on <a href="http://www.opensourcebridge.org/sessions/1394">negotiation</a>, I learned how important it is to practice silence and pause: if you don’t feel informed enough to make a decision, focus on getting your questions answered. It sounds obvious but it’s hard to do in the moment! Similarly, another talk about <a href="http://www.opensourcebridge.org/sessions/1291">influence</a> described how to identify and respond to different communication styles and how to pay attention to the people around us.</li>
<li><a href="http://franceshocutt.com/2014/07/01/why-are-these-people-following-me-leadership-for-the-introverted-uncertain-and-astonished/">Frances Hocutt’s keynote</a> is filled with so much awesome. Go listen to it now. Here’s one of my favourite parts:</li>
</ul>
<blockquote>
<p>Leadership is learned, and learning requires vulnerability, and
vulnerability depends on the safety to be vulnerable. And our communities are
demonstrably unsafe. We are artificially limiting our pool of leaders. […]
We create shame around ignorance. We devalue social and emotional
skills. […] We devalue lived experience and natural variation, and
in its place,
we raise up a false and impossible impartiality, which ends hurting
us all. We experience incompetent and unethical leadership, and
instead of learning to lead better we devalue the idea of leadership.</p>
</blockquote>
<h1>A Delighful Conference Experience</h1>
<ul>
<li>All the food was vegetarian, most of it was vegan. I eat meat but I didn’t miss it at all. It somehow added to the nice atmosphere that everyone could eat everything. </li>
<li>The attendee badges were actually booklets that contained the conference schedule and the code of conduct. One’s name appears on each side of the booklet, so no problem with flipped badges. The lanyards for the badges were colour-coded so you can easily tell who is ok with being photographed and who is not. </li>
<li>There was a quiet room! So nice for overwhelmed introverts like me. Being around people is exhausting. :)</li>
</ul>
<p>The way the conference was organized was considerate, inclusive and thoughtful. I hope to go back in future years. </p>“One and Done” comes to Open Source Bridge 2014!2014-06-20T17:58:00-04:002014-06-20T17:58:00-04:00Maja Frydrychowicztag:www.erranderr.com,2014-06-20:/blog/one-and-done-osbrige.html<p>As part of Open Source Bridge 2014, I’ll be at the Hacker Lounge to show people how to use One and Done to get started with contributions to Mozilla, as well as help interested developers set up an development environment so they can contribute new features to the One and Done tool.</p><p>So, you want to contribute to Mozilla somehow, but you’re not sure where to start or who to talk to. (I felt that way just a few months ago; the anguish is still fresh in my mind.) Fortunately, there’s a tool called <a href="https://oneanddone.mozilla.org">One and Done</a> that can help you complete your first contributions. But wait, there’s more…</p>
<p><em>You can drop by the <strong>Hacker Lounge</strong> at <a href="http://www.opensourcebridge.org">Open Source Bridge</a> to get started with contributions to Mozilla through One and Done, or get started with development of new One and Done features! I’ll be there with <a href="https://twitter.com/lizhenry">Liz</a>, my <a href="https://gnome.org/opw/"><span class="caps">OPW</span></a> internship mentor, on the evening of <strong>Wednesday, June 25th</strong> to guide you and answer questions. Beginners welcome!</em></p>
<p>I’ve been helping out with the development of One and Done for a few weeks now, and it’s great fun. The tool aims to connect new Mozilla contributors to specific tasks according to their interests, skills and available time. It also allows you to keep a record of your accomplishments at Mozilla. We have a lot more planned for it, and if you know Python and want to learn Django, you can help us add new features.</p>
<h1>What to Expect at the Hacker Lounge</h1>
<p>We have two main activities that we’ll be helping out with. </p>
<h2>Use One and Done to Make Contributions to Mozilla</h2>
<p>Dive right into your (first?) Mozilla contribution. You’ll create a One and Done profile, pick some tasks that interest you and record your progress. If you run into any snags, we’ll be there to help you out or show you how to engage with project teams through <span class="caps">IRC</span>, mailing lists and so on. </p>
<p>If you have suggestions for how to make One and Done more useful or if you encounter a bug in the tool, we’ll show you how to use Bugzilla to share your feedback with the One and Done team.</p>
<h2>Work on One and Done as a Python Developer</h2>
<p>There’s lots to do if you want to work on One and Done itself. It’s essentially a Django project and a nice example of how web development is done at Mozilla. The <a href="https://github.com/mozilla/oneanddone">One and Done source code</a> is hosted on Github.</p>
<p>I’ll be happy to help you set up your development environment and build the project. Not a Github person yet? That’s ok, I can help with that, too. :) We can chat about the project structure and some Django basics, as well as where to find project requirements and how to use Bugzilla to collaborate with the One and Done team.</p>Project Management at Mozilla2014-06-05T20:41:00-04:002014-06-18T00:00:00-04:00Maja Frydrychowicztag:www.erranderr.com,2014-06-05:/blog/bugzilla-github.html<p>How One and Done uses Github pull requests and bugs in Bugzilla.</p><p>One of the most interesting aspects of my <a href="https://wiki.gnome.org/OutreachProgramForWomen"><span class="caps">OPW</span></a> internship is learning how contributions and planning are managed in Mozilla projects. So huge! So many contributors! Ahhh! Mozilla is known for giant projects like Firefox, but it also builds lots of supporting tools and websites. All projects, big and small, are managed using Bugzilla and often a mix of other tools, depending on the project team. My observations are based mostly on one project that I’ve become familiar with: <a href="https://github.com/mozilla/oneanddone">One and Done</a>. Warning: I’m going to say “Bugzilla” a lot.</p>
<h1>Bugzilla</h1>
<p><a href="https://bugzilla.mozilla.org/">Bugzilla</a> is used to describe “bugs” and track their status. A “bug” in Bugzilla in not necessarily a problem that needs to be fixed: it can be feature planned by the core project team, a suggestion from a community member, a representation of a project milestone or even a request for new office furniture for a team member. Seriously.</p>
<p>Ideally, all the discussion about a bug takes place publicly in its Comments section on Bugzilla so that everyone can see how the bug is evolving and anyone can join in. (Not all discussion is public: bugs that relate to a security vulnerability can only be viewed by authorized users.)</p>
<p>If the bug represents a new feature, people might use the Comments section to narrow down or adjust requirements, request clarification or feedback, etc. </p>
<ul>
<li><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1020981">Example 1: Profile button bug</a> </li>
<li><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1005082">Example 2: Front page header text bug</a></li>
</ul>
<p>Bugzilla is also where developers submit solutions for <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/How_to_Submit_a_Patch#Getting_the_patch_reviewed">code review</a>. </p>
<h1>Bugzilla with Github</h1>
<p>Like many other Mozilla projects, One and Done has its repository hosted on Github. Github Issues are not used at all since Bugzilla bugs fulfill their role. When a developer makes a pull request on Github, that <span class="caps">PR</span> should refer to a Bugzilla bug <span class="caps">ID</span> and it should be <a href="http://globau.wordpress.com/2013/10/21/github-pull-requests-and-bugzilla/">attached to the bug</a> in Bugzilla with a request for review. (<a href="https://github.com/mozilla/oneanddone/pull/124">Example 3: Completed Tasks <span class="caps">PR</span></a>.) One can also add a brief summary of the <span class="caps">PR</span> in the bug comments. Detailed discussion and feedback about the code takes place in the pull request on Github, but a summary thereof is always included in Bugzilla, often by the reviewer. </p>
<h1>One Source of Information</h1>
<p>Since there are so many tools available to describe and communicate about project progress, it’s a common problem to have project information spread around in many different places, where it may be overlooked or become out-of-date. </p>
<p>In the case of One and Done, although there is a <a href="https://wiki.mozilla.org/QA/OneandDone">project wiki</a>, a <a href="https://github.com/mozilla/oneanddone">Github repo</a>, a <a href="https://mozilla.kanbanery.com/projects/45827/board/?key=fe86e00cb6c613df344772a58b72bd92a0f38995">Kanban board</a> and brief discussion over <span class="caps">IRC</span> and email, all the key project data is in Bugzilla or linked-to from there. As a contributor to the project, there’s really only one place I need to look for definitive information, and that’s Bugzilla. Bugzilla, Bugzilla, Bugzilla.</p>Working Remotely is… Hard? Different? Strange?2014-05-23T15:30:00-04:002014-05-23T15:30:00-04:00Maja Frydrychowicztag:www.erranderr.com,2014-05-23:/blog/remote-work.html<p>My internship with Mozilla involves work within a team whose members are located around the world and not where I am. Some are Mozilla employees or maybe contractors, some are interns like me, some might be volunteers. The key point is that it’s not just me with my mentor …</p><p>My internship with Mozilla involves work within a team whose members are located around the world and not where I am. Some are Mozilla employees or maybe contractors, some are interns like me, some might be volunteers. The key point is that it’s not just me with my mentor, focused on our own area: I need to find my voice in this in group of people, and that’s where working remotely gets hard. </p>
<p>My very initial, tentative impressions of full-time, remote work is that it’s not bad, but it takes more getting used to than I expected. I’ve worked from home a lot, as a developer and as a teacher, but I’ve only ever collaborated on projects with people that I spend time with “at the office” or that I meet with in person from time to time. My current full-time, remote status makes me realize that chatting and getting to know coworkers between bursts of productivity is pretty valuable! Crazy! Time for me to refer you to <a href="http://theoatmeal.com/comics/working_home">Why working from home is both awesome and horrible</a>.</p>Dive In! Applying to the FOSS Outreach Program for Women2014-05-13T13:00:00-04:002014-05-13T13:00:00-04:00Maja Frydrychowicztag:www.erranderr.com,2014-05-13:/blog/applying-to-opw.html<p>I got an internship with Mozilla through <span class="caps">GNOME</span>’s Outreach Program for Women (!). The application process alone taught me a lot about open source contributions and the community around them.</p><p>I just started an internship with Mozilla’s <a href="https://quality.mozilla.org/">Quality Assurance</a> team as part of the <span class="caps">GNOME</span> Foundation’s <a href="https://gnome.org/opw/">Outreach Program for Women</a> (<span class="caps">OPW</span>), but I almost didn’t even apply! In fact, I had decided <em>not</em> to apply for silly reasons, and then a week before the <em>extended</em> deadline I abruptly changed my mind — after some encouragement from the wonderful people at my <a href="https://www.hackerschool.com/">Hacker School</a> batch. (As it happens, I almost didn’t apply to Hacker School either. This is a pattern.)</p>
<p>Of course, I’m very pleased that I did apply in the end, not just because I was ultimately selected for an internship but because the process itself was incredibly instructive. By the time I submitted my application, I felt like I could start contributing to Mozilla on my own regardless of whether I got an internship. </p>
<p>Overcoming initial barriers is built into the <span class="caps">OPW</span> application process: I interacted with people on <span class="caps">IRC</span> and it was scary but productive and good; I posted to project mailing lists and it was scary but productive and good; I added information to bugs in Bugzilla, and it was scary but productive and good. You get the idea… I made mistakes sometimes, but I fixed them and nothing terrible happened and no one seemed to hate me.</p>
<h1>Context</h1>
<p><em>Wait, what’s this Outreach Program for Women?</em> The <span class="caps">GNOME</span> Foundation and many others want more people who identify as women or genderqueer to be involved in contributions to free and open source software (<span class="caps">FOSS</span>). Awesome! So one thing <span class="caps">GNOME</span> does is organize payed internships with various <span class="caps">FOSS</span> organizations especially for this category of humans. To apply, the human in question must learn about the participating organizations and their available projects, choose one or more projects of interest, <em>interact with current contributors</em> and <em>contribute something themselves</em>. Depending on the project, one’s application process might take a few weeks. The project descriptions indicate how you can get started with contributing, and you need to communicate with the project mentor about what you might work on during the application.</p>
<h1>My Application Experience</h1>
<p>I started working on my application a week before the deadline, and I spent 7 full days on my application, setting aside my other Hacker School projects during that time. By the deadline, I wasn’t sure my contribution was substantial enough but I submitted what I had anyway. That being said, the process seems relatively flexible and varies across organizations. I could have expanded my contributions after the application deadline — the reason I didn’t boils down to other time constraints on my side.</p>
<p>My application was for the <a href="https://wiki.mozilla.org/GNOME_Outreach_Summer2014#Mozilla_Bug_Wrangler_.28Desktop_QA.29">Bug Wrangler</a> internship with Mozilla. When I contacted the designated mentor, Liz (<a href="https://twitter.com/lizhenry">@lhenry</a>), I explained that I’m more interested in the software development aspects of the position, and she suggested some more specific projects based on that. </p>
<p>At that point, I only had a few days left to make a contribution to the project in question, so I focused on gaining experience with tasks related to the bug life-cycle at Mozilla: reproducing and adding information to unconfirmed bugs in <a href="https://bugzilla.mozilla.org/">Bugzilla</a>, testing the latest nightly build of Firefox to verify that bugs have been fixed, and so on. I also set up the development environment for a Django project I might work on during the internship. I managed to build the project and wanted to contribute something to it as part of my application, but I felt that I didn’t have enough time to communicate with the project team before the application deadline. (I regret this; maybe I should have just jumped in.)</p>
<p>Here are some examples of how I spent my time during the application process, <strong>aside from making my contributions and writing the application itself</strong>. These small tasks add up, both in terms of time and in terms of making future contributions easier. </p>
<ul>
<li><strong>Reading</strong>, so much panic and reading. Things I read or skimmed: project descriptions, contributor/tool guides/documentation, Bugzilla terminology and Firefox/Mozilla components, bug reports, pull request discussions on Github, mailing-list archives, press releases, blog posts by past interns, project meeting minutes, <span class="caps">IRC</span> etiquette, etc.</li>
<li><strong>Setup and configuration</strong>: registration with things like Bugzilla, <span class="caps">IRC</span>, Mozillians; choosing and setting up an <span class="caps">IRC</span> client; configuring a testing environment for Firefox.</li>
<li><strong>Communication</strong> : emailing and chatting with the project mentor about available projects, my interests, and the application process; asking questions on <span class="caps">IRC</span> when I run into issues; I participated in a small video meeting during a Mozilla Test Day; introduced myself and asked for help on a couple of mailing lists.</li>
<li><strong>Learning</strong> in preparation for more contributions: trying out the specific tools I might work on to gain a user perspective; reading about Django and building one of Mozilla’s Django projects; setting up a database for it.</li>
</ul>
<p>My top tips for applying: </p>
<ol>
<li>Communicate a lot. (I think that’s almost the whole point.)</li>
<li>Keep a brief, informal log of what you do as you do it. The log is handy for when you have questions later, when you come back to the project after a few weeks away, when you’re writing your application, and so on.</li>
<li>Don’t assume there is some universally accepted, perfect approach to your task. There are grey areas. Example: how to classify a particular Firefox bug. So, do ask questions! Remember that “experts” ask questions, too, and interesting discussion among several helpful people many ensue, which is really cool. :)</li>
</ol>
<p>I’ll say it again: the <span class="caps">OPW</span> internship application process is a great learning experience. Although I already knew a good amount about <span class="caps">FOSS</span> before I got started, I now have a much better sense for how a giant, open source project like Firefox grows and stays healthy. </p>
<p>Do apply! Encourage others to apply! </p>