Tag Archives: development

Factory Refactoring – Done!

After over 50 Pull Requests spread over the last 9 months, I’ve finally finished refactoring the openstreetmap-website test suite to use factories instead of fixtures. Time for a celebration!

As I’ve discussed in previous posts, the openstreetmap-website codebase powers the main OpenStreetMap website and the map editing API. The test suite has traditionally only used fixtures, where all test data was preloaded into the database and the same data used for every test. One drawback of this approach is that any change to the fixtures can have knock-on effects on other tests. For example, adding another diary entry to the fixtures could break a different test which expects a particular number of diary entries to be found by a search query. There are also more subtle problems, including the lack of clear intent in the tests. When you read a test that asserts that a given Node or Way is found, it was often not clear which attributes of the fixture were important – perhaps that Node belonged to a particular Changeset, or had a particular timestamp, or was in a particular location, or a mixture of other attributes. Figuring out these hidden intents for each test was often a major source of the refactoring effort – and there’s 1080 tests in the suite, with more than 325,000 total assertions!

Changing to factories has made the tests independent of each other. Now every test starts with a blank database, and only the objects that are needed are created, using the factories. This means that tests can more easily create a particular database record for the test at hand, without interfering with other tests. The intent of the test is often clearer too, since creating objects from factories happens in the test itself and are therefore explicit about what attributes are the important ones.

An example of the benefits of factories was when I fixed a bug around encoding of diary entry titles in our RSS feeds. I easily created a diary entry with a specific and unusual title, without interfering with any other tests or having to create yet another fixture.

def test_rss_character_escaping
  create(:diary_entry, :title => "<script>")
  get :rss, :format => :rss

  assert_match "<title>&lt;script&gt;</title>", response.body
end

All in all, this took much, much longer than I was expecting! Looking back, I feel I might have picked a different task, but I’m glad that it’s all done now. I’m certainly glad that I won’t have to do it all again! Moving on, it’s now easier for me and the other developers to write robust tests, and this will help us implement new features more quickly.

Big thanks go to Tom Hughes for reviewing all 51 individual pull requests! Thanks also to everyone who lent me their encouragement.

So the big question is – what will I work on next?

Steady progress on the OpenStreetMap Website

Time for a short status update on my work on the openstreetmap-website codebase. It’s been a few months since I started refactoring the tests and the work rumbles on. A few of my recent coding opportunities have been taken up with other projects, including the blogs aggregator, the 2017 budget for the OSMF Operations Working Group (OWG), and the new OWG website.

With the fixtures refactoring I’ve already tackled the low-hanging fruit. So now I’m forced to tackle the big one – converting the Users fixtures. The User model is unsurprisingly used in most tests for the website, so the conversion is quite time-consuming and I’ve had to break this down into multiple stages. However, when this bit of the work is complete most future Pull Requests on other topics can be submitted without having to use any fixtures at all. The nodes/ways/relations tests will then be the main thing remaining for conversion, but since the code that deals with those changes infrequently, it’s best to work on the User factories first.

As I’ve been working on replacing the fixtures, I’ve come across a bunch of other things I want to change. But before tackling all that I’m going to mix it around a bit. My goal is to alternate between the work I think is the most important, and also helping other developers with their own work. We have around 40 outstanding pull requests and some need a hand to complete. There are plenty of straightforward coding fixes among the 250 open issues that I can work on too. I hope that if more of the issues and particularly the pull requests are completed, this will motivate some more people to get involved in development.

If you have any thoughts on what I should be prioritising – particularly if you’ve got an outstanding pull request of your own – then let me know in the comments!

Upgrading the OpenStreetMap Blogs Aggregator

One of my projects over the winter has been upgrading the blogs.openstreetmap.org feed aggregator. This site collects OpenStreetMap-themed posts from a large number of different blogs and shows them all in one place. The old version of the site was certainly showing its age. The software that powered it, called PlanetPlanet hasn’t been updated for over 10 years, and can’t cope with feeds served using https, so an increasing number of blogs were disappearing from the site. Time for an upgrade.

My larger goal was moving the administration of the site into the open, in order to get more people involved. Shaun has maintained the old system for many years and has done a great job. Unfortunately there were tasks that could only be done by him, such as adding new feeds, removing old feeds, and customising the site. To make any changes you had to know who to ask, and hope that Shaun had time to work on it for you. It was also unclear what criteria a blog feed had to meet to be added, and even more so, if and when a blog should be removed.

The challenge was therefore to move the configuration to a public repository, move the deployment to OSMF hardware, create a clear policy for managing the feeds, and thereby reduce the barriers to getting involved all round.

Thankfully, other people did almost all of the work! After I investigated the different feed aggregation software options – most of them are barely maintained nowadays – I reckoned that Pluto was the best choice. It turns out Shaun had previously come to the same conclusion, and had almost completed the migration himself. Then Tom put together the Chef deployment configuration, which was the second half of the work. So all I had to do was finish converting the list of feeds, make a few template changes, and everything was ready to go. After a few weeks delay while the sysadmins worked on other tasks, the new site went live on Friday.

If you know of a blog that should be added to the site, please check our new guidelines. If you have any changes you want to make to the site, the deployment, or the software that powers it, now is you chance to get involved!

Refreshing the OpenStreetMap Codebase

The codebase that powers OpenStreetMap is older than any other Rails project that I work on. The first commit was in July 2006, and even then, that was just a port of an existing pre-rails system (hence why you might still see it referred to as “The Rails Port”)

It’s a solid, well-tested and battle-hardened codebase. It’s frequently updated too, particularly its dependencies. But if you know where to look, you can see its age. We have very few enduring contributors, with is surprising given its key position within the larger OpenStreetMap development community. So I’ve been taking a look to learn what I can do to help.

OpenStreetMap-website Contributions

For someone just getting started with Rails, they’ll find that many parts of our code don’t match what’s in any of the books they read, or any the guides online. More experienced developers will spot a lot of things that were written years ago, and would nowadays been done differently. And for some of our developers, particularly our Summer of Code students, they are learning what to do by reading our existing code, so our idiosyncrasies accumulate.

I started trying to fix a minor bug with the diary entries and a bunch of things struck me as needing a thorough refresh. I started the process of refactoring the tests to use factories instead of fixtures – 2 down, 37 to go. I’ve started rewriting the controllers to use the standard rails CRUD method names. And I’ve made a list of plenty of other things that I’d like to tackle, all of which will help lower the barrier for new (and experienced) developers who want to get stuck into the openstreetmap-website code.

I hope that progress will snowball – as it becomes easier to contribute, more people will join in, and in turn help make it even easier to contribute.

But it’s time-consuming. I need help to share these projects around. If you’re interested, please get stuck in!

Getting Involved in the Operations Working Group

For the last few years I’ve been trying to get more people involved in the Operations Working Group – the team within the OpenStreetMap Foundation that runs all of our services. Each time I think “why aren’t more people involved”, I try to figure out some plausible barriers to entry, and then work on fixing them.

One of the reasons is that we’re very quiet about what we do – there’s a pride in keeping OpenStreetMap humming along and not causing too much of a fuss. But the lack of publicity hurts us when we’re trying to get more people involved. Hence this blog post, among other reasons.

I’ve been working recently (as in, for the last few years) on making as much of our OWG activities public as possible, rather than hidden away on our mailing list or in our meetings. So we now have a public issue tracker showing our tasks, and we also publish a monthly summary of our activities.

To make OpenStreetMap work, we run a surprisingly large number of servers. For many years we maintained a list of our hardware and what each server was being used for on the OpenStreetMap wiki, which helped new people find out how everything works. Maintaining this information was a lot of work and the wiki was often outdated. For my own projects I use Jekyll to create static websites (which are easier to find hosting for than websites that need databases), and information on Jekyll sites can be generated with the help of data files. Since OWG uses Chef to configure all of our servers, and Chef knows both the hardware configuration and also what the machines are used for, the idea came that we could automate these server information pages entirely. That website is now live on hardware.openstreetmap.org so we have a public, accurate and timely list of all of our hardware and the services running on it.

Now my attention has moved to our Chef configuration. Although the configuration has been public for years, currently the barrier to entry is substantial. One straightforward (but surprisingly time-consuming) improvement was to simply write a README for each cookbook – 77 in all. I finished that project last week.

Unless you have administrator rights to OSMF hardware (and even I don’t have that!) you need to write the chef configuration ‘blind’ – that is, you can propose a change but you can’t realistically test that it works before you make a pull request. That makes proposing changes close to impossible, so it’s not surprising that few non-administrators have ever contributed changes. I have experience with a few tools that can help, the most important being test-kitchen. This allows a developer to locally check that their changes work, and have the desired effect, before making a pull request, and also it allows the administrators to check that the PR works before deploying the updated cookbook. Both Matt and I have been working on this recently, and today I proposed a basic test-kitchen configuration.

This will only be the start of a long process, since eventually most of those 77 cookbooks will need test-kitchen configurations. Even in my initial attempts to test the serverinfo cookbook (that generates hardware.openstreetmap.org) I found a bunch of problems, some of which I haven’t yet figured out how to work around. There will be many more of these niggles found, but the goal is to allow future developers to improve each cookbook using only their own laptops.

All of these are small steps on the long path to getting more people involved in the Operations Working Group. If you’re interested in helping, get stuck in to both our task list and our chef repo, and let me know when you get stuck.

Coding and writing with the Atom editor

Atom brands itself as “A hackable text editor for the 21st Century”. I first used Atom in August 2014 as an alternative to the Kate editor that comes with Ubuntu and which I had used for many years before that. I was drawn towards Atom by all the publicity around it on Hacker News, and it seemed to have momentum and a few useful features that Kate was lacking. It didn’t take me long to find myself using Atom exclusively.

Much is said online about Atom being slow to respond to key presses, or that it’s built in JavaScript, or that it uses a Webview component to render everything, or some other deeply technical topic, but none of those topics are worth discussing here.

Instead, a key feature for me is that Atom is built on an extensive a plugin architecture, and ships with a bunch of useful plugins by default. There are hundreds more available to download, but trawling through the list is a pain and there’s always a few gems buried in the long tail. To save you some hassle, here’s a list of the plugins that I use. I’ve divided them by the three main tasks for which I use Atom regularly – cartography, coding and writing.

Cartography

language-carto – “Carto language highlight in Atom”. I only found out about this recently from Seth but, as you can imagine, it’s very useful when writing CartoCSS styles! Finding out about this plugin by chance conversation is what sparked the idea of writing this post.

color-picker – “A Color Picker for Atom”. I use Atom to edit all my CartoCSS projects, so this seems like a good idea, but I haven’t actually used it much since installing. I prefer Inkscape‘s HSL colour picker tab when I’m doing cartography, and working in Inkscape lets me quickly mock up different features together to see if the colours balance.

pigments – “A package to display colors in project and files”. This simply highlights colour definitions like #ab134f with the appropriate colour – almost trivial, but actually very useful when looking at long lists of colour definitions.

Development

git-blame – “Toggle git-blame annotations in the gutter of atom editor”. I only started using this last week and it’s a bit rough around the edges, but it’s quicker to run git-blame within my editor than either running it in the terminal or clicking around on Bitbucket and Github. Linking from the blame annotations to the online diffs is a nice touch.

language-haml – “HAML package for Atom”. I was first introduced to HAML when working on Cyclescape and instantly preferred it to ERB, so all my own Ruby on Rails applications use HAML too. This plugin adds syntax highlighing to Atom for HAML files.

linter – “A Base Linter with Cow Powers”, linter-rubocop – “Lint Ruby on the fly, using rubocop”, and linter-haml – “Atom linter plugin for HAML, using haml-lint”. I’m a big fan of linters, especially rubocop – there’s a separate blog post on this topic in the works. These three plugins allow me to see the warnings in Atom as I’m typing the code, rather than waiting until I save and I remember to run the tool separately. Unfortunately it means I get very easily distracted when contributing to projects that have outstanding rubocop warnings!

project-manager – “Project Manager for easy access and switching between projects in Atom”. I used this for a while but I found it a bit clunky. Since I’m bound to have a terminal open, I find it quicker to navigate to the project directory and launch atom from there, instead of clicking around through user interfaces.

Writing

Zen – “distraction free writing”. This is great when I’m writing prose, rather than code. Almost all my writing now is in Markdown (mainly for Jekyll sites, again, a topic for another post) and Atom comes with effective markdown syntax highlighting built-in. I’m using this plugin to help draft this post!

That’s it! Perhaps you’ve now found an Atom plugin that you didn’t know existed, or been inspired to download Atom and try it out. If you have any suggestions for plugins that I might find useful, please let me know!