Quick and dirty usability testing of OSM

Last week I joined Ant and Deb from MapQuest in order to help out with the UCL mapping party. On the Wednesday I went out with some new Masters students and got soaked in the rain around Camden, but the main interest for me was the following day when we all gathered in the computer lab to uploaded the newly collected data. While I was helping out I was also scribbling furiously whenever I found someone stuck on some aspect of OSM that I hadn’t expected.

UCL student mapping party

I was briefly worried that there would be a flurry of activity while they logged on and that I’d miss most of it, but actually the account creation was so long and tortuous that it gave me plenty of time to watch. Silver linings, etc, I guess. I took notes, and so here they are, in the order I wrote them down.

  1. Where did the email go? – The biggest hurdle and the one that spread them out was confirming their email. Given that the OSM servers are on the same campus as we were, it took an extraordinary amount of time for them to appear. But the issue here was that on the user signup page there was no indication as to which email address the confirmation email was sent to, and one person was worried there was a typo. It also made it impossible for me to check that there wasn’t a typo in their (to them) brand new address.
  2. Nobody reads the CTs, and everyone ticks the PD box without reading it either – I’ll win no friends with this observation, but I saw nobody scrolling the CTs box, and everyone reflexively ticked the box beside the agree button. I’m guessing they all thought it was a “have you read the above legal stuff” which you normally get on such forms.
  3. Send another confirmation email – There’s no way to trigger sending another copy of the confirmation email. Sometimes they go missing, and at least if there was a button the frustration levels would go down.
  4. Not obvious what the settings page is for – After confirming their email the users end up on the settings page, where almost the first thing it shows you is your email address and a box to put a new email address into. That confused a lot of people. Things like add a friend, set a home location, read some getting started notes etc would be more useful
  5. Highlight unrecognised tags – I found one guy who had, and it’s not clear how, ended up with all his name tags with a capital N. These would be better highlighted while editing that it’s an unexpected tag.
  6. Anxiety over tags missing from autocomplete lists – on two occasions I had people worried that what they were typing (in both cases “office”) wasn’t in the autocomplete list. I had to explain that there are things on Map Features (and elsewhere on the wiki) that aren’t in the list, and that’s not a problem.
  7. Confusion over the preset dropdown (10a and 10b on this image) Three people struggled to make it stay open (i.e. click – hold – move – release). One guy kept selecting different things, and didn’t realise it was adding more tags and changing one (amenity) that he’d already set, until I pointed it out. I had to explain the small icon (10a) was a button that changed what was on the dropdown. Most of the icons used in 10a weren’t understood (car and bike were good, the football and postbox less so). Many people made the same mistake of adding a POI, adding the correct tags, and then worrying that it said “(no preset)” and tried to find the correct thing in the menu – i.e. misunderstood the purpose of it.
  8. Couldn’t find double-click – Since they were entering POIs they’d already collected, they rapidly found themselves without an appropriate one on the POI panel and searched the wiki. With the tags in hand, they were then stumped on how to add a blank POI. One guy worked out he could change the tags on an existing one, but either instructions (“double click”) or a multi-purpose / “blank” POI icon would be better.
  9. Couldn’t add extra tags – three or four times people needed the + icon pointed out to them
  10. Map Features – long descriptions – most people found themselves on Map Features reading the key, value and short description, but I didn’t see anyone realise that they could click the value for more details. This should perhaps go (automatically) onto the end of the short description text as a “More details…” link.
  11. Confusion with abandoned features – repeatedly people found proposed and/or abandoned features, and similar wiki-works-in-progress. As well as not understanding, they also didn’t care, and didn’t read the page either – they were just skim-reading to find the tags they needed. I’d lean strongly towards clearing off the 3-year-old abandoned pages, but I realise there are “wiki-historians” who want to keep everything for posterity.
  12. Search beyond Map Features – most people searched up and down the Map Features page using the browser-based search (Ctrl+F). They were then stuck when they couldn’t find the thing they were looking for, and had to be pointed towards the search box to search the rest of the wiki. Again, it wasn’t clear that there are plenty of things obscure enough to not be on the main list. Also, “Also known as” and “similar to” and “see also” sections of the tag documentation are worth their weight in gold. A surprising number of pages don’t have them.

A lot of the most interesting stuff I found was regarding Potlatch 1, and (fortunately?) very little of it applies to Potlatch2 since the UI has been overhauled. I’d love to also work on the Friends functionality of the website, since when the students started “friending” each other, pretty much nothing happened. We could show friends edits, diary entries etc. One thing that stood out for me though, was we should remove the PD tickbox from the CTs. It’s added confusion if you read it, and most people don’t so the point of it is moot. It’s not on the critical path for signup so it shouldn’t be in the signup flow at all. It can live in the user settings page or somewhere similar. It’s not legally binding and it’s not working a straw poll either. Finally, it would be great if there was more stuff possible before the email was confirmed, like adding friends – or even links to introduction videos or something like that.

I’ll leave you with the best and least-expected I-never-thought-of-that example of the day. I watched one student find the entry in Map Features for the shop that he wanted to add. He highlighted the icon, right clicked and selected Copy, then changed tabs to Potlatch and right clicked in order to paste the icon where he wanted it to appear.

If only, my friend, if only.

Thanks to Muki Haklay and Thomas Koukoletsos from UCL for inviting us along. If anyone has any similar opportunities for me to come and watch people learning OSM, please get in touch.

13 thoughts on “Quick and dirty usability testing of OSM

  1. Richard Fairhurst

    Of the Potlatch issues (5-9), happily 7 and 9 are explicitly fixed in Potlatch 2, and 6 and 8 are largely irrelevant in P2 (far more draggable POIs). That just leaves 5, which would be solved as outlined at http://lists.openstreetmap.org/pipermail/potlatch-dev/2010-October/000168.html . Maybe you might beat me to coding it. 🙂

    So we have site usability issues (1-4) which are fairly easy to fix; editor issues (5-9) which are already fixed or being fixed; and “our documentation is appalling” issues (10-12) which remain entirely unresolved. And to be a bit harsher, some of the ostensible editor issues are at least documentation issues too; users can pick things up when they’re explained to them, but if the documentation doesn’t explain it in the first place…

    Who’s going to step up to fix the docs?

  2. Tom Chance

    Great work all, and thanks to Andy for the writeup.

    On no. 5 (unrecognised tags), why not force all keys to be lowercase to deal with situations like Name=something?

    Something else I have noticed with friends is that the green circle representing unknown nodes isn’t very obvious, so a friend I was introducing to OSM added a duplicate point without checking that little green dot. Perhaps, like the drag-and-drop generic POI, it needs a question mark icon or similar?

  3. Damouns

    An other problem is how to find all the comments I did on other users’ diaries. Just to check if they answered me.

  4. Martijn van Exel

    Andy — Great read, and great work. So good to see usability testing happening. I know many have a gut feeling about some aspects of OSM being just too hard to grasp when you first get started. Good to see that turn in to something that we can actually work on. This will sure help to reduce that rediculously high churn rate we’re still dealing with.

  5. Frank M. Eriksson

    Potlatch might not be perfect (Man, it is made in Flash; Only that is a downside) but it is a hell a lot easier than JOSM (which by the way works worse than before on my Ubuntu 10.04, might be better on 10.10 – but I have not had the time to do a backup and migrate yet). Potlatch 2 does indeed have some features that make me boot up JOSM anyway sometimes, like straighten out edges on buildings and stuff. But the UI is actually worse, It does not update it’s info as it should and on many places there is lack of “Ok”-buttons and that makes one wounder if it actually swallows whatever one types in and just because of that one goes and recheck if it is correct. And the dialogues are not resize-able, otherwise the UI looks very swell and marvelous – but I hope that performance is not related to that, but something else.

    It would also be good if one could make Potlatch able to have big images and WMS/KML (From example from http://warper.geothings.net/ ), There is no way for me to make a custom background come up in Potlatch (either 1 or 2). And I have really tried hard. I managed once to split up a large image in GIMP and save the tiles one by one; But I got a few ones misaligned. And I can’t get the generate_tiles.py from the OSM SVN to work either… I’m not that good with python, really… I’m more int C-style languages like JavaScript, HaXe and C. Other langs I know is a few assembly dialects (I does really connect with those, they are just so logic!) And a few scripting languages, both common and strange (like esoteric) or uncommon ones.

    For a few days I’ve been tinkering about writing my own editor in HaXe and compile for the Flash/AWM2 target, but I’m not really good at the ActionScript API. I fear that I might not pull that one off and just waste a lot of time and energy.
    Would be great to know if someone would like to help me out with that, you can answer here – I’ll return and read this blog a lot – and my own site is down for maintenance right now, so you cannot contact me there for a while (a month or two tops).

    » » Cheers!

  6. Cycle

    Potlatch 2 is better than Potlatch 1, but where is the “repeat tags from previous element” button?

    What about in-browser editor in Javascript, instead of Flash?

    The documentation is a mess; I hope I don’t make the map a mess, too.
    Like, there are two roads without sidewalks. There is a footway connecting them (without it you would have to walk ten times more). Is it correct to bind the footway to nodes of the road? It seems the only way how a routing algorithm could find it.

    “Make circular” in Potlatch 2 doesn’t work for me at all.

  7. Andy

    For javascript vs flash there’s little advantage in developing a javascript-based editor. Between cross-browser incompatibilities and the two or three orders of magnitude difference in performance between browser versions, it’s highly unlikely to give a consistent experience between browsers. I’d love for someone to prove me wrong, but until that point I’ll be continuing to develop in flash. I spoke more on this subject at the recent WhereCampUK conference.

    As for the support issues around potlatch2, you’re best off asking your questions on http://help.openstreetmap.org

  8. Pingback: Shine » OpenStreetMap usability revisited :: GravityStorm

  9. Clare

    Hi Andy,
    We may have an opportunity to get involved with new users again, with a specific focus on transport related information (including walking and cycling) for journey planning.
    I’ve tweeted you about how to measure completeness of Cycling Networks and how to work it out – if you have any info that would help in our initial research that would help lots.

  10. Pingback: Shine » Make the hard things simple, and the simple things occasionally surprisingly hard :: GravityStorm

Comments are closed.