New OpenCycleMap tileserver

It’s been about a month since I launched with the new OpenCycleMap tileserver. The transition was actually much more hectic than I expected (I’ll describe the EC2 failure some other time) but things have stabilised a bit since and are humming along nicely.

So what’s new? Almost everything, actually, although mostly things you don’t get to see. New hosting, for a start, on a dedicated server with both huge SATA disks (mirrored 1.5Tb) and also a pair of blindingly fast 128Gb striped SSD drives. The postgis database is on the fast disks, everything else being on the SATAs. The speed of the SSD drives really comes into its own while rebuilding the indexes during hourly updates, rather than being directly related to rendering.

I’ve also upgraded all the software behind the scenes – new versions of mod_tile (with much better monitoring), renderd, mapnik, osm2pgsl. The biggest change that you’ll all notice is the tiles are now much smaller – around a third of the size – due to using palletised .png files. The new version of mapnik gives me the raster-handling needed to keep the background colouring at all zoom levels, which is a huge improvement. The biggest change I notice is that we’re now using osm2pgsql in slim mode – allowing incremental updates to the database instead of weekly drop/reload – and all of the fancy route-relation handling is now built into osm2pgsql instead of being post-reload SQL queries. It’s all much easier to deal with.

The whole upgrade project was delayed somewhat by the initial new server not working out. I’d specced out what I though was an appropriate machine, and everything worked fine – loading the database, testing rendering, the works. Unfortunately the diff updates were, on average, taking around “equal time” – an hour and a bit for an hours-worth of OSM changes, about 20 hours for a days worth of changes – and this with the machine doing nothing else! Disaster. So I scrambled to find a new provider with better hardware (at a reasonable price) and started again with the setup. Hat tip to Hetzner.de for their server offerings, and of course my ex-employers CloudMade for their continued sponsorship of the hardware.

Finally, I managed to sneak in some cartography changes while I was waiting for various things to complete (importing contours for the whole planet takes a few days!). Most of them were fairly subtle, and a fair few have gone unnoticed. Buildings are my new favourite, along with finally making new shield icons to replace the (ugh) nasty old ones (there’s a story behind them, as with most things). Can you spot any of the dozen other changes?

What’s next? I’d like to say more cartography updates, since there’s lots more I’d like to change. But cartography always seems to take a backseat to server administration, and that’s, as usual, starting to cause headaches. The huge numbers of people using the opencyclemap tileserver means that there’s many millions of tiles to render, and the server is only – just about – keeping up with refreshing them as the underlying data changes. Something needs to be rethought about this, and I’m hoping to find some speed improvements from somewhere to render the tiles faster and let the server keep its head above water. The server troubles are massively exacerbated by thoughtless people out there (abetted by thoughtless software developers) who are bulk-downloading, or scraping, the tileserver for tens of thousands of tiles at a time – approximately none of which they ever view. Plans are afoot to deal with them though!

13 thoughts on “New OpenCycleMap tileserver

  1. GrahamS

    Care to name and shame any of the developers that are hitting your server so hard?

    I know that the MotionX GPS iPhone app uses OpenCycleMap data and allows caching of large areas of map data. But I’m not sure if they hit your servers for the tiles or they maintain their own tile server (as they should).
    The thing is, having the tiles cached on the phone is pretty essential as it means less mobile data (important for those on limited contracts or using maps abroad) and it means app users are not reliant on a good phone signal when out in the hills.

    I guess you could block bulk downloads, then offer “Premium” access to your tile server for developers that need it?

  2. Matt Toups

    Hi Andy, thanks for the update on the tileserver, I’m definitely excited to see the improvements! I did notice some subtle rendering changes also, thanks for that. 😉

    Unfortunately the multipolygon rendering bug (which you are aware of from http://trac.openstreetmap.org/ticket/1656) is still causing problems for me, particularly due to relations of waterway=riverbank ways rendering as floods. Thanks for looking into it!

  3. Andy

    GrahamS – I’m trying to work with the developers behind-the-scenes, but I’ll certainly name and shame any that are unwilling to help.

    MotionX are the complete opposite of “bad behaviour”, in that we’ve been working for years together to make sure everything runs smoothly. They’re very nice guys to work with.

  4. Jeremy Young

    Hi maybe this is not exactly the right place but I wanted to leave a bit of feedback after using open cycle map in the field. I’m just back from a geology/palaeontology fieldtrip to the Samarinda area in Kalimantan (Indonesian Borneo- there’s a blog at http://www.nhm.ac.uk/natureplus/blogs/borneo).
    Before going I downloaded the open cycle map terrain into my iPhone via MotionX GPS. This proved absolutely invaluable, even though there is virtually no road information in the area. In fact it gave not just the best GPS map we had of the area but actually the most useful map of any type in the area. It was especially useful since this is an area of steeply dipping rocks so the terrain closely reflects the geology (except where coal mines and limestone quarries have removed the hills). More generally though the ready availability of high quality terrain maps this way is an immensely useful resource.
    Many thanks
    Jeremy Young

  5. Dimitrios

    Hi Andy, and thank you for the invaluable service you are offering.
    I’ve noticed some changes in the rendering of some islands, where contours seem to have disappeared, e.g. http://opencyclemap.org/?zoom=10&lat=36.22124&lon=22.95575&layers=B000 or http://opencyclemap.org/?zoom=10&lat=37.03567&lon=25.50046&layers=B000

    Looking closer, it looks as if the contours still exist but are covered by the filled outline of the land mass: http://opencyclemap.org/?zoom=14&lat=36.19587&lon=23.04841&layers=B000

  6. Paul Johnson

    The hourly updates don’t seem to be working…I’ve been adding cycle routes in Oregon and Oklahoma, and it’s been over a week now…still no dice. Also, zooms 8 and lower are hopelessly outdated in North America at this point.

  7. Andy

    They haven’t been hourly for a long time – while the database can keep up, the tile refreshing can’t. They run every 1-2 days now.

    However, I made a configuration error at the start of March and the database was falling further and further behind. For the last week or so it’s been catching up, and it should have caught up by the end of the week.

  8. Richard Stallard

    How is the tile server coping these days? I started doing some updates re bike paths, etc. in my local area (see http://www.openstreetmap.org/?lat=-31.98821&lon=115.81168&zoom=15&layers=M) about 3 weeks ago, but I haven’t seen any updates on OCM yet. The updates flow though pretty quickly to Open Street Map, usually well within an hour or two. They are also flowing though to the OCM database as Garmin files I have downloaded from the OSM Australia site, derived from the database, include details which I have entered recently.

    Is the slow update because not many people are using OCM in my area? I would like to promote OCM amongst the local cycling community including benefits such as being able to do DIY updates, but I am reluctant to do so until I see some evidence that updates are visible. A classic Catch 22 situation I suppose.

    Is there anything I can do to force an update for the area in which I am working?

    Thanks

  9. Richard Stallard

    Some progress to report – most of the Open Cycle Map changes I did 3-4 weeks ago have appeared on the OCM. I can see updated tiles!!

  10. Pieter Kuiper

    This is an invaluable resource, and I am working on writing an article for the Swedish publication “Cykling”. However, updating is indeed behind. For the low zoom levels, it is more than a month.

Comments are closed.