The Freemap blog, revisited

Have now finished with my initial experiments on home tile creation and can now give you an idea of what will be on the new Freemap. I’ve come up with a style I like which combines the existing Freemap style for rights of way and contours with the current full OSM style – so for the first time, Freemap will show the full range of landuse shading shown in the main OSM tiles.

Bad news perhaps is the area of extent. It takes five hours on this quite powerful machine to generate the tiles in the bounding box 2W, 50N, 0, 52N for zoom levels 10 to 15, and then almost an hour to upload the 500MB odd of data to the Freemap server. This basically equates to an area which just about includes Swindon, Oxford, London, Brighton, Bournemouth. This is therefore the absolute limit of data I can deal with. However, what I’ll probably do is ditch zoom level 15 (as that takes the majority of tile generation time and file upload size) and expand the area of coverage a little, west to 2.5W to cover a bit more of Wessex and the Cotswolds. As indicated earlier I may well also add in the Lake District and Snowdonia. Sorry guys if you’re out of this area but with the limited processing power available to me, that’s all I can do! However, as said earlier I hope to cover the rest of the country via the Swansea University Computing Society webserver – though as I don’t have admin rights it may be a little more restricted.

And if you do have the technical know-how (and if you follow the wiki, it’s really not hard to do your own tile generation, though it does require familiarity with a command-line) I’d urge you to set up your own Freemap server for your own part of the country. This seems to me the most effective way to deal with resource issues: distribute the load to several machines, each covering a small part of the country but when combined together, able to produce full maps of the whole UK. Do get in touch if you’d like more help on this!

Another update from last night. Have got contours working! The standard contour-importing procedure into PostGIS was completely painless and now I’ve got some nice tiles being rendered. Just want to tweak the Mapnik rules and it should be ready!

One issue though is the large amount of data I have to upload onto the Freemap server. Luckily my broadband usage is unmetered after 6pm (soon to be 7pm while we get a useless free hour between 6am and 7am when we’re all still in bed … nice one, BT!) so that’s not too much of a problem – but the bzipped tar file of a significant area of S England between zoom levels 10 and 16 still came to 250MB odd. So, I’m sorry to say, I am likely to have to restrict rendering to certain areas, probably at this time southern England plus one or two other interesting areas like Snowdonia or the Lake District. Nonetheless the rest of the country will hopefully continue to be served, either by the SUCS site or by the old tiles on Freemap. Sorry if this isn’t absolutely ideal but it’s an unfortunate side effect of this being a hobbyist site and I think it’s more important to do a small area well rather than a large area badly. Anyway, a) it’s not that hard to set up your own server for your own area (honest!) and b) who knows, if I develop Freemap for S England it might attract sources of income to expand the area covered so be patient!!!

More soon but look out for a relaunch and new style tiles in the near future!

These memory issues on Freemap have really been bothering me, so spent much of the day trying to find a way round it. First of all I used Osmosis to try and reduce the amount of data in the .osm file to what was absolutely necessary for Freemap. I did this and successfully imported a recent (March 1st) planet extract to the Freemap database, however the file size was not that much reduced and therefore this is unlikely to provide a long-term solution.

So I finally took the plunge and decided to install a full Mapnik setup on my home machine – PostGIS and all – with the intent of rendering the tiles at home, where I have the memory and processing power, and upload them to the server, where I don’t. This actually wasn’t quite as painful as I remember from last time (when I set it up on Freemap more or less three years ago to the day) – I encountered one or two problems (setup: Ubuntu 9.10, Postgres 8.3, Mapnik 0.6.1, and latest SVN version of osm2pgsql) but both were resolved by looking at the wiki. Two problems might not be immediately apparent on this setup (they caused me difficulty at first) so I’d like to draw attention to them: section 5.1.7 (Authentication failed) on the Mapnik OSM wiki page, and the need to run the 900913.sql file included in osm2pgsql (see the osm2pgsql README).

So have successfully rendered some tiles using the default Mapnik style. Next task will be to tweak them for the Freemap style and add in contours a la CycleMap. One problem of course will be the size of the archive I have to upload to the server, but for some of the UK at least, with luck the memory problems will be solved! In terms of the database on Freemap, I’ll probably keep it for selected, relevant POIs only (to allow the maintenance of the search facility) but the ways will be dropped.

So overall a good day! Only thing was, it was 3pm by the time I finished that, so while I did manage to get out for about three hours, I couldn’t get to a place where I could do some new mapping! Still, only three weeks till BST and we can finally lay the winter to rest…

A bit of an update on the Footnav project, which aims to be a 3D mobile navigation tool for countryside users. In the past few months Android has become a serious contender in the mobile world, and being the most viable mobile operating system if you don’t want someone restricting redistribution of your app, has become my main focus of interest in terms of mobile development. Got myself an Android phone recently too so fancy having a play!

Therefore I’m shifting Footnav development from C++/Qt to Android. Haven’t done any Android development before, though, so as a first step I’m aiming to produce a simple Android app called “Fix My Paths!” The aim of this will be to allow users to report footpath problems using their phone and either upload them direct to the Freemap server in the field, or save on the SD card for upload at home. Then I’ll probably port Freemap Mobile to Android, before converting the WebGL 3D code (something that should now be a lot easier now I’ve had a taste of the sometimes-esoteric ways of OpenGL ES…)

Would also like to keep the WebGL 3D viewer going, though as always my time is limited and it’s a case of which project is interesting me the most at the moment….

First of all, I don’t know about you but I’m glad winter’s finally on the way out! I’m not sure how many more of those dark, dismal days I could put up with

Anyway, a further update on Freemap. I’m hoping to get the rendering sorted, or at least partly sorted, on my account on the Swansea University Computer Services server (thanks to Chris Jones!) There are a couple of issues with this: firstly, this server is used for other things, so might occasionally go down, and secondly, the thing that wasn’t working last time when I had a play on the SUCS server was the contours, something which was failing for apparently random reasons and I was having great difficulty with sorting out. So no guarantees on this, but at the very least updated tiles for the whole country minus contours should be available.

Meanwhile, I want to focus the Freemap server itself on a relatively small area of central southern England, and try and use it as a showcase for countryside mapping using OpenStreetMap. Hampshire and its neighbouring counties are one of the best mapped areas for footpaths in the UK (though there’s still lots to do!) so in the face of limited server capabilities, it makes sense to focus on this area. I’ll probably set up a system where Hampshire and its surroundings is rendered on Freemap and the rest is rendered via the SUCS server. And remember of course that Freemap is open source so (though I have to admit configuring PostGIS is a bit of a pain….) if you fancy setting up a server yourself to do your part of the UK, you can download the source and setup!

I also want to develop the 3D and path problem reporting stuff. More of that in another post…

Just after writing the last post I realised it hopefully shouldn’t be so difficult to deal with the memory problem. Namely, I don’t need *all* the information in the planet file, so it should be possible to use osmosis to filter out just those things that I need. Of course the difficult bit will be working out what I need and what I don’t, but hopefully there should be a way forward…

The other thing I possibly want to look into is to ditch the postgres database altogether (except for POIs) and render the tiles off-site using custom software based around the Mapnik OSM plugin. More of that later should I decide to go down that route…

Meantime, back to dealing with an even bigger problem: working out which footpaths in south Hampshire haven’t been mapped yet

Sorry it’s been a while since I’ve been posted last! Just been busy with other stuff, as you are..

Anyway, unfortunately a little bit of bad news – I’m having difficulties updating the Freemap database. Problem seems to be that recent UK extracts are causing the import process to go out of memory. I’ve tried adding the “-s” option to osm2pgsql but that has no effect.

This means that for the moment, Freemap is stuck with a database from mid December 2009. Unfortunately, memory issues with servers are difficult to deal with – other than cutting back the amount of data in the database. That said I do have some other webspace on the Swansea Computing Society webserver which has more memory – though I experienced difficulties with getting mapnik talking to the postgres database on that. I hope to find a resolution to this issue but it does very much depend on whether I have the time to spare which,as always is at a premium.

On a brighter note I’ve now got hold of an Android phone and hope to do some interesting things with that, in particular develop a version of Freemap 3D for it eventually. Watch this space…

Just a quick update before Christmas.

Firstly just one small tweak to Freemap 3D. Variable width lines are now handled by the gl.lineWidth() method (thanks to Coolcat for info, not sure how I missed that…) One remaining issue I do have though is that when you rotate the camera to certain angles, the lines are not rendered correctly – can anyone shed any light on this?

Secondly there seems to have been a problem with the latest import of OSM data for the Oxford/Cotswolds area (bounds approximately Bath, Worcester, somewhere east of Northampton, and the Chilterns) into the Freemap database. I’ve tried importing it twice with no luck, so obviously there’s a serious issue with the data. Will try and investigate further before Christmas though time is a bit tight for me in the next couple of days. However all other parts of England and Wales seem to be fine.

If I don’t post back before, have a happy Christmas!

After quite a struggle (see last post!) I’ve managed to get terrain showing on “Freemap 3D”, my test WebGL-based OSM 3D viewer. However it was not easy. Ironically the hardest bits – importing the NASA SRTM height data from the server, calculating vertex normals, and doing the lighting – were actually quite straightforward as I’d done it all before with footnav, but what was very awkward was trying to set up a boolean uniform variable in the shader to determine whether to do lighting or not (yes for the terrain, no for the ways, because the latter do not need lighting effects). It was just not having it, and I haven’t yet figured out why.

But anyway, it is kind-of working:-) (see here) I’ve had to generate normals for all my OSM ways for the moment, so it’s scarcely the most efficient code in the world, but its performance is acceptable on my system at least. The SRTM height data is fetched on demand from the server along with the OSM data; my SRTM height server now generates JSON rather than binary data, easing its parsing on the client considerably. Problems still remain with rendering the ways; since WebGL does not have a method for drawing lines with a width greater than 1, I had to do it manually and the results aren’t always great. Also with it being near Christmas I maybe haven’t spent as long on it as I might otherwise so you might get the odd bug occurring, please let me know if so. Still it’s (hopefully) an interesting demo app showing what you can do with WebGL and OSM data.

This might be just about it as Christmas approaches and other things start taking priority, so if I don’t post again have a happy Christmas In the new year (as always, work and other commitments permitting) I hope to develop it further, so suggestions are welcome Am also considering possibly getting an Android phone so that’ll give me something else to possibly play with…

Have been trying to get SRTM terrain lighting working in “Freemap 3D”. That in itself actually works (you can kind-of see the results on the site now, which is temporarily non functional as far as OSM data goes however) but I’m having huge, huge difficulty trying to show SRTM *and* OSM data together the way I want it to.

The reason? I want the SRTM landscape to be lit, but not the OSM ways as this would be a waste of resources; namely creating a normal for every way vertex when the OSM ways do not need a lighting effect would be highly wasteful. My first thought to counter this was to create a uniform variable in the shader which could be set to either 0 or 1: 0 would mean no lighting, 1 would mean lighting. The fragment colour would then be calculated differently depending on the value of the uniform variable.

So, turning OSM off and just showing SRTM terrain, I had code something like:

if (uDoLightingCalculation==1)
{
do the lighting;
}
else
{
use a default colour;
}

I set the uniform lighting variable to 1. Problem – the terrain did not appear *at all*. Not even in the default colour. Worse, this code did not give the desired result (!)

if (uDoLightingCalculation==1 || uDoLightingCalculation!=1)
{
do the lighting;
}

However, this did:
if (true)
{
do the lighting;
}

So who knows what’s happening here…. The uniform variable did seem to get allocated a valid (>=0) ID so I’m really not sure. If anyone could clarify this it’d be most welcome but maybe it’s best to assume it’s a bug in WebGL for now?

The second way I tried was to have *two* shader pairs: a fragment and vertex shader pair for the SRTM data, and another for OSM. But this failed in various ways depending on whether the shader variables in the first pair and the second pair were named the same or differently.

Looks like I’ll have to resort to giving all the ways normals I would be interested to know if this is my bug, though, because if not, a bug report could be sent off to the Firefox WebGL maintainers. So if I am doing anything wrong, please let me know…