The Freemap blog, revisited

As mentioned in my last post I was aiming for March 1st to make the kothic-based Freemap 0.6 the current version. It is indeed in more or less a working state, and is compatible with all the main current browsers (even IE9). However, I have made a reasonable amount of progress on routing in the last week or so and would like to implement the “intelligent walking routes” functionality in which you can draw a walking route on the map and, using Freemap annotations and description tags in the underlying data, auto-generate a walking route.

In order for 0.6 to be a significant advance over the current version, I am therefore delaying moving 0.6 over to www.free-map.org.uk until a first pass at the walking route functionality is completed. This is likely, hopefully, to be during March. However in the meantime, you can access 0.6 at www.free-map.org.uk/0.6; bear in mind it is a work in progress and may occasionally not function correctly if I’m testing something.

BTW I’m likely to be using libosmscout rather than pgRouting; tests on pgRouting have revealed one or two issues, for example you cannot route from point to point, only segment to segment (try the “walk route” option on 0.6 as it is at the moment to see what I mean). libosmscout, however, allows node to node routing and is quite amazingly fast, calculating a route from Fernhurst to Southampton, some 30 miles, instantaneously.

Have been continuing to do a few things with the new, kothic-js based Freemap 0.6, which, like the current version of Freemap (0.5) now features search and annotation facilities. There are still a few issues, rendering can be rather slow if the server is loaded or the client machine is busy, and some odd rendering artefacts occasionally occur (such as the underlying canvas tiles occasionally not being blanked if you change location): but the underlying map data is being kept up to date and a kothic-based system will be more sustainable than the raster solution and so overall I think it’s time to switch.

This means that from March 1st, barring disaster, Freemap 0.6 will become the default version and the 0.5 source code will be archived as a downloadable tarball. Being canvas-based it will require an HTML5-compatible browser, but this does include current versions of Firefox, Chrome, Opera and Internet Explorer (and at this point I would like to comment on what an improvement IE9 is over earlier versions). I would also like to thank Chris Jones at SUCS for helping to keep Freemap going in the past year or two through his very generous offer of hosting the tiles. With improvements to kothic-js doubtless in the pipeline (there is talk of client-side caching which should resolve performance problems) I’m now fairly convinced that this is the way to go.

This will of course not be a final version of 0.6; many features in 0.5 such as walking routes will not initially be present. It is my hope to add in a new, pgRouting based walking routes feature, time permitting as always.

With the move to vector rendering on the web client, I am also looking to make changes to Freemap’s Android app OpenTrail too and switch from osmdroid to the vector-based Mapsforge. At the moment this does not allow custom styles but the upcoming 0.3 release, due at the end of this month apparently, will.

Have finally got round to moving Freemap source to github. Would not say I’m an expert on it by any means, as I’ve only just started using git, but you can find the repository here: https://github.com/nickw1/Freemap. This means that the existing SVN repository will no longer be maintained.

A few small updates to Freemap 0.6, the new experimental version of Freemap using kothic-js rendering, since I last posted. Bridges and railways now appear, but the main feature is coastline. How was this done?

It was quite simple but thought I’d post it here anyway. Firstly, the standard processed_p.shp file used for Mapnik rendering was converted to SQL using the shp2pgsql tool (part of the PostGIS distribution, I believe). This was then imported into the PostGIS database. The server-side script which generates the GeoJSON data for kothic-js was then altered to return the ST_Intersection() of the current bounding box and any coastline polygons in the area. This then returns a square of land in inland areas, or a complex polygon of land in coastal areas.

As is done in Mapnik, the canvas background was set to light blue. The land polygons from the previous step were tagged – arbitrarily – server side with “natural=land”, and then a natural=land rule added to the MapCSS. So pretty straightforward really! The updated code is now in the SVN repository.

In other developments, I have updated Freemap’s PostGIS database with OSM data current as at last weekend. However – and this will impact upon the search facility on the main, current, Mapnik-based Freemap site (0.5) as well as 0.6 – *only* data from certain counties is included. I think I have finally found a way to make Freemap sustainable with kothic-js (so many, many thanks to the kothic team!) but only if I restrict coverage to certain parts of the UK. In Southern England this includes Hampshire, West Sussex, Surrey, Wiltshire and Somerset (but not those parts which used to be Avon); the whole of Wales is covered; and in Northern England, Cheshire, Derbyshire, Lancashire and Cumbria. In this way, I have covered both my local patch and a significant proportion of the national parks and popular walking areas in the rest of England and Wales. I might experiment with adding further counties but I’d prefer to have a Freemap covering some of the UK only and working reasonably well, rather than unsustainably covering the whole country. Chrome still appears to give the best performance.

What of future plans? Freemap 0.5 (the Mapnik version) is now in feature-freeze so no new features will be added, I’m hoping to make 0.6 the default version by the spring.

With some time off over Christmas I’ve managed to put together a demo of client-side rendering using kothic-js, available here. This will hopefully form the basis to the next revision of Freemap (0.6), which should also feature pgRouting-based walking routes. As you will see if you look at the demo, it is styled similarly to current Freemap and incorporates Ordnance Survey LandForm PANORAMA contours. Currently it covers certain parts of the UK only (in southern England, W Sussex, Surrey, Hampshire, Wiltshire, Somerset, in northern England, Derbyshire, Lancashire and Cumbria, and the whole of Wales); due to server constraints this state of affairs is likely to remain.

So how easy was this to do? Well firstly, OpenLayers has been replaced by Leaflet as the slippy map library, as Leaflet integrates well with kothic-js. Am impressed with Leaflet’s simplicity and clean API I have to say, it seems a very straightforward library to use and doesn’t try to do too much; in that respect it feels rather similar to the FLTK C++ GUI library that I used a number of years ago for the first Freemap editor, way back when Freemap was an entirely separate project to OSM.

Data is sent back from the server in “Google XYZ” tiles in a slightly-modified GeoJSON format, another recent development in geodata which has somehow passed me by until now, and another piece of the system which is impressive in terms of its cleanness and simplicity. The kothic-js Leaflet layer basically works by requesting data tile-by-tile on demand. I did have to make some small changes to the default L.TileLayer.Kothic class to allow user-specified URLs; the modified code is available here. The server-side code is available from Freemap SVN (can be browsed here).

What of the styling? This is done by yet another clean and simple (at least so far!) format, MapCSS. Used by an increasing number of OSM projects, this allows you to specify the styles of map elements using CSS-like syntax and is significantly easier to work with than the Mapnik XML format, for example. The stylesheet so far is very basic, just enough for a demo, but can be found here. The MapCSS then needs to be compiled into a JavaScript file to work with kothic; instructions on how to do this can be found on the kothic-js website.

How were the contours done? Basically they were sourced from the standard DXF files distributed with the OS LandForm Panorama download package, reprojected into Google Spherical Mercator and added to a PostGIS database with the lfp2pg tool, available from Freemap SVN (source code here). The intensive import was done at home, then a database dump was done and the SQL imported back into the Freemap database.

So what of the demo? Well it’s obviously a bit slower than static PNG tiles but the performance does not seem to be too bad, and it’s certainly usable. Of the browsers I’ve tested on (Firefox 8, Chrome 11.0, Opera 11.6) Chrome gives the best performance as it appears to cache the canvas tiles whereas Firefox and Opera appear not to. Some server-side caching occurs (namely with the contours, which will never change) which improves things; due to server space constraints I am unable to cache everything, and querying the database for contours appears to be the slowest part of the whole process. Also, it’s significantly faster than doing live Mapnik rendering from the database server-side; with contours, that is unacceptably slow, taking 5 seconds or so to render one tile.

Aside from performance there are one or two little issues at tile boundaries. As described on the kothic site, coordinates of features have to be supplied from the server relative to tile boundaries, where (0,0) is the bottom left and (granularity,granularity) the top right. However, line features with coordinates of 0 or the granularity appear to give some bizarre tile rendering artefacts, with features becoming disjointed and not rendering at their correct position. Consequently, coordinates of 0 are adjusted to 1, and coordinates of (granularity) adjusted to (granularity-1). This means some small gaps at tile boundaries; if anyone can shed some light on this issue I’d be grateful! The ST_Intersection PostGIS function is used to “crop” features at tile boundaries BTW.

So overall I’m quite satisfied with this experiment, and it’s likely that a kothic-based renderer will become the default view for Freemap 0.6. The SUCS tiles and, possibly, home-rendered Mapnik tiles will be still available as alternative layers but the kothic tiles will be most frequently updated and importantly, will exactly reflect Freemap’s underlying database.

As you can probably appreciate, it’s always something of a challenge running a not-for-profit site on limited memory and disc space, and due to needing the space for some new things I want to do with Freemap, I’ll unfortunately have to stop offering the demo overlaying LandForm Panorama contours and OSM footpaths on VectorMap District raster tiles from January 1st. Apologies for this, but it is occupying a whopping 33% of the entire disc space on the Freemap server.

However, the tiles – or at least the LFP contours overlaid on the VMD raster tiles, which were the trickiest to produce – are available for download from the dev server. See here; there are 5 .tar.bz2 files representing 100-km wide strips from west to east.

Keep a look out, hopefully, for more news on using kothic-js for rendering and have a happy Christmas!

As I said in my last post, I’m looking to move to using a pre-built routing library for Freemap walking routes, and of the various options, pgRouting seems to work best for a web app as it’s based on PostgreSQL and PostGIS. Managed to get it properly installed now, and have populated the routing database with OSM data in Hampshire, Wiltshire, West Sussex and Surrey using the osm2pgrouting tool. Performance is encouraging (as in, instantaneous) for trying to find a route from Fernhurst, W Sussex to Older Hill, two and a half miles or so away.

There are, however, one or two “gotchas” with using osm2pgrouting to import the data. Firstly, the pgRouting documentation doesn’t mention the need to load the topology functions. Consequently, the import process fails (silently) at the topology stage as assign_vertex_id() is missing. You need to do this once you have created your database, with the additional command:

psql -U gis -f /usr/share/postlbs/routing_topology.sql routing

(assuming a user of “gis” and database of “routing”).

The other issue is that backslashes in OSM way names cause an error in the import process. You have to modify the source of osm2pgrouting to deal with this; see, for example: this post.

Would like to announce an update to the walkers’ Android OSM app OpenTrail. Other than the augmented reality stuff (now separated out, see recent posts), I haven’t really had the chance to add any new features to OpenTrail until now; the last release back in March included Freemap and OS out of copyright maps and the ability to add annotations in the field.

However I’ve made a start on adding new features to OpenTrail with a view to developing it into a full-featured walkers’ app. You can now search for selected nearby points of interest (pubs, restaurants, places and peaks); the POIs are downloaded from the Freemap server and then cached on the device. Also, existing annotations and walk notes are also now overlaid on the map.

I’m hoping to do some more work on OpenTrail in the coming weeks/months, the long-term aim being an open-source version of something like Memory Map. One feature I’ve had in mind for a while is for the app to display annotations and notes to the user in the field as alert boxes (probably accompanied by some form of alert sound) as they pass its location. This should be fairly simple to implement and could have a number of uses, for instance, a user could be presented with some information about a view or a place of historical interest as they pass it, or could be presented with directions at a location where it’s easy to get lost. I can envisage this working as follows. Someone follows a path, loses their way temporarily, finally finds the path, and then records instructions in OpenTrail and uploads them to the server. A later walker could then be presented with the same instructions automatically as they pass the same spot.

I’m also aiming to integrate with Freemap walking routes, so that, for example, a user could search for walking routes in their area. However, I’m planning (time permitting of course, as always) on overhauling walking routes as part of a new and quite significantly different “development” version of Freemap (running alongside the main site) in which I aim to experiment with client-side rendering using kothic-js (see last post), so addition of walking routes to OpenTrail will be in conjunction with this. The current walking routes code in Freemap is slow and inefficient, particularly now that the volume of data has increased. So I started looking round for pre-built solutions and came across pgRouting, which adds routing functionality to a PostGIS database. Got step one done here: it’s built and installed successfully and I’ve managed to import some OSM data using osm2pgrouting. Next step will be to see if it actually routes between two points quickly and efficiently; if so I plan on using it to work out a walking route of underlying OSM ways given a set of user-specified points. However, to avoid the DB getting too big (and thus, possibly, slow) the walk route functionality might be restricted to selected areas of the UK initially.

Anyway, where is OpenTrail? It’s available here, if you find any bugs let me know.

Am started to get interested in Kothic-JS (see here) following a tip-off from Richard Fairhurst on the dev mailing list this week. Looks an exciting project, allowing you to render OSM data client-side using HTML5 canvas, thus cutting down on one of the most awkward tasks to deal with when running your own server: rendering tiles. Performance, on the demo showing Minsk and surrounds, looks pretty impressive, and given that these days, client-side resources are much, much less expensive than server, anything which moves the load from server to client must be worthy of further attention.

So I’ll almost certainly, given the time, be investigating kothic-js further as a more sustainable rendering solution for Freemap: currently I’m relying on the goodwill of Swansea University Computer Society and home-rendering selected Southern England counties.

However there still needs to be a lot of *data* stored server-side, in other words, everything you want to render. Up until today (for the purpose of the walking-route facility) Freemap stored rights-of-way and permissive paths for the whole of England and Wales in its postgis database. However this would obviously not be sufficient for providing data for rendering. Consequently, it now stores *all* highways, but *only* for a subset of English counties, and for all of Wales. The counties available are: Hampshire, W Sussex, Wiltshire, Surrey, Bucks, Cambridgeshire, Cumbria, Essex, Herefordshire, (now that’s interesting, why does the wordpress spellcheck recognise some of these counties and not others?), Norfolk, Suffolk, Somerset, Shropshire, Oxfordshire and Kent. Reason being that those are the ones for which geofabrik extracts are available, which allows county-by-county import and makes the osm2pgsql import less memory-intensive. I’ve asked Frederik for Cheshire, Derbyshire and N Yorks extracts too to try and complete the list of popular walkers’ destinations.

So what will the immediate impact on Freemap be? None unless you use the walking route facility or the data API. These will now only work in those counties, plus Wales, but it will recognise routes which don’t have a designation tag. It also means that due to the increased volume of data, the walking route generation is rather slow now – but we’ll have to live with that for the moment. Since rendering is not done on the server, rendering will be no slower and POI search doesn’t seem to be affected.

Back soon with some news of OpenTrail, the Freemap Android app.

Since I last posted I’ve been doing some initial experiments on streaming the camera feed into a texture within the OpenGL view in Hikar, thereby attempting to avoid the problem of overlapping GLSurfaceViews (one for the camera feed, and one for the OpenGL layer).

Following examples on nhenze.net (here) and looking at the AndAR source
code (see here), I managed to get it working, at least in black-and-white (the problem is that the camera feed is delivered in YUV format, whereas OpenGL textures expect RGB and there is a frequently-quoted-as-expensive conversion between the two, for which native code is recommended, e.g. at nhenze.net). So for the moment it’s black-and-white only.

However unfortunately that doesn’t quite solve the problem. You set up a PreviewCallback to receive the camera feed and implement the onPreviewFrame() method, which takes an array of YUV bytes and a Camera object. So far, so simple. But… to send the camera feed to this method, you need to have a visible camera preview in a GLSurfaceView, which is what I was trying to avoid! Various postings on the web confirm that this indeed appears to be the case: you can’t capture the feed without having a preview open.

So it seems that this is a practically insoluble problem, barring re-implementing GLSurfaceView from scratch to somehow incorporate a camera feed. So for the moment, it looks like Hikar will continue to have to do the augmented reality the “wrong way”, overlaying two SurfaceViews on top of each other. (Incidentally the open-source Android AR apps and libraries I’ve seen seem to do this way anyway). Luckily, an API is available to send a camera feed direct to a texture from Android 3.0+. Hmmmm…. time methinks to buy an ice cream sandwich.