The Freemap blog, revisited

A few small updates to Freemap 0.6, the new experimental version of Freemap using kothic-js rendering, since I last posted. Bridges and railways now appear, but the main feature is coastline. How was this done?

It was quite simple but thought I’d post it here anyway. Firstly, the standard processed_p.shp file used for Mapnik rendering was converted to SQL using the shp2pgsql tool (part of the PostGIS distribution, I believe). This was then imported into the PostGIS database. The server-side script which generates the GeoJSON data for kothic-js was then altered to return the ST_Intersection() of the current bounding box and any coastline polygons in the area. This then returns a square of land in inland areas, or a complex polygon of land in coastal areas.

As is done in Mapnik, the canvas background was set to light blue. The land polygons from the previous step were tagged – arbitrarily – server side with “natural=land”, and then a natural=land rule added to the MapCSS. So pretty straightforward really! The updated code is now in the SVN repository.

In other developments, I have updated Freemap’s PostGIS database with OSM data current as at last weekend. However – and this will impact upon the search facility on the main, current, Mapnik-based Freemap site (0.5) as well as 0.6 – *only* data from certain counties is included. I think I have finally found a way to make Freemap sustainable with kothic-js (so many, many thanks to the kothic team!) but only if I restrict coverage to certain parts of the UK. In Southern England this includes Hampshire, West Sussex, Surrey, Wiltshire and Somerset (but not those parts which used to be Avon); the whole of Wales is covered; and in Northern England, Cheshire, Derbyshire, Lancashire and Cumbria. In this way, I have covered both my local patch and a significant proportion of the national parks and popular walking areas in the rest of England and Wales. I might experiment with adding further counties but I’d prefer to have a Freemap covering some of the UK only and working reasonably well, rather than unsustainably covering the whole country. Chrome still appears to give the best performance.

What of future plans? Freemap 0.5 (the Mapnik version) is now in feature-freeze so no new features will be added, I’m hoping to make 0.6 the default version by the spring.

With some time off over Christmas I’ve managed to put together a demo of client-side rendering using kothic-js, available here. This will hopefully form the basis to the next revision of Freemap (0.6), which should also feature pgRouting-based walking routes. As you will see if you look at the demo, it is styled similarly to current Freemap and incorporates Ordnance Survey LandForm PANORAMA contours. Currently it covers certain parts of the UK only (in southern England, W Sussex, Surrey, Hampshire, Wiltshire, Somerset, in northern England, Derbyshire, Lancashire and Cumbria, and the whole of Wales); due to server constraints this state of affairs is likely to remain.

So how easy was this to do? Well firstly, OpenLayers has been replaced by Leaflet as the slippy map library, as Leaflet integrates well with kothic-js. Am impressed with Leaflet’s simplicity and clean API I have to say, it seems a very straightforward library to use and doesn’t try to do too much; in that respect it feels rather similar to the FLTK C++ GUI library that I used a number of years ago for the first Freemap editor, way back when Freemap was an entirely separate project to OSM.

Data is sent back from the server in “Google XYZ” tiles in a slightly-modified GeoJSON format, another recent development in geodata which has somehow passed me by until now, and another piece of the system which is impressive in terms of its cleanness and simplicity. The kothic-js Leaflet layer basically works by requesting data tile-by-tile on demand. I did have to make some small changes to the default L.TileLayer.Kothic class to allow user-specified URLs; the modified code is available here. The server-side code is available from Freemap SVN (can be browsed here).

What of the styling? This is done by yet another clean and simple (at least so far!) format, MapCSS. Used by an increasing number of OSM projects, this allows you to specify the styles of map elements using CSS-like syntax and is significantly easier to work with than the Mapnik XML format, for example. The stylesheet so far is very basic, just enough for a demo, but can be found here. The MapCSS then needs to be compiled into a JavaScript file to work with kothic; instructions on how to do this can be found on the kothic-js website.

How were the contours done? Basically they were sourced from the standard DXF files distributed with the OS LandForm Panorama download package, reprojected into Google Spherical Mercator and added to a PostGIS database with the lfp2pg tool, available from Freemap SVN (source code here). The intensive import was done at home, then a database dump was done and the SQL imported back into the Freemap database.

So what of the demo? Well it’s obviously a bit slower than static PNG tiles but the performance does not seem to be too bad, and it’s certainly usable. Of the browsers I’ve tested on (Firefox 8, Chrome 11.0, Opera 11.6) Chrome gives the best performance as it appears to cache the canvas tiles whereas Firefox and Opera appear not to. Some server-side caching occurs (namely with the contours, which will never change) which improves things; due to server space constraints I am unable to cache everything, and querying the database for contours appears to be the slowest part of the whole process. Also, it’s significantly faster than doing live Mapnik rendering from the database server-side; with contours, that is unacceptably slow, taking 5 seconds or so to render one tile.

Aside from performance there are one or two little issues at tile boundaries. As described on the kothic site, coordinates of features have to be supplied from the server relative to tile boundaries, where (0,0) is the bottom left and (granularity,granularity) the top right. However, line features with coordinates of 0 or the granularity appear to give some bizarre tile rendering artefacts, with features becoming disjointed and not rendering at their correct position. Consequently, coordinates of 0 are adjusted to 1, and coordinates of (granularity) adjusted to (granularity-1). This means some small gaps at tile boundaries; if anyone can shed some light on this issue I’d be grateful! The ST_Intersection PostGIS function is used to “crop” features at tile boundaries BTW.

So overall I’m quite satisfied with this experiment, and it’s likely that a kothic-based renderer will become the default view for Freemap 0.6. The SUCS tiles and, possibly, home-rendered Mapnik tiles will be still available as alternative layers but the kothic tiles will be most frequently updated and importantly, will exactly reflect Freemap’s underlying database.

As you can probably appreciate, it’s always something of a challenge running a not-for-profit site on limited memory and disc space, and due to needing the space for some new things I want to do with Freemap, I’ll unfortunately have to stop offering the demo overlaying LandForm Panorama contours and OSM footpaths on VectorMap District raster tiles from January 1st. Apologies for this, but it is occupying a whopping 33% of the entire disc space on the Freemap server.

However, the tiles – or at least the LFP contours overlaid on the VMD raster tiles, which were the trickiest to produce – are available for download from the dev server. See here; there are 5 .tar.bz2 files representing 100-km wide strips from west to east.

Keep a look out, hopefully, for more news on using kothic-js for rendering and have a happy Christmas!

As I said in my last post, I’m looking to move to using a pre-built routing library for Freemap walking routes, and of the various options, pgRouting seems to work best for a web app as it’s based on PostgreSQL and PostGIS. Managed to get it properly installed now, and have populated the routing database with OSM data in Hampshire, Wiltshire, West Sussex and Surrey using the osm2pgrouting tool. Performance is encouraging (as in, instantaneous) for trying to find a route from Fernhurst, W Sussex to Older Hill, two and a half miles or so away.

There are, however, one or two “gotchas” with using osm2pgrouting to import the data. Firstly, the pgRouting documentation doesn’t mention the need to load the topology functions. Consequently, the import process fails (silently) at the topology stage as assign_vertex_id() is missing. You need to do this once you have created your database, with the additional command:

psql -U gis -f /usr/share/postlbs/routing_topology.sql routing

(assuming a user of “gis” and database of “routing”).

The other issue is that backslashes in OSM way names cause an error in the import process. You have to modify the source of osm2pgrouting to deal with this; see, for example: this post.

Would like to announce an update to the walkers’ Android OSM app OpenTrail. Other than the augmented reality stuff (now separated out, see recent posts), I haven’t really had the chance to add any new features to OpenTrail until now; the last release back in March included Freemap and OS out of copyright maps and the ability to add annotations in the field.

However I’ve made a start on adding new features to OpenTrail with a view to developing it into a full-featured walkers’ app. You can now search for selected nearby points of interest (pubs, restaurants, places and peaks); the POIs are downloaded from the Freemap server and then cached on the device. Also, existing annotations and walk notes are also now overlaid on the map.

I’m hoping to do some more work on OpenTrail in the coming weeks/months, the long-term aim being an open-source version of something like Memory Map. One feature I’ve had in mind for a while is for the app to display annotations and notes to the user in the field as alert boxes (probably accompanied by some form of alert sound) as they pass its location. This should be fairly simple to implement and could have a number of uses, for instance, a user could be presented with some information about a view or a place of historical interest as they pass it, or could be presented with directions at a location where it’s easy to get lost. I can envisage this working as follows. Someone follows a path, loses their way temporarily, finally finds the path, and then records instructions in OpenTrail and uploads them to the server. A later walker could then be presented with the same instructions automatically as they pass the same spot.

I’m also aiming to integrate with Freemap walking routes, so that, for example, a user could search for walking routes in their area. However, I’m planning (time permitting of course, as always) on overhauling walking routes as part of a new and quite significantly different “development” version of Freemap (running alongside the main site) in which I aim to experiment with client-side rendering using kothic-js (see last post), so addition of walking routes to OpenTrail will be in conjunction with this. The current walking routes code in Freemap is slow and inefficient, particularly now that the volume of data has increased. So I started looking round for pre-built solutions and came across pgRouting, which adds routing functionality to a PostGIS database. Got step one done here: it’s built and installed successfully and I’ve managed to import some OSM data using osm2pgrouting. Next step will be to see if it actually routes between two points quickly and efficiently; if so I plan on using it to work out a walking route of underlying OSM ways given a set of user-specified points. However, to avoid the DB getting too big (and thus, possibly, slow) the walk route functionality might be restricted to selected areas of the UK initially.

Anyway, where is OpenTrail? It’s available here, if you find any bugs let me know.

Am started to get interested in Kothic-JS (see here) following a tip-off from Richard Fairhurst on the dev mailing list this week. Looks an exciting project, allowing you to render OSM data client-side using HTML5 canvas, thus cutting down on one of the most awkward tasks to deal with when running your own server: rendering tiles. Performance, on the demo showing Minsk and surrounds, looks pretty impressive, and given that these days, client-side resources are much, much less expensive than server, anything which moves the load from server to client must be worthy of further attention.

So I’ll almost certainly, given the time, be investigating kothic-js further as a more sustainable rendering solution for Freemap: currently I’m relying on the goodwill of Swansea University Computer Society and home-rendering selected Southern England counties.

However there still needs to be a lot of *data* stored server-side, in other words, everything you want to render. Up until today (for the purpose of the walking-route facility) Freemap stored rights-of-way and permissive paths for the whole of England and Wales in its postgis database. However this would obviously not be sufficient for providing data for rendering. Consequently, it now stores *all* highways, but *only* for a subset of English counties, and for all of Wales. The counties available are: Hampshire, W Sussex, Wiltshire, Surrey, Bucks, Cambridgeshire, Cumbria, Essex, Herefordshire, (now that’s interesting, why does the wordpress spellcheck recognise some of these counties and not others?), Norfolk, Suffolk, Somerset, Shropshire, Oxfordshire and Kent. Reason being that those are the ones for which geofabrik extracts are available, which allows county-by-county import and makes the osm2pgsql import less memory-intensive. I’ve asked Frederik for Cheshire, Derbyshire and N Yorks extracts too to try and complete the list of popular walkers’ destinations.

So what will the immediate impact on Freemap be? None unless you use the walking route facility or the data API. These will now only work in those counties, plus Wales, but it will recognise routes which don’t have a designation tag. It also means that due to the increased volume of data, the walking route generation is rather slow now – but we’ll have to live with that for the moment. Since rendering is not done on the server, rendering will be no slower and POI search doesn’t seem to be affected.

Back soon with some news of OpenTrail, the Freemap Android app.

Since I last posted I’ve been doing some initial experiments on streaming the camera feed into a texture within the OpenGL view in Hikar, thereby attempting to avoid the problem of overlapping GLSurfaceViews (one for the camera feed, and one for the OpenGL layer).

Following examples on nhenze.net (here) and looking at the AndAR source
code (see here), I managed to get it working, at least in black-and-white (the problem is that the camera feed is delivered in YUV format, whereas OpenGL textures expect RGB and there is a frequently-quoted-as-expensive conversion between the two, for which native code is recommended, e.g. at nhenze.net). So for the moment it’s black-and-white only.

However unfortunately that doesn’t quite solve the problem. You set up a PreviewCallback to receive the camera feed and implement the onPreviewFrame() method, which takes an array of YUV bytes and a Camera object. So far, so simple. But… to send the camera feed to this method, you need to have a visible camera preview in a GLSurfaceView, which is what I was trying to avoid! Various postings on the web confirm that this indeed appears to be the case: you can’t capture the feed without having a preview open.

So it seems that this is a practically insoluble problem, barring re-implementing GLSurfaceView from scratch to somehow incorporate a camera feed. So for the moment, it looks like Hikar will continue to have to do the augmented reality the “wrong way”, overlaying two SurfaceViews on top of each other. (Incidentally the open-source Android AR apps and libraries I’ve seen seem to do this way anyway). Luckily, an API is available to send a camera feed direct to a texture from Android 3.0+. Hmmmm…. time methinks to buy an ice cream sandwich.

Just a quick update on the last post. “Hikar”, the augmented-reality stuff from OpenTrail, is now in Freemap SVN at http://www.free-map.org.uk/svn. It basically works (in the areas for which there is data, i.e. footpaths in England and Wales). There is however an issue if you go to the preferences screen and go back to the main screen – the camera feed disappears. This is probably due to something I’ve mentioned before: the way in which the OpenGL vector data is overlaid on the camera feed is not “the way you’re supposed to do it”. I am looking into this though it’s likely to be tricky: either adding the camera feed to the OpenGL view as a texture (doable but slow due to YUV->RGB conversion, may need to use some native code someone’s written for this) or actually modifying GLSurfaceView (the OpenGL view class) itself to load a camera feed. Or, in view of the fact that camera feed textures are in the 3.0+ API anyway, maybe it’s just time to get a new Android 4.0 phone!

Am finally getting round to doing a bit more work on OpenTrail and the AR stuff. However there’s a bit of a change. The current OpenTrail code base was becoming rather unwieldy, with the AR bolted onto the standard “mobile slippy map” view as a second activity. So I’ve decided to take a leaf out of the UNIX “do one thing, and (hopefully!?) do it well” app philosophy and split off the AR stuff from OpenTrail. Meantime, the aim will be to develop OpenTrail as a general purpose simple and lightweight walkers’ app. So the slimmed-down version currently shows Freemap and out-of-copyright OS maps as well as allowing users to add annotations. The APK is still available here. Another thing is that the AR is very much “researchware” at the moment – it basically works, but comes with health warnings for casual users.

So what of the AR? Following the typical naming conventions of these sorts of things, I’m proposing the title “HikAR” (hikers’ augmented reality) for the app. It’s temporarily unavailable in SVN as I need to tidy up the code and get it in a state where it’s runnable without the need for any OpenTrail code. However I’m aiming to do more on it in the coming three dark months when there’ll be less opportunity to get out and about actually mapping – in particular I need to get the current problem where two SurfaceViews are overlaid (which you’re not supposed to do) fixed. More on that later…

It’s that time of year again, Dark Sunday is imminent. For those that enjoy getting out and about at the weekends, more so mapping when you’re often out for most of the day (or at least I am) and want a sensible amount of time to do it in without stumbling out of bed bleary-eyed at some godforsaken hour on a Saturday morning and skipping breakfast, this is one of the less-liked annual events. So it was good to hear that this government might finally be doing something something sensible. See here.

While I would certainly not say this about most government policies, all I can say is “bring it on”! It’s quite frankly ridiculous that on Sunday, the sun will be rising before 7 but setting before 5. What sort of timezone is that? Clearly the wrong one, that’s for sure. Apologists for British Winter Time say that GMT+1 is “Berlin Time”, but that’s quite frankly rubbish. Berlin is arguably on the wrong timezone (it ought to be in EET) and so are we. The modern working day does not centre round 12 noon, but around 1pm. So our timezone should be such that solar noon is as close to 1pm as possible. On that measure, it would be more sensible to describe Winter Time as Lisbon Time or Reykjavik Time, as it’s more appropriate for those sort of longitudes.

Year-round BST would achieve that. I’m not necessarily a supporter of GMT+2 in the summer (being still light at 10.30pm would be a bit bizarre and disrupt people’s sleep patterns, I suspect) but I do think we should do something about winter time. Granted, by the time we get to December you’re squeezed both ways, morning and evening, and after Christmas solar noon shifts forward by about 30 minutes,. so I can grudgingly see the need for Winter Time in the latter half of the darker months, but the imbalance in November with wasted morning daylight and early evening darkness is ridiculous. Oh, and just getting up and going to bed an hour earlier is a non-starter, unless you want to be anti-social and leave the pub an hour earlier than anyone else. Not to mention train services on Sunday morning not getting going an hour earlier either…

So, bring it on! It would certainly make a difference to winter mapping…