The Freemap blog, revisited

As you may be aware, I’ve been thinking quite a bit about the future of the main Freemap site lately and thinking about ways forward. Basically, it’s always been something of a struggle running Freemap on limited resources, and the site suffers from a combination of not enough time spent on it and limitations in server power. Even with the use of kothic-js and server-side caching the rendering can be rather slow and it’s quite a bit of effort maintaining the map data. The other issue is that, with an increased number of OSM-based walking and biking sites, Freemap has less of a niche to fill than it once did.

Consequently, and as a result of feedback gathered on the site, Freemap in its current form will no longer be available from May 1st. That said I have had one or two generous offers of hosting from members of the OSM community but obviously they themselves, like myself are very busy and it depends on them having the time to setup and configure Freemap on their own machines.

I do feel that at this point I do have to address recent criticism of Freemap on the mailing list by an OSM contributor. Yes, it does have a number of limitations, yes it doesn’t cover all the UK, yes it only supports a subset of OSM data – but what people do have to remember is that it is, and always has been, a hobbyist site. For hobbyists. I am not being paid to produce a full-featured site, it’s something I do in my own time, for fun, and for precisely zero – in fact negative, if hosting costs are considered – financial reward. If time and hardware specs allow I will add features and data but at the end of the day, I cannot make any form of guarantee that it will do or feature this, that or the other. Freemap is not the only thing I do in my own time, and it’s not even the only OSM-related project I do in my own time – never mind anything else! That said, my feeling is that unfortunately I feel I have neither the time nor the resources to make significant improvements to Freemap and therefore maybe it’s best to call it a day.

Onto more positive things though and what will take its place? Basically the main problem, apart from time, is the sheer volume of data, and thus I aim to revert the main maps to an earlier experiment in which I combined OS LandForm PANORAMA and Vector Map District data with footpaths from OSM, to basically produce a kind-of Landranger-lite effect. As now I cannot guarantee complete UK coverage, but happily for some, major cities in England probably will be covered. I’m also developing FixMyPaths, fixmypaths.org with the hope of eventually producing a nationwide path-reporting system for councils. There’s also the augmented reality Android stuff too. If I get the time to do all that…. would like to, but as always, no guarantees

As you may know I’ve recently been working on Freemap’s Android app, OpenTrail. I think it’s finally feature-complete enough to be considered an 0.1 release. So what can OpenTrail do?

* Display maps of the majority of England (roughly south of Hadrian’s Wall anyhow) using OSM data, with rights of way (using the designation tag) highlighted in the same colour scheme as Freemap. Contours are included, generated from Ordnance Survey LandForm PANORAMA data. Wales and Scotland not yet covered, but probably will be: the maps were generated on my laptop so server power is not required. Maps are generated using the Mapsforge library.

* Add annotations to the map in-the-field. These can be anything you like (within reason!) and are added to Freemap’s annotations database. So you can share points of interest, interesting views, path directions in awkward places, and path blockages with other users. Then, when someone else uses OpenTrail later on, details of that annotation will be presented to them in the field. It’s my aim with Freemap and OpenTrail to produce a crowd-sourced source of information for walkers throughout the country. It might also be nice to send problem reports to the relevant council, adding in the functionality already provided by the Hampshire FixMyPaths app. Note that you have to provide your Freemap login details in the app preferences, otherwise the annotations will have to be authorised before going live.

* Download walking routes from Freemap. As you may know, you can create walking routes on Freemap itself; these can be downloaded into OpenTrail (as GPX files with waypoints for each stage of the route) and “played” on your device. So when you get to the start of the next stage of the walk, the instructions will be presented to you live in-the-field.

* Record new walking routes and save them as GPX files. These can be uploaded to Freemap and shared with other users. It is not yet possible to add text descriptions of each stage to the route; this will probably be left to the 0.2 release unless I get enough requests. Theoretically this functionality can of course be used to survey new data for OSM, but I have not yet tested its robustness over several hours or its effect on the battery, so it is not recommended just yet.

* Find nearby points of interest (but that’s pretty standard for an OSM mobile app). The POIs are obtained from the Freemap server and consequently this functionality is only available in areas with Freemap coverage. (Hants, W Sussex, Surrey, Wilts, Somerset, Devon, Oxon, Bucks, Shropshire, Derbys, Cheshire, Lancs, N Yorks, Cumbria). It will work in Wales as well but there’ll be no map!

* Save annotations on the device for later upload, reducing the need for network access in-the-field. Also, annotations and POIs are cached on the device, meaning that after initial network access, the app can be used completely offline if desired.

It’s available at http://www.free-map.org.uk/downloads/android/opentrail.apk and further documentation is
here. Please note that this is BETA software; I have endeavoured to keep bugs to a minimum and it seems stable in current tests but I cannot guarantee that it will not crash. To encourage bug fixing it includes the ACRA bug reporting system: as a consequence, if it crashes it will send a report over the web to my Google Drive and log the details in a spreadsheet. Please do not install if you are not happy with this!

I’m considering the app to be at feature-freeze now so unless they are small, trivial things, no new features will be added until the 0.2 release. Please email me on freemapinfo at gmail dot com if you do have any suggestions, however.

Next thing I want to do is get back to the augmented-reality stuff I was doing a couple of years ago and continue to develop “Hikar”, the AR app for walkers. Development was suspended on this around 18 months back due to being unhappy with the facilities provided by Android 2.1 to combine the camera feed and an OpenGL view (it was a complete faff) but Android 3.0 introduced some much nicer APIs to deal with this (basically, using the camera feed as a texture) and also the compass on the Hero was a bit dodgy. Now I have an Android 4.x device (SIII) with a decent graphics card this will hopefully reduce these problems.

Finally, how were the maps generated? It was a bit of a faff as I wanted to generate maps by 100x100km grid square, rather than some huge map of the whole of England which might be slow to render. Basically, the OSM data had to be reprojected into OS British National Grid projection to allow easy combination with the LandForm PANORAMA data, which was provided using BNG tiles. The LFP data was converted to OSM using the “lfp” tool which I’ve mentioned a couple of times in the past ( source code)Then the OSM data had to be sliced into BNG-based tiles and combined with the LFP data… and then the whole lot had to be reprojected back into WGS84 lat/lon.

Comments are now enabled again BTW… I’ve installed a captcha so hopefully I won’t get anymore ****** spam! (If all the sad losers who spend their time creating spam created something actually useful to society, maybe we’d be advanced well beyond where we are now, but I won’t launch into a rant )

As you may know I have added a few more counties to Freemap recently, namely Devon, North Yorkshire, Oxfordshire, Buckinghamshire and Shropshire. As a result of this I had to import more OS LandForm PANORAMA contour data to cover some of these areas. However, this has caused some problems.

Seeing as some of the contour data on the boundaries of the new areas was cached (contours are cached by XYZ tile, but are imported by OS grid square) there were some odd artefacts in these boundary areas where an old XYZ tile with formerly partial coverage had been cached resulting in missing contours even with the new contour data.

Therefore, to deal with this – and not wanting to find the “problem” tiles on a case-by-case basis, a tedious task – I am regenerating *all* contour data from scratch. This means that next time you view the map, it might be a few seconds for your area to render, but once this is done, it will be much faster as the data will have been cached.

More generally, I am now caching not only contour data but the actual OSM data itself. This appears to be using less disc space than I anticipated (presumably because many tiles at zoomed-in levels are never looked at) and is therefore more feasible, giving a much faster experience than formerly. It also means that I am able to cover more of the country, so if you have any requests (predominantly rural counties only!) let me know.

First of all Happy New Year everyone! Quick update on Freemap: the JSON data sent back to the browser for rendering using kothic-js is now cached server-side, speeding up performance. Note that due to disc space constraints this is something of an experimental feature depending on how much space there is to cache the data, but for the moment it should hopefully lead to a smoother experience. More on OpenTrail soon….

Sorry, been some time since I’ve posted anything here so I thought that seeing it’s the Christmas period and I’ve got more free time than normally, I thought I’d post an update on what I’ve been doing with Freemap.

Firstly, I’ve just updated Freemap’s database for the first time since September which means that the data is now available under the ODBL rather than CC-by-SA. I’ve also expanded the list of counties to include North Yorkshire and Devon, and so include almost all the national parks of England and Wales, albeit at some performance cost. However I have a possible very generous offer of improved hosting, so watch this space (and if I ever get round to it I’d like to add client-side caching to kothic-js).

Most of my recent development however has been on the Android app side. As you may know already via talk-gb, I’ve launched a side project to Freemap, FixMyPaths, which aims to use county councils’ open rights of way data to allow users to report problems on rights of way which will then be emailed to county councils. FixMyPaths comes with a companion Android app available through the site, which allows you to report problems in the field while out walking. Currently only Hampshire is covered but it’s the aim to expand to other counties for which open data is available.

Also on the Android side I’m continuing to develop OpenTrail, a general-purpose app for walkers which allows users to report not just problems but also path directions, interesting features and so on – and allows users to download walking routes from Freemap. This is coming close to a feature-freeze for an “0.1″ version now, so more on that in a separate post when it’s available. Another thing I’d like to do in the new year is more on 3D visualisation for walkers, either a 3D navigation app or resume development of “Hikar” now that devices with hopefully-improved compass hardware are available.

Having a lot of problems with people sending spam comments. All comments are moderated before publishing – so cannot be seen – but even still it’s a right pain in the backside to continually delete all this cr*p. I’m going to investigate ways of dealing with this – i.e install a CAPTCHA or something of that nature – but while I do so, comments are temporarily disabled… sorry about this.

As you may have heard, recently Hampshire County Council opened up their rights-of-way data (see here) and published it under the OS OpenData licence. It’s looking good for the possibility of adding this to OSM though there seems to be one or two ODBL issues which need to be resolved so holding fire for the moment – though I’m somewhat inclined, if things drag on too long, to add it anyway with a clear “source=HCC” tag so it can be removed if it has to be.

But for the meantime I’ve been working with the data and have produced “OpenHants”, a simple prototype site which overlays HCC rights-of-way data on a kothic-js rendered OSM map (basically the Freemap map, minus the designation layer). This can be seen here; because it uses kothic-js (and therefore canvas) it requires an up-to-date browser, so old IE versions prior to 9 are out.

How was this done?

* The HCC shapefile was downloaded and reprojected to “Google Spherical Mercator” (not strictly necessary, but done for consistency) with ogr2ogr (GDAL conversion/transformation tool):

ogr2ogr -t_src EPSG:3857 Hampshire_RoW_BNG.shp Hampshire_RoW_sphmerc.shp

* The reprojected shapefile was converted to SQL suitable for PostGIS with shp2pgsql (postgis tool):

shp2pgsql -s 3857 Hampshire_RoW_sphmerc hampshire > hampshire_sphmerc.sql

* The data was imported into a PostGIS database.
* A PHP script was used to retrieve data by bounding box from the database as GeoJSON (see here); this actually shares code with Freemap as kothic-js also requires GeoJSON from the server.
* Leaflet was used to overlay the GeoJSON feed on the map (see here)

What I’m hoping to do as well is allow users to report problems in-the-field with a simple Android app. Mind you I’m also planning to work some more on a new mapsforge-based version of OpenTrail (Freemap Android app) so as always this will be dependent on having the time.

Sorry it’s been a while… been busy with other things lately. Anyway just a quick one for now. As you probably noticed, 0.6 became the default version a while back, however I’ve been having problems with people adding “silly” markers and what looks like advertising spam. As a result of this you must now be logged in to add, move or delete a marker. Got to go now but back soon with something on Hampshire County Council open footpath data.

As mentioned in my last post I was aiming for March 1st to make the kothic-based Freemap 0.6 the current version. It is indeed in more or less a working state, and is compatible with all the main current browsers (even IE9). However, I have made a reasonable amount of progress on routing in the last week or so and would like to implement the “intelligent walking routes” functionality in which you can draw a walking route on the map and, using Freemap annotations and description tags in the underlying data, auto-generate a walking route.

In order for 0.6 to be a significant advance over the current version, I am therefore delaying moving 0.6 over to www.free-map.org.uk until a first pass at the walking route functionality is completed. This is likely, hopefully, to be during March. However in the meantime, you can access 0.6 at www.free-map.org.uk/0.6; bear in mind it is a work in progress and may occasionally not function correctly if I’m testing something.

BTW I’m likely to be using libosmscout rather than pgRouting; tests on pgRouting have revealed one or two issues, for example you cannot route from point to point, only segment to segment (try the “walk route” option on 0.6 as it is at the moment to see what I mean). libosmscout, however, allows node to node routing and is quite amazingly fast, calculating a route from Fernhurst to Southampton, some 30 miles, instantaneously.

Have been continuing to do a few things with the new, kothic-js based Freemap 0.6, which, like the current version of Freemap (0.5) now features search and annotation facilities. There are still a few issues, rendering can be rather slow if the server is loaded or the client machine is busy, and some odd rendering artefacts occasionally occur (such as the underlying canvas tiles occasionally not being blanked if you change location): but the underlying map data is being kept up to date and a kothic-based system will be more sustainable than the raster solution and so overall I think it’s time to switch.

This means that from March 1st, barring disaster, Freemap 0.6 will become the default version and the 0.5 source code will be archived as a downloadable tarball. Being canvas-based it will require an HTML5-compatible browser, but this does include current versions of Firefox, Chrome, Opera and Internet Explorer (and at this point I would like to comment on what an improvement IE9 is over earlier versions). I would also like to thank Chris Jones at SUCS for helping to keep Freemap going in the past year or two through his very generous offer of hosting the tiles. With improvements to kothic-js doubtless in the pipeline (there is talk of client-side caching which should resolve performance problems) I’m now fairly convinced that this is the way to go.

This will of course not be a final version of 0.6; many features in 0.5 such as walking routes will not initially be present. It is my hope to add in a new, pgRouting based walking routes feature, time permitting as always.

With the move to vector rendering on the web client, I am also looking to make changes to Freemap’s Android app OpenTrail too and switch from osmdroid to the vector-based Mapsforge. At the moment this does not allow custom styles but the upcoming 0.3 release, due at the end of this month apparently, will.