A Note About JOSM

Over the past couple of months I have been slowly working to add support for OSM notes into JOSM. Back in July 2013, Ian Dees and I initially did a rewrite of the OpenStreetBugs plugin to support notes. We actually both started out in a friendly competition to see who could get a working version out first as we traded words on IRC. But soon Ian gave me access to his GitHub repository where we continued to work on the plugin for a while.

This plugin basically worked but I decided that since notes are now a core feature of OSM, they should also be more integrated into JOSM. In particular I wanted the workflow of notes to follow the standard JOSM download-work-upload sequence. The plugin was designed to be always online and issued a new query against the notes API every time you panned or zoomed the map. It had an "offline" mode which would stop this behavior and allow you to work without constant network traffic. However in this mode, changes to notes and newly created notes were not shown on the map which was a bit confusing. And unfortunately it was not a simple fix because the plugin just wasn't designed that way. In offline mode, it held actions in queue which were then executed when you went back online. So there was no note object in queue to draw on the map until after the upload happened and another download was initiated from the API. Also, the most useful way of using it was to zoom out, let it download notes and then go into offline mode to work. This was not at all intuitive.

After receiving a favorable response from the josm-dev mailing list, I began submitting patches in September 2014. Throughout the rest of 2014 I slowly built up behind-the-scenes functionality to download, parse and display notes in JOSM. As of the December release, the notes functionality is actually usable. It is not turned on by default yet because there is still some polishing that needs to be done, especially in the user interface department. But I would love to get some more people testing it. The secret handshake to enable notes in JOSM is as follows:

0. Download some data. Anything, anywhere. This just initializes a setting that you will change
1. Open JOSM preferences
2. Go to the advanced preferences
3. Search for "notes"
4. Change the preference osm.notes.enableDownload to "true"
5. Restart JOSM

In graphical format: 

When you open the download dialog after restarting JOSM, you will notice a new "Notes" checkbox alongside the "OpenStreetMap Data" and "Raw GPS data":

After your first download of notes, you will also need to open the note panel or you won't be able to manipulate notes. Click the note icon in the left hand panel. If you still have the notes plugin installed, there will be two identical buttons. Either figure out which one is which or uninstall the plugin. Here is a view of things:

The area download limit for the notes API is far greater than the limit on downloading OSM data. You can download 1 square degree at a time if you are only downloading notes. The limit for OSM data is 0.25 square degrees (although realistically you usually run into the 50,000 object limit before you run into the square degree limit). The above screenshot was downloaded in one request. It is about half the state of Kansas.

Protip: By default, JOSM will only download 1,000 notes per download request. The OSM API has a hard limit of 10,000 per request. What JOSM asks for is controlled by the osm.notes.downloadLimit setting in the advanced preferences. The other setting worth mentioning is osm.notes.daysClosed. This determines whether or not closed notes are downloaded. If set to -1, it will download all notes ever created within the requested bounding box. If it is set to 0 it will only download currently open notes. If it is set to a positive number, it will download notes which have been closed for that many days. By default, osm.org shows notes that have been closed within the last week. To copy this behavior, this setting should be set to 7. I believe the default right now is 1 day.

There is some discussion around these options as well as some of the other polishing that is still needed in the JOSM bug tracker. And of course that is also where any new issues can be reported.

In particular, one of the open tickets which I have already submitted a patch for, but hasn't been accepted yet is the ability to download notes via the note search API. I use this periodically to download all the notes created by Craigslist users (they all start with the word "bounds" so that's the query I send to the note search API) and close all the useless ones.

One last screenshot. One of the features of the notes code is that it can read note dump files. It is actually a byproduct of being able to save and load the note layer. Anyway, here is a screenshot of the latest note dump from http://planet.osm.org/notes/ loaded into JOSM. All 309,000 of them.

Happy noting!

P.S. Dear iD: Your move!

Workflow For Fixing County Borders

Well, my last post got some requests for workflow and I got another snow day off from work today so why not! Follow along as I fix Wyoming's county borders.

Two things you will need to have open: JOSM with the mirrored_download plugin installed and the Wikipedia page for the list of counties in the state. Do a search on wikipedia for "List of counties in <state>" to find it. Here is the one for Wyoming.

Next you need to download all the county relations and their members. I do this using the Overpass API. JOSM has a "Download from Overpass API" option in the File menu (Update: turns out this is a feature of the mirrored_download plugin, not in JOSM core) but we aren't going to use that one just yet because it always adds a bbox parameter to the query which we don't want in this one. Instead, use the "Open Location" option and paste in the following URL after changing the 56 to the appropriate FIPS code for your state which should be listed on the Wikipedia page.

http://overpass-api.de/api/interpreter?data=relation["nist:state_fips"="56"];>>;out meta;

This tells the Overpass API to return all relations with a nist:state_fips=56 tag plus all of the relation members (>>) and finally to add OSM metadata that is needed to edit and upload (out meta). This includes things like version number, last edit timestamp and last user to touch the object. By default it excludes this metadata to reduce download size.

After that finishes you should have a nice outline of the state and all its counties. Make sure the number of downloaded relations matches the number of counties in the state. If not, it probably means the nist:state_fips tag is missing from some counties. If you can mange to find the relation through another means, download it by ID using the "Download object" option in JOSM. Or... I don't know... Ping me on IRC I guess :)

Next, let's load up all the place nodes in the state. For this I use the "Download from Overpass API" option in JOSM. Start off by drawing a box around the desired state on the map. So far I have only done mostly square states. If your state is an odd shape then you can download the bounding box and then select things outside of your state and use the JOSM "Purge" option in the edit menu to remove them from your dataset. Or just ignore them. Then paste this in this text into the query box:

[timeout:300];node["place"];out meta;

Now the screen will look a little busy so I set up a filter to only show me place=county nodes until I need the other stuff. To do this, create a new filter with the filter string

place=county or type:way or type:relation

This will initially hide the county borders and nodes that you want to see so after the filter is created, make sure all three checkboxes in the filter panel are checked. The far right one will invert the filter so instead of hiding whatever matches the filter string it will hide everything except the things that match the filter string. The second checkbox on the left makes them completely disappear instead of just making them inactive but still visible. The first checkbox can be used to quickly disable the filter when you need to see everything.

At this point my JOSM window looks like this:

Note the "Relations: 23" on the relation toolbox matches the expected number of counties in Wyoming.

At this point I usually remove the is_in tag from all the place=county nodes just to get that out of the way. Do a search (CTRL-F) for "type:node place=county" and it should select all of the county nodes. If you have some for a neighboring state loaded that you don't want to touch you can deselect them by holding CTRL while clicking. Then just delete the is_in tag.

Now for the real meat. I just go down the list of counties in Wikipedia. Bring up the first county's Wikipedia page and copy the page title for use in the wikipedia=* tag and note the name of the county seat.

Now in JOSM hit CTRL-F to bring up the search dialog and search for the county seat. If you are lucky it will find exactly one match. Note that because of the filter, you won't actually see the selected object but it is still selected! This could be considered a bug but in this case it works out. If the search found multiple things then you will have to disable the filter and find the right one.

Now:

• Hold down shift and select the county node in addition to the city node that is already selected
• Select the county relation in the relation list and click the "Edit" button
• Add the two selected nodes to the bottom of the relation member list using the 4th button down on the lower right panel of the window
• Add the role "admin_centre" to the city node
• Add the role "label" to the county node
• Add the alt_name tag to the relation in the top panel
• Add the wikipedia tag. Remember how you copied the page title from Wikipedia? Just type "en:" and then paste the title.

Here is a shot of Albany county (Wyoming, not New York!) when I'm done with it:

And that's pretty much it. Now just repeat the process through the list of counties.

After I'm done I go back and look for any place=hamlet or place=village county seats and bump them up to place=town. I do this by doing CTRL-A in the relation list, then right clicking and doing a "Select members" which selects all the relation members. Then I use the find dialog to find all place=village and make sure to select the "find in selection" radio button so you are only finding things that are already selected. Otherwise you will promote ALL villages in the whole state to towns! As a sanity check, use the selection toolbox to make sure the number of objects selected matches your expectation. It should be fewer than the number of counties in the state.

I will leave you with a quick video of me editing the next county in Wyoming.

County Borders in OpenStreetMap

I have had a fairly steady relationship with county borders in OSM. They were originally imported before I joined the project in 2010. However they were imported as overlapping closed ways which is not ideal. In some places people had started working on de-duplicating the ways and turning them into relations. However this is tedious work and some of the people who attempted it didn't quite understand how relations worked so it was pretty much a big mess.

Throughout 2011 and 2012 I periodically came back and worked on county borders and eventually got to a place where there was a relation for every county in the nation. There are still some minor differences in tagging but I wasn't as concerned with that and tried not to change the work of other mappers as long as the relation existed and had valid geometry. At the same time I also reduced complexity at state borders. There used to be a mess of admin boundary ways. One for the state, one for each of the counties on either side of the state border and possibly also some city boundaries. They usually didn't match up very well either so it looked terrible. I didn't mess with cities but at least I made the state and county borders share a single way to reduce clutter.

At some point I also noticed (thanks to the MapQuest Open rendering which renders county names) that the place=county nodes were all placed at the extreme eastern border of all the counties. This caused the county names to be rendered in unexpected locations. So I ended up fixing those as well. Here is a screenshot of the last changeset where I moved over 1,100 nodes to the middle(ish) of their respective counties:

The Effects

So that was nice. But there was still some weirdness with counties, particularly when geocoding in Nominatim. Especially if there was a city with the same name as a county. For example, I searched for "Kearney, NE" and the first result was the place=county node for Kearney County in Nebraska. What I was looking for and expected to find first was the city of Kearney which is in Buffalo county. Then there is the issue of Nominatim finding both the county node and the boundary relation when you search for a county. The same thing often happens with cities. Really, they are the same thing and should show up as one result. And sometimes it might. Nominatim tries to be smart about this and does remarkably well but sometimes it needs help.

You might say we should just delete the node since we already have the boundary information. But very few renderers use boundary relations to render place names from. They all rely on these nodes. The same goes for cities, states and even countries I believe. There is also the oddness of what to use for county name. The relations all have a "name=Kearney County" tag while the nodes just have "name=Kearney".

After poking around a little and bugging lonvia (maintainer of nominatim.osm.org) about it a couple of times, I learned a few things about Nominatim and was able to make some fixes. It was my intent to come back to this some day and figure it all out and fix up county relations once and for all. I still hadn't gotten around to that but just in the last few days, new user revent was getting annoyed at the geocoding problems in Texas and really dove into Nominatim to figure out the best practices. He wrote up a diary entry on the subject and we have since then talked together some more on IRC and since I had work cancelled today because of snow I thought I would write up this post about my current understanding of things:

• Nominatim is able to "link" multiple elements into a single feature. It tries to do this automatically but doesn't always get it right.
• Nominatim uses Wikipedia to help determine the importance of a place.
• Nominatim makes use of the alt_name tag.
• After editing the map, Nominatim does not always fully reindex an area. It might depend on current workload of the server so things can seem inconsistent. But a manual reindex can be forced by an admin after a large area has undergone a lot of edits.

The linking is important because this allows the county border and the node to be linked into a single feature so that only one result is returned when you search for a particular county. It also improves addressing since Nominatim no longer has to choose between two different things to associate addresses with. This linking can be accomplished by adding the node to the county border relation with a role of "label".

The use of Wikipedia is interesting and (I think) the source of my problems with Kearney, NE that I mentioned above. Nominatim was linking the county node to the Kearney, Nebraska Wikipedia page based on name tag. This helped bump its importance up to where it was returned before the city. Now that I have added the Kearney county node to the Kearney county relation and specified a "wikipedia=en:Kearney County, Nebraska" tag, it links things to the correct Wikipedia page and the search results return the city first, as expected.

The Fix

So to summarize, I believe revent and I have settled on the ideal way to map a county. Using Kearney county as an example:
One relation with the following tags:

type=boundary
boundary=administrative
admin_level=6
name=Kearney County
alt_name=Kearney
wikipedia=en:Kearney County, Nebraska

In addition, the following tags are in widespread use:

border_type=county
nist:state_fips=31
nist:fips_code=31099

I'm not sure the border_type tag is actually used for much but it has a fair amount of existing use and seems to further differentiate borders that are the same admin_level based on local conventions - county vs borough and such. The two nist: tags were part of the import and I have actually used the state_fips as an easy way to download all county relations in a given state to work on. These FIPS codes have been withdrawn by NIST recently but are still commonly used. We may need to look at replacing or augmenting these tags with an ISO standard at some point but for now they work.

In addition to the relation tags the following relation memberships are recommended:

• Obviously the ways that make up the border with a role of "outer" in the correct order so that they form a closed ring
• The node with the following two tags: place=county and name=Kearney with a role of "label"
• The place=city/town node of the county seat with a role of "admin_centre"

The county seat city node as an admin centre (yes, use the British spelling) is kind of a new thing to me. I'm not 100% sold on it but it does seem like useful information to have.

If you have looked at very many county nodes you might notice that the "is_in" tag is conspicuously absent. It turns out, it causes more problems than it solves and should probably be left off or removed. I have always found it a bit silly anyway. OSM is a geodatabase - there is no need to indicate location in a tag.

I have also been promoting any county seats that were place=village to place=town. I know I saw this as a thing on the OSM wiki at one point although I can't find it right now. While they may not meet the population guidelines of a "town" according to the wiki, I think county seats carry extra importance due to their governmental functions. It also means that at least some city names are rendered on maps at low zoom levels in low population areas. Without this promotion you could pan around in the Kansas/Nebraska/Dakotas area for several screens at zoom level 9 or 10 and not see a single city name on the map on osm.org. In some ways this is a rendering issue but I don't feel like this is an egregious case of tagging for the renderer.

Your Turn!

So if you are looking for something to do, jump in and make sure your county is correctly linked and tagged. I have already done Kansas, Nebraska and South Dakota but I probably won't have time to do everything in the near future and I will concentrate on lower population areas first so please feel free to jump in! If there is interest, I may do another post about my workflow to do a state-wide edit. Otherwise you can always find me on the #osm or #osm-us IRC channels if you have questions.

UPDATE! I have since written another post detailing my workflow: http://ksmapper.blogspot.com/2014/02/workflow-for-fixing-county-borders.html

Incidentally, a lot of the same concepts apply to cities as well. If you do a Nominatim search for "Los Angeles, CA" right now it will return two results. One for the node and one for the boundary relation. Making the node a member of the relation should fix that.
(sigh. In doing that search I just saw that the relation for LA is completely broken anyway. Admin boundaries suck.)

In closing, here is a screen shot of the completed county relations in South Dakota:

Too Much of a Good Thing

What makes OpenStreetMap great? Well there may be multiple answers from different people. But one thing is certain: Our geodata is awesome. I mean it is kind of the reason that we exist. So more is better, right? We are, after all, trying to map the entire planet.

But it turns out, there can actually be too much of a good thing. One of my first encounters with such a situation was while doing some fixing up of imported TIGER data somewhere out west. I came across this:

Notice the scale. We are looking at a little over 100 meters of country highway and there are over 300 nodes in this image. This is obviously beyond ridiculous and I can assure you that this particular instance was cleaned up long ago and those nodes met their appropriate end in a special level of /dev/null.

But this is not an isolated incident. I keep coming across such over-noding. So I decided to use my local planet database (which I previously blogged about setting up) to try and find out how much this really happens and where such things come from.

Preparing for Operation Cowboy

If you haven't heard, there's a major mapping event about to happen. Dubbed Operation Cowboy, this will be the second event of its kind, the first being Night of the Living Maps which happened back in February. As described on the wiki, these events are kind of like a global LAN party for maps. For one weekend people who choose to participate will focus their mapping efforts on a common goal. This time the organizers decided to focus on the USA.

We of course have plenty of mapping to do around here. Cleaning up and updating the TIGER road import is kind of a never-ending task. In some of the sparsely populated areas of the country there aren't really any local mappers to speak of so there is ample work for "armchair mappers" to come in and do some map maintenance.

Considering the audience and the event, I decided to try and get a few things tweaked before Operation Cowboy goes down.

TIGER 2012 road name tiles

First up was improving the TIGER 2012 road name tiles being served up by the OSM-US tile server. They were originally set up by Ian Dees and are a great resource for tracing roads that were built since the original TIGER data was imported. The only issue I had with them is that they show the street name in the Census Bureau's abbreviated format. In OSM we prefer to use unabbreviated names because it reduces ambiguity. Does "St" mean "Street" or "Saint"? While that one might not be too hard to figure out and expanding "Ave" to "Avenue" might be trivial, there are a lot of abbreviations in the data and some of them are not easy to guess if you aren't familiar with the area.

Any guesses as to what the following mean? Brg. Cll. Natl Lkshr. Ofc. Pso. RM. BIA Rte? (see answer key at end of post) Most people in the US can probably guess some of them but there are definitely some regional ones in there that many people may not be familiar with. Then add in abbreviated directional prefixes. And now suppose you are a European trying to figure all of this out during Operation Cowboy. It's going to look like a secret code.

I won't go into the technical details in this post (might save that for another one after I clean up the process doumentation) but it did involve 27 GB of data in 3,200 .dbf files, about 10 different command line utilties and several large SQL queries. The end result was a mapping of 10.8 million line IDs (linearid field in TIGER speak) to fully expanded road names which I applied to the rendering database. The result can be seen on this tile preview page.

Imagery preset updates

Related to the road name tiles, I noticed that Potlatch2 had a TIGER road name tiles preset in the "Background" menu but it was still pointing at tiles with 2011 data in them. Thanks to the magic of github, I was able to get that fixed easily.

Then I noticed that the JOSM imagery sources page didn't have this tile set listed at all. That one was a little trickier to fix. It involved coming up with a simplified polygon of the US border to define the available extent of the tile layer as well as putting a base64 encoded image into an XML tag on the JOSM wiki. The image+base64+XML+wiki thing seems a little nasty but the JOSM functionality that this enables is actually pretty neat. If you download OSM data inside of the US, it will pop up a notice that informs you that that there is additional background imagery available and lets you add it to the "Imagery" menu with a single click. The base64 encoded image is displayed next to the menu option. In this case it is the TIGER logo:

RAWR!

TIGER tag removal

The TIGER road import put some tags on imported objects that weren't really necessary. As people edit roads, some of the tags become even less useful as the information from the tiger tags diverges from the "normal" OSM tags. Some of the tags were useful at the time for the upload process itself but are now just annoying bloat that confuses new users.

Back in July I started a mailing list thread about having editors automatically drop some of these tags like they already drop the created_by tag. Out of that discussion came a JOSM trac ticket and patch that does exactly this. Right before uploading, JOSM checks to see if any of the modified objects it is about to upload have "discardable" TIGER tags on them and removes them.

In September, I opened a ticket against Potlatch2 with the same feature request. Since nothing had happened with that ticket and I already had the Potlatch2 repository cloned from the tile URL change pull request, I decided to take matters into my own hands. I had never so much as glanced at ActionScript but fortunately P2 was a breeze to get running in a development environment. The developer documentation page on the wiki got me up and running in a matter of minutes. Although I did have to give ant more PermGen space to get through compilation. I should probably update the wiki with that tidbit...

After a little over an hour I had another pull request fashioned which Richard was kind enough to merge the next day. I believe this went live on osm.org within the last few hours. So now every way that is touched during Operation Cowboy will help to clean this clutter out of the database without the user having to do anything.

YEEEEHAW!

So that's how I prepared for Operation Cowboy. Unfortunately the actual event will take place over the Thanksgiving weekend which means I will be in Nebraska with very limited internet access so I won't be able to participate much with actual mapping. I will probably manage to hang out in the IRC channel a bit to try and field some questions if they come up. Please join me! (#opc2012 on irc.oftc.net)

Answer key for TIGER abbreviations:

Brg = Bridge

Cll = Calle (Spanish)

Natl Lkshr = National Lakeshore

Ofc = Office

Pso = Paseo (Spanish)

RM = Ranch to Market Road

BIA Rte = Bureau of Indian Affairs Route

Licensed to Map (What happened to Los Angeles?!)

I recently got back from the State of the Map - USA conference. It was great and you should have been there! But for those who weren't... I presented one session on Saturday afternoon with the same title as this post. Instead of just putting my slides out there I thought I would write a blog post to tell more of the story. This is a fairly long post but don't worry, it has a lot of pictures! (click to see full resolution)

So, the license change. It happened. We lost some data. But what happened before that to try and save as much as we could? And what exactly did we lose?

First, a brief timeline:

• Many moons ago, the OSM Foundation voted to change our license from Creative Commons to the Open Database License. This actually happened before I knew that OSM existed.
• In order to make the change, permission had to be secured from everyone who had contributed map data to the database.
• Contributions from anyone who did not agree to the new terms had to be removed from the database.
• At the end of 2011 the board set April 1st, 2012 as a target date for the license switch. This turned out to be wildly optimistic but at least it was a concrete goal to shoot for instead of just an ongoing process with no end in sight.
• The process of removing non-relicensable data was done by a bot that went through the entire database in the 2nd half of June
• After a little more cleanup, the first ODbL planet file was finally delivered in September.

Even though the April 1st target date was not really reasonable to hit, (especially in hindsight of course) it still gave the community something to work towards. And work we did.

Contacting inactive mappers

The first priority was to contact undecided users to make them aware of the change and that they needed to log in and indicate a decision. There were several rounds of emails sent out from the foundation to undecided users, trying to get them to respond. In addition to the emails from the foundation, users actively contacted other people in their area to try and make them aware of the decision. Lastly, the foundation supplied the account email address of some of the accounts with the most map data to a small group of volunteers for more targeted contact.

The result of this contact effort can be seen in this graph that I have been presenting since the beginning of the process:

The green line corresponds to the scale on the left while the red line follows the scale on the right. Of those who responded, it was a 99.5% landslide in favor of re-licensing their data. You can see that the people who were opposed to the change were very quick to log in and enter their decision. The big bumps in the green line clearly show when the mass emails were being sent out by the foundation.

Remapping using TIGER 2011

The license change mapping process I talked about in my previous post works well where there is still some clean data to go off of and you're just having to redo some work that a decliner did in splitting ways or adding some nodes to refine the geometry. But what about when an entire area is on the chopping block? I present the OSM Inspector view in Glendale, CA:

The area I worked on was just south of here and was actually even worse although I didn't get a "before" screenshot. Less yellow, more red. Here is what it looked like on Simon Poole's "badmap" rendering. Pretty much the entire road network is going to get nuked:

Most of this data is from a single, very prolific contributor who is not likely to agree to the new terms. It is also highly unlikely that anyone in the area is going to be able to redo all of his work before April 1st. So I reached for a bigger paintbrush: TIGER 2011.

License Change Mapping

So far I have avoided saying anything about the OpenStreetMap license change on this blog. It is a very sensitive subject for a lot of people for a million different reasons. This is not a post about how right/wrong the license change is. I personally have agreed to the new terms but then again I don't really care what license my data gets redistributed under. While I prefer (in most situations) a share-alike clause, if we were switching to public domain instead of ODbL, I would probably still agree. Whatever.

But the license change is happening and we must prepare the map for it, ahead of the April 1 cutover date set by the license working group. Today I decided to take a look at what it would take to get I-70 in Kansas cleaned up, license-wise, and thought I'd share my findings. Note that this particular process may or may not apply to other areas. The interstate system is kind of special in that it was imported from a public domain source and has very long stretches that are identical except for tags that don't really matter. This means that there isn't really very much unique or unrecoverable information present. If there is a small section of road (say a bridge) that is dirty, it can just be removed and replaced from clean data that surrounds it plus imagery to verify that it is indeed a bridge. However whenever possible it is preferred to maintain the history of objects in OSM so I don't want to just nuke and replace large sections of I-70. Plus, deleting and recreating ways gets more complicated because of the route relations that they may be a part of.

A majority of I-70 in Kansas is already clean but there is still a noticeable fraction of dirty objects. The goal is to end up with all objects being "clean" to re-license under ODbL. First task: find dirty objects. For this I use the "License Change" view in the OSM inspector tool created by Geofabrik: http://tools.geofabrik.de/osmi/?view=wtfe

The Most Interesting Mapper

The Most Interesting Mapper in the World

In case you live under a rock, this particular meme has its origins in a set of beer commercials that I find kind of amusing. Here is a decent compilation of them.

<sorry to interrupt this completely awesome post but quick note: This plugin has been turned into a core JOSM feature. See "Update 2" at the bottom of this post>

But seriously, the ImproveWayAccuracy plugin is a wonderful tool. Perhaps the prime use case to show it off is a smoothly curving road that was originally either mapped from low resolution imagery or (around here) imported from TIGER data that may have used far too few nodes to make a smooth curve. To fix this by hand is highly annoying. You have to drag nodes around and then grab the + between two existing nodes to create a new one and drag it to where you want it, repeating that as often as needed to achieve a smooth enough curve. Each node requires several precise mouse movements. 

It is much easier to draw a way from scratch where you just move your mouse along the curve and click as needed to form a smooth looking curve. Well, the ImproveWayAccuracy plugin makes this possible to do with existing nodes/ways!

Simple OpenStreetMap tile rendering with Maperitive

Lately there have been a lot of people popping into the OSM IRC channel (#osm on oftc.net) because they have been getting the "you've been bad" tile from the OSM tile server:

The reason is that they are using tile scraping applications that violate the OSM tile usage policy. Typically people just want to download tiles for their city or a vacation destination or travel route in order to pre-load them on a mobile device for offline use.

This is of course a perfect use case for OSM. The problem is that OSM is a project run by volunteers and with limited server capacity. Our primary product is geographic data, not map tiles. So when these tile scraping programs start hitting high zoom levels, it can cause serious problems for the tile server. The solution is to download OSM data instead of rendered tiles. Then you can render your own tiles or even use offline vector rendering.

One tool to make your own tiles is Maperitive. It is pretty easy to use and by default generates tiles that are fairly similar in style to the default mapnik style you see on osm.org although it is very customizable. It is a .NET application although it claims to work in Linux/OSX with mono. Also, while it is free to download, it is not open source which is kind of rare for the OSM world. These two things may cause some people to have an allergic reaction (I'm looking at you, Serge) but if you're still reading, here is a quick lesson on how to use it to render your own tiles. If you keep reading, you should be able to start rendering your own tiles in about 10 minutes. There is a tutorial video and some detailed documentation on the Maperitive website but I don't see a basic text based tutorial that walks you through the common "render some tiles for this area" use case.

Getting Started
Installation is pretty straightforward. Just head over to http://maperitive.net/ and find the "Download" link. Grab the latest .zip file (Maperitive-1228.zip right now) and download it. Once the download finishes, just unzip the Maperitive folder inside of the zip file somewhere and you're done. Open the Maperitive.exe file to fire it up.

Cleanup on the Map Aisle

Spills happen. A sticky honey jar in the grocery store. A glass of orange juice at breakfast. Oil spills in the Gulf of Mexico. Node spills in OpenStreetMap. Some can be cleaned up with a mop and some water. Others require a more technical solution. What is a "node spill" you ask? It's when nodes get uploaded to OSM that don't have any tags on them and aren't part of a way. These tagless, unconnected nodes add no useful information to the map and are just dead weight in the database. Where do they come from? There are at least a couple of common sources of node spills. One is editor bugs and simple user error. These are generally pretty small spills of 10s or maybe 100s of nodes. The bigger problem is imports.

The topic of imports deserves its own post. For now I'll just say that badly performed imports and insufficient checking after imports can lead to 10s of thousands of empty nodes. The basic problem is that nodes get uploaded first and don't become part of a way until the way is uploaded later. So if something goes wrong with the way upload, the nodes end up in an orphaned state.

I have come across a few of these nodes before but what really caught my attention was a failed NHD (National Hydrography Dataset) import in Oklahoma. It happened to poke up into Kansas a little bit where I noticed it next to a state highway I was editing. After a lengthy thread on the talk-us mailing list I eventually found a good way to detect these nodes. Since then I have made my way across the US and some of Canada deleting these useless nodes from the database. As detailed in this message it goes something like this:

• Use the XAPI to perform a query of the form /node[not(way)][bbox=a,b,c,d]
• Open the result of this query in JOSM and apply a filter to hide all nodes with tags
• Perform some checks to make sure there really is no useful data
• Delete nodes and upload

The Main Attraction

I have mentioned the TIGER import several times. The initial import tagged roads with abbreviations in their name. For example "Ave" instead of "Avenue" or "St" instead of "Street." However the general rule in OSM is to not abbreviate things. So a script was devised to automatically expand abbreviations. This script was run by the user balrog-kun. Until tonight I assumed that it had been run over all of the TIGER data. But then I saw a few abbreviations while working on some highways in Kansas and decided to dig a little deeper. I did some XAPI queries (using the database I talked about in a previous post!)  to see if anything interesting stood out. I did one query for all streets named "Main St" and another for all roads named "Main Street" and here is what I got back.

Green is "Main St" and the rest is "Main Street"

One Mapper, One Year

Today is the one year anniversary of my first OpenStreetMap edit! I thought I would take the opportunity to reflect on how OSM and myself have had an effect on each other. My intention is not to brag about all the mapping I have done but rather to point out some different ways in which contributions can be made and some of the tools I use when mapping.

My very first edit was to add a foot/bike path near my apartment so that it would show up on OpenCycleMap. Unfortunately the imported TIGER roads were so far off that I had to realign them before I could add the path in the correct location. And of course correcting the ones right around the path led me to correct the ones connecting to those and the ones connecting to those and so on... At the time I didn't even know what TIGER was or what all those tiger:* tags were on all the roads.

Some highlights since then:

• GPS trackpoints uploaded: 283,271 in 75 traces (most from my Garmin Edge 305)
• 410 changesets created using Potlatch, JOSM, OsmAnd and Vespucci
• Pictures taken: over 2,000 (4.5 GB saved)  Most were geotagged to a GPS trace to aid in mapping features (uploaded some of them to openstreetview.org)
• Some of the locations I have edited: 
• My university campus
• My city
• My state
• My country
• My high school
• My childhood home
• Some random place where there was an earthquake
• Map objects affected (according to hdyc.neis-one.org - last updated in January)
• Nodes: about 165,000
• Ways: about 24,500
• Relations: about 500 (I created over 200 county relations in 3 states since the last update)
• Contacted multiple government agencies in Kansas to get explicit permission to use their data to assist in mapping. Documented these data sources on the Kansas page in the wiki
• Started a blog (self referring link!)

Now for a pretty picture. Here are all the GPS traces in my city. I think there may be one or two from other people but most of it is me.

Keeping a database up to date with osmosis

This is a follow-up to my previous post about importing a planet file into a pgsnapshot database schema. However the same basic process applies to other schemas that osmosis can write to.

To briefly set the stage: After downloading the weekly planet file and processing/importing it into a database, the data is now probably 3 or 4 days old. OpenStreetMap provides both minutely and hourly change files that can be applied to the database to bring it up to date with what is live right now in the master OSM database. If you think about it, that is actually a pretty sweet deal. Someone on the other side of the planet can spot a new restaurant while on their lunch break, use an application on their phone to upload it to OSM and you will get the change applied to your local database in less than  120 seconds. Does this blow anyone else's mind just a little? Open data is awesome like that.

Parse the Planet! (into a pgsnapshot database)

What makes OpenStreetMap... well... Open? In a link: http://planet.openstreetmap.org/

Yes, of course there are the legal issues of having an open license, the transparent community processes to govern the project and a whole host of open source software to create, manipulate, consume and display map data. But at the end of the day, none of that would matter if the raw data wasn't available.

As is obvious from my previous posts, I have dabbled in manipulating this raw data in the past, with varying degrees of success. After learning from my previous attempts and hanging around on the mailing lists and IRC channel, picking up information here and there I decided to take another crack at processing a planet file. I ended up finding a partner in crime: Paul Norman. He was wanting to import a planet file for use in a private XAPI server using the new java based XAPI implementation written by Ian Dees. This sounded like a good idea so I decided to follow along. We bounced commands and results off of each other for a while. I ended up going a slightly different route in some of the processing because I had some different hardware available to me.

The first thing to know about processing planet data is that there are several different database schemas that can be used to hold the data. This topic probably deserves its own post at some point. Here is a brief list of the ones I am at least minimally familiar with. I know there are some others (like the gazetteer database used by nominatim):

• API database: Creates a duplicate of the master OSM database. Does not use any GIS features. Nodes are stored with lat/lon values as simple numbers. Contains tables for: ways, tags, users, changesets. Can track history of objects once you start applying updates.
• "pgsnapshot" schema: creates a schema that utilizes the GIS features of postgis that allows you to run spatial queries against OSM data. Still maintains all tag information but they are in a single "hstore" column, not in their own relational tables. Changesets are not stored nor is any information about users except the ID and username. History is not maintained as updates are applied. There is also a related "pgsimple" schema that seems to be an older version of the pgsnapshot schema that may have some advantages. Differences are discussed in the "pgsnapshot_and_pgsimple.txt" file in the osmosis script directory.
• osm2pgsql rendering schema: This schema is created by the osm2pgsql tool and used by mapnik to render map tiles. It uses GIS features as well but it is lossy and very de-normalized. Only objects and tags that you care about rendering are imported. This means it is also smaller and faster to get set up.

The XAPI uses the pgsnapshot schema so this is what I set up. The instructions on how to get going are provided in the xapi readme file but I will go ahead and copy it here with some additional notes.

Dual Carriageways

When I first started editing OSM, I was mostly doing residential roads and bike trails. Eventually I worked my way to major roads. Some of them had dual carriageways. In my case these were also US/State highways. Somehow I got the two mixed up in my head and thought that mapping any dual carriageway road required relations. Since I didn't know much about relations at the time I was kind of scared to touch them. Turns out relations ARE used to describe highways however they are not required for dual carriageway roads.

Since I've recently been working on "dualizing" roads in the Kansas City area, I thought I would throw up a quick post about the process. It has probably already been done (I seem to recall a "TIGER fixup" tutorial that described the process) but here is my version.

To Start off with, let's define a dual carriageway road. There may be some difference of opinion (there usually is in OSM) but in general it is any road where the opposing streams of traffic are separated by a physical barrier. A grass median (think interstates) or a strip of trees in the middle of a boulevard are prime examples. Sometimes the barrier is just a small strip of raised pavement or bricks. Regardless, it is something that would make your car go "bump" or even cause damage if you were to try and drive through/over it.

Airports, Part 2

In my first installment of the airport series I presented some basic mapping techniques I used while mapping rural airports in Kansas. One comment I got on IRC suggested using a slightly different tagging scheme for displaced thresholds. The suggestion (I believe it was from pnorman) was to tag it with two tags. One being aeroway=runway and the other being runway=displaced_threshold. While I haven't gone back and changed my previous work, this may be a better way of doing it. First of all, they would get rendered on the default map. I know, I know... "don't tag for the renderer!" But here's the thing: This is technically not incorrect. Displaced thresholds are considered part of the runway and are included when the runway length is specified. So tagging it as a runway but then specifying that it is a specific type of runway is not incorrect tagging. On standard maps it would be rendered as part of the runway but someone making an aeronautical themed map could use the runway=displaced_threshold tag to differentiate it and produce a customized rendering.

Before I go on, I am just going to repost a link to the OSM wiki page for Airports as a reference.

Now on to some new material. I was moving west-to-east through Kansas so the first "big deal" airport I came to was the Salina Municipal Airport. Its 17/35 runway is a massive 12,300 feet (3.8 kilometers) long. Any aircraft currently in service could land here with room to spare. Why does a municipal airport in a city of under 50,000 people in the middle of Kansas have such a massive runway? The military, of course! This facility was built at the beginning of World War II and was used to train heavy bomber crews. Then during the cold war it became part of Strategic Air Command operations until it was closed and turned over to the city of Salina in 1965. (source: wikipedia) Now it serves mostly general aviation but is also used by Kansas State University's aviation program which is based at the Salina campus. A few years ago it was used by Steve Fossett as the start and end point of his solo, nonstop flight around the globe. And I got to bill them for the telecommunications services they used while doing it! (My current day job deals a lot with the telecom billing system at KSU)

I promise, I'll get to the mapping in this post, eventually. But I should point out that this is one thing I love about contributing to OSM. Seeing something unusual in aerial imagery can often lead to an hour spent on wikipedia or a fun discussion on IRC involving people from around the world or, occasionally, a field trip to go see what the heck is happening "on the ground" to cause it to show up a certain way in aerial imagery.

Airports, Part 1

For the past few days I've gone on an airport binge in OpenStreetMap. I ended up learning a lot about airports. It was fun to start mapping, then come across something different or notice a new detail and realizing that this too can be mapped! Here are my musings on the subject. Feel free to tell me I did it all wrong in the comments :)

The recent import of airport nodes from www.ourairports.com and the subsequent discussion sparked the idea to map easily identifiable runways from aerial imagery. So I downloaded all "aeroway=aerodrome" nodes within Kansas from the XAPI. In addition to the above mentioned import, a lot of airports were imported from USGS GNIS (Geographic Names Information System) back in early 2009. In total there were just under 600.

This being Kansas, a vast majority of those 600 "airports" are nothing more than a flat spot in a field where farmer Joe lands his crop duster. They have such names as "Lundgren Hereford Ranch Airport" or "Flying N Ranch Airport." They should probably be downgraded from "aerodrome" to something else and not rendered at zoom level 10 like they are now. Right now looking at western Kansas at z10 reveals a ridiculous number of airports being rendered. I have yet to see any consensus on a tagging scheme for smaller airports so I mostly just ignored them. Additionally, the aerial imagery I was using (USGS NAIP) wasn't really good enough to reliably pinpoint these grass strips in the middle green fields. It also didn't help that the GNIS information is often off by quite a bit so I wasn't sure if the runway was just blending in or if there really wasn't one in the field I was looking at but rather 2 fields over.

The more interesting airports are the municipal air fields. Most of these have a paved runway, sometimes in addition to a grass one. Even small airports stick out when they have a paved runway:

Updated TIGER map

For a full history of this post you may want to read these three posts first: 1, 2 and 3.

When I first presented my nationwide TIGER map at wherecamp5280, Samat mentioned that he was surprised that his county wasn't more heavily edited since he has done a lot of work there. A few days ago we were chatting on IRC (#osm on oftc.net) and he asked a question that made me realize that the map did indeed have a serious flaw in it. Earlier today a discussion came up on the talk-us mailing list that made me notice that the same error exists in the TIGER edited map that MapQuest (re)set up. So I decided to look at correcting it.

The error is that I was only looking at the latest version of the ways. This means that any edits between the original import and the mass edit that expanded street names were not taken into account. Both the import and the abbreviation expansion happened well before I started mapping so I didn't even consider this fact. As Antony Pegg says later in the mailing list thread, it is too expensive to go back and look at all previous versions. Especially since you would have to use a FULL planet file since the regular ones only contain the current version of objects, not a full history.

But we don't really care about the contents of the edits or who performed them... we just care that they happened. The initial import obviously created version 1 of the ways and if nothing else changed, then version 2 was created by balrog-kun in the name expansion edit. If any edits happened between the import and the name edit, then the ways will be on version 3 or higher. Thus any TIGER way with a version higher than 2 must have been edited by someone other than these users, even if one of them was the last to touch a way.

So here is the updated map which takes this into consideration:

Two more TIGER maps

After my last post there were some requests for me to do additional states. So I imported Colorado and New Mexico since they are kind of one big block. However doing 50 states a couple at a time will be rather cumbersome so I think after this I will try to coax osmosis into doing a more fine-tuned import so that I can import only the things I need but do it for the entire US at once.

I also learned some more quirks about ArcMap. The program is amazing in some ways but user interface is not one of them. I was also unable to figure out how to use two different fields to join my data to the counties shapefile. So that was slightly frustrating but I was able to work around it.

Anyway, until I get around to poking osmosis again,
I give you Colorado: