OpenStreetMap.org maps on aprs.fi

Today I added OpenStreetMap maps on aprs.fi. They can be selected using the map type drop-down menu in the top right corner of the map.

OSM support has been on my things-to-do list for a couple years, but since then, it had become much easier to do, and the map coverage of OSM had increased greatly.

The OSM maps have data in many places where there's no Google Maps coverage at all (for example, take a look at Tbilisi, Georgia). Around my home there's more accurate coverage of walking paths in addition to roads.

If you'd like to improve the OSM maps in your area, just go to openstreetmap.org and click on Edit. Please see the wiki for more information on how to get started. If you feel the Google Maps aren't so good in your area, editing OSM maps can become a nice long-lasting hobby project!

Heard & Gated statistics: 6 months of history accessible

The "who heard who" statistics on the info pages of APRS igates and digipeaters now have a pull-down selection for the past 6 months of data. The same selector was also added for the gated and heard maps. The feature had been requested by quite a few users in the past, and it's been on my mind from the moment I added these statistics in the first place. The data was there in the database, I just hadn't gotten around to adding the user interface for jumping between months before now.

Tracking multiple stations in embedded map

A little Sunday morning coding brought the embedded map up to par with the real-time map's callsign search code. Embedded maps allow you to put a real-time aprs.fi map on your own web page.

To track multiple stations, separate them with a comma (','):

he_track = 'CALL1,CALL2';

This now uses the same search routine as the real-time map. If there are tracking targets in different target classes (APRS, AIS, web) having the same name, you might have to add the class prefix:

he_track = 'a/aurora'; // APRS object Aurora, not the AIS ship Aurora

Sharing your position on the map without GPS

The aprs.fi site now supports sharing your current position from the convenience of your web browser by simply pointing at the map, without using a GPS or web browser Geolocation API support.

1. Sign up to the service, if you haven't done so already. Your account's nickname will, by default, be used to mark your position.
2. Log in!
3. Find your current position on the map.
4. Zoom up close for more accurate pointing.
5. Right-click the position and select Upload my position. On a Mac, press ctrl to simulate right mouse button. Left-handed mouse setup will probably use the left button, but if you have one, you know better.
6. If you didn't get the position quite right, simply move the symbol image by dragging and dropping it.
7. When you've moved, right-click and upload the position again.

To change the symbol icon, the web station's name, or the comment text, click on the Favourites (star) tool button in the top right of the screen, and select My stations and bookmarks, and then Settings from the default My web stations item.

I also upgraded the web server software to a new major release. Nice surprise – it didn't require any configuration changes, it just worked. Or so it seems.

Outages due to upgrades during early December

The aprs.fi service might have some small service breaks during the following few days. I'm upgrading the master server with 4*1TB disks (RAID10, 2TB usable capacity) and doing some other upgrades while I'm at it (including but not limited to the new operating system kernel, switching filesystems, LVM, database engine upgrade, etc). The first two disks already went in tonight and replaced half of the old ones, and the process will continue with data migration from the old disks to the new ones.

The first reboot (due to kernel upgrade) will be on Wednesday morning (around 0600 UTC probably). The web service should keep on running happily, but if the reboot will only take a short while, I probably won't bother to move the APRS data collection master function to the second server during the reboot, and some data will be lost during the boot.

Sorry for the trouble!

Clickable track lines, center & zoom links, etc

I've just installed a few small improvements on aprs.fi.

The track lines are now clickable, allowing for easier track line identification and jumping to the current position of that station.

The info balloons of stations (and the track lines) now have "center" and "zoom" links. There's also a new separator line in the info balloon, indicating the track line color used by that station.

The track waypoint hover-on callsign labels now also show the symbol of the station.

The track line drawing code received a nice little bug fix: the station symbol image is only really drawn at the last position within the selected time range. The symbol image will no longer jump and run away when you scroll the map along a track line towards the current position of the station.

Date/time range lookups on the map

This isn't, technically, a big feature, but I expect it to make a lot of people happy. I should have done it a couple years ago.

The map's user interface for looking up historical tracking data used to only allow fetching a single UTC calendar day at a time. I've now implemented arbitrary date / time range searches, where you can select up to 14 days of track data at a time. It also allows for more exact time lookups, such as 2010-10-20 05:58:00 to 2010-10-20 06:38:00. Click the calendar icon to open up the new search form.

I've also switched to a tabbed view for the year ranges, and brought back a bit of December 2008 data. 2008 will go away over time, but the tabbed view fits at least 3 years without taking an awful lot of valuable screen estate.

The user interface still needs some tuning. The time ranges are expressed in UTC regardless of your time zone settings. The UI doesn't say that yet. It also fails to mention the 14 day limit (but it will complain with a nice red error message box if needed).

I might raise or lower the 14 day limit, depending on the amount and magnitude of server load peaks it generates. We'll see. I'd like to make it 30, but it might have some adverse effects.

I've also made small changes to the real-time map's update API, making the communication between the JavaScript running in the web browser and the server a bit quicker and smaller, saving some bandwidth. This change made it necessary to take the web service into maintenance mode for a few minutes, forcing all current viewers of the map to load the new JavaScript components of the page.

How do you like it? Feedback in the comments, please!

KML overlay support

Tonight I implemented KML overlay support in aprs.fi, which have been requested by some users. You can now place any KML file on top of the real-time map, which opens up a great number of possible applications and mash-ups. The feature is used by adding a 'kml' URL parameter to any real-time map view. Please note that the URL must be encoded properly (/ becomes %2F, : becomes %3A).

For example, here's a simple polygon example made by Google, the Pentagon with a Pentagon-shaped overlay: http://aprs.fi/?lat=38.8720&lng=-77.0570&z=15&kml=http%3A%2F%2Fhe.fi%2Fmisc%2Fpentagon.kml (see the KML file: pentagon.kml)

Here's a KMZ file generated by the Export tool of aprs.fi, stored on another server and overlaid back to aprs.fi: http://aprs.fi/?kml=http%3A%2F%2Fhe.fi%2Fmisc%2Ftest.kmz&lat=60.2326&lng=24.8364&z=12

An external KML file can be used to highlight areas of interest (such as SAR search areas), projecting high-altitude balloon track forecasts, and displaying live data generated by other web services with KML output features. The opportunities should generate lots of fun.

Facebook application: Last call for users

Facebook is dropping support for application tabs on user profiles. Starting from Wednesday this week, users will not be able to add the APRS tab using the aprs.fi APRS application to their FB profile, or publish any other application data in their public profiles. For details, please see the Facebook developers blog.

If you think this is stupid and wrong (like I do), please complain to Facebook. If you are a Facebook user and still wish to try out the APRS FB application and haven't done that before, please do so before Wednesday.

Sorry for the late notice, but I didn't expect them to actually remove it. They promised 4 months of lead time, but now they're pulling the plug after only 1.5 months.

If you ever consider developing a Facebook application, please remember that Facebook can pull the plug for you at any time they wish. It's a gamble, and you can certainly loose the time you've invested in developing a Facebook application.

There is plenty of discussion and complaints about the matter on the FB developers forum

Additionally, if you have any smart ideas on what the APRS application could do (instead of publishing your APRS position to your Facebook friends using your profile tab), please let me know.

Youtube: aprs.fi bookmark feature demonstration

Today I did a little exercise in creating an instructional video. In this Youtube video I demonstrate the new bookmarking feature.

IE7 bookmarks fixed, and help texts translation

The bookmark feature now works on IE7. There was a little bug in there, or maybe it actually is in Microsoft's Internet Explorer 7, since it worked with all other browsers I tried. Workaround implemented.

Additionally, I've finally moved the few help texts which can be opened by the '?' links to templates, so that they can be translated to other languages. This has been requested quite a few times. They're quite long, and I should probably write more of those, which will mean more work for the volunteer translators. Again, a big THANK YOU to all of you – translations are a very valuable contribution and they do help a lot of users to understand the site!

Bookmarking APRS views!

I've just upgraded aprs.fi with a generic view bookmark feature. It allows you to save a real-time map view, together with all of it's settings (tracked callsigns, map type, time range, map center, zoom level...) to a named bookmark, and then recall that same view using the bookmarks (star) button. You can even save a bookmark when tracking multiple stations, or when the "show all" mode is activated while tracking.

Bookmarks are stored together with your aprs.fi user account, so you need to be a registered user and logged in to use this feature. The upside of this is that the bookmarks are shared across all the web browsers and computers you use, as long as you are logged in using the same user account.

The feature is especially handy on mobile devices such as the iPhone and the Androids, because it can be a little tricky to navigate to a specific location or enter a bunch of callsigns on the road.

As a little additional fix, most views now have the primary search input field in focus when the page is opened, so that the searched callsign can be entered without first selecting the input field using the mouse.

Outage on Sunday (502 Bad Gateway)

One of the two aprs.fi web servers got stuck on Sunday afternoon (around 1300 UTC 2010-09-26) and started giving out 502 Bad Gateway responses. About half of the web requests for aprs.fi ended up on the stuck server, so sometimes you got the correct response page and sometimes you didn't. For some reason the automatic failover did not work – in exactly this sort of situation it should magically disable the second server and point all requests to the working one.

I happened to be driving back from the countryside when the problem appeared. After getting home in the evening and analyzing the situation I disconnected the server from the network remotely by shutting down the switch port, so that it would not interfere and that the other web server could handle all of the site visitors. At this point, when the server disappeared completely from the network, the other server automatically took over it's IP address and the service started working perfectly.

The next day, 15 hours after the disconnection, I went to visit the hosting site to bring the server back online, and took a couple of iphone video clips of the servers. First, the primary server (at the moment running the whole aprs.fi site and serving all visitors), and then the secondary server which had just been brought up and was receiving the past 15 hours worth of APRS data from the primary server. It was pretty busy at the time, but it took only an hour to copy over all the missing data from the primary.

After the replay was completed I started up the web service on the second server, too, and the service is now again running in a redundant configuration.

Root cause analysis continues. It was definitely a software glitch, probably something to do with me running a system process under gdb to analyze a SIGSEGV it gets every now and then. It might well have caused some services to freeze. The database didn't hang and continued to replicate until the disconnection, but the web service got stuck and I couldn't log in to the box using ssh, or through the serial console.

I'll also have to fix the health check used to trigger the automatic failover procedure, so that it will work next time this sort of problem appears.

The trip to the computer room gave me an excuse to quickly try out Apple's iMovie for composing a little video. Appears to work, but I guess I still prefer Vegas. There isn't much point in this silly video, but maybe it's interesting for some aprs.fi fans. :)

Ham::APRS::DeviceID published

The Ham::APRS::DeviceID module, written by myself and used by aprs.fi, has been uploaded to the CPAN. The perl module makes a fierce attempt to detect the make and model of the APRS device which has transmitted an APRS packet.

The detection is based on the APRS destination callsigns listed at http://aprs.org/aprs11/tocalls.txt and mic-e type codes documented at http://www.aprs.org/aprs12/mic-e-types.txt. If you wish to have your APRS software or tracker device detected by this module (and aprs.fi), please get in touch with Bob Bruninga to have it added in the relevant lists, and after that, please notify me that they've been updated. Thanks.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

IPv6 support in APRS packet parser - FAP version 1.17

Version 1.17 of the Ham::APRS::FAP APRS packet parser has been released and uploaded to the CPAN. It should appear at the mirrors during the next 24 hours or so. The main feature is IPv6 support in APRS packet path parsing.

What?!? IPv6 support in APRS?

aprsd, and sometimes other software too, sends position packets to the APRS-IS using the qAI Q construct. qAI triggers packet path tracing in the APRS-IS. Each APRS-IS server inserts the hexadecimal source IP address of the previous server to the packet. Now that quite a few Tier2 APRS-IS servers have IPv6 connectivity, even some T2 hubs, IPv6 hex addresses can appear, and they are a bit longer than regular callsigns, and were getting rejected by the parser.

SRCCALL>DSTCALL,qAI,IQ3VQ,THIRD,92E5A2B6,T2HUB1,200106F8020204020000000000000002,T2FINLAND

In this case 92E5A2B6 is the IP address of THIRD, and 200106F8020204020000000000000002 (2001:6f8:202:402::2 in normal IPv6 printed representation) is the IP address of T2HUB1 (T2FINLAND happened to be connected to T2HUB1 at the time).

Here's the complete list of changes:

• Remove / or ' ' from beginning of comment after parsing away PHG, altitude and other optional data
• Allow hexadecimal IPv6 addresses in APRS-IS paths after Q construct
• Added tests for uncompressed packet altitude (negative, too)
• Fixed destination callsign based symbol selection for 'BC'
• Updated URLs to aprs.org, etc

aprs.fi is also available using IPv6 at ipv6.aprs.fi.

API URLs changed

The aprs.fi API URLs have been updated. They all now point to http://api.aprs.fi/api/get instead of aprs.fi.

If you have written an application using the API, you need to upgrade it to use the new links. The old URLs pointing to aprs.fi will stop working in December 2010. This change has been made to make it possible to move the API code to another server or web service process set, if necessary.

Thank you, and happy API hacking!

Mobile signup and password recovery fixed

I've fixed the captcha code (recaptcha) to be valid xhtml, so that mobile browsers with proper strict XHTML parsing (such as the iPhone/iPad family) accept it. It's now possible to sign up and do password recovery using those devices.

blog.aprs.fi !

This aprs.fi news blog has, as of today, moved to the address where it should have been born in the first place: http://blog.aprs.fi/ !

Please update your links and bookmarks, if you have any. Thanks.

IE6 no more.

Users of Internet Explorer 6 get to see, since yesterday evening, a slightly annoying popup telling them to upgrade to one of the more modern browsers. The popup is shown on the live map page, and can be hidden by clicking a little 'x' button in the corner.

The concept, graphics and layout come from the IE6 no more site. It also addresses corporate users who cannot upgrade their own browsers. Believe me, I know your pain, I also have a corporate IT supported laptop at work. Luckily they offer a recent Firefox, too. Supporting IE6 is simply too much of a pain in the arse, it generates way too much extra work for me. I'd rather use that time for developing something new.

Besides, Google has dropped IE6 from the list of supported browsers, too, for Google Maps API v3 and some other products. It will stop working sooner or later, so at least the users should be warned.

Sorry!

IE6 and IE7 broken for a day

Sorry, I broke the real-time map yesterday for the users of Internet Explorer 6 and 7. It did work using IE8 and all other browsers. Fixed this morning.

My excuse is that my home vmware installation, which I use for running IE6 and IE7, was somehow broken, so I didn't test those browsers.

Thanks to Phillip ZL2TZE for bringing this to my attention.

iPad location sharing, bugfixes and Android support

I've installed a couple of fixes in the web locations updating code now. It failed to work for about 6 unlucky users due to a little bug which didn't take database replication delays into account. But that's sorted now.

I've also put the 'share location' button on the non-mobile site, so that it can be used on the iPad and other larger devices with GPS. We'll see what effect that has!

The good news is that it seems to work on several Android phones:

• Google Nexus One
• HTC Dream
• HTC Espresso
• Samsung Galaxy S
• Motorola Droid
• LG Ally

It'll probably work on the rest of the new ones with GPS, too.
The new tool buttons are now documented in the manual.

Web location uploads, better iPhone support, and favourite stations

I've just installed a rather large upgrade on the production aprs.fi servers. As usual, some things might break, in which case you'll get to keep both pieces, but it should be worth it.

The system now supports uploading positions directly to aprs.fi using a web browser which supports the W3C Geolocation API. In practice this means that you can update your location using a mobile phone which has a GPS and a modern web browser. No native application or purchase is required. I have only tested this with the iPhone (OS 3.0 and newer), but there is a good chance that Android phones will work, too. To try this out, create an user account on aprs.fi, log in from the phone, and click on the transmitting tower button to upload your position.

You can also click on the crosshair button to center the map on your current GPS coordinates. Naturally, your web browser will ask you whether your position can be revealed to aprs.fi before either of these functions are enabled.

The positions are not transmitted to or via the APRS-IS, so it is OK for users without amateur radio licenses to upload their positions, too. The stations are identified as web stations on aprs.fi, for example: oh7lzb föni (my iPhone). Web stations names are not restricted by AX.25 / APRS limitations, and can contain spaces and international characters.

The names of the web stations live in a separate namespace from APRS and AIS stations. There can be a 'N0CALL' phone as well as a 'N0CALL' APRS station, and a 'N0CALL' AIS ship. When names overlap, the web site will let you pick the right station from a list. Multiple web stations may also share the same name, which should help avoid collisions.

The mobile version of the map has been improved to better support the iPhone. With some luck, these improvements should also help on Android and other new powerful mobile devices, but I haven't tested it on one yet. I really haven't used anything iPhone-specific which would intentionally break on other devices (I have tested the mobile version of the site on Firefox to validate this), but only testing will tell. Feedback is welcome.

I've also added a favourite stations list. The 'star' button to show the bookmark list is visible, but the editing functionality has not been published yet. Stay tuned for more!

aprs.fi in Chinese

BA5AG has done a lot of work to translate aprs.fi in simplified Chinese (简体字). The simplified characters are currently used in Mainland China, Singapore and Malaysia, and this translation makes aprs.fi and APRS in general accessible to a huge new audience (besides adding one more way to confuse myself while using my own software). A big thank you to Weng Kai for his volunteer effort!

Device Identification, and a rate advisor fix

I've just installed support for APRS device identification. The info page now makes an attempt to show the type of tracker or software being used by the station. The identification data collection started just a few minutes ago, so most stations are still not identified, but they're getting identified as packets are transmitted. The identification happens when position packets are received, so you'll need to send some packets to update the identification. For example, here's an VX-8G:

http://aprs.fi/info/a/KE6SLS-7

As a nice side effect the comment fields of mic-e stations are getting cleaned up from the device identification characters.

I've also updated the packet rate advisor to use an arithmetic mean of the packet transmit interval instead of a median. Some stations which sent two packets in a row (with 2 seconds between the 2 packets) every 30 minutes (giving an average interval of 15 minutes) sometimes quite correctly got a 2-second median interval, and got a "very high rate" warning.

The APRS packet parser was upgraded to Ham::APRS::FAP version 1.16 (to support the device identification).

Class-based links, transmit rate advisor, SSL and other features

I'm having a few days off from work, so I've just installed a rather big upgrade to aprs.fi. There will be bugs and complaints from these, but I think I'll have time to look into them during the following days. So, what's new?

Class-based links and support for targets having the same name

aprs.fi has for a long time had serious problems when multiple tracking targets share the same name. There are an awful lot of AIS vessels having the same name, and those always had to be looked up by their MMSI. APRS stations (and objects and item names) could also overlap with the AIS vessels. I've now implemented proper searching functions to look up these stations and select the interesting ones from a list. To make it possible to link to these stations I've had to add a target class identifier in the links.

• a stands for APRS
• i stands for AIS

Let's use Aurora as an example.

• http://aprs.fi/Aurora will search for all stations having the name Aurora, and show a list. Since the real-time map can show multiple targets at the same time, you can check many checkboxes and get a view tracking those stations.
• http://aprs.fi/a/Aurora jumps directly to the APRS station.
• http://aprs.fi/i/306784000 is a link to an AIS ship having the name Aurora.

If the station's name is unique, old links with no class identifier will continue to work as before: http://aprs.fi/OH7RDA

SSL for sensitive pages

The sign-up, password recovery, login and account pages are now protected by SSL/TLS (2048-bit RSA + up to 256-bit AES). This keeps your password safe from eavesdropping when you're using this site over an untrusted network. I believe the password safety to be on a good level now, since they're also stored in the database using an industry standard safe method (PBKDF2).

I used a free SSL certificate provider, and some more marginal or old browsers might complain about the certificate not being trusted. Sorry about that, please disregard the warning or use a browser which trusts StartSSL's certs.

Packet transmit rate advisor

This has been suggested by several users. The info page now calculates a median interval between packets transmitted. If the median falls below 25 seconds, the following advice is shown:

This station is transmitting packets at a high rate, which can cause congestion in the APRS network.

If the median falls below 13 seconds, the following advice is shown:

This station is transmitting packets at a very high rate, which causes serious congestion in the APRS network. This could be considered an abuse of the network resources.

Up to last 50 packets from a station are considered, but the backwards-going lookup stops at the first 1-hour gap between packets, the intention being to look at the last or current trip of a vehicle. The median is only calculated if there are at least 8 packets in the last trip. Because a median is used, this shouldn't trigger too easily from bursts generated by Smart Beaconing.

There are some ideas how to make this more meaningful. I've been thinking of an algorithm to calculate some sort of fuzzy "network loading" value by multiplying the packet rate by the amount of digipeater hops requested in the packets and using the result to trigger the nagging. This would allow a higher rate to be used with no digipeater path, but trigger more quickly when a 3-hop path is used. It would also handle proportional pathing by looking at the path of each packet separately.

I'm sure some people will be annoyed a bit by this and some rock-throwing might occur. Please be gentle and pick some smaller rocks first. The algorithm is new and it'll be revised. The intention is good – to provide advice in sensible tracker configuration.

Amount of persons on a ship shown when available

The most recent SVN versions of gnuais can receive and upload this information.

Some smaller fixes

• Cursor focus is moved to callsign input field when the real-time map is loaded
• Fixed lat/lng in XML export
• Better string translation moderation tool
• Google maps API is no longer loaded in translation tool

O2 UK mobile users - your operator is breaking this site for you.

In a comment to a previous blog post on iPad/iPhone, Steve notified us that aprs.fi often does not work on the iPhone when used on the O2 UK 3G mobile carrier. It works fine on the same iPhone when he switches from 3G to a Wi-Fi connection.

Well, it turns out that O2 in the UK is running a transparent proxy server which compresses and modifies the web pages in many ways. Their intention is to make the pages smaller so that they would use less of their valuable radio bandwidth, but they ending up breaking the web experience really badly for their users. Here's what they do (information from Stuart Roebuck's blog and O2 user forums):

They re-compress images so that they look ugly on a device which has a bigger screen (such as the iPad, or a laptop connected to the iPhone).

Their proxy downloads CSS and JavaScript files and inlines them inside the HTML page before returning the HTML to the customer's phone. Normally the CSS and Javascript would be downloaded to the phone once when the first page from aprs.fi is downloaded, and after that they would be cached in the phone's memory so that they would not be downloaded again with the following pages. Now they're downloaded again and again with every page.

The CSS and JS code is actually much larger than most of the HTML pages downloaded from this site!

The inlining actually increases bandwidth usage (thus increasing your mobile data charges), slows down this web site, and often makes it unusable, because it breaks the HTML encoding and quoting rules for characters like < >. Steve's iPhone usually refuses to render the pages from aprs.fi due to this.

The inlining also removes comments, which might contain things like copyright clauses and software licenses. I suppose it's illegal to remove those.

For more information, please see Stuart's fine blog posts regarding the matter. I suppose the best options right now are (1) complaining to O2 about this, and (2) switching to another carrier, such as Three.

Translating fixed

The aprs.fi translation tool was broken when the Google Maps API v3 integration was installed on July 5th. I've fixed it now to not use some API functions which were used from the maps API, and translation work can continue. There are some new strings which have been added lately for Street View integration, tooltips, data exporting and other new features, so most languages need some updating.

Again, I wish to thank everyone who has submitted translations – they are really very useful for a lot of the users around the world!

Unfortunately I did a small mistake during the installation of this bug fix, and the real-time map would not load for several minutes right after the initial installation tonight. Sorry about that.

Google fixed the Opera issues

I wrote about the Opera issues recently.

It turns out that Google has fixed polylines with opera yesterday, and track lines are drawn just fine now. Heard maps, phg circles, rulers and locator lookups seem to work, too. Opera is still not officially supported, so further issues may appear, but at least a bunch of very visible bugs were fixed.

Problem and Solution, part 2

The Solution to the Problem has finally been installed in production. It took a while since I wanted to benchmark different configurations well before doing the actual installations. Enough space for backups for some time! I'm now able to store database dumps and transaction logs for much longer than a week, and there's also plenty of room for database growth.


The new disks are faster then the old ones, and they're a bit cooler, too:


4 1TB SATA disks set up as a RAID10 array, giving 1862.62 true gigabytes of space.

I've also ordered more of the same disks for the live aprs.fi servers.

Heard and Gated maps with multiple stations

The 'heard' and 'gated' maps got an upgrade yesterday. They can now plot the combined coverage area of multiple stations. For example:

• http://aprs.fi/heard/OH2TI,OH2NJR
• http://aprs.fi/gated/OH2RCH,OH3RBR

Zoom out to see the whole coverage. The user guide has been updated to reflect the changes.

Video: Using aprs.fi to trace packet errors

radionerd1 has uploaded a video on Youtube, instructing how to trace down a faulty digipeater.

Opera and Google Maps API v3

As you have probably noticed, the Street View feature has made it to the main aprs.fi site, and the beta site has been shut down (as it didn't have anything special any more). aprs.fi is now running with Google Maps API version 3 (up from version 2).

Besides an all-HTML Street View (which works on mobile browsers without Flash, like the iPhone), the version 3 API has other improvements, and all of the improvements made by Google in the future will go to version 3, not version 2. Stay tuned for more! With the new version the map loads faster on many platforms (especially mobile browsers). There are still some bugs in v3, and some compatibility problems with specific browsers, but the bigger ones should be fixed already.

One currently visible bug is that polygons are not rendering in the Opera browser. What this means for aprs.fi is that things like the track lines, APRS packet path lines and PHG circles do not show up. Symbol graphics are shown, and I suppose that everything else is working with Opera.

This problem could be fixed by either the guys at Opera (since it works in the other browsers), or the guys at Google (since they have managed to make a workaround for Opera in API v2). Now, it seems like Google isn't going to do it, so the Opera fans should hope that Opera will do the fixes instead. Or maybe they can use some other browser in the mean time. Here's a quote from a Google employee in the bug ticket for the polygons on opera:

Opera is not currently a supported browser for the Maps API. Supporting a browser is not just a policy decision. To support a browser we must add it to our automated test suite, and then ensure it passes all tests now and ongoing. So there is an overhead associated with supporting additional browsers. We have to balance that overhead against the volume of requests we see from that browser and decide if the adoption amongst users merits the cost.

We can not support Opera Mini, because it does not have a sufficient level of JavaScript support. This leaves Opera for Desktop and Opera Mobile. Together these two browsers account for less than 1% of the requests we receive for the Maps API v3. Perhaps the strongest argument for supporting Opera is that it would open up the API to a number of new platforms (eg. Symbian, WinMo). However when we exclude requests from platforms for which we already have a supported browser, that number drops to 0.05%. So right now supporting Opera is simply not a good investment of our engineering time compared to working on features that will benefit 100% of the developer and user base.

Consequently I am going to collapse all bugs relating to Opera into this one, and reclassify this as a Feature Request to support Opera. We will continue to keep an eye on this issue and on the adoption numbers, and if a new strong argument arises, or adoption increases, we will revisit it.

The statistics for aprs.fi show quite a bit more Opera users than 1%, but it's still not a lot. I could either throw out Street View and go back to v2 for some time, or stay with gmaps API v3 and ask the Opera users to use some other browser for now. I think I'll go for the latter option. I sure hope that one of the players involved does the required fixes soon.

Street view & iPad/iPhone beta test

I've been doing a rather big upgrade on the real-time map code. I have it using version 3 of the Google Maps Javascript API, which better supports the Apple iPad, and there are fixes for the iPhone, too.

Another big thing is that I've added Street View support. The Track in Street View link in the info balloon of a moving vehicle will give you a first-person view of how things look from the driver's point. The camera is pointed to the direction where the vehicle is moving, if it's course is available.

If the station is not transmitting a course, the camera will point from the panorama picture's center coordinates (where Google's camera was when the panorama was taken) to the exact coordinates transmitted by the stations. This implies that clicking on a house symbol of someone's home station will often open up a photo of that house (with antennas on top).

These features are now being tested on the beta.aprs.fi site. There are probably some bugs left – please let me know if you find any! The intention is to fix them before the new (substantially different) code is installed on the production aprs.fi servers.

More details can be found in the user guide.

aprs.fi user guide

To follow up on my previous posting, I've started working on an user guide for aprs.fi. It's a Wiki page, so if you feel you're able to write somewhat good English (or even better, you can correct my grammar), you're welcome to contribute directly.

Plug-n-play APRS (not!) and lack of APRS documentation

Gary Hoffman, KB0H writes on the ARRL web site (The Amateur Amateur: APRS, the Beginning):

"I was still interested enough in APRS that I Googled it the next day. There were a great many sites dedicated to the field, so I skimmed a random sampling of them. Wow! Everything I read seemed highly technical, and none of it described what APRS actually was. The only solid piece of information I was able to garner was that if you called it Automatic Position Reporting System, the APRS Police would come knocking at your door. But just what was APRS? I still didn’t know.

Skip forward a few years.

The term APRS caught my notice from time to time, so I periodically went back to the Internet to see if I could figure it out. I didn’t have much luck. I did find a commercial Web site, byonics.com that sold APRS equipment, but all of the products seemed to be geared toward hams who already had a pretty good grasp of the subject."

I have to agree with him – the obvious places to find documentation for beginners (the Wikipedia page and, especially, aprs.org) do not have it. The wiki page goes quickly pretty deep into a technical network and protocol overview right after the introduction. aprs.org is a mess – it spends a lot of time complaining how APRS really isn't what most people are using it for, and doesn't really have any starting points. And there are broken links. The APRS Wiki has something, but it isn't very strong or up-to-date either.

G4ILO has written an article, APRS - Putting ham radio on the map, which I find pretty good – probably not the least because he says aprs.fi is "the best". It would be good to have this on the APRS Wiki.

If you manage to actually buy some hardware to use APRS, the configuration programs and menus are also very hard to use. I heard or read somewhere that some amateur clubs now have monthly meetings on the topic of programming radios (which actually means going through the menus to get settings right). The Byonics and Argent configuration tools show an awful lot of settings to start with. Most of the defaults are OK and you only need to fix your callsign and the audio levels, but it's way too easy to make configurations which break things for yourself (for example, by selecting invalid symbol codes instead of picking the symbol image from a table). Many misconfigurations actually break the network by transmitting way too often, or with abusive long paths.

I strongly think that the configuration programs should be improved to not accept any abusive or broken configurations. Also, they should initially only show the settings you really, really need to change (your callsign, comment text, TX audio level, and "pick a symbol image from this graphic table of images"). Most other settings should go to an "advanced settings" menu of some sort. Think Apple and the iPhone – to get regular users started it needs to be simple and lean. Path setting should, even in the Advanced menu, consist of a drop-down selection of "1 hop", "2 hops", or "3 hops" which would translate to something like WIDE1-1,WIDE2-2 somewhere behind the scene (with maybe a checkbox for "use TRACE instead of WIDE"). Free-form path entry should be in the Very-Super-Advanced configuration file, and even then the tracker firmware should refuse a WIDE3-7,TRACE9-9 path.

If you're looking for a little programming project, I'd guess Scott wouldn't mind if someone would improve the OpenTracker Windows config program a bit. It's Open Source, after all.

I suppose I should look at my software, too. aprs.fi doesn't have a proper beginner's guide, either. Or an user's manual. There are only 4 tooltips, and 14 help links (which are not localised). Awful.

theborg989 has uploaded a nice introductory aprs.fi video on Youtube. It could use a sequel which would describe things like tracking multiple stations and using the tools in the top right corner of the display.

Tooltips, center map here, quick refresh, and some bugfixes

Here are a few improvements from the past couple days which were installed this evening:

The top right corner buttons of the real-time map now have hover-on tooltips. The right-click context menu of the real-time map now has a 'center map here' option.

The station info balloon will only pop up automatically in the real-time (and embedded) maps, if the map is at least 300 pixels wide and 260 pixels high. This should keep the tracked station in view even on a small view.

The directional PHG cardioids on the real-time map were fixed, they were broken for a while due to a little bug. Same goes for clicking on the Google Earth KML tracking link when tracking a single station.

The static map positioning was fixed on the info pages with Internet Explorer.

The real-time map refreshes quickly after panning the map, instead of waiting for the regular update timer.

AIS ship callsign display is now more correct in the info balloon (no separate tracking link for the callsign).

APRS track export tool

Please try out the new data APRS export tool. There are no links to it from the web site yet. User account and login required. Feedback welcome, as usual!

It can export to KMZ (zip'ed KML + embedded symbol graphics in the same file, for Google Earth), CSV (opens up nicely in Excel, but seems to require some tricks in OpenOffice), JSON and XML (for the programmers out there).

This feature is aimed at people who wish to download their balloon's track afterwards, or their path to Dayton and back.

It's not aimed at people who wish to download the whole aprs.fi database, so it's rate limited quite heavily. Also, it's not an API. I'll probably set the limit to 3 downloads per day per user, although for the beta test, it's currently set to 10 downloads per day per user.

Map on info page, with symbols, and other small things

The info pages now have a little static map showing the station's position. I hope you like it this way. The static maps have also been upgraded to show proper APRS symbol graphics! For example: http://aprs.fi/info/SP3WBX-9

Some of the pages now have an "add this" button which allows you to easily share the page on your favourite social networks.

The raw packet display was fixed to be really, really binary clean. It used to choke on some high binary values.

Weather packets with an unrealistic high or low temperature are now ignored.

Ham::APRS::FAP 1.16 released

Version 1.16 of the APRS packet parser was uploaded to CPAN last night, and according to the first few CPAN tester reports it seems to be working as expected. Here's a list of changes:

• Added a couple of test cases
• Added an example script to parse a whole file of timestamped APRS-IS packets
• Fixed peet WX packet parsing for Perl 5.8 and older, which don't support n! packing
• Minor speedups by reducing amount of regular expressions, extra variables, reordering things to catch the most common case first, and tightening a loop
• Put '' around hash key strings (style issue)

The main change was the Perl 5.8 wx parsing fix. I couldn't have noticed it without the CPAN testers and the unit tests included with the module, since I don't run on 5.8 myself any more.

KP4AO Arecibo Observatory EME reception using a 10-element yagi

This is old news, but anyway... I've just forgotten to upload the videos from my iphone.

On Sunday, 24th of April, I received CW and JT65B transmissions from the Arecibo Observatory in Puerto Rico. Reflected via the moon (EME, Earth-Moon-Earth). Using a very small directional antenna - a 10-element yagi, a scanner-type receiver (Icom IC-R2500) and a small but good preamplifier at the antenna.

To get a good view of the moon we drove to the side of a nearby field with OH2GEK. He even tried to get a two-way contact with the Arecibo station using his TS-2000 and a longer yagi.

Here's a video of the KP4AO CW reception:



And here's one of the JT65B digital signals:

Problem and Solution

This, my friends, is a problem.


And here, my friends, arrives the solution:


Well packaged.


4 units of 1 terabytes each. After RAID10, that's a nice 2 TB of storage for aprs.fi.


This is just for the backups. I'll need to get 4 more of these within a couple of months to upgrade the live servers.

Thanks go to an anonymous Finnish site user for the donation of 2 disks (and organizing delivery in 26 hours), and all other donors for the other two!

Ham::APRS::FAP 1.15 released

Version 1.15 of Ham::APRS::FAP, the APRS packet parser module used by aprs.fi, was uploaded to CPAN yesterday evening. It should appear on mirrors today. This update contains all the changes implemented since November 2009.

Like most of you know by now, the parser is released in Open Source terms (GPL or Artistic license, like most of Perl) so that others can freely use it to implement APRS systems. I believe it is the most complete APRS decoder available as open source.

Here's the change log:

1.14 Tue May 4 20:31:33 EEST 2010 - Hessu, OH7LZB

• Fixed humidity parsing, h0 means 100%
• Fixed peet bros $ULTW packet parsing, integers are signed (not unsigned), negative fahrenheit temperatures are now correctly parsed (and converted to Celsius)
• Parse comment from end of weather reports - if it's short enough treat it as the wx station/software type string
• Parse weather data sent after a compressed position
• Parse snowfall from normal wx packet
• Added test for positionless wx packets
• Added position packet format in hash ('format' => 'compressed')

1.15 Tue May 4 20:44:31 EEST 2010 - Hessu, OH7LZB

• Added a couple of missing tests and 'format' => 'nmea'

POCSAG encoder and 200W transmitter

Lately I've been writing some more code for the amateur paging network project. On Sunday morning I wrote a new POCSAG encoder module in Perl, and on Sunday evening and Monday evening I wrote a simple "modem" for an Arduino card to drive the encoded messages at 512 (or 1200, or 2400) bit/s to an FSK transmitter.

I intend to release most of the bits and pieces as open source, starting with the POCSAG encoder and modem bits. Here's a little video showing the working setup:

As you can see, I haven't grown up enough to not put too many blinking LEDs in my designs. Unfortunately my electronics design skills are mostly limited to calculating the series resistor needed for the LED, so I have to focus my efforts in driving the LEDs from the software in a fancy way. The blue LED does serve a purpose - it shows that the FSK keying timer interrupt routine is working, and indicates it's state. Here's a photo of the Arduino card:


The Perl POCSAG encoder module (POCSAG::Encode) takes one or more messages and encodes them in a binary string. Unlike the other encoders we've found, this one can actually create a long burst with a number of messages, saving a lot of channel time. Only a single preamble is needed, and a lot of idle codewords can be saved between the messages, too.

The burst string is transmitted over USB serial to the Arduino board (the board has an USB-serial chip between the USB port and the Atmel chip) using a protocol which, in theory, resembles the KISS (Keep It Simple Stupid) protocol used with AX.25 packet radio. There are some differences so I've named it PISS, for POCSAG.

The "modem" software on the Arduino waits until the channel is clear, keys the transmitter, and simply sends the bits out through the transmitter's TX data (FSK in) pin. It currently has the usual txdelay and txtail settings hard-coded in the source code. It sends a report back to the host software when the burst string has been received by the modem, and when it has been transmitted, so that the host knows to send the next burst. The Arduino board costs some 20-30€ - if that's too much per transmitter, it's straightforward to just put the Atmel chip on a board alone with an RS-232 or USB transceiver. There are some plans to do just that and design a custom board for the purpose.

The traditional method would have been to drive the FSK out on one of the RS-232 or parallel port handshaking pins. During these days of multitasking and multi-user operating systems it requires a kernel driver, and I didn't quite feel like doing that right now.

The transmitter is a surplus 200W paging transmitter with real FSK keying, configured to run on 144.975 MHz. It happens to have TTL I/O pins, so it can be directly attached to the Arduino.

Erik, OH2LAK, the local repeater builder and coordinator, is also writing about the paging project in his blog. Check out the D-star repeater from a surplus VHF base station radio!

UPDATE 2010-01-31: I finally finished packaging and releasing the software last weekend, and documented it a bit.

Easy map links with labels, dynamic AIS site list, linking enhancements and API terms updated

You can now easily create a link to a position with a label. Right-click a spot on the map, select Add marker to create a temporary marker, drag the marker to the correct spot if it's not there yet, click the marker to open the info balloon, and click on Link to this position. At this point you can edit the label - the shown link will be updated to contain the label (with the correct URL encoding for special and international characters).

Multiple temporary markers are also supported, if you want to mark spots in the map. They only live in the memory of your browser, and are lost as soon as the map view is reloaded or the browser is closed.

The real-time map loading is now much faster by several seconds, or even tens of seconds in some cases. The speed-up was accomplished by not really initialising the date browsing menu before it's actually clicked and opened. The downside is that it now always shows the current and previous year, even if there is no data available for the previous year.

The AIS sites list is now generated automatically. There is a bug in there, causing some stations to not show up in the list. Still investigating that one.

When linking to a callsign, you can now add a parameter others=1 to show other stations, too.

When linking to coordinates, you can now specify a label, which will cause a marker to be shown at the coordinates, and a pop-up balloon showing the label to open.

The API terms have been updated, and a new FAQ item in the end describes the conditions for downloading data from aprs.fi.

Receiver range estimations, colour gradients and AIS receiver list updates

Here's a little summary of changes done during the past weekend and week.

I've changed the colour scale of activity maps to a logarithmic one. This makes the receiving and gating activity maps much more readable:

http://aprs.fi/heard/OH2RCH, http://aprs.fi/gated/OH2RCH, and http://aprs.fi/heard/hkiais - be sure to zoom well out to get the full picture!

The estimated receiving range is now shown for igates and digipeaters. For example, the info page for oh2rch says:

Normal receiver range estimate: 80 km (Updated: 2010-04-12 19:36:27 UTC)

This means that OH2RCH doesn't usually hear packets transmitted much farther than 80 km away. It does hear something from 80-90 km away, and maybe very rarely from much farther away. These statistics have 10 km resolution for up to 200 km, 100 km resolution until 2000 km and 500 km resolution from there on. A monthly receiver performance graph can be seen on the graphs page, with a dotted vertical line marking the estimated normal receiving range. The longest distances are usually caused by bad GPS fixes or corrupted packets. The range is only calculated when the receiver has directly heard at least 3000 packets during the month, so that the estimate is somewhat accurate. This is a new algorithm which will require some fine tuning after some experience is gained.

Last heard a station directly: 2010-04-12 19:36:27 UTC (31s ago)

This indicates that the receiver of the igate is currently working - it has added it's callsign to the path of some APRS packets recently. Please note that because of APRS-IS duplicate filtering only some packets received by the given igate will update this timer.

Owners of AIS receivers can now update their receiver location strings and also configure a home page URL in their user account's AIS settings. These will be used in a new automatically updating page which will replace the old AIS sites list.

On the testing front, I've add system-level tests for the API and the KML interfaces, so that their basic operation is tested every time before building the software installation package.

aprs.fi API, optimisations and rate limiting

The programmers out there might want to try out the aprs.fi API. The code has been there for a while, but I've now polished and documented it a bit. As usual, feedback is welcome.

This evening I've added some optimisations which should make the site load faster, especially over slow connections. It will still be slow on slow mobile devices, I'm afraid.

I've also added rate limiting to protect the service against automated queries by unknown spiders and harvesters. Please let me know if it hits you during normal use. Right after it's installation on Tuesday afternoon it hit all users for a few minutes due to a little installation-time quirk, but that was, sort of, expected. :)

Improved igate and digipeater coverage statistics

This morning I installed an upgrade which I have been working on for quite a while. I've completely reimplemented the way "heard by" and "igated by" data is stored, allowing for faster lookups, smaller on-disk storage, and per-month views.

As a result, the info page, the heard map, the gated map and station statistics graphs will be able to show coverage statistics for complete months. This will make the views much more useful in areas with little traffic.

At first it'll only show the current month (making the view not very useful during the beginning of the month), but I'm going to make it possible to select previous months, too.

As a little bonus, space characters (0x20) now show up as non-breaking space in the "normal" raw packets display, which won't be condensed to a single space in the HTML.

Slowdown on Friday 26th

Yesterday the aprs.fi APRS feed was slow for about an hour, between 15:00:55 and 16:11:03 UTC. One of the WXQA servers the service connects to was down, and the connection attempts timed out. The timeout was too long, and the connection retry timer was too short, and the connect() attempt is a blocking call, resulting in slow processing of packets. I knew about the potential problem, but hadn't bothered to fix it until now.

In the evening I implemented a 2-second connect timeout and an exponential backoff for the retry timer. First reconnect attempts will happen within seconds, but they will slow down to about 2 minutes between retries. Using a non-blocking connect() would have been the correct fix, but this was a bit quicker. The problem should not appear again in this form.

It seems like no APRS data was lost or missed, it was just collected in a buffer, and processed once the connect attempts started working again. The following graph gives some idea of the relative processing rate changes. At peak about 10 megabytes of data was in the buffer.

VRRP failover - HA for the web service

This evening I've set up keepalived on the two aprs.fi front-end web servers. The program automatically manages the IP addresses of the web service. If one server goes down (due to a hardware failure, operating system or web server software hang, or for a maintenance reboot), the other server will now automatically bring up the service IP address of the first one. The fail-over happens within seconds.

The very same VRRP method is used by the routers serving the hosting network to keep the .1 default gateway address available.

There haven't been any hardware problems so far which would have made this necessary, but now I don't have to go through so much reconfiguration every time I want to reboot a server for a kernel upgrade, or take one down to add some memory. I can also shut down the web server processes on one of the servers (for a more complicated reconfiguration) and keepalived will quickly point all users to the other servers.

Project: POCSAG paging network infrastructure

Like OH2LAK reported on his blog, I've started implementing infrastructure code for an amateur radio paging network. Erik (OH2LAK), Weltto (OH7JEV) and Lasse (OH3HZB) got the POCSAG modem running, interfaced it to a Motorola radio, and configured a couple of pagers to receive it on 144.975 MHz.

I got the bits and pieces yesterday for development and debugging, started working on the server software at 6.30 PM yesterday evening, and now I have a working client-server paging network infrastructure with strong authentication, queuing, and support for multiple transmitters. I grabbed quite a bunch of code from the aprs.fi code base, of course.



The system consists of:

• A server component which receives messages from authenticated message source clients, queues them, and provides them to the transmitter clients
• A client library (Perl module) which is used to communicate with the API of the server
• A transmitter client, which fetches messages from the server using the client library and feeds them to the POCSAG modem (and, optionally, transmits CWID at regular intervals)
• And, for demonstration purposes, a DX cluster client which maintains a connection to a DX cluster server, filters DX spots, and sends them to the paging server using the client library. I think I've made a full circle now, over the past 15 years. Feels like home.

The next thing I'll write is an APRS-IS client to collect APRS messages from the APRS-IS and gateway the to the paging server (for those recipients who have a pager configured on the server).

The server is designed so that it's possible to run multiple instances of it at diverse network locations, so that losing one server will not break the whole network.

The network will probably be national, within Finland, coordinated by RATS (Radio amateur technology society of Finland). The system will be demonstrated at the RATS technology day in Otaniemi, Espoo, 20th of March, 2010 (that's next Saturday). I suppose the code and documentation will be published so that you can set up your own networks.

Decoded and hex raw packets, default coordinate format, and eesti keel

The raw packets view can now show the packets in hexadecimal format (revealing all unprintable packets and exact amount of whitespace) and decoded format. The decoding is done using the free Ham::APRS::FAP parser module, as usual.

Developers: Please note that the raw packets display is an user interface, not an API. Don't use it for downloading packets to your applications. Please get them directly from the APRS-IS and decode them yourself. The same goes for other views on the service, too.

The default coordinate format is now degrees and decimal minutes (60°10.92' N 24°31.86' E), which is the default format used on most APRS applications. aprs.fi used to default to degrees, minutes and seconds.

ES1TFJ and ES3AT have started a translation to Estonian, and the language has now been enabled on aprs.fi. Thank you!

I've also added a bunch of basic system-level tests which are run always before the software installer is built ('make bdist'). They feed data to the system using a simulated APRS-IS server and a simulated AIS receiver, and see that the uploaded data appears in different views on the site. I can also run just the system-level tests ('make testsys') which takes about 8 seconds for 93 test cases, or the whole test set ('make test': unit + system tests) which takes about 12 seconds. The validators of the tests are not very strong, but at least they check that the data is updated and that the views don't crash.

ipv6.aprs.fi, and receiver detection tuning (again)

After working on some Other Stuff for about a month, I did a little upgrade for aprs.fi.

I fixed receiver detection for packets with a path like DIGI1*,WIDE1*,DIGI2*,WIDE2* (and added an unit test to keep it working from now on). This bug was reported on the discussion group by a few users.

The status message of a station is no longer shown on the real-time map view if it's more than 24 hours older than the last position report from the station.

I did a couple of very small fixes (disabled IPv4-only GeoIP and increased database column length), recompiled web server software with IPv6 support, and added some configuration, and made aprs.fi available over IPv6 at http://ipv6.aprs.fi/! The whole effort took under an hour. Like the guys at Google say, it's surprisingly easy to set up. After all, it's pretty old technology – I first played with it about 10 years ago.

After some time I'll probably add AAAA records for http://aprs.fi/ so that everyone with IPv6 connectivity will automatically use IPv6 instead of IPv4. Most big sites still use separate hostnames like ipv6.google.com because there are some buggy operating systems and bad ipv6 network configurations which make the end-user hosts believe they have global IPv6 connectivity while it is, in fact, broken.

Windows Vista and Windows 7 have IPv6 connectivity built in and enabled by default, so the amount of IPv6 clients is going to grow quickly in the near future. Your network provider doesn't need to support IPv6, since 6to4 and Teredo tunneling are also enabled in those operating systems by default.

The Other Stuff included setting up native IPv6 routing for the network where aprs.fi servers are hosted. We got two /48 IPv6 PI prefixes, one for each AS. I set up BGP routing on 5 routers (Cisco, Juniper) over those two ASes and made native (not tunneled) IPv6 available to the users of the network. The ISP I'm using (Nebula) also offers native IPv6 to residential ADSL customers.

There's not much value in this to most of you right now, but at least it's cool and future-proof. Makes the service available for those of you who will end up with IPv6-only connections when we run out of IPv4 addresses in a few years.

Receiver performance metrics tuned for APRS

I've improved the "first receiver" detection algorithm a bit. It's now a bit tougher and shouldn't get as many false positives. For example, the packet:

SRCCALL>DST,DIGI1*,WIDE2*,qAR,IGATE:data

is no longer considered to be heard first by DIGI1. Often in these cases the packet was transmitted as SRCCALL>DST,WIDE2-2, and then digipeated by a non new-N digi which changes that to WIDE2-1, and then by DIGI1 which changes that to DIGI1*,WIDE2*. And DIGI1 gets a too big range estimate for that.

On the other hand, the following packet:

SRCCALL>DST,DIGI*,WIDE2-1,qAR,IGATE:data

will give the receiving point to DIGI1. That packet is probably an original WIDE2-2 packet with a single used hop, or a WIDE1-1,WIDE2-1 packet which has been digipeated by a callsign-substitution fill-in DIGI1.

Because this changes the receiver performance metrics heavily (in the good direction), I've deleted the old "heard" databases and receiver histogram data of APRS igates and digipeaters. Sorry for that. They'll grow back soon.

The time to test has arrived

Okay, I'm now officially fed up. Fed up with my own bugs caused by the complexity of the aprs.fi software. Every now and then I change something in one corner, maybe to fix a bug or to add a little feature, and that breaks something small in another corner of the project. Because I fail to notice the bug, it might be broken for days until someone actually tells me. Sometimes it's the embedded maps, sometimes it's the Facebook integration, sometimes it's AIS feeding using one of the three methods. And usually they're broken because I changed something that was quite far from the broken part.

It's time to do some automatic testing. It's no longer feasible to manually verify that things work after making a change and before installing the software on the production servers - there are too many things to test. It takes too long, and something is easily forgotten.

Writing automatic tests in hobby projects like this one is usually not done, because it generally feels like the time spent on writing testing code is wasted - hey, I could be implementing useful features during that time. But on the other hand, once some testing infrastructure is in place, it's much quicker and safer to implement changes since it takes only one or two commands to run the test suite and to see that the change didn't break anything.

A little terminology:

Unit tests execute small parts of the code base (usually a single function/method, or a single module/unit/class). They feed stuff to that little piece of code and see that the expected results come out. They're often run before actually compiling and building the whole application. As an example, I can write a test to run the APRS packet digipeater path parser with different example paths, and check that the correct stations are identified as the igates and the digipeaters.

System tests run the complete application, feeding data from the input side (for example, APRS or AIS packets using a simulated APRS-IS or JSON AIS connection) and checking that the right stuff comes out at the output side (icons pop up on the map, updated messages show up on the generated web pages).

The open-source Ham::APRS::FAP packet parser, which is used by aprs.fi, already has a fairly complete set of unit tests. After changing something, we can just run the command "make test" and within seconds we know if the change broke any existing functionality. If you follow the previous link to CPAN, and click View Reports (on the CPAN Testers row) you'll get a nice automatically generated report from the CPAN testers network. The volunteer testers run different versions of Perl on different operating systems and hardware platforms, automatically download all new modules which are submitted to the CPAN, run the unit tests included with the modules, and send the results to the cpantesters.org web site. Thanks to them, I can happily claim that the parser works fine on 8 different operating systems (including Windows), a number of different processor architectures (including less common ones like DEC Alpha and MIPS R14000 in addition to the usual 32-bit and 64-bit Intels), and with all current versions of Perl, even though I only run it on Linux and Solaris myself.

Last Friday SP3LYR reported on the aprs.fi discussion group that negative Fahrenheit temperatures reported by an Ultimeter weather station were displayed incorrectly by aprs.fi: -1F came up as 1831.8F and 999.9C. I copied a problematic packet from the aprs.fi raw packets display and pasted it to the testing code file in the FAP sources (t/31decode-wx-ultw.t), and added a few check lines which verify the results. Sure enough, the parsed temperature was incorrect, and "make test" failed after adding a test with a low enough temperature. There were a couple of test packets in there before, but none of them had a temperature below 0 Fahrenheit.

Only after adding a test case for this bug I started figuring out where the actual bug was. After fixing the bug the "make test" command passed and didn't complain about the wrong parsing result any more. I committed the changes to the SVN revision control system, and then installed the fixed FAP.pm module on aprs.fi. Because none of the other tests broke after the fix I can be sure that I didn't break anything else with the fix. And because there's now a test in the unit test suite for this potential bug, I'm sure that same bug will not accidentally reappear later.

This is called test-driven development, and it can be applied to normal feature development just as well. First write a piece of code which verifies if the new feature works, and then write the code which actually implements the functionality. When the test passes, you're done. You need to write a bit more code, but it's much more certain that the piece of code works, and won't break later on during the development cycle.

None of this is news to a professional programmer. But from now on I'll try to apply this approach to this hobby project too, at least to some degree. Yesterday I added a few unit tests to the code to get started:

$ make testperl
--- perl tests ---
PERL_DL_NONLAZY=1 /usr/bin/perl \
"-MExtUtils::Command::MM" "-e" \
"test_harness(0, 'libperl', 'libperl')" \
tests/pl/*.t
tests/pl/00load-module.......ok
tests/pl/11encoding..........ok
tests/pl/20aprs-path-tids....ok
All tests successful.
Files=3, Tests=83, 1 wallclock secs ( 0.26 cusr + 0.06 csys = 0.32 CPU)

It ran tests from 3 test files, and the files contained 83 different checks in total. The first file makes sure all Perl modules compile and load. The second file tests the magic character set converter using input strings in different languages and character sets, checking that the correct UTF-8 comes out. The third one runs 24 example APRS packets through the digipeater path inspector. By comparison, the Ham::APRS::FAP module's test suite has 18 files and 1760 tests, and it's just one component being used by aprs.fi.

In the near future I'll try to implement a few system tests which automatically reinstall the whole aprs.fi software in a testing sandbox, feed some APRS and AIS data in from the different interfaces, and see that they pop up on the presented web pages after a few seconds. I want to know that the live map API works, the embedded maps and info pages load, and that the Facebook integration runs. With a single 'make test' command, in 30 seconds, before installing the new version on the servers.

But now, some laundry and cleaning up the apartment... first things first.

Temporary markers and AIS receiver position uploading

Here's the list of changes in today's upgrade. A few of the smaller bug fixes were already deployed on Friday and Saturday.

You can now drop temporary markers on the map by right-clicking the map and selecting "Add marker". The markers can be moved around, just drag-and-drop.

It's now possible to conveniently upload the position of an AIS receiver. The instructions are on the AIS feeding page, step-by-step instructions, step 8.

At this point you probably figured that the context menu (which opens up on right-click) and the marker feature add a good bit of infrastructure, and can be later used for a bunch of other nice features like object/item uploading and home position marking. Stay tuned.

The Facebook app's canvas page failed to load since a couple weeks. Fixed!

Humidity parsing of h0 (100%) was fixed for normal APRS weather packets. Negative Fahrenheit temperature parsing was fixed in peet bros ULTW packets. These fixes went to the Ham::APRS::FAP svn trunk, and will be included in the next FAP release.

AIS receiving statistics and some small enhancements

The receiving performance statistics are now generated for (directly connected) AIS receivers too, but only if the position of the receiver is known. There is currently no other method of uploading the position to aprs.fi besides sending an APRS packet (object, item, or normal position packet) on behalf of the AIS receiver. One packet is enough, there's no need to send it all the time. I'm going to implement a method to drop items/objects (and positions of other things like AIS receivers) on the map some time in the near future.

The service status page now shows the time the aprs.fi software was upgraded on the server.

Wildcard callsign lookup result and moving stations tables are now sortable. Callsign sorting was fixed to be alphabetic in a few other places, too.

AIS receivers now have their own green tower symbol to distinguish them from AIS base stations.

The background colour of APRS item labels is now a bit lighter, to make the name of the item more readable.

A couple of smaller bug fixes were also done, and some code re-factoring to break things into smaller parts.