Improved igate and digipeater coverage statistics

This morning I installed an upgrade which I have been working on for quite a while. I've completely reimplemented the way "heard by" and "igated by" data is stored, allowing for faster lookups, smaller on-disk storage, and per-month views.

As a result, the info page, the heard map, the gated map and station statistics graphs will be able to show coverage statistics for complete months. This will make the views much more useful in areas with little traffic.

At first it'll only show the current month (making the view not very useful during the beginning of the month), but I'm going to make it possible to select previous months, too.

As a little bonus, space characters (0x20) now show up as non-breaking space in the "normal" raw packets display, which won't be condensed to a single space in the HTML.

Slowdown on Friday 26th

Yesterday the aprs.fi APRS feed was slow for about an hour, between 15:00:55 and 16:11:03 UTC. One of the WXQA servers the service connects to was down, and the connection attempts timed out. The timeout was too long, and the connection retry timer was too short, and the connect() attempt is a blocking call, resulting in slow processing of packets. I knew about the potential problem, but hadn't bothered to fix it until now.

In the evening I implemented a 2-second connect timeout and an exponential backoff for the retry timer. First reconnect attempts will happen within seconds, but they will slow down to about 2 minutes between retries. Using a non-blocking connect() would have been the correct fix, but this was a bit quicker. The problem should not appear again in this form.

It seems like no APRS data was lost or missed, it was just collected in a buffer, and processed once the connect attempts started working again. The following graph gives some idea of the relative processing rate changes. At peak about 10 megabytes of data was in the buffer.

VRRP failover - HA for the web service

This evening I've set up keepalived on the two aprs.fi front-end web servers. The program automatically manages the IP addresses of the web service. If one server goes down (due to a hardware failure, operating system or web server software hang, or for a maintenance reboot), the other server will now automatically bring up the service IP address of the first one. The fail-over happens within seconds.

The very same VRRP method is used by the routers serving the hosting network to keep the .1 default gateway address available.

There haven't been any hardware problems so far which would have made this necessary, but now I don't have to go through so much reconfiguration every time I want to reboot a server for a kernel upgrade, or take one down to add some memory. I can also shut down the web server processes on one of the servers (for a more complicated reconfiguration) and keepalived will quickly point all users to the other servers.

Project: POCSAG paging network infrastructure

Like OH2LAK reported on his blog, I've started implementing infrastructure code for an amateur radio paging network. Erik (OH2LAK), Weltto (OH7JEV) and Lasse (OH3HZB) got the POCSAG modem running, interfaced it to a Motorola radio, and configured a couple of pagers to receive it on 144.975 MHz.

I got the bits and pieces yesterday for development and debugging, started working on the server software at 6.30 PM yesterday evening, and now I have a working client-server paging network infrastructure with strong authentication, queuing, and support for multiple transmitters. I grabbed quite a bunch of code from the aprs.fi code base, of course.



The system consists of:

• A server component which receives messages from authenticated message source clients, queues them, and provides them to the transmitter clients
• A client library (Perl module) which is used to communicate with the API of the server
• A transmitter client, which fetches messages from the server using the client library and feeds them to the POCSAG modem (and, optionally, transmits CWID at regular intervals)
• And, for demonstration purposes, a DX cluster client which maintains a connection to a DX cluster server, filters DX spots, and sends them to the paging server using the client library. I think I've made a full circle now, over the past 15 years. Feels like home.

The next thing I'll write is an APRS-IS client to collect APRS messages from the APRS-IS and gateway the to the paging server (for those recipients who have a pager configured on the server).

The server is designed so that it's possible to run multiple instances of it at diverse network locations, so that losing one server will not break the whole network.

The network will probably be national, within Finland, coordinated by RATS (Radio amateur technology society of Finland). The system will be demonstrated at the RATS technology day in Otaniemi, Espoo, 20th of March, 2010 (that's next Saturday). I suppose the code and documentation will be published so that you can set up your own networks.

Decoded and hex raw packets, default coordinate format, and eesti keel

The raw packets view can now show the packets in hexadecimal format (revealing all unprintable packets and exact amount of whitespace) and decoded format. The decoding is done using the free Ham::APRS::FAP parser module, as usual.

Developers: Please note that the raw packets display is an user interface, not an API. Don't use it for downloading packets to your applications. Please get them directly from the APRS-IS and decode them yourself. The same goes for other views on the service, too.

The default coordinate format is now degrees and decimal minutes (60°10.92' N 24°31.86' E), which is the default format used on most APRS applications. aprs.fi used to default to degrees, minutes and seconds.

ES1TFJ and ES3AT have started a translation to Estonian, and the language has now been enabled on aprs.fi. Thank you!

I've also added a bunch of basic system-level tests which are run always before the software installer is built ('make bdist'). They feed data to the system using a simulated APRS-IS server and a simulated AIS receiver, and see that the uploaded data appears in different views on the site. I can also run just the system-level tests ('make testsys') which takes about 8 seconds for 93 test cases, or the whole test set ('make test': unit + system tests) which takes about 12 seconds. The validators of the tests are not very strong, but at least they check that the data is updated and that the views don't crash.

ipv6.aprs.fi, and receiver detection tuning (again)

After working on some Other Stuff for about a month, I did a little upgrade for aprs.fi.

I fixed receiver detection for packets with a path like DIGI1*,WIDE1*,DIGI2*,WIDE2* (and added an unit test to keep it working from now on). This bug was reported on the discussion group by a few users.

The status message of a station is no longer shown on the real-time map view if it's more than 24 hours older than the last position report from the station.

I did a couple of very small fixes (disabled IPv4-only GeoIP and increased database column length), recompiled web server software with IPv6 support, and added some configuration, and made aprs.fi available over IPv6 at http://ipv6.aprs.fi/! The whole effort took under an hour. Like the guys at Google say, it's surprisingly easy to set up. After all, it's pretty old technology – I first played with it about 10 years ago.

After some time I'll probably add AAAA records for http://aprs.fi/ so that everyone with IPv6 connectivity will automatically use IPv6 instead of IPv4. Most big sites still use separate hostnames like ipv6.google.com because there are some buggy operating systems and bad ipv6 network configurations which make the end-user hosts believe they have global IPv6 connectivity while it is, in fact, broken.

Windows Vista and Windows 7 have IPv6 connectivity built in and enabled by default, so the amount of IPv6 clients is going to grow quickly in the near future. Your network provider doesn't need to support IPv6, since 6to4 and Teredo tunneling are also enabled in those operating systems by default.

The Other Stuff included setting up native IPv6 routing for the network where aprs.fi servers are hosted. We got two /48 IPv6 PI prefixes, one for each AS. I set up BGP routing on 5 routers (Cisco, Juniper) over those two ASes and made native (not tunneled) IPv6 available to the users of the network. The ISP I'm using (Nebula) also offers native IPv6 to residential ADSL customers.

There's not much value in this to most of you right now, but at least it's cool and future-proof. Makes the service available for those of you who will end up with IPv6-only connections when we run out of IPv4 addresses in a few years.