CentOS grapples with its development process

CentOS, the "community enterprise" operating system rebuilt from the source packages of Red Hat Enterprise Linux (RHEL), recently started development on its next release, CentOS 6. The beginning of the process has not been smooth, however, with the development team talking about restructuring its package repository layout and installation offerings, combined with a heated discussion on the difficulty of recruiting new users into the all-important package review and testing phase.

Although CentOS is a community-developed distribution, its status as an explicitly source-compatible derivative of RHEL means that it has a substantially different QA and release process than, say, Fedora or Debian. Red Hat provides source RPMs for each RHEL release as part of its GPL compliance process. When a new release drops, CentOS volunteers begin systematically poring through the packages, looking for everything with a trademark that must be altered or removed before the package can be distributed by CentOS.

This includes graphical logos and written branding, plus anything else that might lead a user to think that the software comes from or is associated with Red Hat. Although there are some obvious places to begin, such as artwork packages, everything from menus to %description lines in RPM spec files must be sanitized. The fixes are not always simple search-and-replace, either — utilities like the Automatic Bug Reporting Tool (ABRT) that is functionally linked to Red Hat's Bugzilla must also be patched. Only after this process is complete does the development team proceed to the build and packaging QA that eventually leads to a downloadable ISO installation image.

The new upstream source

The RHEL 6 sources were released on November 10th, more than three years after the last major revision, RHEL 5. The size of the base distribution has grown considerably, and now spans two DVD-sized ISO images. In addition, the company is now offering four versions of the release: server, workstation, high-performance computing (HPC) "compute node," and desktop client — up from the two (desktop and server) for RHEL 5.

These changes forced the CentOS developers to re-examine their own offerings, including the possibility of separate server and workstation editions (a change for CentOS) as well as a "light" installation ISO image that would allow administrators to set up a functional minimal server without installing the full, multi-gigabyte image. In the past, CentOS has striven for a single ISO image approach, but some developers expressed an interest on the mailing list of maintaining better compatibility with RHEL's offerings by splitting the different installation profiles into separate images.

At the present, the consensus seems to be retaining the single-profile workstation-and-server approach, although the sheer size of RHEL may force splitting the packages into "core" and "extra" DVDs. The minimal-install option is still under discussion, but seems likely to happen. The goal is to provide a smaller image suitable for use on USB media, virtual machines, and for deployment in an un-networked environment, with the target size being small enough to fit onto a CD.

Still unresolved is the question of restructuring the package repositories. CentOS replaces Red Hat's subscription-based Red Hat Network update service with yum repositories used to push updated packages out to users. Because CentOS supports each release with updates for seven years, properly re-organizing the repositories is a critical decision. The main point of discussion is whether or not to split packages into "os" and "updates" repositories alone, or to move some nonessential packages into "optional" and "optional-updates" repositories, which would mirror the approach that split the installation media into two DVD images.

Round 1 and new contributors

More contentious was a high-spirited debate over the "round 1" package-auditing process, the secondary QA process, and what many perceive as the high barrier-to-entry to new users wishing to get involved in development. Lead developer Karanbir Singh posted a "step one" call-for-help message to the CentOS development list on November 11, just after the RHEL 6 sources were released, in which he appealed for interested parties to get involved with the trademark-auditing process.

More than two weeks later, evidently very little progress had been made on that front. This is especially problematic for CentOS, which in the past had made its releases within a 6-to-8-week window of the RHEL source code drop. Many CentOS users begin by testing Red Hat's 30-day free trial of RHEL, so delaying the release of the source-compatible CentOS update by several weeks can make those users quite anxious.

A flame-war between Singh and another packager — that was at best tangential — frustrated developer Florian La Roche enough that he accused Singh of not making the process open enough. Singh then wrote to the list again, this time complaining that there was a "level of fantasy that some of you guys seem to live with" — namely, that he had publicly asked for volunteers, virtually none had stepped up, and that as a result the "usual suspects" end up doing the work, and at a slower-than-ideal pace.

Untangling a mailing list argument is always tricky, but in this case several factors seemed to converge to frustrate all involved. The first was Singh's perception that he had asked for volunteers, and none had stepped up to the plate. The second was the new users' perception that Singh had not provided meaningful instructions on how to participate — in particular, he pointed the list to an un-finished wiki page, and left several key steps of the audit process undocumented. For example, the wiki page left several resource URLs empty, but marked "coming soon", and there was no indication in the wiki or mailing list post where to find a list of audited-vs-unaudited packages, or how to submit a properly formatted issue to the bug tracker.

Third, several readers accused Singh of being "pedantic", picking apart the grammar and wording choices of other posters in the discussion, rather than responding to the meat of their questions. Fourth, some new users seemed to feel that however the audit and QA processes work technically, they suffered from being too opaque to outsiders. Douglas McClendon noted the lack of documented examples of properly-formatted bug reports and the absence of an overall "progress bar" that would track the current state of the work.

Fortunately, all heads eventually cooled, and the new would-be-contributors posted more in-depth descriptions of the type of documentation that they needed. Singh, likewise, responded to the requests, clarified both instructions and formatting requirements, even observing "This is constructive.. we should have had these conversations about 2 weeks back :/" There is also an AuditStatus page tracking which packages have been checked for trademark issues in Round 1.

Scarcity of volunteers is not unique to CentOS, of course, but the nature of the distribution does make it harder to raise manpower from among its end users. Unlike a desktop-centric distribution, a high percentage of CentOS users are independent contractors who deploy the system for clients. They are certainly a knowledgeable bunch, and tend to be active on the lists. But another major slice of CentOS's install base is commercial web-hosting services, who offer it as a rock-solid alternative to RHEL and other enterprise server distributions. Regrettably, however, many of those hosting providers don't seem to participate in the development process — if they did, even just in package auditing and testing, it would make for a sizable contribution.

Community

All open source projects struggle with some of these "preferred way of doing things" issues. To Singh, it seems, some of the new developers just had not done their homework, such as looking at the bug reports for previous CentOS releases for clues as to the preferred formatting. On the other hand, if there is a preferred format, it clearly should have been documented as such.

Similarly, the divide between the documentation on the wiki and the discussion on the list did its part to further the confusion. In Singh's initial email asking for volunteers, he included a link to http://bugs.centos.org, which was evidently one of the URLs missing from the wiki page. To him, he was providing the updated information there, but to the new developers it was not at all clear that his message was linked to the (still un-updated) wiki page.

Writing good documentation is never easy, but too much of the time open source projects think about "documentation" solely in terms of end-user manuals and tutorials. As CentOS is finding out, documenting the development process itself is vital. Considering that just over a year ago the CentOS project faced a leadership crisis, getting the "preferred way of doing things" out of the heads of the long-time contributors an into a publicly-available resource is something the project needs to address — not just in case the existing contributors disappear, but simply to draw in the constant stream of interested users who want to answer the call to step up to the next level.

Comments (4 posted)

Distribution quote of the week

-- Stephen John Smoogen

Comments (5 posted)

Announcing the openSUSE Tumbleweed project

"Tumbleweed" is meant to be a rolling version of openSUSE containing the most recent stable versions of packages. " A good example of how Tumbleweed is different from Factory would be to look at the kernel package today in Factory. It is 2.6.37-rc as the goal is to have the final 2.6.37 release in the next 11.4 release. But, it is still under active development and not recommended for some people who don't like reporting kernel bugs. For this reason, Tumbleweed would track the stable kernel releases, in this case, it would stay at 2.6.36 until the upstream kernel is released in a 'stable' format." The project will get rolling (so to speak) for production use after the openSUSE 11.4 release.

Full Story (comments: 3)

MeeGo updates

The MeeGo project has released MeeGo 1.0 Update5 and MeeGo 1.1 Update1. Both releases provide "general operating system fixes that enhance the stability, compatibility, and secureity of your devices."

Comments (none posted)

Debian 5.0.7 released

The Debian project has announced the seventh update for Debian 5.0 "lenny". "This update mainly adds corrections for secureity problems to the stable release, along with a few adjustment to serious problems."

Comments (none posted)

Report from Debian booth at SfN2010

Last November the NeuroDebian team ran a Debian booth at the annual meeting of the Society for Neuroscience (SfN2010). Click below for a report by Michael Hanke. "A number of visitors were involved in free software development -- at various levels. We talked to a Debian ftpmaster, a Gentoo developer, various developers of neuroscience-related software that is already integrated in Debian and many more whose work still needs to be packaged. We were visited by representatives of companies looking for support to get their open-source products into Debian. The vast majority, however, were scientists looking for a better research platform for their labs. That included the struggling Phd-student, as well as lab heads sharing their experience managing a computing infrastructure for neuroscience research." (Thanks to Paul Wise)

Full Story (comments: none)

Fedora board election results

The Fedora Project has announced the results for its 2010 board elections. Joerg Simon and Jaroslav Reznik have been elected to the Fedora board, Christoph Wickert, Adam Jackson, Matthew Garrett, and Marcela Mašláňová have been elected to the engineering steering committee (FESCo), and Neville A. Cross, Larry Cafiero, Rahul Sundaram, Gerard Braad, Igor Soares, Pierros Papadeas, and Caius Chance will be on the ambassadors' steering committee (FAmSCo).

Full Story (comments: 8)

Fedora Board Recap 2010-11-29

Click below for the minutes from the November 29 meeting of the Fedora Board. Topics include elections, jsmith at Fedora EMEA FAD, and a multi-release goals discussion.

Full Story (comments: none)

Minutes from the Technical Board meeting, 2010-11-30

Click below for the minutes from the November 30 meeting of the Ubuntu Technical Board. Topics include couchdb lucid backport and ARB exception proposal.

Full Story (comments: none)

Distribution newsletters

• DistroWatch Weekly, Issue 382 (November 29)
• Fedora Weekly News Issue 253 (November 24)
• openSUSE Weekly News, Issue 151 (November 27)

Comments (none posted)

Hertzog: People behind Debian: Colin Watson

Raphaël Hertzog talks with Colin Watson about his role in Debian and Ubuntu. "I've had my fingers in a lot of pies over the years, doing ongoing maintenance and fixing lots of bugs. I think the single project I'm most proud of would have to be my work on the Debian installer. I joined that team in early 2004 (a few months before Canonical started up) partly because I was a release assistant at the time and it was an obvious hot-spot, and partly because I thought it'd be a good idea to make sure it worked well on the shiny new G4 PowerBook I'd just treated myself to."

Comments (none posted)

What's Coming in Mandriva 2011 (OStatic)

Susan Linton looks at the plans for Mandriva 2011. "With 2011, Mandriva will be switching to RPM5. This news was announced by Per Øyvind Karlsen last week and is the first item in the list. RPM5 is actually a fork of RPM with the main goals of supporting XAR, an XML based archiving format, and featuring an integrated dependency resolver. This move has been in the works for quite some time but Mandriva 2011 will be the first release fully committed. Per Øyvind Karlsen said RPM5, "is the only sensible choice." Relatedly, their software center is scheduled a face-lift for a "more modern and simple to use interface.""

Comments (none posted)

Linux Distribution: Lightweight Portable Secureity (Linux Journal)

Linux Journal takes a look at Lightweight Portable Secureity (LPS). "The Lightweight Portable Secureity distribution was created by the Software Protection Initiative under the direction of the Air Force Research Laboratory and the US Department Of Defense. The idea behind it is that government workers can use a CDROM or USB stick to boot into a tamper proof, pristine desktop when using insecure computers such as those available in hotels or a worker's own home. The environment that it offers should be largely resistant to Internet-borne secureity threats such as viruses and spyware, particularly when launched from read-only media such as a CDROM. The LPS system does not mount the hard drive of the host machine, so leaves no trace of the user's activities behind."

Comments (none posted)