Testing in the Yocto Project
The ever-increasing complexity of the software stacks we work with has given testing an important role. There was a recent intersection between the automated testing being done by the Yocto Project (YP) and a bug introduced into the Linux kernel that gives some insight into what the future holds and the potential available with this kind of testing.
YP provides a way of building and maintaining customized Linux distribution; most distributions are one specific binary build, or a small set of such builds, but the output from YP depends on how you configure it. That raises some interesting testing challenges and the key to that is automation. The YP's build processes are all automated and its test infrastructure can build compilers, binaries, packages, and then images, for four principal architectures (ARM, MIPS, PowerPC, and x86) in 32- and 64-bit variants, and for multiple C libraries, init systems, and software stacks (no-X11, X11/GTK+, Wayland, etc.). It can then build and boot-test them all under QEMU, which takes around six hours if everything needs to be rebuilt; that can drop to under two hours if there are a lot of hits in the prebuilt-object cache.
Not content with that, YP has been adding support for running the test suites that many open-source projects include on a regular and automated basis. These are referred to as packaged tests or "ptests" within the project. For example, a ptest might be what would run if you did "make check" in the source directory for the given piece of software, but packaged up to be able to run on the target. There are many challenges in packaging these up into entities that can run standalone on a cross-platform target and parsing the output into a standard format suited to automation. But YP has a standard for the output and the installed location of these tests, so they can be discovered and run.
While all architectures are boot-tested under QEMU, and those tests are run on batches of commits before they're merged into YP, right now only architectures with KVM acceleration have the ptests run. Also, the ptests are run less regularly due to the time they take (3.5 hours). This means ptests are currently run on 64-bit x86 a few times a week and aarch64 is in testing using ARM server hardware.
When YP upgraded to the 5.0 Linux kernel recently, it noticed that some of its Python 3 ptests were hanging. These are the tests from upstream Python, and the people working on YP are not experts on Python or the kernel, but it was clear there was some problem with networking. Either Python was making invalid assumptions about the networking APIs or there was a kernel networking bug of some kind. It certainly seemed clear that there was a change in behavior. The bug was intermittent but occurred about 90% of the time, so it was easy to reproduce.
Due to that, YP developers were able to quickly bisect the issue down to a commit in the 5.0 kernel, specifically this commit ("tcp: implement coalescing on backlog queue"). The problem was reported to the netdev mailing list on April 7.
Nothing happened at first, since the YP kernel developers and Python recipe maintainers didn't have the skills to debug a networking problem like this and there wasn't much interest upstream. On April 23, though, Bruno Prémont also ran into the same problem in a different way. This time, the original patch author was able to figure out the problem. There was an off-list discussion about it and a patch that fixes the problem was created; it has made its way into the 5.0 stable series in 5.0.13.
The problem was in the 5.0 changes to the coalescing of packets in the TCP backlog queue, specifically that packets with the FIN flag set were being coalesced with other packets without FIN; the code paths in question weren't designed to handle FIN. Once that was understood, the fix was easy. This also highlighted potential problems with packets that have the RST or SYN flags set, or packets that lack the ACK flag, so it allowed several other possibly latent problems to be resolved at the same time.
To YP, this is a great success story for its automated testing as it found a real-world regression. YP has had ptests for a while, but it has only recently started to run them in an automated way much more regularly. It goes to show the value in making testing more accessible and automated. It also highlights the value of these existing test suites to the Linux kernel; there is a huge set of potential tests out there that can help test the kernel APIs with real-world scenarios to help find issues.
The Yocto Project would welcome anyone with an interest in such automated testing; while it has made huge strides in improving testing for its recent 2.7 release, there is so much more that could be done with a little more help. For example, the project would like to expand the number of test suites that are run, improve the pass rates for the existing tests, and find new ways to analyze and present the test results. In addition, with more people available to triage the test data, the project could incorporate some pre-release testing to help find regressions and other problems even earlier.
[Richard Purdie is one of the founders of the Yocto Project and its
technical lead.]
| Index entries for this article | |
|---|---|
| GuestArticles | Purdie, Richard |
Posted May 18, 2019 16:31 UTC (Sat)
by pj (subscriber, #4506)
[Link]
Posted May 18, 2019 16:39 UTC (Sat)
by dvdeug (subscriber, #10998)
[Link] (29 responses)
Posted May 18, 2019 18:20 UTC (Sat)
by mtaht (subscriber, #11087)
[Link] (7 responses)
I think having a diverse automated testbase so one group with an automated test suite could get verified by another group with a different automated testsuite, could possibly evolve out of all this. Ultimately we'll end up with AI's stacked upon AIs competing to keep the computing environment clean...
... duking it out with AIs stacked upon AIs competing to find holes and exploit them before the other AIs do... and other AIs ripping out the work of other AIs to find a place for themselves to execute....
... and befuddled humans as the bystanders in that rapidly escalating war hoping that no AI notices that the AIs don't actually need to keep the humans alive and around in order to thrive in the complexity collapse.
I just finished reading a pretty decent new SF book out about this sort of stuff: https://www.amazon.com/Genesis-Geoffrey-Carr-ebook/dp/B07...
Posted May 19, 2019 2:54 UTC (Sun)
by roc (subscriber, #30627)
[Link]
Why would you need that? I can't think of any project that requires a bug to be confirmed in two independent test suites before investigating it. I don't even know how you would know it was the same bug.
> Ultimately we'll end up with AI's stacked upon AIs competing to keep the computing environment clean...
Maybe, but the kernel community is still far behind what other big projects consider basic testing requirements. (And those basics don't require AI.)
Posted May 19, 2019 4:19 UTC (Sun)
by dvdeug (subscriber, #10998)
[Link] (5 responses)
Posted May 19, 2019 9:20 UTC (Sun)
by knuto (subscriber, #96401)
[Link] (4 responses)
Posted May 19, 2019 10:37 UTC (Sun)
by roc (subscriber, #30627)
[Link] (3 responses)
> That way the author of a change keeps the burden of proof for the change, instead of the test owner.
That's a good point, but if you're a kernel developer and you can keep the burden of testing, triaging, diagnosing and reporting bugs on other people, it makes sense to carry on with that system. Even if in many cases you're shifting work from paid kernel developers to unpaid open source project maintainers. Even if it means you're forcing downstream consumers to duplicate tons of work maintaining multiple test infrastructures and running the same sorts of tests, or even exactly the same tests. Even if it strangles the feedback loop between detecting and fixing regressions.
Posted May 20, 2019 9:16 UTC (Mon)
by metan (subscriber, #74107)
[Link] (2 responses)
Well in SUSE we do run LTP compiled with -m32 on x86-64 which catches bugs in the compat layer from time to time, but the coverage of ioctls() is sparse at best, we do have some tests for /dev/tty*, /dev/net/tun, block device ioctls, /dev/random, namespace ioctls and some Btrfs ones and that's it.
Posted May 20, 2019 13:14 UTC (Mon)
by roc (subscriber, #30627)
[Link] (1 responses)
Posted May 20, 2019 13:21 UTC (Mon)
by metan (subscriber, #74107)
[Link]
Posted May 20, 2019 15:35 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (20 responses)
Posted May 20, 2019 16:04 UTC (Mon)
by mathstuf (subscriber, #69389)
[Link] (1 responses)
See this bug report to git: https://marc.info/?t=151242550100001&r=1&w=2 And the followup patch after just one day: https://marc.info/?t=151251216400002&r=1&w=2. Could I have come up with that `read_cache()` line? Probably, but not in that timescale. The commit message would also certainly have been far inferior.
Posted May 20, 2019 17:35 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link]
Apart from that, paraphrasing a famous quote, "the kernel is just a program".
Posted May 20, 2019 18:14 UTC (Mon)
by dvdeug (subscriber, #10998)
[Link]
That philosophy works great for code that somebody threw up on GitHub because someone else might find it useful. But if maintainers get dismayed that a feature is being dismissed as broken, with no bug reports, or that some group is running an ancient version of the kernel, this is why. The fact that updating the kernel could cause random problems, and even dedicated searching for the bug, where you can provide a reproducable test case and the commit that's causing the problem, is not going to help, is a reason to lock the kernel to one version or use a local patch if one can figure out what's wrong.
I see the problem. But if you want to make a program that's used by a wide audience, sometimes you have to respond to the needs of that wide audience, and when they can give you a test that's failing and even a commit that made the change, perhaps you should take a look and fix it. Otherwise, people will fork or stay on older versions or switch to another program that does the same thing. As you say, the kernel is just a program, and if Linux's rapid changes are a downside, no other Unix kernel is changing nearly as fast.
(I'd point out that there's not this attitude inside the kernel community; if a commit won't bootstrap on ARM, it's not the problem of the ARM community, that's the problem of the commit submitter. There's a good argument that a problem like this doesn't need time to work out what's going on; once a particular commit was known to be causing a userspace problematic change, just rollback the commit and if the submitter of the commit wants the change, they get to make it work right or explain why the problem is not a kernel bug.)
Posted May 20, 2019 20:51 UTC (Mon)
by roc (subscriber, #30627)
[Link] (16 responses)
The Linux kernel is special. Linux is so dominant that downstream users are more or less stuck with it, so kernel development is structured to offload as many costs as possible to downstream. Even though it's a terribly inefficient setup and painful for downstream, downstream users have no leverage to alter the deal.
This will continue to make sense until Linux has serious competition, and then it won't.
Posted May 20, 2019 21:33 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (13 responses)
If *you* think this is terrible, you're perfectly entitled to that but other people who are also "downstream users" might disagree. The kernel is really just a program and as such, it can be made to dance to whatever tune someone currently prefers. It's even a pretty organized program, hence, changing it is reasonably easy compared to other open source software (I happen to have encountered).
Posted May 20, 2019 23:16 UTC (Mon)
by roc (subscriber, #30627)
[Link] (12 responses)
Imagine if Firefox and Chrome didn't do their own testing, and instead Linux distros, device vendors and Web developers felt obliged to set up independent browser-testing projects with duplicated but not always identical tests, their own triage and false-positive screening, various different channels for reporting test failures upstream (many of which are ignored), etc. It would obviously be insane. Yet the kernel does this and everyone seems to accept it.
Posted May 20, 2019 23:30 UTC (Mon)
by corbet (editor, #1)
[Link] (6 responses)
The problem with the kernel is that so many of the problems people run into are workload and hardware dependent. Developers increasingly test for the things they can test, but they can never test all of the available hardware, and they can never test that your workload, in particular will not regress. So, while the project still very much needs to improve its act in this regard, I'm not sure I see an alternative to a lot of downstream testing anytime soon.
(And yes, a better answer to bug tracking is sorely needed, but there are many ghetto areas in kernel development that nobody wants to pay for, and that is one of them).
Posted May 21, 2019 0:06 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted May 21, 2019 0:20 UTC (Tue)
by roc (subscriber, #30627)
[Link]
The Linux Foundation seems to have taken on the role of trying to ensure that organizations that profit from Linux put money back into it. I'm not privy to the workings of LF so I don't know whether they can take this on. Still, if there was a consensus in the kernel community that central testing infrastructure is needed and we just have to figure out how to pay for it, that would be a big step forward.
One thing that might help: the kernel community could try to normalize the expectation that writing tests, building better test frameworks, and investigating failed tests is something every kernel developer needs to do (including the expectation that every bug fix and every new feature comes with automated tests). This is normal for a lot of other projects, from tiny no-money projects to large big-money projects. Then the companies paying people to do kernel development would effectively be taxed to support the testing effort.
> The problem with the kernel is that so many of the problems people run into are workload and hardware dependent.
The kernel workload is highly variable. Then again, for browsers your workload is *every Web page in the world*, from gigantic static pages to GMail to twitchy 3D shooters and everything in between. Mostly written in obfuscated code by tens of millions of developers with ... a range of skill levels ... whom you mostly can't talk to. Browsers are sensitive to hardware variation too, though less so than the kernel, sure. But desktop applications have their own special challenges: myriad combinations of software that your software depends on and interacts with, including ultra-invasive stuff like antivirus and malware that patches your code, all changing underneath you day to day, almost entirely closed-source and often controlled by your competitors. Mobile has similar issues.
I talk a lot about browsers because that's what I know. I don't think kernels and browsers are uniquely difficult.
I get that testing the kernel is hard. That's why greater cooperation and more efficient use of testing resources is so important.
> I'm not sure I see an alternative to a lot of downstream testing anytime soon.
Sure but so far I don't even see that the kernel community sees lack of upstream testing as a problem that needs to be solved.
Look at it this way: kernel people emphasize how important it is for people to upstream their code --- "upstream first", etc. That reduces duplication of effort, ensures things stay in sync, etc. But for some reason* they don't apply the same logic to tests. Tests, test harnesses, and test infrastructure should be upstream-first too for exactly the same reasons. Yes, there will always be stuff that is hard to upstream.
* Come on, we all know what the real reason is. Testing is less fun that coding features or fixing bugs. Figuring out test failures is even less fun (though I'm working on that).
Posted May 21, 2019 0:26 UTC (Tue)
by roc (subscriber, #30627)
[Link] (3 responses)
You're paying for poor bug tracking already, the cost is just smeared out as people waste time finding and diagnosing bugs people have already found and diagnosed.
I assume the actual hardware/software running costs would be insignificant, so is the problematic cost what you'd have to pay people to do triage or what? I always assumed the barrier to better bug tracking was the difficulty of coming up with a single system and getting people to pay attention to it.
Posted May 21, 2019 0:31 UTC (Tue)
by roc (subscriber, #30627)
[Link] (2 responses)
Posted May 23, 2019 4:25 UTC (Thu)
by tbird20d (subscriber, #1901)
[Link] (1 responses)
Posted May 23, 2019 21:42 UTC (Thu)
by roc (subscriber, #30627)
[Link]
Posted May 20, 2019 23:47 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Even in this case some duplication will still inevitably take place, but at least it can be greatly minimized.
Posted May 21, 2019 9:04 UTC (Tue)
by metan (subscriber, #74107)
[Link] (1 responses)
Posted May 23, 2019 4:33 UTC (Thu)
by tbird20d (subscriber, #1901)
[Link]
I'm aware of 3 different get-togethers to continue working on pushing stuff forward: A testing microconference at Plumbers, a kernel testing summit immediately after Plumbers, and the 2019 Automated Testing Summit after OSSEU/ELCE in France. Some of these have not gotten much visibility yet, but hopefully that will change shortly.
Posted May 25, 2019 17:14 UTC (Sat)
by flussence (guest, #85566)
[Link] (1 responses)
That insanity is already old news. Are you completely unaware of the long tail of blogs posting cargo-culted tweaks to get things like basic GPU acceleration and sound to work in browsers?
“This web browser best viewed with Nvidia on Windows 10” is not a good look, but everyone seems to accept it.
Posted May 27, 2019 23:11 UTC (Mon)
by roc (subscriber, #30627)
[Link]
I'm aware that real-world combinations of software and hardware vastly exceed anything you can test in any reasonable test farm. Firefox's upstream tests aren't perfect, and can't ever be, so tweaks to Firefox's configuration in unusual situations remain necessary. (Keep in mind, though, that many of those "cargo-culted tweaks" aren't addressing Firefox bugs but are choosing different points in a tradeoff, e.g. choosing performance over stability by using GPU drivers with known bugs.)
That doesn't affect what I wrote, though. It is still true that, unlike the kernel, Firefox and Chrome upstreams do massive testing in a way that's tightly integrated with the development process, and take responsibility for detecting, diagnosing and fixing regressions.
Posted May 23, 2019 22:01 UTC (Thu)
by roc (subscriber, #30627)
[Link] (1 responses)
Posted May 27, 2019 23:11 UTC (Mon)
by roc (subscriber, #30627)
[Link]
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
> structured to offload as many costs as possible to downstream. Even though it's a terribly inefficient setup and painful for
> downstream, downstream users have no leverage to alter the deal.
Testing in the Yocto Project
Kernel testing needs to improve a lot — and has moved significantly in that direction. But I do wonder how you might envision this working; what organization, in particular, would do the testing that you are looking for? Who should pay for it?
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Please come to the Automated Testing Summit in Lyon, France, in October. It's a new event, where I hope we can talk about (and listen to testing experts on) exactly the issues you raise. The CFP should be available shortly, and personally I'd love to hear from you and other testers about your experiences. I'll announce the CFP on the automated testing e-mail list when it's ready (which should be within a few weeks).
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
See https://elinux.org/Automated_Testing_Summit_2018 and https://lwn.net/Articles/771782/
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project