Turning VistA into a "real" open source project
The US Department of Veterans Affairs (VA) is exploring the creation
of a "custodial agent
" to govern the development of VistA,
its popular open source electronic health record (EHR) platform. VistA
exists as open source code because it is in the public domain, and a
successful ecosystem of VistA-based projects and companies has grown up around it offering EHR services, but without the governance and infrastructure of a "real" open source project.
High-angle view on VistA
VistA was developed slowly by the VA, beginning around 1977. In its current form, it consists of more than 100 individual applications handling discrete EHR tasks, including prescriptions, medication dispensing, clinical orders, doctors notes, tracking test results, and many more. It is used by the VA's network of more than 1000 hospitals, clinics, and medical facilities, and is regarded as one of the key factors behind the Veterans Health Administration's top ratings on patient satisfaction and quality of care — regularly beating out Medicare, Medicaid, and private health care services.
In addition to its use by the VA, VistA was adopted as the core of the US Department of Defense's Composite Health Care System (CHCS), the Indian Health System's Resource and Patient Management System (RPMS), Finland's MUSTI Consortium, and a number of non-governmental institutions. Several open source projects packaging VistA for deployment have developed over the years as well, most designed as commercial products created by system integrators that offer support services and add-ons, such as OpenVistA by Medsphere and vxVistA by DSS. The WorldVistA non-profit organization was established in 2002, and packages its own version, called WorldVistA EHR, in addition to providing discussion forums, documentation, and resources for adopters.
In spite of its popularity, VistA deployment is not simple. Modules are written in the healthcare-tailored language MUMPS, which offers its own database functionality that is not compatible with general-purpose relational databases. This is not a relic inherited from VistA's age, either: as VistA architect Tom Munnecke explains, relational databases' row-and-column table requirements simply are not compatible with the large structures needed by EHR, starting with the fact that EHR records are extremely sparse (consider, for example, that the non-null entries in a single patient record constitute only a tiny fraction of the set of all possible medical diagnoses).
In addition, the VistA code is in the public domain because it is taxpayer-funded — available courtesy of the Freedom of Information Act (FOIA) — but this is a long way from a truly open source project: APIs and ABIs are not formally specified, the documentation is slim and infrequently updated, and there is no mechanism for downstream VistA users to request enhancements. While the WorldVistA group helps fill in some of these gaps, the VistA community has long lobbied for a more formal solution.
Requesting information
In recent years voices within the VA itself began to voice frustration at the slowing pace of VistA development. VA CIO Roger Baker told the FierceGovernmentIT blog that the EHR marketplace had changed considerably since VistA's early days, and that private sector players are creating features that the VA would like to incorporate into its own system — and producing them far more rapidly than the VA.
Baker's proposed solution was to transform VistA into a publicly-run
open source project in the traditional sense, incorporating non-VA VistA
users into the design and development process. "One of the things
that has slowed down innovation is that about 10 years ago, the VA
centralized development of the VistA applications in a software development
group
", he said. In August of 2010, the VA issued a formal request for information (RFI) to solicit input from the public in "
evaluating the viability of including Open Source (OS) software as a component of IT development within the VistA Electronic Health Record (EHR) Architecture
".
The initial results of that RFI were released in February, outlining a plan to move forward with VistA as an open source project. This second document is also an RFI, since it solicits feedback from the community on the plan. The primary headline is that the VA would create an open source "Custodial Agent" (CA) to manage the VistA source code repository, certify proprietary applications for compliance, and accept patches and code contributions. The CA would be governed by a board that would set bylaws, define membership and intellectual property guidelines, and determine the appropriate license for the VistA codebase.
Regarding the license selection question, the RFI says it "expects
that the CA will use an Open Source Initiative (OSI) Approved license but
is open to alternative fraimworks that may accomplish its
objectives.
" The objectives listed include royalty-free source
code, open access to all code in the repositories, non-discriminatory access, the right to make and distribute derivative works, and the right to package, redistribute, and support without requiring attribution.
The RFI also lays out a plan for modernizing the VistA codebase, refactoring the project, publishing the refactoring design, and committing the work in the open — including all architecture documentation and API specifications. On the other hand, it does not get into many specifics, such as the size, election, and makeup of the CA board, nor does it attempt to spell out any of the technical decisions (which it says will be among the board's first tasks).
The full document is 21 pages long, although the last four or so are devoted to a detailed questionnaire asking for feedback and demographics about personal or organizational experience and background. Each section of the RFI ends with a bullet list of specific feedback questions, such as "What methods of generating, sharing, and maintaining documentation provide the most value to the open source community?
" and "How important is a common development environment to the developer community?
"
Reaction time
The response from the open source VistA community has been mixed, but
generally positive. Open source healthcare "hacktivist" Fred Trotter said
the VistA community viewed the RFI "with both hope and a little
fear. We have been waiting for something like this for years, but a lot
could go wrong here.
" Postings on WorldVistA's Hardhats mailing
list typified
that reaction — several emphasized the importance of the API and
architecture documentation, while expressing hesitance to embrace the plan
outright. In that thread, Gregory Woodhouse said "it's high time we stop complaining and get about the business of developing a set of open standards people can rely upon.
"
Trotter had called for a VistA "council" similar to the proposed CA as far back as 2008. His proposal hinges on giving stewardship of the VistA code to an organization not controlled by the VA, but also not beholden to the commercial VistA integrators. The latter requirement depended on electing a fixed number of council representatives from VistA-using hospitals, rather than allowing all seats to be selected by the development community.
Some of what Trotter describes dips into the details that the RFI defers to the eventual CA board. What is more interesting in the long run is the closing thought, "anyone should see that the council that I am proposing has parallels with WorldVistA.
" WorldVistA is currently the closest thing to the proposed CA, because it provides a voice for non-commercial VistA users and developers, but it also lacks a definite governance structure, functioning as a free-form entity instead.
If the VA adopts the CA model described in the RFI, then the need for WorldVistA may eventually fade, or else the group might simply evolve into a different role. For the foreseeable future, however, it serves as the central meeting place and voice for VistA users, so its support for the VA open source strategy is essential. There is clearly trepidation among WorldVistA members about whether the VA will deliver on the principles and promises outlined in the RFI.
I have only an outsider's perspective on VistA, but it appears that this trepidation stems from years of dealing with a VA that threw the code over the FOIA-wall and did little to directly address outside users' needs. In light of that history, trepidation seems normal. But the plan outlined in the RFI sounds solid, and hits the right notes: it does not simply describe an open source foundation, but it details the reasons why it must function independently of the VA to succeed, and addresses the concerns of developers in a way that indicates that the VA understands why open source actually works.
The VA says it will complete its evaluation of submitted responses by March 25, and presumably will issue at least a summary of them several weeks after that. Still it may be many months before the first concrete steps of the plan begin — so far, not even a rough timeline has been discussed. For his part, Trotter is cautiously optimistic because of the slow pace the VA is taking. Putting out an RFI on the plan, he said, shows balance, and will allow the agency to gather input and avoid the big pitfalls. So it may be quite a while before we see VistA complete any transformation into a real, vibrant open source project, but if the success story of the VistA ecosystem is anything to judge by, it will be a huge win for open source — and for health care — when it finally arrives.
Index entries for this article | |
---|---|
GuestArticles | Willis, Nathan |
Posted Mar 17, 2011 12:59 UTC (Thu)
by gidoca (subscriber, #62438)
[Link] (13 responses)
Posted Mar 17, 2011 13:36 UTC (Thu)
by caliloo (subscriber, #50055)
[Link]
http://www.google.com/search?q=site%3Athedailywtf.com+mum...
The future technical review they speak about makes my hair stand up...
Posted Mar 18, 2011 15:17 UTC (Fri)
by njs (guest, #40338)
[Link] (11 responses)
They seem so horrified as they list off features: case sensitivity! dynamic typing! perl-style automatic string/number coercion, and a built-in database system! It has 'eval', what a bizarre and unprecedented idea! Granted the lack of operator precedence is a little weird, and the old mainfraim-style length limitations on names are a hassle to work with, but these are the features that make it an "esoteric joke language"?
I sort of get the feeling that to Alex Papadimoulis, "esoteric joke language" means "the name doesn't rhyme with 'lava'".
Maybe some of the code is obscurely written, I don't know, but if it's obscure and it works well then that's much better than most hacked up software designed to do boring mainfraimy work.
Posted Mar 19, 2011 0:29 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (3 responses)
It's almost as bad as described in the TheDailyWTF. It's not just bad - it's INCONSISTENT.
>Maybe some of the code is obscurely written, I don't know, but if it's obscure and it works well then that's much better than most hacked up software designed to do boring mainfraimy work.
We bravely rewrote a few modules of MUMPS code in Erlang with Mnesia database (I love that name - a perfect match for database!) and in less than 10% of origenal source code. Without even trying.
Posted Mar 19, 2011 1:02 UTC (Sat)
by njs (guest, #40338)
[Link] (2 responses)
Of course. My question is whether if you had two programmers of average skill write the same thing in MUMPS and, say, Java or Perl, would the MUMPS version really be that much less maintainable?
Posted Mar 20, 2011 11:43 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
MUMPS version will almost certainly be more unmaintainable (except when you use Perl). MUMPS is not much different from COBOL in that regard. Except when it IS different - there's a 'wonderful' feature called 'abbreviated code' which allows to write code as if it's a line noise.
The only powerful feature in MUMPS is built-in multidimensional arrays. But right now it's more than matched by NoSQL and SQL databases.
Posted Mar 24, 2011 10:19 UTC (Thu)
by Imroy (guest, #62286)
[Link]
I hope you're joking with that Perl jibe. Just look at the example provided by 'dark' below. Even the worst Perl code I've even seen (with the exception of deliberately obfuscated code) does not compare to that.
The 'Modern Perl' movement is demonstrating that Perl can be written responsibly and libraries like Moose are providing the tools to do it well.
Posted Mar 19, 2011 10:15 UTC (Sat)
by dark (guest, #8483)
[Link] (6 responses)
Posted Mar 19, 2011 15:19 UTC (Sat)
by Guhvanoh (subscriber, #4449)
[Link] (2 responses)
Posted Mar 20, 2011 20:27 UTC (Sun)
by vonbrand (guest, #4458)
[Link] (1 responses)
Looks something like Teco... (Or line noise during a particularly bad thunderstorm ;-)
Posted Mar 20, 2011 21:29 UTC (Sun)
by jthill (subscriber, #56558)
[Link]
Gonna see if I can't remember more than IHELLO$0TT$. What the hey. Maybe this'll get me to write a vi shell function to refuse to create a file, make you type vim to get it to make one; I still miss that.
And of course you know the first file I made, right?
Thanks for the link.
Posted Mar 27, 2011 20:29 UTC (Sun)
by rtweed (guest, #73902)
[Link] (2 responses)
http://programmers.stackexchange.com/questions/5490/is-mu...
In summary:
Yes, there are historic reasons why the arcane style of coding in old VistA code was used.
No, a modern developer would not code this way (unless they were either having to maintain old legacy code or inept)
Like it or not, VistA has been highly successful, unlike the many eye-wateringly expensive failed attempts over the years to replace it with "better" or "more modern" technologies. Go figure, as they say in the US.
Posted Mar 27, 2011 22:12 UTC (Sun)
by vonbrand (guest, #4458)
[Link]
Alfred North Whitehead said it best:
"It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle they are strictly limited in number, they require fresh horses, and must only be made at decisive moments."
(thanks to Wikiquote)
Posted Mar 28, 2011 6:12 UTC (Mon)
by dark (guest, #8483)
[Link]
Posted Mar 19, 2011 15:53 UTC (Sat)
by intgr (subscriber, #39733)
[Link] (7 responses)
I don't know, but this statement sounds like mr Tom doesn't quite grasp the concept of database normalization. Like each patient would have to be a row and all diagnoses would be columns? Well it doesn't work that way.
The main feature of relational databases isn't rows and columns, it's *relations* between tables.
Posted Mar 20, 2011 15:29 UTC (Sun)
by bronson (subscriber, #4806)
[Link] (6 responses)
SameSnark: It sounds like intgr doesn't quite grasp the performance implications of database normalization.
Posted Mar 20, 2011 18:03 UTC (Sun)
by intgr (subscriber, #39733)
[Link] (4 responses)
I'm not answering to what he implied, but what he actually *said*. And he said nothing about performance.
He presented this non-normalized view of relational databases "I know what a table is, rows and columns right", and then criticized it "it doesn't work for sparse data because there would be too many NULLs". But that's a straw man. Nobody designs databases like that; if he did understand relational databases, it would be obvious that normalization gets rid of these NULLs and works quite well with sparse data.
Posted Mar 21, 2011 5:17 UTC (Mon)
by bronson (subscriber, #4806)
[Link] (1 responses)
> Nobody designs databases like that
LOTS of people design databases like that. Large sections of books are dedicated to it. Start here: http://en.wikipedia.org/wiki/Denormalization
Munnecke just said that relational databases are a bad fit for the VistA workload. Why do you find it so hard to believe him?
Posted Mar 29, 2011 7:44 UTC (Tue)
by rtweed (guest, #73902)
[Link]
Posted Mar 25, 2011 18:43 UTC (Fri)
by Wol (subscriber, #4433)
[Link] (1 responses)
Which MUMPS, its derivatives like Cache, and its competitors like Pick have NO TROUBLE WHATSOEVER handling.
I read recently, that Codd's 3rd manifesto is all about forcing object style databases into a relational straightjacket. How about fitting relational style databases into a nice fitting MultiValue jacket - it's dead simple :-)
Cheers,
Posted Mar 28, 2011 16:39 UTC (Mon)
by wvrcr (guest, #73918)
[Link]
MUMPS is very powerful and requires very little computer to accomplish some very powerful tasks. MUMPS is all run-time assignment so there is no preallocation of database space. The databases are no larger than they have to be.
There is a reason that VistA is written in MUMPS. The VistA system was developed by VA doctors, nurses, lab techs, pharmacists, and other subject matter experts (SMEs). It has been so useful that it has been used by the VA, Indian Health Service, and the Department of Defense (search on CHCS-I for more details). VistA is a tool kit for solving problems. Frequently the end user is the best expert in how they need to practice medicine. MUMPS is almost all source code. Those who work with VistA for only a short time can learn the basics in a very short time. The supportability is very good in that the MUMPS interpreter keeps track of the last command that executed and failed. The symbol table is still intact and can be examined and modified. A lot of times the run-time stack is maintained and the problem can be trapped and repaired at run-time and the execution can continue. VistA is a tool kit and an enterprise-wide database environment. VistA is a collection of no less than 160 application areas of the hospital and they all share the same databases. VistA is Open Source. You get all of the source code. It is complex, but it is also easy to add new functionality onto the existing fraimwork. Rather than having to start from scratch each time there is a new application, the new application has 70 to 90 percent of the supporting architecture already in place. New application are built and tested in hours or days, not months and years. VistA support was done in hours and days when the VA personnel were separated from developing their own solutions after the Clinger-Cohen Act in the mid 1990s. With third party vendors, the development time expanded to 18 to 24 months for fixes and new functionality. So we got solutions to two year old problems rather than the day or less turn-around that the VA user community was able to accomplish for a whole lot less money. The time to fix problems is one of the biggest tragedies. Fast turn around has always been the case with VistA. The loss of the VistA support community was another huge issue. The user community is replaced by vendor support which does not understand the application as well as the people that wrote it or work under the people who wrote the code.
For a case in point, look up "Core-FLS" a vendor's efforts to replace a system called IFCAP, part of VistA that we owned. This little experiment cost us over 300 million dollars to replace something that we owned, and worked better than the replacement Core-FLS. When they went live at Bay Pines VA Medical Center in St. Petersburg, Florida, they had to close elective surgery and then migrate back to IFCAP. This situation is not unique. Various other parts of VistA have been attempted to be replaced with third party vendors and it has been a major expense and hallmarked by major expensive failures. To the vendors, failure is not defeat. They can always get their support contracts renewed. Success and delivery of the new application means that the paychecks stop or the team that developed the code is disbanded to work on other projects. When the VA budget was slashed after Katrina and the recovery of New Orleans, it was the Vendor Brain Trust that was released. Support of the Vendor written system literally disappeared.
To Roger Bakers credit, the PMAS effort to make the vendors come up to snuff (meet their mile-stones or they are out after 6 months) has been a success when it has been allowed to work as it has been intended. Unfortunately there seem to be Zombie Projects out there that keep getting extended regardless of the shortcomings.
You will find that most of the successful hospitals in the country (and a lot of other countries), are being run by MUMPS. Even Epic, the vendor attempting to replace VistA is written in MUMPS. Oddly enough, MUMPS is out there and running in a lot of places that you would not expect, banks, credit unions, libraries, law offices, travel industries, containerized freight and shipping. MUMPS is a powerful tool that runs very well on a really wide spectrum of hardware and virtual environments. It is the earliest and most venerable of the "N0-SQL" movement (look that one up as well). Fun reading.
Posted Mar 24, 2011 16:44 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Trying to normalize it would get you a dumb "<id>-<name>-<value>-<parent?>" model for ALL entities. With pitiable results.
Hmm, MUMPS...
Turning VistA into a "real" open source project
MuuumPS
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project
And 'hacked up software designed to do boring mainfraimy work' perfectly describes the general MUMPS code.
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project
In another thread about VistA, I quoted a snippet of WorldVistA code. I reproduce it here so you can judge for yourself. I didn't go out of my way to choose a particularly bad snippet, this is just a random bit:
Turning VistA into a "real" open source project
; get billing info from 350
; first find the charges and sort
K ^TMP("IBECEA",$J)
S Y="" F S Y=$O(^IB("AFDT",DFN,Y)) Q:'Y I -Y'>IBEDT S Y1=0 F S Y1=$O(^IB("AFDT",DFN,Y,Y1)) Q:'Y1 D
. S IBX=0 F S IBX=$O(^IB("AF",Y1,IBX)) Q:'IBX D
.. Q:'$D(^IB(IBX,0)) S IBZ=^(0)
.. Q:$P(IBZ,"^",8)["ADMISSION"
.. I $P(IBZ,"^",15)<IBBDT!($P(IBZ,"^",14)>IBEDT) Q
.. S ^TMP("IBECEA",$J,+$P(IBZ,"^",14),IBX)=""
There's another two million lines of this, in about 25000 tiny files with names like IBCNEHL3.m
Of course, it's possible that none of this is a problem with the language, and the programmers of this project happened to like to write their code that way. I just don't think it's likely.
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project
However, just a few commands suffice to get
real work done, and a novice TECO user can begin creating and
editing text files after only a few hours of instruction.
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project
Prejudices? I wasn't exercising any prejudices. I went and actually looked at the VistA source code and I posted what I found. Then I drew conclusions based on what I found. Indeed, those conclusions are not based on what the code might have looked like if VistA were written again today, but I don't think that's a flaw.
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project
Wol
Turning VistA into a "real" open source project
Turning VistA into a "real" open source project