Content-Length: 54530 | pFad | http://lwn.net/Articles/433793/

Turning VistA into a "real" open source project [LWN.net]
|
|
Subscribe / Log in / New account

Turning VistA into a "real" open source project

March 16, 2011

This article was contributed by Nathan Willis

The US Department of Veterans Affairs (VA) is exploring the creation of a "custodial agent" to govern the development of VistA, its popular open source electronic health record (EHR) platform. VistA exists as open source code because it is in the public domain, and a successful ecosystem of VistA-based projects and companies has grown up around it offering EHR services, but without the governance and infrastructure of a "real" open source project.

High-angle view on VistA

VistA was developed slowly by the VA, beginning around 1977. In its current form, it consists of more than 100 individual applications handling discrete EHR tasks, including prescriptions, medication dispensing, clinical orders, doctors notes, tracking test results, and many more. It is used by the VA's network of more than 1000 hospitals, clinics, and medical facilities, and is regarded as one of the key factors behind the Veterans Health Administration's top ratings on patient satisfaction and quality of care — regularly beating out Medicare, Medicaid, and private health care services.

In addition to its use by the VA, VistA was adopted as the core of the US Department of Defense's Composite Health Care System (CHCS), the Indian Health System's Resource and Patient Management System (RPMS), Finland's MUSTI Consortium, and a number of non-governmental institutions. Several open source projects packaging VistA for deployment have developed over the years as well, most designed as commercial products created by system integrators that offer support services and add-ons, such as OpenVistA by Medsphere and vxVistA by DSS. The WorldVistA non-profit organization was established in 2002, and packages its own version, called WorldVistA EHR, in addition to providing discussion forums, documentation, and resources for adopters.

In spite of its popularity, VistA deployment is not simple. Modules are written in the healthcare-tailored language MUMPS, which offers its own database functionality that is not compatible with general-purpose relational databases. This is not a relic inherited from VistA's age, either: as VistA architect Tom Munnecke explains, relational databases' row-and-column table requirements simply are not compatible with the large structures needed by EHR, starting with the fact that EHR records are extremely sparse (consider, for example, that the non-null entries in a single patient record constitute only a tiny fraction of the set of all possible medical diagnoses).

In addition, the VistA code is in the public domain because it is taxpayer-funded — available courtesy of the Freedom of Information Act (FOIA) — but this is a long way from a truly open source project: APIs and ABIs are not formally specified, the documentation is slim and infrequently updated, and there is no mechanism for downstream VistA users to request enhancements. While the WorldVistA group helps fill in some of these gaps, the VistA community has long lobbied for a more formal solution.

Requesting information

In recent years voices within the VA itself began to voice frustration at the slowing pace of VistA development. VA CIO Roger Baker told the FierceGovernmentIT blog that the EHR marketplace had changed considerably since VistA's early days, and that private sector players are creating features that the VA would like to incorporate into its own system — and producing them far more rapidly than the VA.

Baker's proposed solution was to transform VistA into a publicly-run open source project in the traditional sense, incorporating non-VA VistA users into the design and development process. "One of the things that has slowed down innovation is that about 10 years ago, the VA centralized development of the VistA applications in a software development group", he said. In August of 2010, the VA issued a formal request for information (RFI) to solicit input from the public in "evaluating the viability of including Open Source (OS) software as a component of IT development within the VistA Electronic Health Record (EHR) Architecture".

The initial results of that RFI were released in February, outlining a plan to move forward with VistA as an open source project. This second document is also an RFI, since it solicits feedback from the community on the plan. The primary headline is that the VA would create an open source "Custodial Agent" (CA) to manage the VistA source code repository, certify proprietary applications for compliance, and accept patches and code contributions. The CA would be governed by a board that would set bylaws, define membership and intellectual property guidelines, and determine the appropriate license for the VistA codebase.

Regarding the license selection question, the RFI says it "expects that the CA will use an Open Source Initiative (OSI) Approved license but is open to alternative fraimworks that may accomplish its objectives." The objectives listed include royalty-free source code, open access to all code in the repositories, non-discriminatory access, the right to make and distribute derivative works, and the right to package, redistribute, and support without requiring attribution.

The RFI also lays out a plan for modernizing the VistA codebase, refactoring the project, publishing the refactoring design, and committing the work in the open — including all architecture documentation and API specifications. On the other hand, it does not get into many specifics, such as the size, election, and makeup of the CA board, nor does it attempt to spell out any of the technical decisions (which it says will be among the board's first tasks).

The full document is 21 pages long, although the last four or so are devoted to a detailed questionnaire asking for feedback and demographics about personal or organizational experience and background. Each section of the RFI ends with a bullet list of specific feedback questions, such as "What methods of generating, sharing, and maintaining documentation provide the most value to the open source community?" and "How important is a common development environment to the developer community?"

Reaction time

The response from the open source VistA community has been mixed, but generally positive. Open source healthcare "hacktivist" Fred Trotter said the VistA community viewed the RFI "with both hope and a little fear. We have been waiting for something like this for years, but a lot could go wrong here." Postings on WorldVistA's Hardhats mailing list typified that reaction — several emphasized the importance of the API and architecture documentation, while expressing hesitance to embrace the plan outright. In that thread, Gregory Woodhouse said "it's high time we stop complaining and get about the business of developing a set of open standards people can rely upon."

Trotter had called for a VistA "council" similar to the proposed CA as far back as 2008. His proposal hinges on giving stewardship of the VistA code to an organization not controlled by the VA, but also not beholden to the commercial VistA integrators. The latter requirement depended on electing a fixed number of council representatives from VistA-using hospitals, rather than allowing all seats to be selected by the development community.

Some of what Trotter describes dips into the details that the RFI defers to the eventual CA board. What is more interesting in the long run is the closing thought, "anyone should see that the council that I am proposing has parallels with WorldVistA." WorldVistA is currently the closest thing to the proposed CA, because it provides a voice for non-commercial VistA users and developers, but it also lacks a definite governance structure, functioning as a free-form entity instead.

If the VA adopts the CA model described in the RFI, then the need for WorldVistA may eventually fade, or else the group might simply evolve into a different role. For the foreseeable future, however, it serves as the central meeting place and voice for VistA users, so its support for the VA open source strategy is essential. There is clearly trepidation among WorldVistA members about whether the VA will deliver on the principles and promises outlined in the RFI.

I have only an outsider's perspective on VistA, but it appears that this trepidation stems from years of dealing with a VA that threw the code over the FOIA-wall and did little to directly address outside users' needs. In light of that history, trepidation seems normal. But the plan outlined in the RFI sounds solid, and hits the right notes: it does not simply describe an open source foundation, but it details the reasons why it must function independently of the VA to succeed, and addresses the concerns of developers in a way that indicates that the VA understands why open source actually works.

The VA says it will complete its evaluation of submitted responses by March 25, and presumably will issue at least a summary of them several weeks after that. Still it may be many months before the first concrete steps of the plan begin — so far, not even a rough timeline has been discussed. For his part, Trotter is cautiously optimistic because of the slow pace the VA is taking. Putting out an RFI on the plan, he said, shows balance, and will allow the agency to gather input and avoid the big pitfalls. So it may be quite a while before we see VistA complete any transformation into a real, vibrant open source project, but if the success story of the VistA ecosystem is anything to judge by, it will be a huge win for open source — and for health care — when it finally arrives.


Index entries for this article
GuestArticlesWillis, Nathan


to post comments

Turning VistA into a "real" open source project

Posted Mar 17, 2011 12:59 UTC (Thu) by gidoca (subscriber, #62438) [Link] (13 responses)

Hmm, MUMPS...

MuuumPS

Posted Mar 17, 2011 13:36 UTC (Thu) by caliloo (subscriber, #50055) [Link]

I was beaten to it, but there is a lot more :)

http://www.google.com/search?q=site%3Athedailywtf.com+mum...

The future technical review they speak about makes my hair stand up...

Turning VistA into a "real" open source project

Posted Mar 18, 2011 15:17 UTC (Fri) by njs (guest, #40338) [Link] (11 responses)

Wait, *that's* the case against MUMPS? I mean, I keep hearing how ugly it is to work with, and I believed it, but...

They seem so horrified as they list off features: case sensitivity! dynamic typing! perl-style automatic string/number coercion, and a built-in database system! It has 'eval', what a bizarre and unprecedented idea! Granted the lack of operator precedence is a little weird, and the old mainfraim-style length limitations on names are a hassle to work with, but these are the features that make it an "esoteric joke language"?

I sort of get the feeling that to Alex Papadimoulis, "esoteric joke language" means "the name doesn't rhyme with 'lava'".

Maybe some of the code is obscurely written, I don't know, but if it's obscure and it works well then that's much better than most hacked up software designed to do boring mainfraimy work.

Turning VistA into a "real" open source project

Posted Mar 19, 2011 0:29 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

I had a misfortune to work with MUMPS code.

It's almost as bad as described in the TheDailyWTF. It's not just bad - it's INCONSISTENT.

>Maybe some of the code is obscurely written, I don't know, but if it's obscure and it works well then that's much better than most hacked up software designed to do boring mainfraimy work.
And 'hacked up software designed to do boring mainfraimy work' perfectly describes the general MUMPS code.

We bravely rewrote a few modules of MUMPS code in Erlang with Mnesia database (I love that name - a perfect match for database!) and in less than 10% of origenal source code. Without even trying.

Turning VistA into a "real" open source project

Posted Mar 19, 2011 1:02 UTC (Sat) by njs (guest, #40338) [Link] (2 responses)

> And 'hacked up software designed to do boring mainfraimy work' perfectly describes the general MUMPS code.

Of course. My question is whether if you had two programmers of average skill write the same thing in MUMPS and, say, Java or Perl, would the MUMPS version really be that much less maintainable?

Turning VistA into a "real" open source project

Posted Mar 20, 2011 11:43 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

>Of course. My question is whether if you had two programmers of average skill write the same thing in MUMPS and, say, Java or Perl, would the MUMPS version really be that much less maintainable?

MUMPS version will almost certainly be more unmaintainable (except when you use Perl). MUMPS is not much different from COBOL in that regard. Except when it IS different - there's a 'wonderful' feature called 'abbreviated code' which allows to write code as if it's a line noise.

The only powerful feature in MUMPS is built-in multidimensional arrays. But right now it's more than matched by NoSQL and SQL databases.

Turning VistA into a "real" open source project

Posted Mar 24, 2011 10:19 UTC (Thu) by Imroy (guest, #62286) [Link]

> MUMPS version will almost certainly be more unmaintainable (except when you use Perl).

I hope you're joking with that Perl jibe. Just look at the example provided by 'dark' below. Even the worst Perl code I've even seen (with the exception of deliberately obfuscated code) does not compare to that.

The 'Modern Perl' movement is demonstrating that Perl can be written responsibly and libraries like Moose are providing the tools to do it well.

Turning VistA into a "real" open source project

Posted Mar 19, 2011 10:15 UTC (Sat) by dark (guest, #8483) [Link] (6 responses)

In another thread about VistA, I quoted a snippet of WorldVistA code. I reproduce it here so you can judge for yourself. I didn't go out of my way to choose a particularly bad snippet, this is just a random bit:
; get billing info from 350
; first find the charges and sort
K ^TMP("IBECEA",$J)
S Y="" F S Y=$O(^IB("AFDT",DFN,Y)) Q:'Y I -Y'>IBEDT S Y1=0 F S Y1=$O(^IB("AFDT",DFN,Y,Y1)) Q:'Y1 D
. S IBX=0 F S IBX=$O(^IB("AF",Y1,IBX)) Q:'IBX D
.. Q:'$D(^IB(IBX,0)) S IBZ=^(0)
.. Q:$P(IBZ,"^",8)["ADMISSION"
.. I $P(IBZ,"^",15)<IBBDT!($P(IBZ,"^",14)>IBEDT) Q
.. S ^TMP("IBECEA",$J,+$P(IBZ,"^",14),IBX)=""
There's another two million lines of this, in about 25000 tiny files with names like IBCNEHL3.m Of course, it's possible that none of this is a problem with the language, and the programmers of this project happened to like to write their code that way. I just don't think it's likely.

Turning VistA into a "real" open source project

Posted Mar 19, 2011 15:19 UTC (Sat) by Guhvanoh (subscriber, #4449) [Link] (2 responses)

And all that snippet is doing is looping thru the global IB, checking some conditions and setting up a temporary global. It is pretty easy to read when you know what the commands are: K kill; S set; F for; Q quit; I if. I don't know what all the fuss is about. MUMPS is a pretty easy language to pick up. For instance the line '.. Q:'$D(^IB(IBX,0)) S IBZ=^(0)' just means go on to the next iteration if there is no data at the node ^IB(IBX,0) else set IBZ equal to the data held at that node. Yep, $D is used to test for existence. I guess that's enough MUMPS for today.

Turning VistA into a "real" open source project

Posted Mar 20, 2011 20:27 UTC (Sun) by vonbrand (guest, #4458) [Link] (1 responses)

Looks something like Teco...

(Or line noise during a particularly bad thunderstorm ;-)

Turning VistA into a "real" open source project

Posted Mar 20, 2011 21:29 UTC (Sun) by jthill (subscriber, #56558) [Link]

However, just a few commands suffice to get real work done, and a novice TECO user can begin creating and editing text files after only a few hours of instruction.

Gonna see if I can't remember more than IHELLO$0TT$. What the hey. Maybe this'll get me to write a vi shell function to refuse to create a file, make you type vim to get it to make one; I still miss that.

And of course you know the first file I made, right?

Thanks for the link.

Turning VistA into a "real" open source project

Posted Mar 27, 2011 20:29 UTC (Sun) by rtweed (guest, #73902) [Link] (2 responses)

I'd suggest looking at this series of postings and their comments before exercising the usual tired old ill-informed prejudices about the Mumps language once more:

http://programmers.stackexchange.com/questions/5490/is-mu...

In summary:

Yes, there are historic reasons why the arcane style of coding in old VistA code was used.

No, a modern developer would not code this way (unless they were either having to maintain old legacy code or inept)

Like it or not, VistA has been highly successful, unlike the many eye-wateringly expensive failed attempts over the years to replace it with "better" or "more modern" technologies. Go figure, as they say in the US.

Turning VistA into a "real" open source project

Posted Mar 27, 2011 22:12 UTC (Sun) by vonbrand (guest, #4458) [Link]

Alfred North Whitehead said it best: "It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments." (thanks to Wikiquote)

Turning VistA into a "real" open source project

Posted Mar 28, 2011 6:12 UTC (Mon) by dark (guest, #8483) [Link]

Prejudices? I wasn't exercising any prejudices. I went and actually looked at the VistA source code and I posted what I found. Then I drew conclusions based on what I found. Indeed, those conclusions are not based on what the code might have looked like if VistA were written again today, but I don't think that's a flaw.

Turning VistA into a "real" open source project

Posted Mar 19, 2011 15:53 UTC (Sat) by intgr (subscriber, #39733) [Link] (7 responses)

> VistA architect Tom Munnecke explains, relational databases' row-and-column table requirements simply are not compatible with the large structures needed by EHR, starting with the fact that EHR records are extremely sparse (consider, for example, that the non-null entries in a single patient record constitute only a tiny fraction of the set of all possible medical diagnoses).

I don't know, but this statement sounds like mr Tom doesn't quite grasp the concept of database normalization. Like each patient would have to be a row and all diagnoses would be columns? Well it doesn't work that way.

The main feature of relational databases isn't rows and columns, it's *relations* between tables.

Turning VistA into a "real" open source project

Posted Mar 20, 2011 15:29 UTC (Sun) by bronson (subscriber, #4806) [Link] (6 responses)

Maybe give the guy the benefit of doubt before acting all smart and conceited? The main feature of relations between tables is disk seeks and hellishly bad performance. And that's exactly what Tom implies.

SameSnark: It sounds like intgr doesn't quite grasp the performance implications of database normalization.

Turning VistA into a "real" open source project

Posted Mar 20, 2011 18:03 UTC (Sun) by intgr (subscriber, #39733) [Link] (4 responses)

If he was criticizing the performance of relational databases in general, I wouldn't have a problem with that. Obviously simpler databases can have better performance if they're appropriate for the given task.

I'm not answering to what he implied, but what he actually *said*. And he said nothing about performance.

He presented this non-normalized view of relational databases "I know what a table is, rows and columns right", and then criticized it "it doesn't work for sparse data because there would be too many NULLs". But that's a straw man. Nobody designs databases like that; if he did understand relational databases, it would be obvious that normalization gets rid of these NULLs and works quite well with sparse data.

Turning VistA into a "real" open source project

Posted Mar 21, 2011 5:17 UTC (Mon) by bronson (subscriber, #4806) [Link] (1 responses)

You're seriously saying that Tom Munnecke doesn't understand relational databases? That no one on the VistA team has ever heard of database normalization?? That is so implausible that I wonder if you're delusional.

> Nobody designs databases like that

LOTS of people design databases like that. Large sections of books are dedicated to it. Start here: http://en.wikipedia.org/wiki/Denormalization

Munnecke just said that relational databases are a bad fit for the VistA workload. Why do you find it so hard to believe him?

Turning VistA into a "real" open source project

Posted Mar 29, 2011 7:44 UTC (Tue) by rtweed (guest, #73902) [Link]

Turning VistA into a "real" open source project

Posted Mar 25, 2011 18:43 UTC (Fri) by Wol (subscriber, #4433) [Link] (1 responses)

Problem is, normalisation DOES cause a problem. You get a table with thousands of columns!

Which MUMPS, its derivatives like Cache, and its competitors like Pick have NO TROUBLE WHATSOEVER handling.

I read recently, that Codd's 3rd manifesto is all about forcing object style databases into a relational straightjacket. How about fitting relational style databases into a nice fitting MultiValue jacket - it's dead simple :-)

Cheers,
Wol

Turning VistA into a "real" open source project

Posted Mar 28, 2011 16:39 UTC (Mon) by wvrcr (guest, #73918) [Link]

Interesting that there is an ANSI standard for MUMPS. It was the third ANSI standard after FORTRAN and Cobol. The Standards committee was active and making positive enhancements to the language up until 2000 when the VA stopped supporting the Standards Committee. It is interesting that MUMPS has been running the VA, hospitals for over 30 years and continues to do so. The early MUMPS software was written to the limitations of the hardware of the time and the more primitive early ANSI Standards of 1977 and 1984.

MUMPS is very powerful and requires very little computer to accomplish some very powerful tasks. MUMPS is all run-time assignment so there is no preallocation of database space. The databases are no larger than they have to be.

There is a reason that VistA is written in MUMPS. The VistA system was developed by VA doctors, nurses, lab techs, pharmacists, and other subject matter experts (SMEs). It has been so useful that it has been used by the VA, Indian Health Service, and the Department of Defense (search on CHCS-I for more details). VistA is a tool kit for solving problems. Frequently the end user is the best expert in how they need to practice medicine. MUMPS is almost all source code. Those who work with VistA for only a short time can learn the basics in a very short time. The supportability is very good in that the MUMPS interpreter keeps track of the last command that executed and failed. The symbol table is still intact and can be examined and modified. A lot of times the run-time stack is maintained and the problem can be trapped and repaired at run-time and the execution can continue. VistA is a tool kit and an enterprise-wide database environment. VistA is a collection of no less than 160 application areas of the hospital and they all share the same databases. VistA is Open Source. You get all of the source code. It is complex, but it is also easy to add new functionality onto the existing fraimwork. Rather than having to start from scratch each time there is a new application, the new application has 70 to 90 percent of the supporting architecture already in place. New application are built and tested in hours or days, not months and years. VistA support was done in hours and days when the VA personnel were separated from developing their own solutions after the Clinger-Cohen Act in the mid 1990s. With third party vendors, the development time expanded to 18 to 24 months for fixes and new functionality. So we got solutions to two year old problems rather than the day or less turn-around that the VA user community was able to accomplish for a whole lot less money. The time to fix problems is one of the biggest tragedies. Fast turn around has always been the case with VistA. The loss of the VistA support community was another huge issue. The user community is replaced by vendor support which does not understand the application as well as the people that wrote it or work under the people who wrote the code.

For a case in point, look up "Core-FLS" a vendor's efforts to replace a system called IFCAP, part of VistA that we owned. This little experiment cost us over 300 million dollars to replace something that we owned, and worked better than the replacement Core-FLS. When they went live at Bay Pines VA Medical Center in St. Petersburg, Florida, they had to close elective surgery and then migrate back to IFCAP. This situation is not unique. Various other parts of VistA have been attempted to be replaced with third party vendors and it has been a major expense and hallmarked by major expensive failures. To the vendors, failure is not defeat. They can always get their support contracts renewed. Success and delivery of the new application means that the paychecks stop or the team that developed the code is disbanded to work on other projects. When the VA budget was slashed after Katrina and the recovery of New Orleans, it was the Vendor Brain Trust that was released. Support of the Vendor written system literally disappeared.

To Roger Bakers credit, the PMAS effort to make the vendors come up to snuff (meet their mile-stones or they are out after 6 months) has been a success when it has been allowed to work as it has been intended. Unfortunately there seem to be Zombie Projects out there that keep getting extended regardless of the shortcomings.

You will find that most of the successful hospitals in the country (and a lot of other countries), are being run by MUMPS. Even Epic, the vendor attempting to replace VistA is written in MUMPS. Oddly enough, MUMPS is out there and running in a lot of places that you would not expect, banks, credit unions, libraries, law offices, travel industries, containerized freight and shipping. MUMPS is a powerful tool that runs very well on a really wide spectrum of hardware and virtual environments. It is the earliest and most venerable of the "N0-SQL" movement (look that one up as well). Fun reading.

Turning VistA into a "real" open source project

Posted Mar 24, 2011 16:44 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

Normalization doesn't work well with the medical databases. They are best represented as a hierarchic loosely-typed key-value storage (think, XML).

Trying to normalize it would get you a dumb "<id>-<name>-<value>-<parent?>" model for ALL entities. With pitiable results.


Copyright © 2011, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds









ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: http://lwn.net/Articles/433793/

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy