Presentation #: 202
The HP3000, Healthcare and the Internet
Dan MacFayden
LPTP
501 – 250 Bloor Street East
Toronto, Ontario Canada
Phone:416-323-9540 ext. 240
Fax: 416-323-9324
Email: macfayden@lptp.on.ca
This presentation covers a Health-Care External Quality Assurance program’s journey from a safe, secure and reliable technological harbour, the HP3000 and its proprietary operating system MPE, onto the turbulent and perilous sea known as the Internet.
I’ll begin by describing why we felt this journey was necessary even though we knew we were venturing into uncharted territory equipped with limited experience, equipment and money. Along the way I’ll review some of the organizational and social conditions that influenced the directions we took and describe the additions and modifications we made to our legacy environment to make it more Internet sea-worthy.
The uncharted nature of our adventure is manifest as I make this presentation at HP-World; I can explain how I got here but I’m not sure this presentation is in the right track. This is supposed to be the ‘Internet and E-Commerce’ track. The emphasis in that title makes sense since the majority of discussions regarding business use of the Internet are centered on ‘E-Commerce’. For the past several months all the airline magazines have contained articles declaring that an E-Commerce presence is necessary for our businesses to survive. (That is assuming our businesses are going to survive the Y2K crisis, but that is a completely different track.) By now every CEO knows they have to have some of that ‘e-commerce’ stuff if they are going to keep up with the competition. Nonetheless there are opportunities to deploy Internet-based applications that are not commercially oriented but still deliver bottom-line cost savings and quantifiable service-level improvements.
Our organization, the Laboratory Proficiency Testing Program, which I’ll refer to from now on by its acronym ‘LPTP’, is a not-for-profit organization that provides external quality assessment to laboratories that provide human in-vitro medical testing services in the province of Ontario. Our role is to insure that when a patient’s test samples are submitted to a laboratory for analysis the patient that sample came from has a reasonable expectation that the results reported to their doctor are accurate. Examples of the kind of testing we monitor include:
Our prime concern and the prime concern of our patient-clients therefore is not ‘E-Commerce’ it’s ‘E-Quality’. Instead of a discussion of E-Commerce this presentation will describe and demonstrate how we have utilized electronic database access, data collection and information dissemination to deliver better services and Quality of information to our customers, the people of Ontario, Canada. That’s how this presentation found its way onto the Internet track of this conference.
But there are other tracks this presentation will travel along. LPTP’s concern for Quality (we are, after all, an External Quality Assessment service) is a prime reason why LPTP’s mission-critical systems have been developed and deployed on the HP3000’s MPE platform, so much of this presentation will be travelling along an MPE track as well. And since we discovered along the way that the HP3000 and MPE weren’t ready to support the type of Internet presence we required when we required it there is an NT front-end to our Internet deployment so we’ll be going down the NT track too.
You may have noticed that it didn’t take very long before I switched metaphors from a nautical to a railroading theme. That wasn’t my original intent but the switch wasn’t accidental. I started out with what seemed like some pretty clever seafaring images but the organizers of this conference had already established the track theme so I had to switch metaphors to fit in. The need to change course is a frequently recurring necessity on an Internet application development journey and you’ll need the controlled flexibility to be able to do this while staying on track, or staying afloat, or both.
In the Beginning or ‘Why We Went To Sea’
LPTP has formally existed for 25 years and operates under authority governed by provincial legislation that authorizes us to carry out proficiency testing on all licensed laboratories in the province of Ontario. It is a condition of holding a laboratory license that the licensee participates in the LPTP proficiency-testing program. (In the United States, the Health Care Financing Administration (HCFA) similarly stipulates that laboratories performing moderate or high-complexity testing must enroll in and successfully complete a Proficiency Testing program.) You could call this a captive market. This would seem to put us in an enviable position since our clients don’t really have a choice, either submit to our testing or don’t open for business in our territory. Nonetheless the stresses on our operating model over the last decade have been severe enough to require significant action if we are to remain a viable program.
Ontario covers an area of 1.1 million square kilometers (413,000 square miles). This is larger than France and Spain combined. The Ontario population continues to grow, mainly through immigration, and is currently around 11.4 million people. While the majority of people live in the major urban concentrations along the north shore of Lake Ontario, the health-care system must provide publicly funded universal access to services for all residents throughout the province. This is a distance that spans more than one thousand miles from North to South and more than one thousand miles East to West.
Health care in Ontario consumes 35% of provincial annual government revenues. Spending increases of more than half a billion dollars per year have not kept pace with the rising cost of providing health-care services to an aging population. The government of Ontario recognized some years ago that the increasing cost of universal, publicly funded health care was unsustainable and began a process of reviewing and restructuring the elements of the health-services system. Many health-care sectors, including acute-care hospitals and support services, such as the laboratory sector, began receiving considerably less funding than they had in the past. So while costs continued to increase, LPTP’s funding did not keep pace.
In spite of the tightening economic situation a significant additional cost had to be incurred in order to upgrade LPTP’s computing capability. A classic fire-fighting environment existed; in the early 1990’s over 85% of I.T. staff resources were consumed maintaining and modifying the patchwork systems that had been developed on a classic HP3000 using the software product ‘Relate/3000’ that was no longer supported (the vendor had gone out of business).
The business-case justifying the upgrade was based on a return on investment (reduced staff maintenance, improved productivity, etc.) of less than 3 years. The ease-of-operations, reliability and lower total-cost-of-ownership than any alternative, made continuing in the HP3000 computing family an easy decision. The classic HP3000 was replaced by another HP3000, a Series 937 LX.
The choice of database software was not as easy. The unique nature of a laboratory Quality Assurance application required a relational model that could not be built on the transaction-oriented architecture of the native HP3000 database system ‘IMAGE’. (The early 1990s was before IMAGE-SQL was ready for prime time.) ALLBASE, Hewlett-Packard’s relational offering, was considered as were two other relational database packages that were available on the HP3000: ORACLE and INGRES. INGRES was finally selected due to a combination of superior capabilities and lower price (more capabilities than ALLBASE, cheaper than ORACLE). At the time, INGRES was an ASK company. In addition to Ingres, ASK also offered the MANMAN applications on the HP3000 and since Hewlett-Packard held an equity position in ASK, Ingres looked like a good long-term choice for HP3000 software development.
The timing for the HP3000 upgrade was premature. Anyone could have anticipated the subsequent events that were not taken into consideration when this decision was made. In the next 2-3 years:
MESSAGE # 1 – If You Can Predict the Future Please Give Me a Call
If the decision to upgrade the HP3000 had been held off a couple of years until the mid-90s LPTP would probably have jumped ship and swum for a UNIX harbour. There, any one of the raft of RDMS offerings (Sybase, Informix, etc.) that were available might have been chosen. If the decision had been held off yet another two years LPTP might have gone directly to a Windows NT platform. But time, tide and e-world wait for no one. The Internet was beckoning and the investment in Ingres and MPE had already been made.
MESSAGE # 2 – You Can Wait Forever To Get The Perfect Solution. If You Need to Do Something, Work With What You Have.
For us, access to the Internet was not on account of a startling realization that a brave new world of technology existed beyond our physically secured HP3000 server. LPTP has committees of experts in the several disciplines of laboratory medicine that review the results of our surveys. These committee members are medical practitioners associated with teaching hospitals and universities around the province. We also exchange information with international health care providers. The Internet has been around for at least twenty years and many of our medical colleagues already had several years experience with Internet access provided through their affiliated institutions. They had already been exposed to such arcane technological wonders as the Unix shell, ftp, telnet and the all-time friendliest e-mail software package ever written, ‘pine’.
LPTP needed to offer a comparable service to our colleagues. Our original Internet relationship was a UUCP (Unix to Unix Communication Protocol) connection via a gateway from QuickMailâ , our Mac-based internal e-mail application, that provided scheduled e-mail exchanges through our ISP’s (Internet Service Provider’s) server. This allowed us to communicate with our remote colleagues more efficiently and economically than via phone, fax or paper. (And it was a lot easier to use than pine.)
E-mail was and still remains the greatest practical benefit of Internet access for most people who use the Internet. E-mail offers the most cost-effective means of broadcasting non-sensitive data for any organization that relies on timely communication and data sharing among individuals whose physical locations are far apart.
MESSAGE # 3 – If You Already Have the Perfect Solution, Something Will Come Along That Will Force You to Change.
This AppleTalk Internet connection was relatively inexpensive to set-up. The Mac’s ease-of-use, like the HP3000’s, made this network easy to support. There was only one Internet service offered, plain vanilla e-mail, no ftp, no telnet, no world-wide-web. No file-attachments. We had no security worries. The AppleTalk network had a limited range of about 200 feet that was not accessible outside the office. Our HP3000 was accessible only via a direct connection and the HP-Support modem that was always on. (And since the only people who could access the system were the on-site staff it was unnecessary to have passwords on all the User Accounts.)
In the first half of the 90s we were in the same economic boat as everyone else. We needed to reduce costs and increase productivity without increasing resources. Since our internal processes had already been perfected during the previous recession technological advancements seemed to offer the only opportunity for achieving these objectives.
We needed to improve our service by improving the Turnaround-Time for reporting our clients’ Proficiency Testing results, and at the same time decrease our costs by reducing the amount of paper and labour that was required to capture data and distribute the subsequent reports.
At that time the cost for direct Internet access via a commercial Internet access provider was still prohibitive to individuals and small organizations. On the other hand, Electronic Bulletin Board Services accessed by low-cost, high-speed 14.4 modems offered the same potential communications and timesavings. We charted a development course to implement a First Classâ electronic bulletin board, hosted on an Apple platform, for our clients.
MESSAGE # 4 – If You Think You Have Found the Perfect Solution, Something Will Come Along That Will Change Your Thinking.
There were two significant sea changes during our evaluation and initial development of the LPTP BBS:
People were now realizing that E-mail’s file attachment capability is a far more convenient method of sharing files than ‘sneaker-net’ (diskette mailers). As more and more colleagues and clients moved to Windows 95 and acquired e-mail it became difficult to take advantage of this file-sharing capability from the Mac side of the exchange.
Access to the World-Wide-Web via a browser meant it would now be possible for our staff and consultant experts to access the vast wealth of medical and laboratory-related information available via the Internet without having to refer to their pocket Internet-reference guides.
Our monochrome, 4 Meg RAM, 20 Meg Hard-drive Apple SE desktop computers needed to be upgraded; overnight we jumped ship and boarded the Microsoft-Intel cruise-line. Like LPTP, Microsoft had also recognized the Internet sea change and bundled their browser (Explorer) with Windows 95 and since we didn’t have an emotional attachment to any other browser software, we opted for the freebie.
For our Network services we opted to replace our AppleTalk network with Windows NT. Version 3.51 of this Network Operating System was almost as easy to set-up and support as our AppleTalk network although not quite as robust. When NT 4.0 was released, and subsequently patched with Service Packs 1, 2, 3 and 4, our network became much more stable.
LPTP is an information-disseminating organization and like many organizations our initial venture onto the Internet was intended to establish a ‘presence’. The advent of the World-Wide-Web afforded us the opportunity to provide access to our services to clients beyond the limited range of our traditional distribution network. While we were upgrading our internal network we were also able to establish our Internet presence with a relatively modest investment in in-house HTML development, a dedicated ISDN Internet connection and a web-hosting arrangement with a regional ISP.
MESSAGE # 5 – What the Point?
This preamble has been provided to explain how we accumulated our current Internet-fleet of hardware and software:
HP3000 Series 929 – MPE 5.5
Ingres RDMS – 6.4/04 for MPE
Windows NT Server 4.0 with IIS 4.0 and Active Server Pages (ASP)
It also shows we hadn’t devised a master plan that was merely waiting for the Internet to show up in order to be implemented. Technology decisions had to be made to insure the continuity of our organization. The concurrent explosion in Internet-based tools and services helped influence our choices but they were not the determining factor. I’m pretty sure there are a lot of organizations that still have to endure the pejorative references to their ‘legacy’ systems who might not be around today if they had neglected their ‘back-ends’ in pursuit of a glossy Internet presence. A back-office suite of robust, mission-critical applications is much harder to establish than an Internet presence. The challenge for many organizations now is to merge Internet ‘presence’ with their line-of-business applications, without compromising those businesses.
Our challenge was to provide access to a ‘legacy’ Ingres database residing on the proprietary MPE Operating System. Like most organizations, we consider ‘our’ data to be sensitive and confidential. The need to insure its integrity and confidentiality is paramount. It is possible to use an HP3000 as a transaction server for web-based activities, for example HP has certified the Apache Web server for MPE/iX 6.0. However, the lack of SSL (Secure Sockets Layer) in the Apache product on the HP3000 should raise a red flag for anyone needing to provide a secure service.
In a physically restricted world, the access control capabilities within MPE give the HP3000 community every reason to feel safe from intruders if a comprehensive security plan is implemented and constantly monitored. We at LPTP used to feel a little smug because our ‘proprietary’ operating system was unfamiliar to our stereotypical image of a hacker who lives and breathes Unix/DOS/NT but has never heard of MPE. In the wired and unrestricted world of the Internet that’s a small deterrent to anyone willing to make an effort to crack your system. There are some very well organized sites that offer advice on how to break into any operating system. For example, there are sites on the Internet that contain documentation on how to recognize MPE and its default accounts.
While Internet security on MPE remains suspect, a general perception also remains that Windows NT is not secure or stable enough to host a mission-critical Internet site. NT 4.0 might not yet offer the mission-critical reliability and dependability of an HP3000, but real-world experience at high-traffic sites has shown that sites built on NT can be as secure and stable as they need to be.
We added an NT-based firewall in front of our HP3000 data source. There are multitudes of Unix or NT firewall products on the market. For anyone selecting the right software there is going to be a balancing act between requirements, ease of management and cost. I’ll leave the evaluation of these alternative to another track; we implemented an AltaVistaâ Firewall and so far it has done the job.
Access to our data is provided by utilizing the Internet services available under MPE 5.5 in concert with tools that are available on our NT platform. The NT platform delivers excellent performance plus it has the widest variety of programming tools and 3rd party add-ons of any Internet Server. Arguably, Windows NT Server 4 along with the Internet Information Server 4 component (IIS) is the best Web front-end platform available today.
A big reason for this is the use of Active Server Pages (ASP) for creating dynamic Web pages. It supports a number of scripting languages such as Java Script and VBScript that make it easy to access software components residing on external servers. This is done using COM, the standard for Windows component interoperability.
If you have an IMAGE, ALLBASE or Oracle Database, you can get an ODBC driver so data exchange between the HP3000 and Windows-clients is relatively straightforward. We were required to develop an alternative data access mechanism since none of the usual ODBC driver providers have seen any profit potential in providing one for Ingres. Fortunately the tools were available for us to do this. A Microsoft COM component was written in Visual Basic that encapsulates the establishment of Telnet and FTP connections with the HP3000. This Connection Object can now be used in any programming language such as Visual Basic, C++, or ASP, that supports the COM object model.
After our client session establishes the connection to the HP3000, the client application composes its database request using SQL. This is passed to the COM object via a command file that is uploaded via FTP to the HP3000. The Telnet portion of the object runs using the redirected input from the SQL command file and redirects its output to another file that is downloaded via FTP back to the client. This file is then parsed by the connection object and returned to the client application program as a standard ADO (ActiveX Data Objects) Recordset. Any errors returned from the Connection Object are raised as standard error events.
This approach is not as quick as native database calls: the initial session connect time is approximately 1.5 seconds and total response time for data retrieval is 2.5 – 3 seconds. For most web-based processing this will still be considered acceptable, particularly for moderate or low transaction volumes. For us it has the advantage of providing a no cost alternative to a $150,000 custom ODBC driver.
Our clients can access our Web-Site to submit responses to surveys and questionnaires instead of a traditional paper-based information exchange. The benefit for us is a direct saving of data entry time and effort. The benefit to our participant laboratories is an immediate and accurate record of their response, unencumbered by the whims of the postal system, the long-distance telephone carriers or the best interpretive efforts of LPTP’s data entry resources.
The following example demonstrates the process we use:
Medical laboratory technologists will use HP3000-based proficiency testing applications to define the parameters for a survey. The parameters determine which discipline of laboratory medicine the survey pertains to, what laboratory tests are to be conducted, and the laboratories (based on their licensing status), that are expected to participate in the survey. This information is used by an NT-based web-application, written in Visual Basic, which generates dynamic web pages representing the electronic survey/questionnaire the participants are to complete. On a date scheduled by the technologist, the Ingres surveys application generates an e-mail message containing a description of the survey along with its due date and a hyperlink to the log-in site. This e-mail message is distributed to the directors of the laboratories participating in the survey.
The laboratories then submit their results via secure (SSL) web pages that do real-time updates to an SQL-Server database on a Windows NT-Server. These results can be modified by the participating laboratory until the survey due date has passed, when they are committed to the HP3000 database for analysis and reporting.
After the survey due date, the ‘correct results’ for the survey, are retrieved from the HP3000 Ingres database and made available via the LPTP web-site. Another trigger sends an e-mail message to the survey participants when these ‘reference’ results are available and includes a hyper-link to the log-in screen for reviewing these results.
Work that is currently in progress will soon permit secure web-access by laboratories to their own historical performance data residing on HP CD-ROM servers installed on our internal network.
Combining the back-office performance, reliability and integrity of HP3000-based applications with a Windows NT porthole to the Internet has provided us with a flexible, reliable, cost-effective and High-Quality foundation to deliver ‘E-quality’ services to our clients.
Our software and hardware are too specific to LPTP’s mandate (i.e. to deliver high quality external quality assessment to laboratory service providers), to provide an off-the-shelf solution to anyone else. I don’t think we’re unique in that regard.
MESSAGE # 6 – Were’re Unique. You’re Unique.
Each enterprise has specific application needs based on the history and the ongoing objectives of the organization, not just the need to be interconnected via the Internet. Our IT challenge, as it has always been, is to provide a stable environment that allows interaction among a variety of constantly-evolving applications built with an increasing variety of tools over ever-changing communication media.
MESSAGE # 7 – Steady As She Goes!
If we were to start now there are so many additional options available that it’s almost certain we wouldn’t have taken the same route. But if we’d waited we wouldn’t be where we are today. And where we are today is not where we’re going to be a year or even six months from now. The journey is far from over – it’s never over until it’s over. We’ve covered a lot of distance and there are open seas ahead.
MESSAGE # 8 – Smooth Sailing!