HP World '99 and ERP World '99 Conference Proceedings

As computer systems continue their march into the production environment towards direct machine control, the demands imposed on these systems frequently run head-on into the enterprise solution. Inevitably, an engineer will insist that only a “real-time” box will work for the application, and that “NT isn’t real-time!”

What is a real-time computer? Is their really anything unique about a real-time computer? More importantly, when does the application of a real-time computer outweigh the costs and support issues associated with these machines? Even the definition of a real-time computer has been blurred by the advance of computer systems. An application that would have required a real-time box several years ago may well be handled by a desk-top machine today. This paper will attempt to take some of the mystery out of these real-time boxes and their use.

Real-time systems are as unique as their system requirements are typically driven by the process or machine they are connected to -- not the users at the other end. These requirements are not merely design decisions, but are driven by very real physical requirements such as the Niquist sampling rate and response times to process events. Unlike a desktop system, a slow response is not merely an inconvenience -- in a real-time system it’s a failure, frequently causing a loss of production or even damage to equipment.

As we will see, a real-time computer has some unique features to satisfy these system requirements such as deterministic process scheduling and fast interrupt response. These unique features do however add to the software development effort as does the general lack of development tools and shrink-wrapped applications common on the desk-top.

The decision to use a real-time system is driven by costs, support and system requirements. These boxes are generally more expensive in both equipment costs and in support, from installation on. Inappropriate use or non-use of these machines can be costly. With the information presented here, a more informed decision about real-time computer can be made.

There are perhaps as many definitions of the term “Real-time Computer” as there are vendors touting their “real-time” solutions. For our purposes, we will define “Real-time computer” as one in which the correctness of the operation of the computer depends not only on the accuracy of the computation, but also on the precise time that the calculated result is available. For example, consider the operation of a VCR, programmed to record a TV show. To record the show, not only must the VCR must record the desired channel, but must do so at a particular time. Failing to record the correct channel or starting to record at the time, either

too early or too late, is equally bad. Only if the VCR records the correct channel at a precise time is the operation considered correct -- hence the concept of real-time. This property is frequently referred to as a deterministic response.

A strict interpretation of this definition implies that failure to deliver a result precisely at a given time would constitute a failure. However in many applications, this timing constraint can be relaxed to some extent. In these applications, providing a result within a particular period of time is all that is necessary for proper operation of the system. Consider our VCR example: is it really necessary that the VCR starts to record a program within 1 uS (millionths of a second) of its programmed start time? In practice, as long as the recording starts within a few 10s of seconds, that should be close enough. This type of system does indeed have very real timing constraints; however, these specifications constrain us to a time window, not necessarily to a precise time.

In each scenario, the timing requirements constitute one of the basic requirements for proper operation of the system. The environments of the system specify these timing requirements, if the system cannot fulfill the needed timing, the system fails as much as if it provided an inaccurate calculation. Exactly what timing requirements the system environment impresses on the system can in of itself be difficult to quantify.

How Fast is Fast Enough?

Most are familiar with one of the more useful control applications of recent years -- cruise control. One merely sets the desired vehicle speed, and the cruise control will maintain the speed, on level terrain and over hills and through valleys. The design of such a function involves an area called control system theory. An in-depth discussion of control theory would, unfortunately, require several volumes more of material; however, some basic ideas can be presented in this context.

One of the basic and most common control concepts is the closed loop system. In this scheme, the value to be controlled is compared to a desired value or setpoint. The output of the control system is based on the error value of the loop, defined as the difference between the desired value and actual value. Various algorithms can be used to calculate the output, one of the most common is called PID for Proportional + Integral + Derivative control. PID control bases its output on the error value of the loop, the time integral of the error value (how long has the error existed) and the time rate of change of the error (derivative). Properly implemented, a PID loop can provide very good control.

A Little History

Engineers have been controlling their machines since the advent of the industrial revolution in the 1800’s. These controls were mostly mechanical in nature, using governors, cogs and various linkages. With the advent of electronics in the first part of this century, a move to analog electronic controls was made for several reasons, including reliability and ease of modifications and support. These machine controls were based on analog signals such as thermocouple temperature measurements and tachometer speed measurements that provided a voltage level that corresponded to the physical quantity. Electromechanical relays were also commonly used for on-off machine functions. These electromechanical elements had very quantifiable timing parameters: the speed of relays was reasonably well known, the bandwidth of analog amplifiers and control circuits was measurable.

With the advent of digital computers in the 1960s, attempts began to use these computers to control machines. The benefits of computers were obvious, modifications of the control system would no longer require the use of hammers, wire and soldering irons! Complex control schemes that were difficult or impossible to achieve with traditional analog controllers were readily implemented with digital computers. For Boolean on-off machine control functions, digital computer support was straight-forward. However, many machine control functions such as temperature or speed control are really analog functions that need to be implemented in the digital domain on a computer.

Digitizing Reality

In order for a digital computer to implement what is essentially an analog function, the input values must be converted from the analog domain to the digital domain by a device called an Analog-to-Digital Converter, or A/D Converter. Analog outputs on a digital computer are driven by a similar device, a Digital-to Analog Converter, or D/A Converter. These converters provide a sampled value of a signal at a particular point in time; however, these analog signals are usually not static. They are changing in value with respect to time. This time component is equally important, and needs to be measured as well. Only the combination of a signal value and its time component can the analog signal be represented in the digital realm. As you may recall, a PID loop derives its output based on this time information as well as the signal magnitude.

The Nyquist Sampling Theorem

As we have seen, the time content of a signal is as important as the value of a signal. To preserve this time information, just how fast do we need to sample the signal? The Niquist Sampling Theorem tells us that we need to sample the signal at twice the rate of the maximum frequency component of the signal in order to be able to accurately reproduce the signal. Lets look at an example to help understand this concept: the audio CD.

Music is stored on a CD in digital format. The source of the music was digitized by sampling the sound at a rate of 44 kHz. The stream of amplitudes is then written in digital format on the CD. To reproduce the music, the digital values on the CD are read and converted back to sequential analog levels at the 44 kHz rate. The Nyquest Sampling Theorem tells us that at a 44 kHz sample rate, the maximum frequency the CD system can reproduce is 22 kHz. But what does this maximum frequency have to do with reproducing the sound of a piano, a complex and rich sound? The Fourier Theory states that any complex signal can be represented as the sum of a series of simple sine waves or individual frequencies of particular magnitudes. Many home stereo systems include a graphic equalizer and a real-time spectrum analyzer consisting of many vertical display bars representing various frequencies in the music being played. This spectrum analyzer is providing the frequency content of our complex music source.

Control system signals are governed by the same principles as the CD player, but the frequencies involved are generally in the 10 Hz to 1000Hz range. Considering these values, real-time systems must accurately schedule processes to run every 500 uS to 100 mS. If the computer fails to run a process in the time required, several errors are introduced into the sampled signal, resulting in signal distortion. Timing errors of 1% are a good target, providing the real-time computer of only a 5 uS window to produce a result if a 1000 Hz signal bandwidth is desired!

As we have seen, computers designed to control a machine has very exacting timing requirements if they are to satisfy the demands of accurately controlling the machines they are attached. It should now be apparent where the definition of “real-time” originates: “the correctness of the operation of the computer depends not only on the accuracy of the computation, but also on the precise time that the calculated result is available.” This determinism is what distinguishes a real-time computer from its brothers.

Hard Real-time vs. Soft Real-time

The most demanding real-time applications such as fly-by-wire aerospace applications require high degree of hardware -- software integration to satisfy the timing requirements of the system. These types of systems have become known as Hard real-time systems. Depending on the exact requirements, few off-the-shelf components may even exist, so much of the hardware and software development ends up custom for the application. This can be costly, but is the only way to fulfill the system requirements.

However, there are many applications where the timing requirements are not as stringent as a fly-by-wire system. As the time window for a system action increase, there is a growing number of off-the-shelf hardware and software solutions available. These systems are commonly referred to as Soft real-time systems. But as there is no exact definition to this term “soft real-time," these systems must be evaluated as to the suitability to the intended application. This will be the focus of this presentation.

How do real-time computers excel?

Real-time computers provide this execution time precision by guaranteeing a maximum interrupt latency, a priority driven context switching mechanism and a high resolution system clock.

Interrupt latency

Most modern computers are interrupt driven, that is, they respond to external inputs is some way. For instance, when a user depresses a key on a desktop keyboard, a signal or interrupt is generated to direct the operation of the computer to read the key that was depressed. The computer responds by interrupting the normal execution of the software and jumping to an interrupt handler to process the interrupt signal. This mode of operation is much more efficient than constantly having to scan the keyboard to see if a key has been typed. However the use of interrupts does add a degree of complexity to the computer, as interrupts can occur at any time, no matter what the computer happens to be executing. Because there are instances where responding to an interrupt could cause the computer to malfunction, interrupt handling is turned off during execution of critical areas of the software, which usually includes executing many parts of the Operating System. For a desktop computer, delaying the handling of an interrupt for a period of time is not usually not even noticeable, but this delay for a real-time application can be disastrous.

For this reason, the very design of real-time computer software minimizes the time interrupt handling is turned off. But the problem of protecting critical areas of the software execution remains, complicating the design of any software for the real-time system. Table 1 lists the published maximum interrupt latency of several operation systems.

Manufacturer	Operating System	Hardware Platforms	Interrupt Latency
Hewlett-Packard	HP-RT	HP 742rt, 743rt, 744rt	100 uS
Silicon Graphics	IRIX	SGI Workstations	200 uS
Lynx	LynxOS	x86 PC, Power PC	14 uS
Microware Systems	OS-9	x86 PC, Power PC	3 uS
QNX Software	QNX 4	x86 PC	4.3 uS
Microsoft	NT 4	x86 PC	Not Specified
Microsoft	Windows CE	x86, Power PC, MIPS	Not Specified
Hewlett-Packard	HP-UX	HP-9000 Series	Not Specified
VenturCom	RTX for NT	x86	50 uS

Table 1

Interrupt Latency for several Operating Systems

Notes:

1. The exact Interrupt Latency depends on many factors, including the speed of the target processor. The reported figures should be considered “typical.”

2. Microsoft NT and CE, and HP’s HP-UX are not marketed as real-time Operating Systems, but are included for reference only.

Interupt Nesting

The interrupt latency times contained in table 1 reflect the maximum time between the occurrence of an interrupt and the execution of the interrupt handler. However, most computer systems contain multiple source of interrupts, from system timers to serial ports. To coordinate all these interrupt sources, a mechanism in hardware is used to prioritize the interrupt sources, so the higher priority interrupts will be handled before the lower priority interrupts. However, once a particular interrupt starts to be serviced, a higher priority interrupt will probably not be serviced until the lower priority interrupt handler finishes and re-enables the interrupt system. For this reason, the device drivers and associated interrupt handlers in real-time systems are designed to execute as fast as possible. This does limits the functionality that can be included, making complex I/O devices drivers like network interfaces more difficult to support if one needs to limit the interrupt latency in a real-time system.

This is where the differences between a real-time operating system and a more general purpose O/S like NT and HP-UX become apparent. The interrupt system supported by NT on a desktop PC is designed to support a wide variety of devices, a feature appreciated by NT system administrators. However, the resulting complexity of interrupt handling makes predicting the interrupt latency in an NT system almost impossible to characterize. Adding an additional device to an NT system or even updating a device driver can have a profound impact on the interrupt latency in an NT system.

Process scheduling and execution

A soft real-time system usually consists of several processes of varying real-time requirements running under an operating system responsible scheduling the processes so they complete their calculations on the required deadlines. The process scheduling mechanism is usually priority based to allow critical task to be completed at the expense of lower less critical tasks. A critical scheduling timing parameter is the Process Switching Time, defined as the amount of time the system takes to stop execution of the currently executing process and start executing a higher priority process. Table 2 contains a summary of the context switching times of several operating systems.

Scheduling of real-time tasks to complete on a deadline depends on knowing how long the task is going to take to execute. However, several features of modern computers can cause a great deal of uncertainty in execution time. The largest contribution to this uncertainty is the use of a virtual memory system, common in operating systems such as NT and UNIX variants. In a virtual memory system, system RAM is allocated to a process as needed, with an area in a hard disk used to store memory contents not immediately needed to execute the current processes. The use of a virtual memory system allows the execution of processes requiring much more memory than is present in the computer. However, if a process executes code that is not currently in RAM, an unexpected delay of several milliseconds will be introduced into the execution of the process that was not accounted for when calculating the execution time of a process. For a process interacting with a human, this delay usually is not noticeable; for a real-time process, it can be disastrous. For this reason, a high priority process usually can lock itself in memory to eliminate virtual memory delays. This however does limit the amount of RAM available for the other processes to execute.

For these reasons, the code size of the operating system and processes in a real-time computer is much more important than more general purpose systems, as all the operating system and real-time processes must fit into available RAM. As can be seen in table 2, the size of the real-time operating systems and the minimum size of processes are considerable smaller than more general purpose operating systems.

Operating System	Context Switch Time	OS Kernel Size	Minimum Process Size
HP-RT	30 - 50 uS	1.5 M to 7.5 M	46 K
IRIX	50 uS	N.A	N.A
LynxOS	4 uS to 19 uS	280K to 4M	1073 bytes
OS-9	3 uS	88K o 156K	10K
QNX 4	1.95 uS	40K to 84K	250 bytes
NT 4	Not Specified	16 Mb min rec.	Not Specified
Windows CE	Not Specified	Not Specified	Not Specified
HP-UX	Not Specified	Not Specified	Not Specified
RTX for NT	N.A.	250K min	N.A.

Table 2

Context Switch Time and Code Size for several Operating Systems

Real-time systems have traditionally been found primarily on the factory floor controlling process temperatures, pressures and speeds. These systems have evolved from large mini-computers to PLCs. As plant information systems have continued to evolve as well, attempts are being made to combine the functions of both systems in a single computer.

The PLC

With the advent of the microprocessor in the 1970’s, early attempts at incorporating these processors into the real-time control of machines evolved into what became known as the PLC, or Programmable Logic Controller. These PLCs were designed to replace the electromechanical relays commonly used to control the operation of machinery. As the capability of the PLC increased, additional functions such as PID control loops were added, making the PLC a very effective tool for machine control.

The PLC is a successful real-time computer system largely because the hardware and software were designed as a single integrated unit by a particular manufacturer. The PLC was designed for a single purpose -- machine control -- which made the design of the real-time aspects of the PLC much simpler. This design constraint however, has made integration of these systems with other factory floor computer systems difficult. Much of the difficulty arises because of the non-deterministic nature of the communication with non real-time systems and the required deterministic response the PLC needs for successful machine control. In many instances, a dedicated computer must be used as an interface between the PLC system and other plant information systems.

The Soft PLC

The integration of PLCs into a factory floor information system has usually involved an addition of a dedicated computer to act as an interface between the deterministic PLC and the non-deterministic information network. Efforts at combining the PLC and dedicated computers have evolved into the concept of a soft PLC, a standard computer system with machine control capabilities. However as we have seen, machine control requires a very deterministic response from whatever is used to control it, a PLC or other computer system. The debate continues over the nature of the computer system hardware and software to use for both machine control and plant integration support.

Real-time hardware

Real-time computers are heavily dependent on the system hardware for the deterministic operation of the systems. There are several commonly used “off the shelf” hardware platforms in use for these systems including VME, PC based and Compact PCI.

VME Based systems are based on the VME bus and a standard card form factor. The speed and properties of the VME bus are well known and easy to characterize. Traditionally, VME has been the bus of choice for high performance real-time systems.

PC Based Systems continue to be attractive due to the cost advantages enjoyed by PCs over other traditional real-time hardware platforms. However, the traditional PC architecture was not as well suited for real-time applications as other platforms. With the advent of the PCI bus, the PC has enjoyed a growing presence in real-time applications.

Concerned with the reliability of PC based factory floor systems, the Compact PCI hardware platform was developed to address several of the hardware shortcomings of PCs while maintaining the cost advantages of a high volume bus architecture. Compact PCI continues to gain market share on the factory floor, and may well be the platform of choice in the near future.

The development of highly integrated I/O system has changed the real-time environment. These I/O systems are capable of many timing dependent functions traditionally handled by the real-time system software. Therefore, the demands on the real-time system can be relaxed to some extent, allowing the use of more off the shelf components in real-time systems design.

Off the Shelf Software

Today, a growing list of off the shelf real-time software packages are available for the factory floor. However, the use of software packages such as Labview and Wonderware must be evaluated with the operating system and system hardware as a whole to determine the suitability for the real-time application. In particular, the operating system that these packages reside on has a very direct impact as to the level of determinism these packages can provide.

Operating System	Timer Resolution
HP-RT	1 uS
IRIX	800 nS
LynxOS	20 nS
OS-9	25 uS
QNX 4	1 nS
NT 4	10 mS
Windows CE	Not Specified
RTX for NT	100 uS

Send email to Interex or to the Webmaster©Copyright 1999 Interex. All rights reserved.

Send email to Interex or to the Webmaster
©Copyright 1999 Interex. All rights reserved.