Writing MP Scalable Modules and Drivers

Modules and drivers that run over UP emulation hardware drivers must run under UP emulation. Before changing STREAMS/UX modules and drivers to be MP scalable, modify hardware drivers to be MP scalable.

You can improve the performance of modules and drivers by using the elsewhere synchronization level. Configure all modules and drivers in a subsystem to be in the same group. They can all share data. However, STREAMS/UX will not synchronize bufcall and timeout callback functions or any non-STREAMS/UX code with the modules or drivers. You may be able to use the streams_put utility described in Chapter 3. In general, UP emulation provides more protection for bufcall, timeout, and non-STREAMS functions.

To change modules and drivers to be MP scalable, analyze how the code shares data structures. Determine which structures are shared and which module and driver entry points read and write to the structures. Using this information, choose synchronization levels for modules and drivers that correctly serialize access to shared data.

If all modules and drivers of a product share the same structure, consider changing the module and driver data structures and algorithms to allow for more parallelism. Alternatively, consider using spinlocks to protect shared structures that are accessed infrequently or for short amounts of time. Using spinlocks is a good way to protect structures which are not accessed on the main read and write paths. You can either use the native HP-UX spinlock primitives or the SVR4 MP LOCK, TRYLOCK, UNLOCK, LOCK_ALLOC and LOCK_DEALLOC utilities. The SVR4 MP utilities are discussed under "HP-UX Modifications to STREAMS/UX Utilities" in Chapter 3 and in the SVR4.2 Driver manual.

Use service routines only for flow control, recovering from resource shortages or executing interrupt completions in a process context. Service routines degrade performance.

Be careful when writing timeout and bufcall callback functions, as well as non-STREAMS code that calls STREAMS/UX utilities or shares data with modules and drivers. See the "Guidelines for MP Scalable Modules and Drivers" section.

It is easier to develop STREAMS/UX-based software that runs completely MP scalable or completely under UP emulation. Try to avoid mixing MP scalable and UP emulation modules and drivers in the same stream or multiplexor.

MP scalable STREAMS/UX modules and drivers cannot call UP emulation software. A put or service routine cannot acquire the I/O semaphore because put and service routines cannot block. This means, for example, that modules and drivers which run over a UP emulation hardware driver must run under UP emulation.

Modules and drivers which can run both MP scalable and under UP emulation must use queue or queue pair synchronization. An example of an MP scalable module which can run in UP emulation mode is timod. Although timod is configured to be MP scalable, it is pushed onto many streams, some of which run in UP emulation mode.

The MP scheduler runs differently from the uniprocessor scheduler. This may affect STREAMS/UX application programs. On multiprocessor systems, the scheduler may not run a service routine before the process which scheduled the routine returns to user level.

A module or driver's synchronization level determines the entities with which it can share data. It also determines the entities with which it can share its STREAMS/UX queues. For example, if a module uses queue pair synchronization, the write put routine can call insq to insert a message onto the module's read queue. But, if the module uses queue synchronization, the write put routine can only call insq to insert messages onto the write queue. The synchronization level determines which queues a module or driver can pass to STREAMS/UX utilities.

In general, a put or service routine can only pass its own queue or queues belonging to entities with which it can share data. The restricted utilities are backq, bcanputnext, canputnext, flushband, flushq, freezestr, getq, insq, putbq, putnext, putnextctl, putnextctl1, putnextctl2, putq, qreply, qsize, rmvq, SAMESTR, strqget, strqset and unfreezestr. The putq utility is not restricted when it is passed a driver's read queue or a lower mux's write queue. Any put or service routine can call putq if it passes a driver's read queue or a lower mux's write queue. However, putq's caller must guarantee that the queue passed in is still allocated.

Some STREAMS/UX utilities, such as canput, are commonly passed a parameter of the form q->q_next. These routines are restricted in a different way from those listed above. A put or service routine can only pass its own queue's q_next field or the q_next field of queues belonging to entities with which it can share data. These requirements apply to bcanput, canput, put, putctl, putctl1, putctl2, and streams_put. These utilities are not restricted when they are passed a parameter of the form q, except that the queue must still be allocated.

Some restrictions exist for timeout and bufcall callback routines as well as non-STREAMS/UX code in the kernel. This software cannot share data structures with STREAMS/UX modules and drivers, unless spinlocks are used to protect critical sections. Also, the code cannot call the following utilities: backq, bcanputnext, canputnext, flushband, flushq, freezestr, getq, insq, putbq, putnext, putnextctl, putnextctl1, putnextctl2, qreply, qsize, rmvq, SAMESTR, strqget, strqset, and unfreezestr.

Callback routines and non-STREAMS code cannot call bcanput, canput, put, putctl, putctl1, putctl2 or streams_put if they pass the utility a parameter of the form q->q_next. They can call these utilities if they pass a parameter of the form q (q must be a valid, allocated queue). Callback and non-STREAMS code can call putq only if they pass it a driver's read queue or a lower mux's write queue. Callback and non-STREAMS code can use the new streams_put utility documented in the section "HP-UX Modifications to STREAMS/UX Utilities" in Chapter 3.

Some restrictions exist on free routines passed to esballoc. A free routine can call STREAMS/UX utilities in the same way as the put or service routine that calls freeb. A free routine can access the same data structures as the put or service routine that calls freeb.

A protect_q parameter can be passed to the weldq utility. The protect_q parameter specifies which queue the func parameter can access safely. The func function can use the same STREAMS/UX utilities as the protect_q put and service routines. Also, the function can access the same data structures as the protect_q put and service routines.

put and service routines cannot be called directly. They must be executed by calling STREAMS/UX utilities such as putnext, put, putq, or qenable. They cannot be called using the function pointer stored in the q_qinfo structure.

STREAMS/UX applications in which multiple processes access the same stream need to know how STREAMS/UX will synchronize operations on the stream. See "Multiple Processes Accessing the Same Stream" in Chapter 3.

Modules and drivers can allocate their own spinlocks to protect data structures. If they do, they should use the lock orders reserved for them in /usr/include/sys/semglobal.h or /usr/conf/h/semglobal.h: STREAMS_USR1_LOCK_ORDER, STREAMS_USR2_LOCK_ORDER, and STREAMS_USR3_LOCK_ORDER.

The lock order is passed in the order parameter of the native HP-UX alloc_spinlock primitive and the hierarchy parameter of the SVR4 MP LOCK_ALLOC utility. The HP-UX kernel uses this information to check for deadlocks when the kernel is compiled with SEMAPHORE_DEBUG. When a module acquires a spinlock, the spinlock's order must be higher than the order of any spinlocks the module already holds. Modules and drivers cannot hold spinlocks when calling some STREAMS/UX utilities. See Table 4-1 “Holding Module or Driver Defined Spinlocks While Calling Utilities” at the end of this chapter for more information. See the SVR4.2 Driver manual for more information about SVR4 MP hierarchies.

To reduce contention and improve performance, you should minimize the amount of time that modules and drivers hold spinlocks.

To improve performance, modules and drivers should verify that they are actually running on a multiprocessor system before calling the HP-UX native spinlock primitives. The SVR4 MP LOCK and UNLOCK routines described in Chapter 3 do this for the caller. If a spinlock is being used only to protect against software running on other processors, but not interrupts, modules or drivers can call the MP_SPINLOCK and MP_SPINUNLOCK macros in /usr/include/sys/spinlock.h (or /usr/conf/h/spinlock.h). These macros obtain only the requested spinlock if they are executing on a multiprocessor system. If a spinlock is being used to protect against both software running on other processors and interrupts, modules and drivers should check the uniprocessor flag and raise the spl level if they are running on a uniprocessor system. Example code is shown below.

if (uniprocessor)
   x = splstr();
else
   spinlock(mylock);

Be careful when choosing a multiplexor's synchronization level. When a driver is linked under a mux, STREAMS/UX changes the driver's Stream head to be the lower mux. STREAMS/UX uses the upper mux's synchronization level for the lower mux. So if the upper mux uses global, elsewhere, or module synchronization, the lower and upper muxes can share data. If the upper mux uses queue or queue pair synchronization, the lower and upper muxes cannot share data.

The synchronization level also influences how messages can be passed across the mux. If the upper mux uses global, elsewhere, or module synchronization, it can pass messages downward by passing the lower mux's write queue to putq, put, or putnext. Likewise, the lower mux can pass messages upward by passing the upper mux's read queue to putq, put, or putnext. If the upper mux uses queue or queue pair synchronization, it can only use putq and put to pass messages to the lower mux. To use putnext, the upper mux must ensure that the driver stays linked under the mux until after the putnext completes. Also, the lower mux can only use putq and put to pass messages to the upper mux. To use putnext, the lower mux must guarantee that the driver stays linked under the mux, that the mux stays open, and that modules are not pushed or popped until after the putnext completes.

No matter which utility is used to pass messages across the mux, you must make sure that the queues passed to the utilities are still allocated. You may also want to check that the driver is still linked under the mux.

Follow the design guidelines in the SVR4.2 STREAMS manual. The guidelines are located at the end of these chapters: Overview of STREAMS Modules and Drivers, STREAMS Modules, STREAMS Drivers, and STREAMS Multiplexing. For STREAMS/UX, you do not need to follow some of these guidelines. However, if you ignore them, your software will not be portable to SVR4 STREAMS. For HP-UX STREAMS, you do not need to call qprocson or qprocsoff as you do for SVR4 MP STREAMS. Also, you can use synchronization levels to protect module and driver private structures instead of SVR4 MP locks and synchronization primitives. Lastly, you do not need to use SVR4 MP canputnext and bcanputnext instead of canput and bcanput on STREAMS/UX.