HP 3000 Manuals HP 3000 Manuals
Understanding the Native Mode Spooler and System Spoolfiles by Thomas McNeal--Commercial Systems Division In the mid-70's, after the release of MPE II, the MPE Spooler was developed to handle simultaneous access to shared peripherals. With the initial releases of MPE XL, the MPE Spooler (SPOOL is an acronym for Simultaneous Peripheral Operation OnLine) remained relatively intact in Compatibility Mode as the CM Spooler. In Release 2.1 of MPE XL, the CM Spooler was replaced by a new Native Mode (NM) Spooler. The functions provided by the CM Spooler and the SPOOK utility were integrated into the MPE XL file system and command interface. Spoolfiles are now permanent, recoverable files, handled by the standard Type Manager/Storage Manager layers of the MPE XL Native Mode file system. SPOOK, the CM Spooler utility, was obsoleted in Release 2.1. A migration path was provided for all commands within SPOOK. These commands were replaced by new CI commands, extensions to old CI commands, or by new utility programs. Some CI commands, such as COPY, or PURGE, will not change syntactically, but semantically extend to operate on spoolfiles. In any case, the CI command syntax will remain fully backward compatible. CM SPOOLER LIMITATIONS The CM Spooler limitations are becoming apparent within the context of MPE XL. Primarily, the creation and management of spoolfiles outside of the MPE XL file system can not meet the requirements necessary to interface with periphrals. This becomes particularly important when considering system performance and reliability requirements. With the CM Spooler, the maximum amount of disc space allowed to spoolfiles must be configured separately, imposing an artificial limit when space requirements are high. Additionally, a 32 extent limit propogated from MPE V/E into the CM Spooler places a burden on single spoolfiles that does not exist for files within the MPE XL file system. As separate entities, spoolfiles cannot be backed up with the rest of the file system, forcing system managers to maintain separate tapes and backup processes. Spoolfile recoverability does not meet the standards of the file system, and any enhancements to file system reliability will not automatically apply to the CM spooler. Access to the spoolfiles through procedures such as searching, browsing, and copying, is also limited, since only the primitive SPOOK commands are available. Other CM Spooler issues such as performance bottleneck with CM/NM switching, and the exposure of CM internals required by third party software compromises system reliability. Interprocess communication also affects reliability as well as security, since the CM Spooler processes implement IPC by depositing messages in the receiving process's stack. All of these issues are addressed within the NM Spooler. NM SPOOLER FEATURES The NM Spooler is now fully integrated with the MPE XL file system. Spoolfiles are accessible through CI commands, through file system utilities such as STORE/RESTORE, and through browsing utilities or editors. They are also programmatically available through the file system intrinsic layer, or through the NM Spooler Architected Interface layer. Within the file system itself, beneath the file system intrinsic layer, the type manager layer has been extended for spoolfiles. The spoolfile type managers handle file creation, access, and deletion, both for implicitly created spoolfiles (created when opening a spooled device directly) and explicitly created spoolfiles (through jobs, the :BUILD or :FILE command, HPFOPEN, STORE/RESTORE, etc.). Beneath the type managers, in the storage management layer, the new printer storage managers handle the three device classes supported by the NM Spooler, which are serial devices, CIPER protocol devices, and the 268x page printers. The NM Spooler is no longer an independent entity using ATTACHIO and the CM file system. The CI contains three new commands: :LISTSPF, :SPOOLER, and :SPOOLF. The :LISTSPF command, replacing the SHOW command found in SPOOK, lists the name and status of output spoolfiles. The :SPOOLER command controls spooler processes, and :SPOOLF controls spoolfile attributes. Both :SPOOLER and :SPOOLF compliment and extend existing commands, such as :STARTSPOOL, :STOPSPOOL, or :ALTSPOOLFILE. The existing commands tied to the CM Spooler and Spooler processes are extended to handle the NM Spooler semantics. For example, the SPOOL option has been added to the :BUILD command for building an output spoolfile, and the PRIVATE and SPSAVE options have been added to the :JOB command to alter the output spoolfile access rights, and to save the spoolfile in the system after it has been printed. As in the CM Spooler, a separate spooler process handles each spooled device. These processes are children of a single monitor process, which in turn is a child of PROGEN. The monitor process creates the spooler processes in response to an MPE XL system port message from either the STARTSPOOL or the SPOOLER (when using the START option) command executor. The spooler processes handles, the spoolfiles through the spoolfile directories. These two directories, an input and an output directory, are generically labelled as the input and output SPFDIR. The entries in these directories contain all the information needed to manage the spoolfiles, including the spoolid, the target device or device class, the total number of copies, and so forth. When a spoolfile is identified by an entry in a SPFDIR, it is considered to be linked to the NM Spooler. The SPFDIR entries and a table of queue header pointers replace the CM Spooler use of the Input Device Directory and the Output Device Directory (The IDD and the ODD remains as CM system tables in order to control non-spooled devices). Each queue header points to the following two subqueues: A short wait subqueue containing SPFDIR entries for spoolfiles which are likely to be accessed immediately, and a long wait subqueue, containing all others. The spoolfile state determines to which subqueue it will be linked. Spoolfiles which are READY, for example, will be in the short wait subqueue, whereas spoolfiles in the DEVER state will be linked to the long wait subqueue. Under most circumstances, once the file has been printed, renamed, or purged, the entry will be removed from the SPFDIR, and from the subqueues. Any other spoolfile creation (as with COPY, or restoring spoolfiles into some other account) is simply considered to be standard file manipulation, and the file is not linked to the NM Spooler subsystem. A new account was created to handle linked spoolfiles. The account, HPSPOOL, contains two groups, IN and OUT, for the input and output spoolfiles, plus a set of device groups associated with the spooled devices. Each device group name is the same as the device name, if that device was monfigured through SYSGEN. If not, the device is identified only by a logical device number, and the device group name is the LDEV, with the first digit replaced by the letter D. All input spoolfiles reside in the IN group of the HPSPOOL account. All output spoolfiles created and linked to the output SPFDIR reside in the OUT group. There may be unlinked output spoolfiles in the system, either in OUT.HPSPOOL, or in some other account. If an output spoolfile is created with the :BUILD or :FILE command, or copied, or renamed, the resulting file is linked to the NM Spooler through an output SPFDIR entry, regardless of its location within the system. If an output spoolfile is created through STORE/RESTORE, the file is linked to the Spooler if it was restored to OUT.HPSPOOL. If not, it is also created as an unlinked spoolfile. The PRINT option of the SPOOLF command will copy an unlinked output spoolfile into OUT.HPSPOOL (if it is not already there), and link the file to the Spooler through an entry in the output SPFDIR. The device groups hold checkpoint files created by the NM Spooler processes to handle device interrupt recovery. Each process creates a checkpoint file for each printed spoolfile, regardless of the number of copies being printed. In all, three new file codes are created to designate the files in the HPSPOOL account. They are: 1515 - Input Spoolfiles 1516 - Output Spoolfiles 1517 - Checkpoint files MAINTAINING SPOOLFILES The HPSPOOL account is built as part of the system startup, if it does not already exist. It resides on the system volume, and cannot be relocated to a private volume. If any of the groups, or the account, are purged, the account structure will be rebuilt by rebooting the machine. The default security levels are: HPSPOOL Account: (R,A,W,L,X:any) IN Group: (R,A,W,L,X,S:any) OUT Group: (R,A,W,L,X,S:any) Device Groups: (R,A,W,L,X,S:gu) The physical location of spoolfiles can, however, be controlled by configuring the volume class SPOOL within the system volume set. If this volume class is defined, spoolfile disc space will be allocated only on the device (or devices) configured as this volume class. If this volume class is not configured, spoolfile disc space will be allocated across the entire system volume set. When an :UPDATE, or a :START command with the NORECOVERY option, is executed at the ISL prompt, all input spoolfiles will be purged from the HPSPOOL account. The input spoolfiles have a one-to-one association with entries in the Job Master Table (JMAT), and this table is rebuilt for every UPDATE or START NORECOVERY. Finally, output spoolfiles will be retained until the output device notifies the spooler that the output has completed and that no errors have been detected. If the SPSAVE option has been requested, the spoolfile state will then be changed to SPSAVE, and its entry left in the output SPFDIR. Otherwise, the entry will be removed from the SPFDIR, and the file will be deleted. Note that the SPSAVE and PRIVATE attributes are mutually exclusive; any private spoolfile will be automatically deleted after printing. SPOOLFILE SECURITY If spoolfiles are permanent disc files, how can they be protected from other users who still have access across account boundaries to the HPSPOOL account? In an attempt to implement security for the new spoolfile environment, Release 2.1 introduced file level security extensions, plus the new concept of private spoolfiles. The name and account of the spoolfile creator is saved in the file label extension, allowing the creator (and the account manager of the creator) access to the spoolfile, even though it may be stored in the separate HPSPOOL account. This enhancement maintains the access model of the CM Spooler. This means users have access to their own spoolfiles, and the account manager has access to all spoolfiles owned by users within the account. For spoolfiles containing sensitive data, the added protection of the 'private' spoolfile attribute is also available. Private spoolfiles are privilege level 2 files which are protected from non-privileged users. All input spoolfiles are created as private, but output spoolfiles may be either private or non-private, depending upon the choice of the user or application when creating the file. Access to private spoolfiles is extremely restricted. The non-privileged user can not open a private spoolfile, either to browse, copy save, store, or print the file. Non-privileged users will at most be able to raise and lower the spoolfile priority, or purge the file (with :DELETESPOOLFILE or :SPOOLF;DELETE) altogether. A new :JOB command attribute, and a new option to HPFOPEN have been added to designate output spoolfiles as private. STORE/RESTORE AND SPOOLFILE BACKUP The integration with the MPE XL file system represents the fundamental advantage of the NM Spooler. Spoolfiles are accessible through the standard Intrinsic/Type Management/Storage Management file system layers, removing disc space limitations, recovery limitations, and file access limitations. Now that spoolfiles are permanent disc files, you may transport non-private output spoolfiles between Release 2.1 MPE XL systems with STORE/RESTORE. Normally, users must be able to store and restore output spoolfiles without being in the HPSPOOL account. STORE/RESTORE uses the file system security enhancement previously mentioned, allowing the creator to access their own spoolfiles, and allowing the account manager to access any files created within the account. When restoring spoolfiles into OUT.HPSPOOL, the output spoolfiles will automatically be linked into the output SPFDIR. Alternatively, since the ;PURGE option of STORE is supported for spoolfiles, an output spoolfile will be deleted from the output SPFDIR and purged from the system when storing the file with this option. The file name of a spoolfile is determined by the file's spoolid. To avoid file name collision, the spoolid and file name of a restored spoolfile will be changed to a value assigned by the spoolfile management routines. DEVICE RECOVERY The checkpoint files created in the device groups of the HPSPOOL account are used for both device and spoolfile recovery, saving page checkpoints and other data for use during recovery following a device interruption, or when the :RESUME command is executed following a :SUSPEND command. If supported by the device (as with CIPER protocol devices), the spooler will recover to a specific page. The HP2680 and HP2688 do not support page checkpoints, requiring a silent-run from the beginning of the spoolfile to the page at which the interrupt occurred. With either type of device, the output will resume correctly after an interrupt. Serial printers, on the other hand, cannot recover accurately to a specific page, and therefore may not resume at the correct page, or may shift the page boundary from its original position. For recovering spoolfiles themselves, all statements concerning recovery of disk files will apply to spoolfiles as well. MIGRATION ISSUES All commands available through SPOOK have been replaced by CI commands, text editors and file browsers, or by two new NM Spooler utilities. A utility created for the NM Spooler, SPFXFER, provides a replacement for the SPOOK INPUT and OUTPUT commands, as well as a tape pathway between the CM Spooler and the NM Spooler. The SPFXFER utility will write NM spoolfiles to tape in a format which can be read by the SPOOK utility on MPE V/E (for release G.02.B0 and after) and all pre-Release 2.1 MPE XL systems. The utility will also be able to read tapes created by SPOOK on any release of MPE V/E and pre-Release 2.1 MPE XL system. SPFXFER may also be used to transport output spoolfiles via tape between Release 2.1 MPE XL systems, but the preferred vehicle for this situation is STORE/RESTORE. SPFXFER has four commands, each of which can be truncated and still be recognized. The commands use the same syntax as do the commands in the SPOOK utility. They are: INPUT - Read one or more spoolfiles previously written to tape by SPOOK or by SPFXFER. The files will be written to the OUT.HPSPOOL group and account, and will be linked into the output SPFDIR. OUTPUT - Write one or more spoolfiles from the OUT.HPSPOOL group and account to tape. HELP - List the SPFXFER commands and syntax. EXIT - Leave SPFXFER Another NM Spooler utility, PRINTSPF, emulates the 'mode c=on' command within SPOOK, and the :PRINT command, a standard text editor, or a file browser may be employed to look at spoolfiles. File browsers can be very useful in this context, allowing the user to read or search through the spoolfile without the overhead of bringing spoolfile copies into an editor's temporary buffer. All other SPOOK commands have been replaced by new commands or existing commands that have been modified. In many cases, the semantic extensions to the CI commands are greater than the syntactic changes, when considering that all commands formerly applicable to normal files also apply to spoolfiles. In particular, :COPY, :PURGE, and :RENAME remain unchanged, but their meaning now extends to spoolfiles, and each command has side effects. When copying or renaming a spoolfile, the resulting file is linked to the NM Spooler. Purging a file obviously has the same effect, if the file is not being printed. If it is in use, an exclusive access violation will occur, and the file will not be purged. The commands actually modified (or, rather, extended) are: :BUILD - A new option, SPOOL, has been added to create an output spoolfile. The spoolfile will not be linked to the output SPFDIR. :FILE - Four new parameters (SPOOL, PRIVATE, FORMID, and SPSAVE) have been added. PRIVATE and SPSAVE are mutually exclusive. :JOB - Two new parameters, PRIVATE and SPSAVE, have been added. PRIVATE will force the job output $STDLIST to be a private (output) spoolfile. SPSAVE will retain the spoolfile in the OUT.HPSPOOL account after the spoolfile has been printed, rather than purging it (by default). The two attributes are mutually exclusive. :LISTEQ - The output of LISTEQ has been extended to display the parameters added to the :FILE command. :OPENQ - The LDEV, device name, or device class name are now all legal parameters. In addition, the SHOW parameter will display the current queue status of the specified device(s). :OUTFENCE - A device parameter, DEV, has been added to allow designation of the LDEV, device name, or device class name when setting the outfence. :SHUTQ - As with OPENQ, two parameters, DEV and SHOW, have been added. DEV allows designation of an LDEV, device name, or device class name, and SHOW displays the current state of the queue. ARCHITECTED INTERFACES AND THIRD PARTY SOFTWARE To remove dependencies upon NM Spooler internal data structures, a set of Architected Interfaces have been provided, along with the AIF's defined for XL Release 2.1. The AIF definitions have already been shipped to a large number of third party vendors, many of whom are now updating their applications to release themselves from CM Spooler dependencies. (AIF:OS is an orderable product. To order, please refer to product number 36374A.) ________________ This article is reprinted from the original article titled "A Native Mode Spooler" from Interact magazine by permission of INTEREX. For subscription information, please write: 585 Maude Court, P.O. Box 3439, Sunnyvale, CA 94088-3439 USA.