MPE/iX Release 5.5 Express 3 CI Enhancements

by Jeff Vance and M. Gopalakrishnan (Gopi)
CSY / ISO

Enhancements Summary:

LISTF/LISTFILE Enhancements

Syntax:

  LISTF [fileset][,listlevel][;listfile]

  LISTFILE [fileset[,fileset[,...]]]
        [[;FORMAT=]format_opt]
        [[;SELEQ=]select_eq  indir]
        [[;NAME=] pattern]
        [;PASS]
        [;{PERM}{TEMP}{PERMTEMP}]
        [;USENAME][;TREE][;NOTREE]

The LISTF and LISTFILE commands allow users to discover who is accessing files on their system, including remote accessors. The existing syntaxes remain the same; however, new format numbers, format mnemonics and selection equations can be specified as a result of this enhancement.

Two new formats have been added: format 8 or "access", and format 9 or "locks". Both the LISTFILE and LISTF commands support these formats, although only LISTFILE is able to use the format mnemonic. Format 8 shows greater access-related details compared to the existing format 3, as well as displaying individual file accessors. Format 9 is a super set of format 8. All of the format 8 data is shown plus specific details about each process accessing the file, including information on which locks a process owns or is waiting for. A detailed description of the items displayed for the two new formats and examples are shown below.

The selection equation parameter of the LISTFILE command is also enhanced. The new SELEQ= choices allow selection of only files that are currently opened, in use, locked, or opened exclusively. If a file is not opened as specified in the selection equation it is skipped. Also, files can be selected based on their file code number or mnemonic. The file code mnemonic "PRIV" includes all privileged files, that is, those files with a negative file code. More details on the access SELEQ choices appear later in this article.

Format 8

Both formats 8 and 9 show the fully qualified filename or absolute pathname followed by the accessor summary line (described below). After the accessor summary line there is one line of output per file accessor, consisting of: Below is a format 8 example:

  :listfile hppxudc.pub.sys,8
  ********************
  FILE: HPPXUDC.PUB.SYS
  15 Accessors(O:15,P:15,L:0,W:0,R:15),Share
  #S265   MIKEP.HPE                  P:2,L:0,W:0,R:2      LDEV: 49
  #S263   JEFFV,MGR.JVNM             P:3,L:0,W:0,R:3      LDEV: 47
  #S261   KROGERS.MPENT              P:2,L:0,W:0,R:2      LDEV: 50
  #S231   SUSANC.MPENT               P:2,L:0,W:0,R:2      LDEV: 46
  #S219   FAIRCHLD.MPENT             P:2,L:0,W:0,R:2      LDEV: 39
  #S214   CATHY,MGR.BOSS             P:2,L:0,W:0,R:2      REM : 15.14.16.198
  #J434   FTPMON,FTP.SYS             P:2,L:0,W:0,R:2      SPID: #O21905

Analyzing the above format 8 example:

1) The filename shown is always fully qualified as it appears in format 6.

2) The line beginning with "15 Accessors..." is the accessor summary line, which shows the total number of accessors of the file across all jobs and sessions. This total includes all processes that have formally opened the file and all processes that have pseudo opened the file. This is the same number seen in the "FLAGS" field when executing LISTF,3. The total number of accessors may not necessarily be derived by adding up the individual accessors displayed. This is because certain file accesses are done without using the file system and these accesses are transient and difficult to detect. An example of this kind of access is when STORE pseudo opens a file to read its contents to backup.

The "O:" field is the actual number of processes that formally opened the file (e.g., called FOPEN, HPFOPEN, open, etc.). The O: number can be lower than the total number when the file has been pseudo opened, as done by STORE. Another factor is that the system protects certain files at boot time by pseudo opening them, as is done on NL.PUB.SYS. Also, a LISTF ,8 done at the same time on the same file as a LISTF ,2 (for instance) will show the LISTF ,2 users as an accessor that has not formally opened the file. Typically, however, the O: count and total number of accessors match.

The total number of processes (P:) accessing the file should generally match the O: count.

The number of processes that have the file locked (L:) is seen next. The number of lockers includes all file system locks (semaphores) held by all detectable processes that are accessing the file. A single process can obtain multiple locks, and each time the count is incremented. When a process releases a lock the count is decremented. The locks that are considered are the FLOCK semaphore (used by the FLOCK intrinsic), the OPEN semaphore (used by the file system) and the GUFD semaphore (used to control I/O).

The number of writers (W:) and readers (R:) are displayed next. The writer and reader counts can be greater than the number of accessors since a single process can open the same file multiple times. Also, when a process opens a file for append, update, or save (read/write) access both the writer and reader counts are incremented.

After all of the counts are shown the file sharing mode is displayed. This is either "Share" for files opened shared, or "Exclusive" for files opened exclusively. Typically when a file is opened exclusively there is only a single accessor; however, the POSIX fork() and dup() functions are exceptions.

The last item displayed in the access summary is the backup state. If the file is being stored then "Storing" is shown. If the file is being restored then " Restoring" is seen. If the file is being concurrently backed up then "ConcurrBackup" is displayed. Otherwise, nothing additional is shown.

3) The lines after the accessor summary line reflect each job and session accessing the file. There is one line displayed for each job and session accessing the file, even if there are multiple processes within the job accessing the same file.

The first field shown is the job/session number. If the accessor is a system process then "system" is seen instead of a job ID. Following the job ID is the user name associated with the job. This consists of the USER.ACCOUNT preceded with an optional job name. The logon group name is not seen.

The same counts displayed in the accessor summary line are shown for each individual job accessor. The definitions are identical. Unlike the summary line, the reader and writer fields are truncated if the process and locker counts are extremely high. In this event the right-most field is the first to be truncated and an "$" indicates that truncation ocurred.

The last field displayed depends on whether the accessor is a job, a remote session or a local session. For jobs the spoolfile ID is shown. The LDEV number is displayed for local sessions. If the accessor is remote the IP address is shown.

Format 9

In addition to the format 8 data, format 9 shows more details such as: Below is a format 9 example:

  :listfile hppxudc.pub.sys,9
  ********************
  FILE: HPPXUDC.PUB.SYS
  5 Accessors(O:5,P:5,L:5,W:0,R:5),Share
  #S263   JEFFV,MGR.JVNM             P:3,L:3,W:0,R:3      LDEV: 47
   #P147   (LFCI.PUB.SYS)
     ACCESS: R-excl            REC#: 0                    FNUM: 13
      LOCKS--Owner--   --Waiter--
           FLOCK
           OPEN
   #P154   (CI.PUB.SYS)
     ACCESS: R-excl            REC#: 0                    FNUM: 13
      LOCKS: none
   #P86    (JSMAIN.PUB.SYS)
     ACCESS: R-excl            REC#: 336                  FNUM: 16
      LOCKS--Owner--   --Waiter--
                       FLOCK

  #J434   FTPMON,FTP.SYS             P:2,L:2,W:0,R:2      SPID: #O21905
   #P79    (CI.PUB.SYS)
     ACCESS: R-excl            REC#: 0                    FNUM: 14
      LOCKS: none
   #P47    (JSMAIN.PUB.SYS)
     ACCESS: R-excl            REC#: 336                  FNUM: 15
      LOCKS--Owner--   --Waiter--
           OPEN        FLOCK

Analyzing the above format 9 example:

1) all of the format 8 data is seen.

2) The PIN (Process ID Number) begins the next line, followed by the process filename.

3) The next line shows the type of access which includes: "R" for read, "W" for write, "X" for execute, "A" for append, "L" for lock, "S" for save, "U" for update, "RD" for directory read access. The file sharing mode is next. Values include: "sysexc" for system exclusive (rarely seen), "excl" for exclusive, "ear" for semi-exclusive (exclusive-allow- read), "shr" for shared, "multi" for shared job, or "gmulti" for system shared. Next the current record number is seen, and last is the file number.

4) If the process does not have any locks then "NONE" is displayed. If the PIN owns or is waiting on one or more of the relevant semaphores then the semaphore name is shown under either the "Owner" or "Waiter" column. There are three semaphores (locks) considered.

The FLOCK semaphore, which is obtained when a processes calls the FLOCK intrinsic. This lock is exclusive meaning that only one process at a time can own the FLOCK semaphore for a file. Occasionally the FLOCK semaphore is used by the file system to to synchronize directory access.

The OPEN semaphore is locked by the file system when a file is being formally opened. In this usage the OPEN semaphore is locked exclusively. It is also used by the operating system to protect certain file system data structures when they are being read. This usage allows the OPEN semaphore to be locked in a shared fashion. The OPEN semaphore can be locked shared by multiple processes, but once it is locked exclusively all other PINs trying to lock it will wait.

The GUFD semaphore is locked by the file system when it does I/O on the target file. It is locked exclusively.

"Phantom" Accessors

It was mentioned earlier about processes that pseudo open (call sm_open) a file. STORE and PROGEN are examples of processes that bypass the file system by not calling FOPEN or HPFOPEN to open a file. This type of access is usually only evident by noticing that the total accessors count is greater than the open count (O:), and format 9 output typically will not show these phantom accessors.

There are also cases where a process does not open the file in any way at all, yet this process can lock one or more of the semaphores, and thus needs to be accounted for. LISTFILE and MPEX (when calling AIFSYSWIDEGET item 2065 to obtain a file's accessors) are examples of these kinds of phantom accessors. In this case the total accessor count is lower than what may be revealed by counting individual accessors shown in format 9.

If any of these phantom accessors lock one of the three important semaphores then LISTFILE can detect the process and report reduced information about the access. If the phantom accessor process does not lock either the FLOCK, OPEN or GUFD semaphores then its access is undetected. Below is a format 9 example of a file that another user is also executing a LISTF,3 against. Notice that the total accessor count is one, yet we see two accessors. PIN 58 is the process that is doing the LISTF,3 and thus is seen as a phantom accessor.

  FILE: GOPI.PUB.SYS
  1 Accessor(O:1,P:2,L:2,W:0,R:1),Share
  #S3     JVREM,MANAGER.SYS          P:1,L:1,W:0,R:1      REM : 15.28.88.18
   #P62    (LOCKP.PUB.SYS)
     ACCESS: R-excl            REC#: 0                    FNUM: 11
      LOCKS--Owner--   --Waiter--
           FLOCK

  #S2     JVREM,MANAGER.SYS          P:1,L:1,W:0,R:0      REM : 15.28.88.18
   #P58    (CI.PUB.SYS)
     ACCESS: n/a               REC#: n/a                  FNUM: n/a
      LOCKS--Owner--   --Waiter--
           OPEN

Restrictions

Formats 8 and 9 are available to all users, however only more privileged users get to see individual accessor information. If the user has SM or OP capabilities then she can see all accessors of a file - assuming she has TD access to the file itself. If the user does not have SM or OP capabilities but does posses AM capability, and the file's GID matches the user's GID then he can see all accessors too. If the user lacks SM, OP and AM capabilities but is the owner of the file being listed then she can see individual accessors. Otherwise only the accessor summary line is visible.

Even if the user is able to see individual accessors, certain format 9 fields have additional restrictions for security concerns: the program name and IP address are protected.

To see the program name the user must meet the same rules that apply to the SHOWPROC command. Specifically,
1) the process is within the user's logon job/session, or
2) the process' user and account names match the user's user and account names and the system's JOBSECURITY is set to LOW, or
3) the user has OP or SM capability.

To see the IP address the user must meet the SHOWCONN utility's rules, namely: the user must have SM or OP or PM or NA or NM capabilities.

Selection Equations

Selection equations for the LISTFILE command have been expanded. It is now possible to select files by whether or not and how a file is accessed, and by file code. The ACCESS= criteria supports 4 types of access, as shown below:

    SELEQ=[ ACCESS = INUSE | OPEN | LOCK[ed] | EXCL[usive] ]

"inuse" - is true if any processes are accessing the file. This includes normal FOPEN-like accessors and sm_open accessors. Phantom accessors (like LISTF,2) that don't open the file at all are not noticed and thus "inuse" evaluates to false. "inuse" examines the total number of accessors field described above.

"open" - is true if there are any processes that have formally opened the file by calling FOPEN, HPFOPEN, reset, open, etc. "open" uses the same field that the "O:" count is derived from.

"lock" - is true if any process has locked (even if they are waiting) the FLOCK, OPEN or GUFD semaphores, or has called the HPFLOCK pseudo intrinsic (for POSIX record level locking).

"excl" - is true if the file was opened for exclusive access. "excl" is true when LISTF,3 shows "EXCL" or LISTFILE,8 shows "Exclusive".

The CODE= criteria supports file code numbers and mnemonics as shown below:

    SELEQ=[ CODE = number | mnemonic | PRIV ]

The number can be positive or negative. "PRIV" will match all files with negative file codes.

Note: a new FINFO item 61 ("accessors") returns the number of accessors for a file. This represents the total number of accessors, including all processes that open the file in any manner. This is the first value seen in the accessor summary line, which is described under the Format 8 description. It is also the same value shown in the "FLAGS" field of a :LISTF,3.

Note: the end-of-file (EOF) for CM file types, such as RIO and CIRcular, may be overstated when the target file is being accessed. This is due to the fact that LISTF and LISFILE no longer sm_close the file being listed. Previously, the sm_close caused the file label on disk to be updated, and one of the fields updated was the EOF. Now that the listed files are no longer opened and closed the label on disk is not updated by LISTF, thus causing the EOF on disk to be significantly inaccurate for files currently being accessed. LISTF and LISTFILE now calculate the EOF for all reported files that are being accessed by other processes. This calculation is slightly inaccurate for CM files types.

New Variables

Two new variables were added to MPE/iX CI: HPLASTSPID and HPSPOOLID.

HPLASTSPID is a read-only string variable. It contains the $STDLIST spoolfile ID for the job defined by the HPLASTJOB variable, typically the job most recently streamed in the session or job. Since HPLASTJOB allows write access, it can be set to any job number, and then HPLASTSPID will contain that job's $STDLIST spoolfile ID.

HPSPOOLID is a read-only string variable. It contains the $STDLIST spool ID of the currently streamed job. This variable is useful only in jobs -- in a session it is set to "".

FLABELINFO and FINFO Enhancements

The FLABELINFO intrinsic returns information from the file label of a disk file and the FINFO function is a CI interface to FLABELINFO. These APIs have been enhanced to return four new items:
1) the number of sectors occupied by a file,
2) the number of extents occupied by a file,
3) file creation time,
4) the number of accessors to a file.

FLABELINFO:

  Item Num   Type         Item Description

  58         I16          Number of sectors occupied by the file.
  59         I16          Number of extents occupied by the file.
  60         I32          File creation time (CLOCK format).
  61         I32          Number of accessor of the file.

FINFO accepts the following numbers and aliases

  FINFO Num     Mnemonic             Return Type     Notes

   58           SECTORS              Integer
                NUM SECTORS

   59           EXTENTS              Integer
                NUM EXTENTS

   60           CREATETIME           String
                FMTCREATETIME
                CREATION TIME

   -60          INTCREATETIME        Integer
                CREATION TIME INTEGER

   61           ACCESSORS            Integer      This is the same value
                NUM ACCESSORS                     as the total number of
                                                  accessors in the LISTF
                                                  format 8 summary line

  Examples:

        finfo('CI.PUB.SYS',"sectors")     = 192
        finfo('CI.PUB.SYS',"extents")     = 2
        finfo('CI.PUB.SYS',"createtime")  = 11:08 AM
        finfo('CI.PUB.SYS', -60)          = 110808
        finfo('CI.PUB.SYS', "accessors")  = 124

PAUSE Enhancement

Syntax:

    PAUSE [num_seconds]
          [JOB= jobid]
          [INTERVAL=interval_secs]
          [;EXIST | WAIT | NOTEXIST]

The PAUSE command allows the current task to be suspend or "sleep" for a specified number of seconds. PAUSE now supports sleeping until one or more jobs reach a certain state. For example, a script can pause while selected jobs or sessions are executing ("EXIST"). Or, a job can sleep while another job is suspended or waiting ("WAIT"), and as soon as the job start executing or terminates the pause completes. A session can pause while no jobs exist ("NOTEXIST") and wake up as soon as the first job is streamed.

In its simpliest form, the PAUSE command sleeps for "num_seconds", or less if BREAK is pressed. In this simple case no "jobid" is specified and all other command arguments are ignored. If the "jobid" parameter is specified then "interval_secs" and the remaining command parameters are relevant. When "jobid" is supplied PAUSE typically sleeps until the jobs or sessions matching "jobid" have terminated.

Options

EXIST - (default) means to pause while all jobs and sessions matching "jobid" exist. These jobs can be scheduled, waiting, executing, etc., but as long as the SHOWJOB command displays one or more of the jobs defined by "jobid", the pause continues.

WAIT - means to pause while the selected job or jobs are waiting. As soon as all the matching jobs are no longer waiting (meaning all the job states are no longer "introduced", "waiting", or "scheduled") the pause ends. The life cycle of a job is typically: [sched or waiting->] intro-> initializing-> exec-> [susp-> exec->] terminate. Waiting jobs are considered all job states left of and excluding "initializing". Non-waiting jobs are all jobs right of and including "initializing".

NOTEXIST- means to pause while the matching job or jobs do not exist. As soon as any jobs matching "jobid" exist (in any state) the pause completes. PAUSE might miss finding very fast jobs. This is particularly true for a match on a single job/session number. A more practical use might be:

                PAUSE job=@J;notexist

which means to sleep while no jobs exist. As soon as the first job is streamed the above pause stops.

Collectively EXIST, WAIT and NOTEXIST are referred to as the "while_state", since PAUSE sleeps "while" the specified state is true.

PARAMETERS

num_seconds
If "num_seconds" is specified without "jobid" PAUSE sleeps for that many seconds, or until the process issuing the pause is interrupted by the break signal.

If "jobid" is also supplied then "num_seconds" has a different meaning. In this case it indicates the maximum duration for the PAUSE command, such that PAUSE should continue while the selected jobs are in their "while_state" or when "num_seconds" has expired, WHICHEVER IS SHORTEST. Thus, "num_seconds" represents the maximum length of the pause. If PAUSE completes but one or more jobs are still in their "while state" a CIWARN is reported. Note: to pause while a job is in its "while_state" or until "num_seconds" has expired, whichever is LONGEST, one can execute the following two commands:

       PAUSE x
       PAUSE job=y ;z

If after X seconds job Y is still in state Z then the second PAUSE continues while state Z applies. On the other hand, if after X seconds job Y is not in state Z then the pause is complete.

jobid
"jobid" can be one of: [#]Jnnn, [#]Snnn, [jobname,]user.acct, @, @J, @S. Note if "jobname" is included then the "jobid" must be quoted since the comma is a command token delimiter.

If the JOB= parameter is specified then PAUSE sleeps while "jobid" is in its "while_state". "jobid" can be an executing, waiting, scheduled job, or a session. "jobid" can also name many jobs or sessions. Wildcarding is supported, and a non-wildcarded "[jname,]user.acct" can match several jobs or sessions. The job name can be " ," or " @," to match all jobs or sessions without a job name. When more than one job or session matches "jobid" PAUSE sleeps while all matching jobs are in their "while_state". If the job executing PAUSE matches "jobid" it will not be selected.

interval_secs
If "interval_secs" is specified PAUSE sleeps for this many seconds between attempts to see if "jobid" is still in its "while_state". Otherwise, PAUSE sleeps a variable amount of seconds depending on the job state and the number of previous times a particular job has been polled. This computed method favors executing jobs that terminate quickly.

Examples:

The following example pauses while job #J24 exists in the system job table, (JMAT) i.e., it is visible in SHOWJOB output.

        :PAUSE job=#j24

The next example sleeps as long as MANGER.SYS has any jobs or sessions running or waiting.

        :PAUSE job=manager.sys; exists

The next example pauses until the job just streamed starts executing.

        :STREAM myjob
        :PAUSE job=!hplastjob; wait

Or, sleeps until the job you just streamed completes.

        :PAUSE , !hplastjob

The following example sleeps until all jobs have logged off or 5 minutes, whichever occurs first.

        :PAUSE 300, @J
        :IF hpcierr = -9032 then
            # pause terminted but one or more jobs are still running

The next example pauses while all jobs (by naming convention only) in the PROD account are running.

        :PAUSE job="J@,@.PROD"               # note the quotes are required

The next example sleeps while the backup job ("JBACKUP,OP.SYS") has not been streamed. PAUSE reports CIWARN 9032 if the job is not streamed within 30 minutes.

        :PAUSE 1800, job="jbackup,op.sys"; notexist

The final example polls the JMAT every 3 minutes looking for any job or session matching a user name that includes the letters "MGR", and waits for all such job/sessions to terminate before the pause ends.

        :PAUSE , @mgr@.@ , 180

INPUT Enhancement

Syntax:

    INPUT varname
          [PROMPT=prompt]
          [WAIT=seconds ]
          [READCNT=numchars]

The INPUT command was enhanced to accept the number of characters to read from $STDIN. INPUT optionally writes "prompt" to $STDLIST then reads "numchars" from $STDIN into "varname", optionally waiting "seconds" for the user to enter the data.

The READCNT parameter is new. If "numchars" characters are specified then up to "numchars" bytes will be read from $STDIN. Of course the user can press the <return> key at any time to complete the read. "numchars" is most useful for reads of only a few characters and exempts the user from being required to press the <return> key. The default "numchars" is the maximum size of the CI's command buffer, currently 511 bytes. If "numchars" is specified and less characters are supplied as input, the user must still press the <return> key to send the data.

Example:

     INPUT myvar,'Do you want to continue (Y/N)?', 10, 1

prompts to $STDLIST, and does a 10 second timed read on $STDIN of one character. When the user types the first character the read will complete and the read byte will be stored in the variable named MYVAR.

PRINT Enhancement

PRINT command prints truncated records when an UNNUMbered file contains the trailing eight characters as digits. The :PRINT command is not displaying the trailing eight characters as it assumes those digits as line numbers.

The existing rule as of MPE/iX 5.5 for determining whether a file is a numbered file:
If the trailing 8 characters of at least 10 records of a file or all records (in the case of <10 records) are digits, then that file is considered as numbered file, provided the record line numbers (trailing 8 character digits) are in ASCENDING order.

With this new option NONUM, the contents of the file would be displayed as it is, without assuming the trailing 8 characters as line numbers.

With this enhancement the :PRINT command syntax is:

    PRINT filename]
          [OUT=outfile]
          [START=m]
          [END=n]
          [PAGE=p]
          [;UNN | NUM ]
          [;NONUM]

With this new option, the command

[1] :PRINT infile;NUM;NONUM
would print the line numbers as in the case of UNNUMbered files, ie, line numbers starting from 1 for the first record and so on.

[2] :PRINT infile;UNN;NONUM
:PRINT infile;NONUM
would consider the file as UNNUMbered file even when the file is a NUMbered file and the print the contents as it is in the file.

Examples:

   [UFILEYES is an unnumbered file with trailing 8 characters as digits. ]

    :PRINT UFILEYES                          [As of MPE/iX 5.5]
    aaaaaaaaaaaa
    bbbbbbbbbbbb
    cccccccccccc
    dddddddddddd
    eeeeeeeeeeee
    ffffffffffff
    gggggggggggg
    hhhhhhhhhhhh
    iiiiiiiiiiii
    jjjjjjjjjjjj
    kkkkkkkkkkkk
    llllllllllll

Note: The above file was considered by PRINT to be a numbered file and thus the trailing 8 bytes are truncated.

    :PRINT UFILEYES;NONUM
    aaaaaaaaaaaa00010001
    bbbbbbbbbbbb00010002
    cccccccccccc00010003
    dddddddddddd00010004
    eeeeeeeeeeee00020001
    ffffffffffff00020002
    gggggggggggg00020003
    hhhhhhhhhhhh00020004
    iiiiiiiiiiii00030001
    jjjjjjjjjjjj00030002
    kkkkkkkkkkkk00030003
    llllllllllll00030004