⇓ ⇐

The HP 3000--for Complete Novices

PART 13: Files

by George Stachnik

In the last few articles in this series, we have seen how to compile and link application programs on the HP 3000. But as anybody who has ever written an application knows, there's more to an application than a computer program. There's also the data and the linkages between the data and the program. This month we're going to begin exploring how data is stored on the HP 3000 and accessed by application programs running under MPE/iX.
Building Files
On the HP 3000, data is typically stored in files, just as it is on virtually any other computer system. In the simplest sense of the term, a file is a reserved area on a device used to store data. The HP 3000 allows you to create files on a number of different kinds of devices, but disk devices are the ones used most frequently, so that's where we're going to focus in this article.
Files are a part of the HP 3000's file system. The file system was introduced at the beginning of this series of articles. In parts one through three of this series, we explained the framework of accounts, groups, and directories in which files reside, as well as MPE/iX's rules for naming files.
Now we're going to turn our attention to the contents of files. We'll see exactly how files are created, how they are purged, and how data is written to files. We'll also learn how data in files can be accessed on the HP 3000.
Creating a File: A BUILD Command
The first step to learning how to work with files on MPE is to create a file. MPE provides several different ways to create files. Perhaps the easiest and most straightforward way is to use the MPE :BUILD command. There are many alternatives to :BUILD. For example, application programs can create files using programmatic interfaces to the operating system. But for the purposes of learning how MPE files are created, we'll begin by examining the simplest possible example of a :BUILD command.
Suppose you wanted to create a file, and assign it the name MYFILE. The command shown in Figure 1 (:BUILD MYFILE) creates a file and assigns it the name that we specified--MYFILE. It's important to understand that this example makes a large number of assumptions. These assumptions may or may not be appropriate, depending on how you plan to use the file. We're going to spend most of the rest of this article examining those assumptions and exploring a few of the options that you can specify while creating a file, in particular the record size, the record format, and the file type. We'll also introduce a widely used file management utility program called FCOPY.
Note that the :BUILD command does not place any data in the file. Figure 1 shows how you can tell: we have issued a :LISTFILE command to display information about the file that we've created. The option (",2") that is specified in the example tells :LISTFILE to display additional information about the file, including the number of records that it contains. Under the heading EOF (End Of File) the :LISTFILE command displays a zero. This is telling us that this file contains zero records. The file is empty.

Figure 1: Creating a File with the :BUILD Command

Using FCOPY
Let's turn our attention to the task of loading this file with data. Of course, you could write a short program in a language like C or COBOL to write data into the file (and we'll be seeing how that's done in a future article in this series), but for now we're going to learn how to use a more general-purpose tool, MPE's FCOPY utility program. This program resides in PUB.SYS, so you can run it using the command:
    :RUN FCOPY.PUB.SYS
You can also run FCOPY by simply typing the program name as if it were a command:
    :FCOPY
Once it's running, the program will prompt you for FCOPY commands, which you would use to copy, display, print, and manipulate files. FCOPY is a very versatile utility, and we're only going to scratch the surface of its capabilities in this article. The easiest way to enter FCOPY commands is to include them right on the command line. A couple of examples will serve to explain how to do this, and how the commands work.
Suppose there are two files named FILE1 and FILE2, and you want to copy the data in FILE 1 into FILE2. The following command, entered at a colon prompt, will do the job:
    :FCOPY FROM=FILE1;TO=FILE2
There are a couple of things to notice about the above example. First, the two parameters, FROM= and TO=. Each parameter is followed by a filename, and the parameters are separated by a semicolon. The semicolon is mandatory; it must be specified exactly as shown above.
The output file (FILE2 in this example) must already exist before you enter the FCOPY command that we just saw. FCOPY will not create an output file for you unless you specifically tell it to do so. If you want FCOPY to create an output file for you, you must tell it to do so by using another FCOPY parameter: ;NEW.
Here's an example of how you'd tell FCOPY to build a new file called FILE3, and copy the contents of FILE1 into FILE3:
    :FCOPY FROM=FILE1;TO=FILE3;NEW
Once again, the parameters must be separated by semicolons exactly as shown above.
Using FCOPY with $STDIN
When you log on to an HP 3000, MPE/iX automatically assigns a filename to your session's input device. The filename used is $STDIN, and your session's input device is the terminal or PC that you're using to access the HP 3000. If you're logged on at a terminal, you can think of $STDIN as a name representing your terminal keyboard. Similarly the filename $STDLIST represents your terminal's display.
You can use FCOPY to read data from $STDIN or to write data to $STDLIST. If you leave the FROM= or TO= parameters blank, FCOPY will default to $STDIN for the FROM= parameter or $STDLIST for the TO= parameter.
For example, suppose you wanted to create a file called TEXTFILE and load it with a few lines of text. Here's an easy way to copy some text from your keyboard into a NEW file called TEXTFILE:
    :FCOPY FROM=;TO=TEXTFILE;NEW
The NEW option tells FCOPY that the output file (TEXTFILE) hasn't yet been built. FCOPY will create the file for you. The FROM= parameter has been left blank, so it will read the data from $STDIN--that is, it will expect you to type the data at your keyboard.
The way this works can be a little confusing if you haven't done it before. After you enter the FCOPY command, a banner will be displayed. Then the program will appear simply to stop. At this point, new users sometimes get confused and think that something has gone awry. But there's nothing wrong.
FCOPY is waiting for you to type some data. It has posted a READ operation on $STDIN (your keyboard). But there is no prompt or any other indication that it's waiting for you to type something, hence the confusion. But from this point forward, anything that you type at your keyboard is treated as data. As far as FCOPY is concerned, your terminal is a file, and everything you type will be read by FCOPY (as if it were a record in an input file) and written to the output file (TEXTFILE).
If you type MPE commands (:SHOWJOB, LISTFILE, :BYE, HELP), they will not be executed, because FCOPY now "owns" your $STDIN file. Every keystroke that you type is captured and sent directly to the output file (TEXTFILE). So now is the time to type the data that you want to load into the output file. Of course, sooner or later you'll need to get FCOPY to turn your $STDIN file loose so you can start typing MPE commands again. Let's see how that's done.
When FCOPY is reading data from a disk file, it works its way through the data until it reaches the end of the file. Reaching the end of a file triggers an end-of-data condition that tells FCOPY to finish copying the data that it has read to the output file, close all its files, and terminate.
But $STDIN is not a disk file. Instead it's associated with your terminal. There isn't an "end-of-file" as long as you keep typing. If you stop typing, FCOPY will simply wait until you start again. How do you tell it that you're done?
When FCOPY is reading from $STDIN, you can trigger the end-of-data condition by typing the character string :EOD. (Note that the first character in this string is a colon (":"). Figure 2 shows an example of using FCOPY with $STDIN. The FCOPY command is entered at the top of the figure. The parameter "FROM=;" specifies that data will be read from $STDIN. The parameter "TO=ASCFILE" specifies that the data will be written to a file called ASCFILE. And finally, the ;NEW parameter tells FCOPY that ASCFILE doesn't exist yet, and so FCOPY will create it.
After FCOPY displays its banner, we type three lines of text, and then the string ":eod" to tell FCOPY that we are done. FCOPY terminates, and Figure 2 shows that we next use the :LISTFILE command to display the attributes of the output file, ASCFILE.

Figure 2: Using FCOPY with $STDIN

Using FCOPY with $STDLIST
FCOPY also can be used to display the contents of a file by copying a file to your $STDLIST file. When FCOPY's "TO=" parameter is left blank, FCOPY uses $STDLIST as its default output file, causing the data to appear on your terminal screen. An example is shown in Figure 3. In this example, we are using FCOPY to display the three lines of text that we entered into ASCFILE on our terminal display.
You might be thinking that FCOPY, with its rigid rules about semicolons, and the need to specify ";NEW" just to get it to create a file, is a bit klunky and old fashioned. Of course, you'd be right. Ten years ago, FCOPY was the preferred method for manipulating files on the old MPE/V operating system (which was used with the 16-bit "classic" models of the HP 3000). Today, FCOPY is still supported on MPE/iX but only in order to maintain compatibility with the earlier models. On MPE/iX, much of what FCOPY does can be done more easily (and using more intuitive syntax) with other MPE commands.
For example, the COPY command can be used to copy files and the :PRINT command (also shown in Figure 3) can be used to display the contents of files without having to deal with FCOPY's older, more cumbersome syntax.
We've seen that MPE's :BUILD command is used to create files. You might be wondering how to get rid of them. This is accomplished using the :PURGE command. The command is followed by the name of the file you want to delete. An example of the :PURGE command is shown at the bottom of Figure 3.

Figure 3: Using FCOPY with $STLIST

File Structure: Records
On many operating systems (including UNIX and NT), files are byte-oriented. This means that as far as the operating system is concerned, each file is nothing more than a stream of bytes, stored on a device (typically a disk) and assigned a file name. For example, a UNIX application program that reads or writes a file will (by default) read (or write) only a single byte of that file. If the application program wants to read more than one byte, the program must specify how many bytes it wants to read.
MPE/iX files can have this same simple byte-stream structure. This can be useful if you are working with application software that was originally designed to be used on a computer that uses byte-oriented files exclusively, such as an application that is being ported from a UNIX or NT environment.
But by default, MPE/iX endows its files with more complex structural characteristics. An MPE application program that reads or writes a file will (by default) read (or write) one record. A record is a string of bytes. The length of each record does not need to be specified by the application program, because MPE/iX keeps track of file record sizes by storing them in the files themselves.
If you aren't used to record-oriented files, it may help to think of each file on the HP 3000 as a table made up of columns and rows (similar to a spreadsheet). Each row of the table corresponds to one record. The columns correspond to fields in the records (about which we'll have a great deal to say when we discuss the linkages between application programs and files in a future article in this series).
When an MPE application reads a file, it will read the records (rows) one at a time. The application program does not have to specify how many bytes to read (as it would on a UNIX system) because the file "knows" how many bytes are in each record.
The record size is one of a number of file characteristics that are managed by the MPE/iX file system, and which can be displayed using the :LISTFILE command. These characteristics may be specified at the time a file is created.
Referring back to bottom of Figure 2, we used the :LISTFILE command to display the attributes of the file named ASCFILE. The :LISTFILE command has a parameter that specifies what kind of attributes you want to display. (Remember, the number "2" in the command line indicates that we want to see basic information about the file, including its record length.) In the example shown in Figure 2, ASCFILE has a record size of 80 bytes (the :LISTFILE command shows this as 80B under the heading SIZE). This means that every record in this file contains 80 bytes of information. The example also shows us that there are three records in this file. The :LISTFILE command displays the number of records under the heading EOF.
Byte-oriented operating systems (UNIX, NT) typically display file sizes in bytes. By default, MPE/iX displays file sizes in terms of the number of records in the file. The :LISTFILE command does not display the file size in bytes the way UNIX and NT do (at least not by default). File sizes will be discussed in more detail in a future article of this series.
In most cases, MPE files are built so that all records have exactly the same length. That is, if a file has a record size of 80 bytes, then every record in that file will be 80 bytes long. At the bottom of Figure 2, the :LISTFILE command has displayed the character string "FA" under the heading TYP. The "F" in this string tells us that this file contains "fixed-length records"--which is to say that every record has the same size.
MPE/iX also supports files with variable-length records. That is, a file can be structured so that every record is of a different length: one record might be 80 bytes and the next record might be 40, 16, or 200 bytes long. The :LISTFILE command displays such files with a "V" (instead of an "F") under the TYP heading. The important thing to understand is that the record format (Fixed or Variable) must be specified at the time the file is created, and cannot be changed.
Figure 2 also demonstrates a little quirk in FCOPY that sometimes causes some confusion. Referring back to the top of Figure 2, you can see that we wrote three records into ASCFILE (there are three lines of text shown in the figure). This agrees with :LISTFILE's record count of 3 (look under EOF in the LISTFILE output). But notice that after we entered the end-of-file indicator (:EOD), FCOPY displayed a message that says "EOF FOUND IN FROMFILE AFTER RECORD 2." It may seem as if FCOPY missed a record somewhere, but it didn't. FCOPY counts records starting with record number zero, so the three records are numbered 0, 1, and 2.
Padding Records
Referring back to the top of Figure 2, when we entered our three lines of data into FCOPY, we used the RETURN key to indicate the end of each record. For example, the first record is made up of the 14 characters "This is a test." Then we hit RETURN to indicate the end of the first record.
But ASCFILE has fixed-length records of 80 bytes, as we've seen. Therefore FCOPY knows that each record in ASCFILE must be made up of a full 80 bytes of data. Since we supplied only 14 bytes of data ("This is a test"), FCOPY made up the difference for us. It appended 66 spaces to the end of our character string before writing the full 80-byte record to the output file.
There is one situation in which fixed-length records will be padded with something other than spaces. To understand that situation, we'll need to understand the difference between ASCII files and binary files.
ASCII and Binary Files
MPE/iX differentiates between two types of files: ASCII files and binary files. The guideline is that ASCII files are generally used to hold ASCII data, i.e., bytes that have printable ASCII characters. binary files are generally used to hold binary data--bytes for which there may not be any printable ASCII characters.
You can tell whether a file is an ASCII file or a binary file by using the :LISTFILE command. As shown in the example at the bottom of Figure 2, the file type is displayed under the heading TYP. The character string "FA" tells us two things about the file. We've seen that the "F" says that this file has fixed-length records. The "A" says that it's an ASCII file. If this had been a binary file, a "B" would have appeared in place of the "A."
Keep in mind that the guidelines for ASCII data and binary data are just that--guidelines. There's no rule that says that an ASCII file cannot contain binary data, or vice-versa. This might make you wonder why anybody would care whether a file was defined as a binary file or an ASCII file. The reason is shown in Figure 2.
In Figure 2, we wrote records to an ASCII file named ASCFILE that were shorter than 80 bytes (i.e., shorter than ASCFILE's fixed record size). Those records were padded with ASCII space characters (blanks) to bring the length of each record up to 80. Spaces were used because ASCFILE is an ASCII file.
But if the output file had been a binary file, (i.e., if :LISTFILE displayed a "B" under TYP), the records still would have been padded, but with binary zeroes instead of spaces.
Assigning File Attributes
The record size, record format, and file type are assigned at the time a file is created. Figure 4 shows how some of the attributes that we've been talking about are assigned. In the :BUILD command at the top of the example, we've added a parameter that we haven't seen before: "REC=-80." This tells the :BUILD command to create a file with a record size of 80 bytes. There are a number of other things to notice about this example.
First of all, referring to the :LISTFILE command at the bottom of Figure 4, you'll notice that the file that was created is a binary file with fixed-length records. You can tell because the :LISTFILE command displays the string "FB" under the heading TYP. The "F" tells us that this file has fixed-length records, and the "B" tells us that it's a binary file.
We did not have to specify either of these attributes in the :BUILD command, because they are defaults. If we had wanted to create an ASCII file, or create a file with variable-length records, we would have had to specify those parameters in the :BUILD command. We'll see how that's done in a future article in this series.
There's another curious thing about the example shown in Figure 4. Notice that the record size being reported by :LISTFILE isn't 80 bytes: it's displayed as "40W." This is MPE shorthand for "40 words" and to understand what that means, we'll have to go back and revisit the history of the HP 3000.
Remember that the original models of the HP 3000 family were 16-bit machines. The old MPE/V operating system that ran on those machines was a 16-bit operating system. That means that it dealt with data using 16-bit words.
The :LISTFILE command displays the record size for ASCII files in bytes (as we saw in Figure 2). However, the record size for binary files is displayed in words. On the old 16-bit models of the HP 3000, each word was 16 bits (or two bytes) long. The file that we created in Figure 4 was created with a record size of 80 bytes, which is the equivalent of 40 16-bit words. Because it's a binary file, the record size shown in Figure 4 is "40W." If we had created an ASCII file, the record size would have been displayed as "80B" (as we saw in Figure 2).
This has caused some people a good deal of confusion. The HP 3000 is no longer a 16-bit machine. Today, MPE/iX is a 32-bit operating system running on 32-bit and 64-bit hardware platforms. HP has announced plans to make MPE/iX a 64-bit operating system in the future. In spite of this, the :LISTFILE command displays binary record sizes as if it were still a 16-bit machine, purely for the purpose of maintaining compatibility with earlier versions of the operating system.
When we specified the record size in Figure 4, it was specified as a negative number ("REC=-80"). You might have been wondering what the minus sign is for. If we had left it off ("REC=80"), the file would have been built with a record size of 80 words (or 160 bytes). Remember that if you want to specify the record size in bytes, you must specify it as a negative number.

Figure 4: Specifying a Record Size

Blocking Factors
The "REC=" parameter of the build command can be used to specify the file characteristics that we've been discussing: the record size (in bytes or in words), the record format (fixed or variable), and file type (ASCII or binary ). It can also be used to specify something called a file blocking factor. The blocking factor was key to maximizing the performance of the old 16-bit models of the HP 3000.
MPE/iX does not block files in the same way that MPE/V did, so the blocking factor is no longer relevant. It can still be specified (in order to maintain compatibility with the earlier systems), but it is simply ignored.
Putting It All Together
Let's finish up this month's article by looking at how you specify all these file characteristics in a :BUILD command. They are all specified as part of the ;REC= parameter. The keyword "REC=" can be followed by up to four positional parameters. Positional parameters are separated by commas, and must be specified in a particular order. The full syntax of the REC= parameter is as follows:
    REC= record-size,blocking-factor, record-format,file-type
The bottom of Figure 4 shows an example of a :BUILD command in which all of these parameters have been specified. The REC parameter has been specified as follows:
    rec=-60,10,f,ascii
Let's look at the four positional parameters one at a time. Remember that they must be specified in the order shown above: the record size followed by the blocking factor, followed by the record format, followed by the file type.
In the example in Figure 4, the record size is 60 bytes (remember that a negative number indicates bytes, not 16-bit words). The blocking factor is 10, which is irrelevant because MPE/iX ignores it. The file being built will have fixed-length records ("f"), meaning that every record in the file will be 60 bytes long. And the file will be ASCII, meaning that FCOPY will pad short records with blanks if necessary.
We've learned a lot about MPE files in this article. We've seen that MPE's files are record-oriented, rather than byte-oriented, and we've seen how to use FCOPY to do some basic file manipulation. We've also learned some of the basic characteristics of files that are "built into" MPE files, such as the record length, whether the records are ASCII or binary , and whether they have FIXED or VARIABLE lengths.
Next month we'll continue to explore some of the more advanced characteristics of MPE files.
George Stachnik works in technical training in HP's Network Server Division.

←Part 12 Part 14→