FTP/WWW Uploading and Downloading HP3000 Files

How to transfer files from the network to your HP3000

When we leave the realm of HP's proprietary :STORE/:RESTORE utilities, transferring files from one HP3000 system to another becomes a challenge. Unlike Unix and DOS systems which have stream-oriented file systems (no rigid record structure), MPE has a record-oriented file system. When files are exchanged between dissimilar operating systems, you must be very careful to insure that the data remains intact. This problem is compounded by the extended file attributes that an MPE file may posess (file code; fixed, variable, or undefined record structure, etc) as well as the mapping of the resulting file into the Posix (HFS) file system.

ASCII versus BINARY

File transfers between dissimilar systems take on two general forms:

ASCII transfers are used for readable text. Various file systems have their own ways of marking the end of line in text - end of record, a numbered byte count, a linefeed, carriage return, or both. ASCII transfers will translate the end of line into a universally accepted character sequence.
BINARY transfers will literally transfer every byte of the file with no translation and no record markers.

Problems can arise when transferring BINARY data in ASCII mode -- each time the end of line character is encountered, it is translated into it's universal equivalent and you end up with a corrupt file. When using FTP transfers, for example, you have control over the mode of the data transfer; but when using WWW the mode is dictated by the server. NCSA HTTPD, for example, has a configuration file (httpd_1.3/mime.types) which determine the transfer mode based on the file extension. Unfortunately for some, the default transfer mode is ASCII, which is not always the mode of choice. If you are running a WWW server, it is your responsibility to insure that files you make available for download will be served in the proper mode by checking the mime.types file.

To further illustrate the ASCII issue, as well as some of the Posix file quirks, consider the following example. We will create two files with EDITOR, one fixed length ASCII and the other variable length.

:editor
HP32201A.09.00 EDIT/3000 SUN, JUL  2, 1995, 12:57 AM
(C) HEWLETT-PACKARD CO. 1993
/a
    1     This is a test file
    2     //
... 
/k test1,unn
/s variable
/k test2,unn
/e

END OF SUBSYSTEM
:listf test@,2
ACCOUNT=  SYS         GROUP=  MANAGER 

FILENAME  CODE  ------------LOGICAL RECORD-----------  ----SPACE----
                  SIZE  TYP        EOF      LIMIT R/B  SECTORS #X MX

TEST1              72B  FA           1          1   3       16  1  1
TEST2            1276B  VA           1          1   1       16  1  2

Note the radical difference in file characteristics. If we continue one step further and create a TEST3 file in the Posix shell using vi, the result is now:

:listf,2
ACCOUNT=  SYS         GROUP=  MANAGER 

FILENAME  CODE  ------------LOGICAL RECORD-----------  ----SPACE----
                  SIZE  TYP        EOF      LIMIT R/B  SECTORS #X MX

TEST1              72B  FA           1          1   3       16  1  1
TEST2            1276B  VA           1          1   1       16  1  2
TEST3               1B  BA          20 2147483647   1       16  1  *

The TEST3 file is a true bytestream of 20 bytes, the text "This is a test file" plus the end of line character. To complicate matters, the Posix shell views the files differently as well (in spite of what you might think):

shell/iX> ls -l
total 5
-rwx------   1 MANAGER.SYS       SYS           72 Jul  2 04:58 TEST1
-rwx------   1 MANAGER.SYS       SYS         1280 Jul  2 04:58 TEST2
-rw-rw----   1 MANAGER.SYS       SYS           20 Jul  2 05:22 TEST3

TEST1 is seen as 72 bytes since it is fixed length. TEST2 appears as 1280 bytes, which is the 1276 byte record plus 2 byte record length field and 2 byte block length field to account for variable record storage format. Only TEST3 truly appears as the raw 20 bytes which account for the text and end of line. How are these files served by a host in ASCII mode? Transferring from the HP to a Unix system reveals (as of MPE/iX 5.0 Push):

unixbox.utc.edu% ls -l test*
-rw-------   1 jeff     staff         73 Jul  2 01:45 test1
-rw-------   1 jeff     staff         20 Jul  2 01:45 test2
-rw-------   1 jeff     staff         40 Jul  2 01:45 test3
unixbox.utc.edu%

In this example, test1 appears as the original 72 bytes (fixed ASCII) plus the end of line character, and test2 appears as expected, the 19 original bytes of text plus an end of line character. The mystery case is test3 which, by current conventions of FTP/iX, is transferred as 20 records of a 1 byte record with a newline character after each byte. For example:

unixbox.utc.edu% cat test1 test2 test3
This is a test file                                                     
This is a test file
T
h
i
s
 
i
s
 
a
 
t
e
s
t
 
f
i
l
e


unixbox.utc.edu%

You can transfer the bytestream file in BINARY mode to another MPE/iX or Unix system, but this does not work for IBM PCs (different end of line) or IBM mainframes (EBCDIC character set). This problem has been reported but not fixed as of this writing (7 Jul 95), refer to SR 5003-272690.