⇓   ⇐
⇓  Features

FTP on the HP 3000


by Raghuram Bharathan

FTP stands for File Transfer Protocol, a very important Internet protocol, like telnet, SMTP, and HTTP.* But FTP is not only a protocol; it is also one of the most popular applications available on all platforms. FTP is used for transferring files from one machine to another.

FTP makes use of the protocol, which is specified in RFC 959 and Section 4.1 of RFC 1123. (Please refer to the Notes at the end of this article for a little more detail on RFCs.)

The FTP protocol specifies certain minimum requirements to be followed by the implementations. FTP implementations can and do have features over and above the minimum specified.

FTP has been available on HP 3000 for a long time. Since its introduction in the 1980s, it has undergone several changes. Most of them have been to bring FTP on the HP 3000 closer to the specifications in the RFC. (FTP on HP-UX has been used as a reference for some changes, such as FTP messages.) Some of them have been to provide features relevant to the HP 3000 only. All of them have been in response to customer requirements.

FTP is a client-server application. This means there is a client component and a server component. A client on one system must be able to communicate with a server on another system and accomplish the tasks required by the user.

These client-server interactions are achieved by adhering to the protocol specifications. The client communicates with the server by sending certain key words that have meaning for the server. The server replies to the client, using some numbers that have meaning to the client. Thus the server understands messages from the client and vice versa.

FTP on the HP 3000 is a classic example of interoperable software. An HP 3000 FTP client is able to interact with an FTP server on any other platform--HP 3000, HP-UX, NT, etc. Similarly, any FTP client on any platform can connect to an HP 3000 FTP server, and send or receive files.

In the following sections, we will see: We will conclude with a look at what the future holds for FTP.

Recent enhancements to FTP were made in two phases. In the first phase (which is available to customers from MPE/iX 5.5 PP4), the following features were made available: In the second phase the following features are provided. These features are available in FTP on MPE/iX 6.0. As this article is being written, MPE/iX 6.0 is a few days away from release. (See Table 1 for a list of FTP client commands on MPE/iX 6.0.) Table 1: FTP Client Commands Available on MPE/iX 6.0
dir mget recvsystem
appenddisconnectmkdirremotehelptimeout
ascii exit mode renamestruct
binaryexitonerrormodtimereplacesunique
bye form mput reset trace
bytestreamgetopenrmdirtype
case hash promptruniqueuser
cd help put send verbose
close lcd pwd site ?
debug ls quit size
deletemdeletequotestatus

Features Designed for HP 3000 Users

The Ability to FTP Non-Standard Files

Until recently, FTP was used only to transfer standard MPE files with fixed, variable, or bytestream format, and files with either binary or ASCII data types.

The recent enhancement allows the user to transfer the following "non-standard" files from an MPE system to another MPE system.
  1. Privileged files (files with PRIV file code)
  2. Circular files
  3. Message files
  4. RIO files
  5. CM KSAM files
  6. NM KSAM files
  7. HPSPOOL files
The client and server FTP must be compatible to take advantage of this feature. The user need not specify build parameters for these files.

Here is an example: 
FILENAME  CODE  ------LOGICAL RECORD------------- ----SPACE----
                SIZE  TYP      EOF     LIMIT  R/B SECTORS #X MX

BIGRIO1         128B  FAR     8193      8192    3    4128  1  8

ftp> get bigrio1
200 PORT command okay
150 File: bigrio1 opened; data connection will be opened
1054934 bytes received in 3.69 seconds (279.27 Kbytes/sec)
ftp>

Configurable Server Timeout

Timeout has been available for MPE FTP clients for a long time. The TIMEOUT command does not come from the RFCs. This was an invention on the HP 3000 to deal with long delays in sending a TCP packet and not getting an acknowledgement.

The timeout is not for the length of data transfer but for the length of time between a data send and an acknowledgement.

The current implementation of the MPE FTPSRVR has a hard-coded infinite timeout value applied to its data connection.

How nice it would be if we could specify a timeout for the server as well! Then the server need not keep the connection alive if there is some network problem or a problem on the client side. It can disconnect.

To incorporate this feature, a new option was added to the SITE command.

The server timeout can be specified as 
ftp> site timeout <numsec>
This allows the user to set the timeout on a remote MPE FTP server.

Additional CI Variables

CI variables are useful for using FTP in User Defined Commands (UDC), command files, and programs. They allow users to send or receive files in the background while running other applications.

In order to facilitate this feature, FTP provides several environment variables, the values of which can be checked after each command to see whether it was executed correctly or not. Based on the result, FTP can either continue with the transfer or stop. The following are the variables currently available. From MPE/iX 6.0, three more CI variables have been added for programmers:

A Few Existing MPE-Specific Features

In addition to the above, FTP on MPE/iX already provides the following features that make the life of an HP 3000 user easier.

The ability to stream a job on the remote system

Using FTP, the user can stream a file on the remote system. The user can ftp to the remote system, and then issue the following command: 
ftp> site stream <jobfilename>
Of course, the user must have sufficient permissions to do this.

The Exitonerror option

This is an option available on the MPE FTP client. When the user connects to a remote machine and turns on this option, the FTP program terminates as soon as a 5xx message is received from the remote machine. (5xx messages are permanent negative messages from the server, indicating a specific command cannot be executed.) 
ftp> exitonerror

ExitOnError mode on. (FTPINFO 33)

Case option

Since MPE file names are uppercase, when an MPE file is transferred to a non-MPE system, it is transferred as an uppercase file itself. To transfer a file as lowercase, the "case" option can be enabled. Once this is done, an MPE file is transferred as a lowercase file. 
ftp> case

Lower case on. (FTPINFO 43)
This feature, of course, is not applicable in file transfers between MPE systems.

Build parameters for target files

This is a very useful feature that allows the user to decide the format of the file to be created on the MPE system. FTP on MPE supports a subset of the MPE build parameters implemented by the MPE BUILD command. 
ftp> get testfile newtest;rec=-128,,f,ascii;disc=200
Here, the build parameters are specified after the semicolon ";".

FTP messages

The MPE FTP server messages follow the protocol specified by the RFCs. There is a three-digit number, followed by a description, for example: 
226 Transfer complete

550 File not found
But MPE FTP goes much beyond this. There are three kinds of FTP messages returned by MPE FTP (# stands for the error number): In addition, it also returns any error message returned by other subsystems with which it interacts--the File System (FSERR), Transport Subsystem (SOCKERR), and so on.

Thus MPE FTP provides the user with a wealth of status information. These messages are generally used by MPE programmers when they write job files, UDC, or programs.

Other New Features

FTPSRVR Configured to Run Under INETD

At present, we stream the job JFTPSTRT.ARPA.SYS, which starts FTPMON. FTPMON is running on the HP 3000 system, waiting for a connection request from an FTP client. This is dedicated software that waits only for FTP connection requests.

As soon as a connection request comes, FTPMON creates a process, FTPSRVR, under it.

The following is a sample excerpt of showproc 1;system tree: 
D230  0:00.635  WAIT  J3       48   :RUN ftpmon
C152  1:08.360  WAIT  J3       51   (FTPMON.ARPA.SYS)
C153  0:00.080  WAIT  J3       69   (FTPSRVR.ARPA.SYS)
Once the logon is done, a new session is created, and FTPSRVR runs under it. (See Figure 1.) 
B152  0:00.166  WAIT  S919     88   (JSMAIN.PUB.SYS)
C152  0:00.036  WAIT  S919     78   :
C152  0:00.217  WAIT  S919     77   (RPMDAD.NET.SYS)
C153  0:00.071  WAIT  S919     76   (FTPSRVR.ARPA.SYS)
Figure 1: FTPSRVR Under FTPMON
Missing: http://www.interex.org/pubcontent/interact/nov98/f1bharat.gif

In the above example, S919 is the new session created.

There are two disadvantages to this arrangement: FTP in MPE/iX 6.0 runs under INETD.

INETD, also called Internet superdaemon, is a process that checks for connection requests for any services that are present in its configuration file. So, if FTP is added to the configuration file, INETD will check for FTP connections as well.

When a connection is made, no separate session is created. A new FTPSRVR is created under INETD itself. This caters to individual user requests. Once a user closes the connection, this FTPSRVR terminates. (See Figure 2.)

Figure 2: FTPSRVR Under INETD
Missing: http://www.interex.org/pubcontent/interact/nov98/f2bharat.gif

Here is a sample excerpt of showproc 1;system tree: 
D230  0:00.369  WAIT  J8       62   :RUN inetd.net.sys
D230  0:00.928  WAIT  J8       67   (INETD.NET.SYS)
D230  0:00.553  WAIT  J8       63   (FTPSRVR.ARPA.SYS)
An important prerequisite for bringing FTP under INETD was for FTP to shift from NETIPC to BSD sockets for network communication. Until now, FTP has used the HP proprietary NETIPC protocol to do network communication. In order to bring FTP under INETD, the NETIPC code was replaced with BSD SOCKETS code. So FTP on MPE/iX 6.0 uses BSD sockets and not NETIPC code.

Some Other Advantages of Running FTP Under INETD

Enhanced security

INETD has a built-in security feature. It reads the file INETDSEC.NET.SYS, which contains details of permissions for different machines, hosts, and subnets.

By making a suitable entry in this file, we can allow only specific users to FTP to the concerned host, or deny specific users access to the FTP server. The simple entry is of the form 
<service name>   <allow/deny>  <host/network addresses, host/network names>
For example, if you want to keep the host 134.25.199.239 from doing an FTP to your system, you can make the following entry in the INETDSEC file: 
ftp  deny  134.25.199.239
This, coupled with anonymous FTP (described later) allows you to control the FTP to your machine. This was not possible earlier.

Running the FTP server on a non-default port

FTPSRVR, by default, is associated with port 21. This means that any client that wants to do an FTP to a remote system sends a request to port 21.

For enhanced security, some systems run FTP server on a port other than 21. This means no FTP client can make a connection to that FTP server unless it knows the port number on which the server is running.

FTPSRVR on MPE could only be run on port 21. This value was hard-coded and could not be changed.

But with the advent of INETD, all that it takes for FTPSRVR to be run on another port is a change in the entry for FTP in the SERVICES.NET.SYS file.

The default entry in the file is 
ftp          21/tcp
If you want FTP to run on, say, port 289, you just have to change the above line to 
ftp          289/tcp
Restart INETD, and FTP will henceforth run on port 289.

Note: It is one thing to run FTPSRVR on a non-default port. But the FTP clients must have the ability to connect to a non-default port too.

Until recently, the FTP client on the HP 3000 could make a connection only to port 21. A recent enhancement allows FTP clients to specify an optional port number when making the connection.

Anonymous FTP

One of the most useful features of FTP is its ability to allow users to do ftp into a system "anonymously." What this means is, the user need not have a login account to do an ftp to that system.

Anonymous FTP is a configurable feature of all FTP systems. Some machines have it, some don't. This can be set up by the systems administrator.

But what is undeniable is the usefulness of this feature. The most efficient way to share large files with a group of people is to put those files in your anonymous FTP directory. Once the files are put in your anonymous FTP directory, any user can log in to your system as "anonymous" without the need for an account, and retrieve the files. The user can also "put" files into your machine, if required.

FTP on the HP 3000 did not have an anonymous FTP feature. But this has changed in MPE/iX 6.0.

The MPE/iX 6.0 FTP has provisions to configure anonymous FTP. (See the box "How to Configure Your System to Have Anonymous FTP.")

One of the concerns regarding anonymous FTP is the security of the system. When a user can ftp to a system without an account, there is naturally a concern about whether the user can do some harm.

This is taken care of by the fact that the anonymous FTP user can access only the PUB group of the FTPGUEST account of the system. The anonymous FTP user cannot change directory to any other account or group. Moreover, the user does not have permission to stream a job.

Hash Mark Printing

Very often, when we transfer a large file, we would like to know whether or not the file is actually being transferred. We need some feedback from FTP on this.

This is achieved by the "hash mark printing" feature.

Here, the FTP client prints a hash after each kilobyte of file transferred, indicating the progress of the file transfer.

The syntax is simple. 
ftp> hash
Hash mark printing on (1024 bytes/hash mark).
ftp> get tst
200 PORT command successful.
150 Opening BINARY mode data connection for tst (40863 bytes).
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
226 Transfer complete.
40863 bytes received in 0.18 seconds (221.70 Kbytes/sec)

REPLACE Option

Very often, we get or put a file that already exists on our system. By default, FTP overwrites the existing file. One way to avoid this is to specify the following options:

sunique--for store unique--store as a unique file

runique--for retrieve unique--retrieve as a unique file

In this case, FTP checks whether a file of the same name exists. If it does, FTP appends a ".1" to the file and then checks whether a file of that name also exists. This goes on until it finds a unique file name. Then it does a file transfer.

The problem with this arrangement is that the user has no control over the name of the new file. The attachment of the digit after the file name also results in the creation of a POSIX file, which cannot be used in the MPE domain!

The REPLACE option is designed to solve this problem.

REPLACE essentially means, "Do you want the existing file to be replaced?" By default, it is set to ON (TRUE), which means the existing file is replaced automatically. But by giving the REPLACE command, we can toggle the option to OFF (FALSE). Once this is done, FTP checks whether the file exists before each transfer. If it finds the file, it prompts the user either to replace the file or to enter a new file name. In this way, the user can decide whether to replace the old file or enter a new file name. 
ftp>replace
REPLACE option set to off.
ftp> get tst
File tst exists! Do you want to replace it? (y/n)(n) >n
New target file name: tst1
200 PORT command successful.
150 Opening BINARY mode data connection for tst (40863 bytes).
226 Transfer complete.
40863 bytes received in 0.18 seconds (219.26 Kbytes/sec)

MOVE Option

Sometimes, FTP is used for moving files from one machine to another. What this means is, once the files are transferred from the source machine to the target machine, the files in the source machine are no longer required and are deleted. Currently, this is a two-step process.

First, the user has to do a "get" to move the files from the remote system to the user's local system. Then the user must do a "delete" to delete the files from the remote system.

This has been combined into a single step by means of the MOVE option, which can be specified along with any of the following commands: get, mget, put, mput.

When MOVE is specified, once the file is successfully transferred, the source file is purged. However, the deletion of the source file might fail because of protection or similar issues, but that has nothing to do with FTP.

Here is an example: 
ftp> dir te*
200 PORT command ok.
150 File: LISTFILE te@,2 opened; data connection will be opened

PATH= /SYS/RRAM/

CODE  ---------LOGICAL RECORD----------  ----SPACE----   FILENAME
      SIZE  TYP      EOF     LIMIT  R/B  SECTORS #X MX

       80B  FA         5    204800    1     16    1  8   TEST

226 Transfer complete.
226 bytes received in 0.03 seconds (6.69 Kbytes/sec)
ftp> get test movtest;move
200 PORT command ok.
150 File: test opened; data connection will be opened
226 Transfer complete.
400 bytes received in 0.01 seconds (26.04 Kbytes/sec)
ftp> dir te*
200 PORT command ok.
150 File: LISTFILE te@,2 opened; data connection will be opened
te@ not found
226 Transfer complete.
16 bytes received in 0.03 seconds (0.46 Kbytes/sec)
ftp>

Little-Known FTP Facts

mput Using an Indirect File

If you want to "put" more than one file to another system, you can use mput.

The syntax of mput is 
mput local-files
where local-files can contain wildcards that can be expanded.

But if you cannot specify all the files you want to put using a single local file (with all possible wildcards), then you will have to do multiple mputs.

To avoid this, create a local file that contains the names of all the files you want to put on the remote system. Say the file is LOCFILS.

Now, 
mput ^LOCFILS
When FTP sees the "^" symbol, it takes the rest of the parameter as an indirect file. It opens LOCFILS, gets the file names from it, and puts them to the remote.

NETRC File for Frequently Accessed Hosts

If you regularly do an ftp to a particular system, you can create an entry for that machine in a file named NETRC in your home group. The entry must have the following format: 
machine <machine-name> login <loginname> password <password>
Each word can be separated by spaces or tabs. The entries can also be in double quotes. All three entries are compulsory.

For example, if the machine is "testmac," the login "mgr.testing," and the password "netrc,testing," then the NETRC entry is as follows: 
machine testmac login mgr.testing password netrc,testing
If there is no password, you can have a " " to denote that. When you do an ftp, it checks the NETRC entry for the machine name, and if it exists, reads the login and password from that.

What Does the Future Hold?

SAMBA/iX and FTP

Now that Samba is available on the HP 3000, is FTP no longer required? Of course it is! Samba is used to view MPE files from NT and other platforms. It can also be used for doing limited file operations such as rename, copy, and delete.

However, you cannot write job or command scripts to do a file transfer in Samba. Also, it cannot copy non-standard files from one MPE machine to another.

So, we will have Samba and FTP co-existing and complementing each other.

Socksified FTP

Now that FTP uses BSD sockets, it can be enhanced to incorporate features provided by FTP software on other platforms.

One possibility is socksified FTP. This requires the SOCKS library to be ported to MPE/iX. Once that is done, FTP can be socksified by making a few changes and linking with the SOCKS library.

What is the advantage of socksified FTP? It can be used to make a connection to a remote host outside the firewall. On the other hand, a machine outside the firewall cannot access the machines inside the firewall.

Welcome Banner

Some FTP servers display a welcome banner when you make a connection to them. The MPE FTP server can also be enhanced to do the same.

This feature is useful for displaying copyright or other informative messages.

Large File Transfer

At present, you can transfer files up to 4 GB using FTP. As and when required, FTP can be enhanced to transfer larger files. *

Notes

These are documents that describe the various Internet protocols. These documents are widely available for free. Any implementations of any of these protocols is expected to adhere to the standards stipulated in the RFC. Fixes for problems in earlier releases of the operating system and new features are made available to customers through Powerpatch. For example, for MPE/iX 5.5, there have been five Powerpatches.

References

Raghuram Bharathan joined the Hewlett-Packard India Software Operation as a project trainee in January 1997 and became a software engineer in the CSY R&D Lab in Bangalore in July 1997. He has been working with FTP on MPE/iX since December 1997.

* Throughout this article, I'll be using the terms "FTP on the HP 3000" and "MPE FTP" interchangeably to mean the same thing. Similarly, MPE and MPE/iX are used interchangeably. FTP refers to FTP software. Any reference to the protocol is mentioned explicitly.

⇑  Contents
⇑   ⇐   [3khat]3kRanger   [3khat]3kMail   Updated