HP 3000 Manuals HP 3000 Manuals
Structure of Each File Organization
The following sections describe the physical structure of the four data
file organizations.
Sequential Organization
The order of records in a sequential file is set by the order of WRITE
statements when the file is created. Record order does not change once
it is set. New records are added to the end of the file. Each record in
a sequential file (except the first record) has a unique record that
precedes it, and a unique record that follows it (except for the last
record).
Fixed Format Sequential Structure.
In a fixed format sequential file, each record immediately follows the
previous record in the file. Each record is the same length as the
maximum length record.
No additional control characters are written to the file unless you use
WRITE BEFORE or WRITE AFTER
ADVANCING statements. In this case, carriage return and line feed
characters are added as required to ensure correct positioning when the
data file is printed. If a file containing such characters is used as
input, you may find that records are not read as expected. In such
cases, we recommend that you use line sequential organization (the see
section "Line Sequential Organization" later in this chapter).
+--------------------------------------------+
| Fixed length record |
+--------------------------------------------+
| Fixed length record |
+--------------------------------------------+
. .
. .
+--------------------------------------------+
| Fixed length record |
+--------------------------------------------+
Variable Format Sequential Structure.
A variable format sequential file is the simplest form of the variable
structure defined above: each record written is preceded by a record
header containing the length of the record; the record is written at the
length defined in the program; the file contains a standard variable
structure file header record.
+-----------------------------------------------+
| File Header record - 128 bytes |
| |
+--------+-----------------------------+--------+
| Header | Variable length record |
+--------+-----------------------------+------------+
| Header | Variable length record |
+--------+------------------------------------------+
. . . .
. . . .
+--------+--------------------------------------+
| Header | Variable length record |
+--------+--------------------------------------+
Record Sequential Files
Record sequential files are intended to cater for binary data. These
files consist of a series of either fixed or variable length records.
The order of records in these files is set by the order of WRITE
statements when the file is created. The record order does not change
once it has been set. New records are added to the end of the file.
Each record in a record sequential file (except the first record) has a
unique record which precedes it, while each record (except the last
record) also has a unique record that follows it.
Record sequential files that are fixed length and are not destined for
the printer have no record terminator; the end of one record is
immediately followed by the beginning of the next.
Files destined for the printer are record sequential files which you can
redirect to a UNIX file. However, such files cannot be read easily by a
COBOL program if you want to recover print lines.
You can request special processing to take place on the output of print
files. To do this, specify the LINE ADVANCING option in the SELECT
statement. This is performed automatically if the ASSIGN TO PRINTER
clause is used. This causes:
* trailing spaces to be discarded in output records
* each print record to be terminated by a carriage-return character
(hex 0D)
* the OPEN statement to add a carriage-return character (0D) to the
file
* each WRITE statement without any BEFORE or AFTER clause to behave
as if you had specified the AFTER 1 clause.
Each print record has one or more of a line-feed character (hex 0A), a
form-feed character (0C) or a vertical tab character (hex 0B) added
before or after the print record depending on whether you specified the
BEFORE or AFTER clause in the WRITE statement.
We recommend that you use either the LINE ADVANCING option for all files
which you intend to print or, alternatively, specify either the BEFORE or
AFTER clause in every WRITE statement for that file.
Notes:
* You should never use the BEFORE or AFTER clauses for data files
which you do not intend to print.
* You should not open files destined for the printer for either
INPUT or I/O.
Record Sequential Structure.
+-------------------------------------------+--+
| Variable length record |0A|
+---------------------------------+--+------+--+
| Variable length record |0A|
+---------------------------------+--+------+--+
| Variable length record |0A|
+-------------------------------------------+--+
. . .
+-------------------------------------------+--+
| Fixed length record |
+-------------------------------------------+--+
. . .
. . .
+-------------------------------------------+--+
| Variable length record |0C|
+-------------------------------------------+--+
Line Sequential Organization
Line sequential files are intended to cater for text (ASCII) created
using text editors and similar utilities. These files consist of a
series of variable length records. The length or each record is
determined by the length of the longest file description for the file in
the program's FILE-SECTION and also by the length of each text line
which, in UNIX, is delimited by the line-feed character (hex 0A). On
input, tab characters (hex 09) are expanded into spaces so as to move the
following character onto the next 8 byte boundary (that is, 9, 17, 25,
and so on). Line feed characters are discarded.
Where a line is shorter than the record area, the line is padded with
space characters.
Where the line is longer than the record area, the whole line is passed
as a sequence of records.
On output, trailing space characters are discarded and replaced by a
line-feed character. By default, space characters are not converted to
tab characters.
We recommend that you ensure that line sequential records contain only
printable characters in the range hex 20 to hex 7A, which are not used
with a WRITE statement that contains either the BEFORE or AFTER clause.
Line Sequential Structure.
+-------------------------------------------+--+
| Variable length record |0A|
+---------------------------------+--+------+--+
| Variable length record |0A|
+---------------------------------+--+------+--+
| Variable length record |0A|
+-------------------------------------------+--+
. . .
. . .
. . .
+-------------------------------------------+--+
| Variable length record |0A|
+-------------------------------------------+--+
Relative Organization
Relative files enable you to access data randomly by specifying its
position within the file. Data held in these files can be of any type.
These files consist of a series of fixed or variable length records.
Each record is uniquely identified by a record number. The first record
in the file is record number one, the second is record number two, and so
on. The highest record number shows the total length of the file.
The format of relative files is similar to that of sequential files,
except that each record is terminated by a one-byte control field whose
value indicates whether or not that record logically exists in the file.
This control field can have one of the following values:
hex 0A the record exists and can be accessed by the
program
hex 00 the record has been deleted and cannot be accessed
by a program
This control field is not included in the definition of the record length
in the FD entry.
When you delete a record from a relative file, the value in that record's
control field is changed from hex 0A to hex 00. The data itself remains
in the file in its original position. You can read a fixed length record
in a relative file by declaring the file as a record sequential file with
the record length n+1, where n is the record length of the relative file.
This enables you to read records which have been deleted.
If, for security reasons, you want to ensure that deleted data can never
be accessed, you must overwrite the record before deleting it.
Fixed Format Relative Structure.
A fixed format relative file is the same as a fixed format sequential
file, except each record is followed by the two byte marker.
+-------------------------------------------+--+
| Fixed length record - Record 1 |0A|
+-------------------------------------------+--+
| Fixed length record - Record 2 |0A|
+-------------------------------------------+--+
. . .
. . .
+-------------------------------------------+--+
| Fixed length record - Record i deleted |00|
+-------------------------------------------+--+
. . .
. . .
+-------------------------------------------+--+
| Fixed length record - Record j - unused |00|
+-------------------------------------------+--+
. . .
. . .
+-------------------------------------------+--+
| Fixed length record - Record n |0A|
+-------------------------------------------+--+
For relative files in random access, writing records 1, 2 and 9 will
occupy the same disk space as creating a file containing records 1, 2 and
3.
Variable Format Relative Structure.
A variable format relative file follows the basic variable structure
defined earlier in this chapter. However, each record is placed into a
fixed length slot, the length of the slot being the length of the longest
record defined. The record header for each record contains the length of
the logical record written, not the length of the physical fixed length
slot. Each slot is followed by the two byte marker.
+--------------------------------------------------+
| File Header record - 128 bytes |
| |
+-------+--------------------------------+------+--+
|Header |Variable length record-Record 1 | pad |0A|
+-------+--------------------------------+------+--+
|Header |Variable length record-Record 2 |0A|
+-------+--------------------------------+------+--+
|Header |Variable length record-Record 3 | pad |0A|
+-------+--------------------------------+------+--+
. . . .
. . . .
+-------+---------------------------------------+--+
|Header |Variable length record-Record i delete |00|
+-------+---------------------------------------+--+
. . . .
. . . .
+-------+---------------------------------------+--+
|Header |Variable length record-Record j unused |00|
+-------+---------------------------------------+--+
. . . .
. . . .
+-------+-------------------------------+-------+--+
|Header |Variable length record-Record n| pad |0A|
+-------+-------------------------------+-------+--+
Micro Focus Indexed Sequential Organization
Micro Focus indexed sequential (indexed) files consist of a series of
fixed or variable length records. An indexed file is implemented as two
separate files; the data file and the key or index file. Variable length
records are handled by the variable length file handler supplied with
this COBOL system. For Micro Focus format files, both the data and index
files are of the variable structure defined in the section "Variable
Format Sequential Structure" earlier in this chapter.
If you are using C-ISAM, this handles all fixed length indexed records
and the data file is in relative format. If the C-ISAM file handler is
not the default one supplied with this system, or you have substituted
your own file handler for the default as described in Chapter 14 ,
Callable File Handler, the format of the data file is dependent on file
handler you are using. See your Release Notes for details of the default
file handlers supplied with this COBOL system.
It is possible to use an environment variable to specify that the index
and data files should appear in separate directories. See the section
The "&dquote; Character in Environment Variables in Chapter 14 , COBOL
File Handling for further information.
When you name the file, the name is given to the data file; the name of
the associated index file is produced by adding a .idx extension to the
data file name. The name you give can be only one element long; that is,
it cannot contain a "/" character.
For example:
Data file Index file
myfile myfile.idx
clock.fle clock.fle.idx
You should avoid using the .idx extension in other contexts.
The index is built up as an inverted tree structure that grows in height
as records are added. The number of key file accesses required to locate
a randomly selected record depends primarily on the number of records in
the file and the key-length.
File I/O is faster when reading the file sequentially, but only if other
indexed sequential operations do not intervene.
We strongly recommend that you take regular backups of all file types.
There are, however, situations with indexed files (for example, media
corruption) that can lead to only one of the two files becoming unusable.
If the index file is lost in this way, you can recover data records from
just the data file (although not in key sequence) and, therefore, reduce
the time lost due to a failure. As an aid to this, all unused data
records are marked as deleted at the relative file level by adding one
byte to each record that contains LOW-VALUES. For undeleted records, this
byte contains the character hex 0A.
The recovery operation can, therefore, be performed with a simple COBOL
program by defining the data file as ORGANIZATION RELATIVE ACCESS
SEQUENTIAL. The records are then read sequentially, the data MOVEd from
the relative file record area into the indexed (sequential) record area
and written to a new version of the indexed file. Those records with
LOW-VALUES in the last (extra) byte are discarded. Note that this byte
(containing a line feed character in a required record) is not written to
the indexed sequential file on recovery, because of the record length
discrepancy of one byte in the record definitions. The recovery can also
be performed using the fhrebuild utility. See Chapter 13 , File
Handler Utilities for further information on fhrebuild.
Micro Focus Indexed Sequential Index File Structure.
An indexed sequential file can have several keys. For each key defined,
the index file contains an independent index, structured as a
B-Tree. A leaf
node in an index contains a list of consecutive key-values in ascending
order, each of which points to the data record (in the data file) to
which it belongs. A non-leaf node contains a list of key-values which
are non-consecutive, but in ascending order. Each of these key-values
points to a subordinate node in which the key-value is the largest.
This
index structure provides both the fastest possible random access to a
data record using any key, as well as efficient processing of data
records in sequential key order. The
index file starts with the index
File Header
Record, which contains information about the file. It points to the Free
Space Record,which is used to maintain a list of free records in the
index file. The index File Header Record also points to the Key
Information Record,which contains details of every key defined for the
file, and, for each key, points to the root Index Node Record of the
associated index. Each of these records is described in the following
sections.
Index File Header Record
The File Header Record is located at offset 0 within the index file. The
first 128 bytes are the same as a standard variable structure file header
record, except for the fields below:
index File Header Record description:
-------------------------------------------------------------------
Offset Size Description of the field
-------------------------------------------------------------------
0 4 Length of the file header.
39 1 Organization of the file. Always contains value 2
(for Indexed organization).
62 14 Always contains zeros.
76 1 Security Level. 0 is the highest level, 4 is the
lowest level. Always set to 4.
120 8 Address of logical end of the index file.
-------------------------------------------------------------------
The remainder of the index file header record contains the following
fields:
-------------------------------------------------------------------
Offset Size Description of the field
-------------------------------------------------------------------
128 8 Address of logical end of data file.
136 1 Value 2.
137 1 Value 2.
138 1 Value 4.
139 1 Value 4.
140 2 Contains the number of keys defined for the file.
142 1 Value 0.
143 1 Number of bytes used for occurrence numbers in
indexes where duplicates are permitted. Value 2.
144 8 Address of first Key Information Record.
152 8 Address of the Free Space Record for the data file.
For fixed format files, this is a record in the
index file of the same format as the index Free
Space Record, but the addresses point to free
records in the data file. For variable format
files, this is the address in the data file of the
data Free Space Record. This record has a
different structure to the index Free Space Record.
160 8 Address of first Free Space Record in index file.
168 4 Value zeros.
172 4 Index file record length.
176 8 Value zeros.
184 328 Reserved. Value zeros.
-------------------------------------------------------------------
Note:
"Address" is the offset from the beginning of the file of the
first byte of the structure addressed.
Free Space Record
The Free Space record is a record equal to the nodesize of your file and
contains the location of free records in the index file. Continuation
records of the same size and structure are created as needed, each
pointing to the next.
The first Free Space Record is pointed to by the File Header Record.
Free Space Record description:
---------------------------------------------------------------------
Size Description of the field
---------------------------------------------------------------------
2 Bit 15 - Security Flag. Value 0. Bits 14-0 - Pointer to
end of last free record address entry, relative to start of
this record.
4 Address of Free Space continuation record. Zero if no
further continuation records.
4 Address of a free record in index file
. . . . .
. . . . .
4 Address of a free record in index file
2 Bit 15 - Security Flag. Value 0. Bits 14-0 - Reserved.
Value x"7F".
---------------------------------------------------------------------
Key Information Record
The Key Information Record is a record equal in size to the index mode
size for your file. It describes the physical characteristics of all the
keys used in the indexed file, including the length of each key; where
the key is defined within the data record; whether duplicates are
permitted, and so on. The File Header Record points to the Key
Information Record.
Within the Key Information Record structure is a sub-structure, the Key
Block. A Key Block is created for each key defined. The first Key Block
always describes the prime key. Subsequent Key Blocks define the
alternate keys in the order specified when the file is defined.
If the Key Information Record is not big enough to hold Key Blocks for
all the keys defined, equal sized continuation records are created, each
pointing to the next, until all the keys have been defined.
Key Information Record description:
---------------------------------------------------------------------
Size Description of the field
---------------------------------------------------------------------
2 Bit 15 Security Flag. Value 0.
Bits 14-0 Pointer to end of last Key Block entry
in this record relative to start of
this record.
4 Address of Key Information continuation record. Zero if no
further continuation records.
n Key Block for prime key
( . . )
( . . ) One for each alternate key in file
(n Key Block )
1 Reserved. Value x"FF".
1 Reserved. Value x"7E".
---------------------------------------------------------------------
Key Block description:
---------------------------------------------------------------------
Size Description of the field
---------------------------------------------------------------------
2 Length of this entry in bytes.
4 Address of the root Index Node Record for this key
1 Key compression.
Bit 2 - Compression of trailing spaces
Bit 1 - Compression of leading characters
Bit 0 - Compression of duplicates
5 Key-Component Block
(...) If key is split, one block
(...) per component
( 5 Key-Component Block )
---------------------------------------------------------------------
Key-Component Block description:
---------------------------------------------------------------------
Size Description of the field
---------------------------------------------------------------------
2 Bit 15 Duplicates permitted flag. If set,
duplicates are permitted.
Bits 14-0 Length of component in bytes.
2 Offset of component within data record, starting at 0.
1 Component type. Value zeros.
---------------------------------------------------------------------
Index Node Record
For each key defined, a complete and independent index is constructed.
It consists of a tree of Index Node Records, each record being the size
of your index node and containing actual key-values associated with data
records written to the indexed file. Every key-value in a node will
point either to a subordinate Index Node Record or, if it is a leaf node,
to the data record associated with the key. The top level node is called
the root.
Index Node Record description:
---------------------------------------------------------------------
Size Description of the field
---------------------------------------------------------------------
2 Bit 15 Security Flag. Value 0.
Bits 14-0 Pointer to end of last Key-Value Block
in this record, relative to the start
of this record.
n Key-Value Block
. .
. .
n Key-Value Block
1 Index number.
The value is the same for all nodes belonging to the same
index tree. Contains zeros if prime key.
1 Bit 7 Security flag. Should be 0.
Bits 6-0 Level of this node. Leaf nodes are
level 0.
---------------------------------------------------------------------
Key-Value Block description:
---------------------------------------------------------------------
Size Description of the field
---------------------------------------------------------------------
1 (Optional) Compression character count.
This field is present only if compression is enabled for
this key. It contains a count of the number of characters
(leading or trailing) that have been suppressed.
n Key-value.
2 Optional. Duplicate occurrence number.
This field is present only if duplicates are allowed for
this key. It contains the duplicate occurrence count. The
first key stored that is a duplicate has this field set to
1. Second duplicate has this field set to 2, and so on.
4 Bit 31 Reserved. Value 0.
Bits 30-0 Address of the data record in the data
file if this is a leaf node;
otherwise, the address of the
subordinate Index Node Record in the
index file.
---------------------------------------------------------------------
Micro Focus Indexed Sequential Data File Structure.
The
data file of an indexed file is a variable format sequential file. The
structure of such a file is described earlier in this chapter. This file
contains all the data records. It can be processed as a sequential file
by defining the file as ORGANIZATION SEQUENTIAL, ACCESS SEQUENTIAL, and
adding a RECORDING MODE IS V clause to the otherwise unchanged FD. The
file can then be opened and read sequentially. Since the data file is
not ordered in any particular way, the records read should not be
expected in a consistent order.
Information about free records in the data file is maintained so that
space created by deleting records can be re-used, preventing the file
from growing too quickly. In a fixed format indexed file, this
information is held in a
Free Space Record in the index file. This record has the same structure
as the index Free Space Record, except the addresses point to data file
records. In a variable format file the information is held in a system
record in the
data file.
In a variable structure data file, all record slots are a multiple of 4
bytes. For each slot length in a variable format file, a chain is
maintained for all slots of that length that are free. The start of the
chain for all lengths is maintained in the Data Free Space Record in the
data file. This system record is always the same length as the maximum
slot length defined for the file.
Each free slot pointed to contains the address of the next free slot of
the same length in the first four bytes. The last slot in the chain
contains an address of zero.
Data Free Space Record description:
-------------------------------------------------------------------
Offset Size Description of the field
-------------------------------------------------------------------
0 4 Address of first free data slot of length 8 bytes.
4 4 Address of first free data slot of length 12 bytes.
. . . . . .
. . . . . .
n 4 Address of first free data slot of maximum length.
-------------------------------------------------------------------