KSAM EXTRA DATA SEGMENTS

Another factor that may affect performance when KSAM files are used, is the number and size of KSAM extra data segments. An extra data segment (XDS) is an area in memory used as a buffer during KSAM file access. Each extra data segment contains:

Statistical information on file use (listed by VERIFY command);
Control Block and Key Descriptor Block data from the first two sectors of each KSAM key file;
A Working Storage area large enough to hold a data record and two key entries;
A data block buffer large enough to hold a block from the data file;
At least one, and up to 20, key block buffers each large enough to hold one key block.

When the key file is searched for a particular data record, the root block and lower level blocks, as needed, are moved to key block buffers in the extra data segment. Key entries are compared in the working storage area. When the data block is located, it is moved to the data block buffer of the extra data segment, and when the particular data record is located, it is moved to the working storage area.

Since each open KSAM file is allocated an extra data segment and each extra data segment can be up to 32K words long (32,767 words), KSAM files can use a lot of memory. When there is not enough room in memory for all the extra data segments, they must be swapped between memory and disc as needed. This swapping can slow access to KSAM files.

In order to minimize swapping, you can reduce the number of KSAM files by combining several files into one file. This automatically reduces the number of extra data segments, and it can be a very effective way to improve performance, particularly when files are shared by a number of users. (Refer to Number of Extra Data Segments, below.)

Decreasing the overall size of the extra data segment may reduce swapping of extra data segments. However, reducing the number of key block buffers in the extra data segment may increase swapping of key blocks between the key file and the extra segment during a file search. By default, KSAM allocates key block buffers according to a formula that takes into account the type of access for which the file is opened, the number of levels in the key file structure, and the number of alternate keys in the file. Since this formula (see Table B-1 “Number of Key Block Buffers Assigned by Default”) keeps the extra data segment size as small as is compatible with efficiency, the default number of key block buffers should be used except in special cases. (For details, refer to Extra Data Segment Size later in this section.)

NUMBER OF EXTRA DATA SEGMENTS

KSAM assigns an extra data segment to each KSAM file opened by an active process. Since more than one process can use the same file during shared access, one file may require a number of extra data segments. Thus, the number of extra data segments depends both on the number of KSAM files and the number of users concurrently using the file. (Refer to Figure B-10 “Extra Data Segments for Shared Access”.)

Figure B-10 Extra Data Segments for Shared Access

EXTRA DATA SEGMENT SIZE

The size of each extra data segment associated with an open KSAM file is determined by the number of key block buffers it contains, the size of each key block buffer, the size of the data block buffer, and to a lesser extent, the key entry size and the data record size. (Refer to Figure B-11 “KSAM Extra Data Segment”.)

Initially (when a file is opened), 12K words are allocated to the extra data segment. If less actual space is needed, the extra space is not used, but remains in virtual memory. If more is needed, the original extra data segment is released and a new extra data segment is allocated with the actual size needed.

The maximum size of any extra data segment is 32K words. The actual size is calculated from:

The total size of the overhead statistics and working storage area;
The size of the data block buffer;
The size of each key block buffer and the number of buffers allocated.

The overhead statistics and working storage area is approximately 1-1/2K bytes long depending on variables such as the key entry size and the data entry size. The data block buffer size is based on the size of each data block in the data file. Each key block buffer must be large enough to contain all the key entries in a key block plus one key entry used when new keys are added to a full block (as described earlier, see Figure B-2 “Split Causes New Level in Tree”).

The default key block size is 2K bytes (1024 words) and the maximum size of the key block buffer is 4K bytes (2048 words). If a key block is larger than 4K bytes, KSAM reduces the block size so that no block is larger than will fit in an extra data segment key buffer. Thus, the main variable in extra data segment size is the number of key block buffers.

Figure B-11 KSAM Extra Data Segment

NUMBER OF KEY BLOCK BUFFERS.

The number of key block buffers depends on the type of access for which the file is opened, the number of keys in the file, and the number of levels in the tree structure for each key. (Refer to Table B-1 “Number of Key Block Buffers Assigned by Default” for details.) The least number of buffers is allocated for read only access, unless the primary key has many levels in its structure. More buffers are usually required for write only, read/write, or update access. The number of buffers for read only access increases with the number of levels used by the key, but is never less than one. The number of buffers for write only access increases with the number of alternate keys in the file, but is never less than six. The number of buffers for all other types of access increases with the number of alternate keys and with the number of levels for each key, but is never less than four.

Unless you specify a particular number of key block buffers, KSAM allocates buffers in the extra data segment according to the file characteristics as shown in Table B-1 “Number of Key Block Buffers Assigned by Default”.

Table B-1 Number of Key Block Buffers Assigned by Default

Access Type	Buffers Assigned
Read Only Access	1 buffer per level in key with most levels (minimum of 1 buffer up to 20 buffers)
Write Only Access	3 buffers per primary key + 3 buffers per alternate key + 3 buffers (minimum of 3 buffers up to 20 buffers)
Other Access (Read/Write or Update)	1 buffer per level in primary key structure + 1 buffer per level in alternate key structure + 3 buffers (minimum of 3 buffers up to a maximum of 20 buffers)

Note that you can determine the number of levels per key with the KSAMUTIL command, VERIFY.

For example, if the file is opened for read only, and the only key needs two levels, two key block buffers are allocated. If the file is opened for write only, and there is one alternate key in the file, nine key block buffers are allocated. If this same file is opened for update access, the primary key uses two levels, and the alternate uses three, a total of eight buffers is allocated.

If you want to override the number of key block buffers assigned by default, you can use the MPE :FILE command before opening the file, or set the numbuffers parameter of FOPEN when you open the file programmatically.

The file equation is specified as follows:

 

      :FILE filename; DEV=,,#buffers

The KSAM extra data segment will be allocated space for as many key block buffers as you specify, up to a maximum of 20. (Note that the third DEV= parameter is interpreted as the number of key block buffers only when the file name is a KSAM file; for standard MPE files, this parameter indicates the number of list copies of the file.)

Another way to reduce the number of key block buffers is to use fewer alternate keys, or to adjust the blocking factor so that the key file structure uses fewer levels. Either of these methods is effective when the file is written to or updated more than it is simply read.

Note that when you are loading a KSAM file with large amounts of data, you should increase the number of key buffers. The more key buffers in the extra data segment, the more likely it is that, as new data is added, locations for the new key values will be found in memory. This cuts down on disc access and can significantly reduce the time it takes to load the file.

For example, if you are reloading a KSAM file after a system failure, you should use the :FILE command to increase the number of buffers to maximum of 20 buffers. Then, after the file is loaded, you can allow it to revert to the default number of buffers established by KSAM for the particular file.

EXTRA DATA SEGMENTS WITH SHARED ACCESS

The extra data segment allocated to each open file acts as a control block for that file. The extra data segment contains not only the current data block and the current key block buffers, but also the latest control information for the file. This information includes the logical and chronological record pointers that indicate the current record being accessed. Because the current pointer position is not in a "common block", when several programs open the same file, each can alter the key file structure by adding or deleting records so that the pointers set by other programs may point to the wrong record without those other programs being aware of it.

To make sure that the latest pointer position is stored with the file rather than in the separate extra data segments, programs that share the same KSAM file must use a locking scheme. Whenever a program locks a KSAM file, the control information is transferred from the file to the extra data segment; and when a program unlocks the file, the contents of the extra data segment is written back to the file. Thus, each program should lock a KSAM file before executing any procedure that positions a record pointer (pointer-independent procedures), and not unlock the file until all procedures that depend on this pointer position (pointer-dependent procedures) have completed execution. This is true regardless of whether the pointer is chronological (points to a record in the data file) or is logical (points to a key in the key file). Both types of pointer are maintained in the extra data segment for the open file.

Table B-2 “Pointer Dependence” lists all the procedures that affect or are affected by the record pointers.

Table B-2 Pointer Dependence

Pointer-Independent Procedures	Pointer Type	Pointer-Dependent Procedures	Pointer Type
FFINDBYKEY CKSTART BKSTART	Logical	FREAD CKREAD BKREAD	Logical
FFINDN	Logical	FSPACE	Logical
FREADBYKEY CKREADBYKEY BKREADBYKEY	Logical	FREMOVE CKDELETE BKDELETE	Logical
FWRITE CKWRITE BKWRITE	Logical	FUPDATE CKREWRITE BKREWRITE	Logical
FPOINT	Chronological	FREADC	Chronological
FREADDIR	Chronological^[1]
^[1]
^[1]* Each procedure that sets the logical pointer also sets the chronological pointer; but only FPOINT sets the logical pointer as well as the chronological pointer.

The pointer-independent calls position the pointer regardless of its current position. Pointerpendent calls, on the other hand, must know to which record the pointer is currently positioned in order to operate correctly.

All the procedures listed in Table B-2 “Pointer Dependence” affect the pointer in some way. In order to use these procedures correctly, it is important to understand how each moves the pointer, whether it positions the pointer directly or advances it from its current position.

In general, when access to the file is random, the pointer is positioned directly. For example, a call to FFINDBYKEY (or CKSTART or BKSTART) positions the logical pointer to a particular key in the key file based on a key value specified in the call; and a call to FPOINT positions the chronological pointer to a particular record determined by its chronological record number.

When access to a file is sequential or the file is being modified, pointer positioning is not direct but is relative to its previous position. Depending on the sequence in which procedures are executed, the pointer may or may not be advanced to the next record in key or chronological sequence. Internally, a flag is used to indicate whether or not to advance the pointer. This flag, the "Do Not Advance" flag, is set to FALSE if the pointer is to be advanced sequentially, to TRUE if it is not to be advanced. Some procedures never test the flag; these are, in general, the pointer-independent procedures that set the pointer directly. Other procedures test the flag and advance the pointer if the flag is FALSE; generally, these are procedures that read the file or position the pointer sequentially. Table B-3 “Record Pointer Summary” summarizes when the pointer is set or advanced. (Note that only SPL procedures are listed; check Table B-2 “Pointer Dependence” for the equivalent BASIC or COBOL procedures.)

Table B-3 Record Pointer Summary

For example, if you call FREADBYKEY, it positions the pointer to a specified key value. After the call, the logical pointer remains positioned to this key and the Do Not Advance flag is set to FALSE. If the next call is to FREAD, FSPACE, or FREADC, then the pointer is advanced to the next key in key sequence before these procedures perform their other functions. Thus, after FREADBYKEY, a call toFREAD will read the next record, not reread the same record, and a call to FSPACE will move the pointer relative to the record following the record just read.

KSAM EXTRA DATA SEGMENTS

Technical documentation

» Table of Contents

NUMBER OF EXTRA DATA SEGMENTS

EXTRA DATA SEGMENT SIZE

NUMBER OF KEY BLOCK BUFFERS.

EXTRA DATA SEGMENTS WITH SHARED ACCESS