Large Transactions for IMAGE Users
MPE documents | |
 Complete PDF | |
| Table of Contents | |
| Index |
Commercial Systems Division
Background
Of late many of the users of TurboIMAGE and IMAGE/SQL have the requirement of transaction sizes larger than is currently possible today (4 MB). This is true, especially for users who run applications involving bulk puts/ deletes /updates. The situation becomes more difficult with the limitation of the size of the Transaction Manager (XM) userlog, if the transactions are on the same volume set. This integrated solution involving TurboIMAGE/iX, XM, IMAGE/SQL and ALLBASE/SQL addresses these problems. The following sections discuss these issues, and various options available to the users to overcome these issues, as a result of this solution.
Current Limitations
Version C.03.00 of TurboImage introduced the concept of dynamic transaction, meaning the application can rollback a transaction at runtime. Later when the Transaction Manager started handling the dynamic transactions, some customers used to see the stalled transaction message due to the limit on the size of the transaction.
- Transaction Size
Dynamic transactions are the ones which are bracketed by DBXBEGIN and DBXEND or DBXUNDO, which may span a single database or multiple databases. In the case of multiple database transactions the total number of databases which can take part in a transaction is restricted to 15. A dynamic transaction can be rolled back using DBXUNDO or automatically when the application aborts within a transaction. When an application does a bulk insert, delete or update, there is a chance that it hits the current limit of 4 MB on transaction size. Until recently, the process used to abort under such situation, after displaying an error message that the transaction size has exceeded. An intermediate solution has been provided (patch MPEKX35) to rollback the transaction before process abort, so as to leave the database in a consistent state, and user could re-run the application after manually pruning the application. However, this was not a comfortable alternative, and users have been requesting that there be an increase in the transaction size.
- User Log Size
The user log related limit arises due to the fact that XM in MPE is log-based. That is all the transactions, mainly the ones which involve manipulation to the database, are logged on to a circular log file. When there are many applications writing into the XM log file, it could also get filled up easily, again resulting in process abort. The userlog (all references to userlog in this article refers to XM userlog) comes into the picture here because all the changes to IMAGE databases on a specific volume set are written to the same userlog. If only the transaction size limit is increased, because of the increased number of open transactions, the 64 MB limit is easily attained.
Solution Objective
The main objective of this solution therefore has been to increase the transaction size from the existing 4 MB. The second objective is to increase the size of the XM userlog from the current value of 64 MB to support large transactions.The third objective is to ensure compatibility with existing applications, so that they can run without any problems even with these enhancements.Overview of the Solution
In order to achieve the above mentioned objectives, the capabilities of the following products /subsystems of MPE/iX have been enhanced:Transaction Manager
TurboImage/iX
Image/SQL
ALLBASE/SQL
Transaction Manager
A patch MPEKXG8 was released (on MPE/iX 5.5) to cater to the increase in number of transactions by reducing the data stored in a transaction. This was made possible, by keeping only the essential information required for possible rollback in future. A large amount of internal data becomes irrelevant after completion of an extended transaction (we will refer any dbput/ dbdelete/ dbupdate as an extended transaction throughout this article). This allows for more transactions even within the current 4 MB limit. However, the long term solution addressed here, calls for increasing the transaction limit to larger values. Based on current OS limits, the maximum transaction size has been increased to 32 MB in this solution. In addition to this, XM has been changed to send out an early warning message (soft limit warning), before the transaction size limit of 32 MB is hit (i.e. at 28 MB). When, ultimately the transaction size reaches 32 MB, it aborts the process and rollback the transaction. Another enhancement in XM is to increase the userlog. The current userlog size is 64 MB per volume set. Currently, for each transaction, the information that can be contained in the userlog is limited to 4 MB. Due to the increase in the transaction size for each process (to 32 MB), the total userlog file should also be increased. The userlog resides on the master volume of a volume set. User can use VOLUTIL utility to increase the size of the userlog (the default value is 64 MB) with the following,>From VOLUTIL, ALTUSERLOG <volset-name> <new size>This command accepts the volume set name and the new size of XM userlog for it in sectors. It can not be less than 264256 sectors corresponding to 64 MB. One caution is that ldev 1 has limited reserved space and one can not expand the XM userlog beyond that. The actual expansion of the XM userlog happens when user boots the system next time. This is because the XM log files can not be expanded/truncated while in use. During bootup, just after recovery of the userlog, the XM userlog is expanded and a message is written onto the console. One can also truncate the userlog after expansion if there are any disk space or performance issues. The volutil interface is same for this operation too. One can backdate the OS to an earlier version with the userlog in the expanded state. The expansion of the userlog may fail due to one or more reasons. In that case, the system reports it through a warning message on the console. If it fails due to lack of disk space, make sure that there is sufficient contiguous space on the master that would fit the need for the increment in the userlog size. Then reboot the system so as to enable it to complete the expansion of userlog. You must ensure that memory beyond 256 MB is available on the system, before increasing the userlog through VOLUTIL.
 |
NOTE: The userlog size gets scaled between 32 MB and 64 MB if the memory size available on the system is > 64 MB and < 256 MB. |
>From VOLUTIL, SHOWUSERLOG <volset-name>This command accepts a valid MPE volume set name and shows the current size of the XM userlog in sectors. It has to be noted that, a single transaction can grow up to 32 MB now and if the size of userlog is 64 MB, total size of all active transactions on that volume set can be 32 MB (half log size). If the total size of all active transaction exceeds half log size, all the processes which are actively logging into the same userlog are aborted. So, a large transaction may result in all processes getting aborted, if the userlog size is not expanded appropriately. The downside of having a larger userlog is that the XM checkpoint process has more data to write to disk at reaching the end of each userlog half. This can have effect on response time to on-line users.
TurboIMAGE
Support for Large Transaction Size: Image has been enhanced to support the new 32 MB transaction size provided by XM. This means, without doing anything special, one can have approximately more than 8 times the number of extended transactions in a dynamic transaction than possible today (including the XM optimization available with XM patch MPEKXG8, which has been rolled into this solution). User Control of Transactions: In addition to the benefit of transaction sizes up to 32 MB, available across the board to all IMAGE and IMAGE/SQL users, direct IMAGE applications benefit from a new feature. This feature helps a user to judiciously control/manage large transactions. User application can avail of an early warning message feature through which the application can have virtually a full control over the transaction. To avail this facility, a new mode (mode 18) has been added to DBCONTROL intrinsic. If the application makes a call to DBCONTROL mode 18 before the start of a transaction, it sets a flag in IMAGE, to trap the softlimit warning message from XM, and inhibits all new transactions. Now the user application have two options:to commit the transaction (through a call to DBXEND)
to rollback the transaction (through a call to DBXUNDO)
DBCONTROL Mode 18
DBXBEGIN
Intrinsics like DBPUT, DBDELETE or DBUPDATE
Receives soft limit warning (message # -1025) from the modify intrinsics DBPUT/DBDELETE/DBUPDATE, when softlimit is reached.
DBXUNDO / DBXEND
NOTE: DBCONTROL Mode 18, has to be called before the start of a dynamic transaction, that is, before DBXBEGIN. (Otherwise, it returns an error message, and does not have any effect on the transaction manager). However, DBCONTROL Mode 19, can be called anywhere in the program. It turns off the detection of the softlimit warning message from XM, regardless of whether it is set or reset.
Image/SQL and ALLBASE/SQL
For the benefit of users doing bulk deletes, ALLBASE has a "DELETE WITH AUTOCOMMIT" feature for ALLBASE tables. Now this facility has been extended to IMAGE tables as well. IMAGE/SQL and ALLBASE/SQL have been enhanced to support this. The command would be "DELETE WITH AUTOCOMMIT FROM" owner.table [WHERE search conditions]. However, this option has to be exercised with great care, as IMAGE/SQL does periodic commits at internally maintained intervals, and there is not any way to rollback the deletes. For bulk deletes, the autocommit feature is very useful in that, chances of hitting the 32 MB limit is remote.Limitations
If MPE/IX 6.5 is installed on a system, then TurboIMAGE/iX version C.08.00 also needs to be installed. However, this Image version can work with older versions of XM as well. In this case, XM will support transaction sizes only up to 4 MB, and will not issue a softlimit warning message. If there are several processes having large transactions, and if XM user log has not been expanded appropriately, all the processes may get aborted once the log becomes full.Summary of Benefits to the Users
Across the board benefit of transaction sizes up to 32 MB for all IMAGE and IMAGE/SQL users
Can accommodates more number of extended transactions in a top level dynamic transaction
Can accommodates more number of extended transactions in a top level dynamic transaction
No need to change existing applications to get the benefit of increased transaction size
User control of dynamic transaction, if desired, to avoid stalled transaction
Facility of "DELETE WITH AUTCOMMIT" feature to IMAGE tables
User configurable XM userlog
|
Chapter 5 Internet and Interoperability |
XM User Checkpoint Priority Control |