EMEA DEVELOPMENT STANDARD

CHECKPOINT RESTART

Prepared by: Development CentreDate: 15th January 1999

Table of Contents1.0 An overview to checkpoint restart................................................................................................................. 3

1.1 Why it is necessary................................................................................................................................... 31.2 How it works............................................................................................................................................. 31.3 Database Backout..................................................................................................................................... 41.4 ARC......................................................................................................................................................... 51.5 Standard for Checkpointing and Restart....................................................................................................5

2.0 Checkpoint/Restart and Program Structure....................................................................................................52.1 Overview.................................................................................................................................................. 52.2 How to identify the best L.U.W................................................................................................................. 52.3 When to issue checkpoints........................................................................................................................ 62.4 Where to code checkpoint/restart logic......................................................................................................72.5 How to use the checkpoint area itself........................................................................................................ 72.6 Initialising items which are stored in the checkpoint area..........................................................................72.7 Database Repositioning after XRST.......................................................................................................... 8

3.0 Testing a Checkpoint/Restart Program..........................................................................................................83.1 Testing Procedure..................................................................................................................................... 83.2 ARC Pacing Class..................................................................................................................................... 93.3 Use of ARC Online Panels........................................................................................................................ 9

4.0 GSAM Files................................................................................................................................................ 114.1 Accessing GSAM files............................................................................................................................ 114.2 GSAM variable length record files.......................................................................................................... 12

5.0 System Design Considerations.................................................................................................................... 145.1 Drivers.................................................................................................................................................... 145.2 Reports.................................................................................................................................................... 145.3 GSAM to GSAM..................................................................................................................................... 145.4 VSAM files............................................................................................................................................. 145.5 Sorts........................................................................................................................................................ 15

6.0 Checkpoint/Restart with Other Products.....................................................................................................166.1 Director................................................................................................................................................... 166.2 Other Products....................................................................................................................................... 18

7.0 Defunct Checkpoint/Restart Components....................................................................................................197.1 BMP Restart............................................................................................................................................ 197.2 CBLTBMC............................................................................................................................................. 197.3 CBLTDLX.............................................................................................................................................. 197.4 SPLTDLI................................................................................................................................................ 19

Appendix A - DB2 Restart Logic...................................................................................................................... 20A.1 Simple Key........................................................................................................................................... 20A.2 Compound Key - Wrong!!...................................................................................................................... 20A.3 Compound Key - Right........................................................................................................................... 21A.4 Compound Key - Best........................................................................................................................... 21

-2-

1.0 An overview to checkpoint restart

1.1 Why it is necessary There are two main reasons why all batch programs using databases at Brighton should take checkpoints and why some need to be 'checkpoint/restartable'.

Firstly, if a job runs for a long time, then it can be a severe waste of expensive machine resources to have to re-run it from the beginning if, for example, there is a hardware problem. Some batch programs run for many hours: if these abend it would be extremely inefficient to have to re-run them from scratch; what is needed is the ability to make them 'start where they left off'.

Secondly, we operate a block-level data sharing environment where the databases can be accessed in parallel both for read and update access across online and batch systems. The system software which controls access to databases has to ensure that databases which are being used simultaneously do not get corrupted. Further, even if this is achieved, it also has to make certain that programs do not get inconsistent views of the databases because, say, some segments have been updated by a program which is still in the process of updating others.

IMS makes this happen through the use of locks which are managed by the IMS/VS Resource Lock Manager (IRLM). The IRLM actually supports eight levels of locking, but for simplicity’s sake we can think just in terms of update locks and read locks.

As an application program updates a DL/1 database, all of the segments in that database record (i.e. the root segment and all of its dependants) are locked up so that no other program can look at them. (The lock may be effected at the higher level of physical block: not only the database record in question, but any others sharing the same block in the database, would then be locked up). As the program proceeds, updating other segments in other database records, more and more of the database is being locked up so that other programs cannot read it. The updates to the database will be happening physically, but the other programs are barred from looking at the results of those updates.

In the same way, a program which is reading a DL/1 database will acquire read locks. These locks will allow other programs to read the same data but will prevent parallel update access until the lock is released.

The thing that stops this process continuing until the entire database is unavailable to any other program is that the updating program reaches a 'sychronization point'. This happens when a CHKP call is issued in a batch program.

Similarly, DB2 establishes locks in order to serialise access to data. These locks will be released when the application issues a COMMIT call. Both IMS and DB2 data may be accessed and updated by the same program. When the program issues a CHKP call then a DB2 COMMIT will also be invoked. The two-phase commit process ensures that the data base updates are synchronised between IMS and DB2. This ensures that the integrity of the data is maintained and that both the DB2 updates and the IMS updates in a unit of work are both either committed or backed out. Should it be necessary to recover the IMS data bases to a point in time, the DB2 tables will need to be recovered to the same point in time for the data bases to be consistent

There are a number of problems associated with a program which accesses large areas of a database between synch points:

It will slow down or lock out other programs which want to access the same data. This is particularly undesirable if the other programs are online transactions.

The more locks a program holds, the more likely it is to encounter ‘deadly embrace’ problems in which two programs mutually lock each other out, each of them holding locks that the other is waiting for.

The more locks that the lock managers are having to manage at any one time, the greater will be the system overhead. This problem can be very significant in the case of DB2 in a parallel sysplex environment, where the presence of large numbers of unnecessary read locks can cause serious performance degradation to the whole DB2 subsystem.

There is a particular problem in the situation where one of the subsystems fails. As DB2 restarts it will back out updates for all tasks which were in-flight at the time of failure, back to their last commit point. If a program has run for two hours without taking a commit then it will typically take two hours to back out the updates, thus delaying DB2 availability across the entire system. A similar problem will affect IMS as it backs out in-flight updates for BMP’s and online transactions.

For all of these reasons it is necessary to have batch programs which a) issue DL/I CHKP calls at 'frequent' intervals and b) are capable of restarting processing from the last checkpoint taken. The precise rules for this are given in section 1.5 below.

-3-

1.2 How it works There are two types of checkpointing: (which cannot be mixed in the same program) basic and symbolic. A basic checkpoint call has only one function: it is a synch point, and will cause locks to be released so that other programs can use the affected segments.

A symbolic checkpoint call also does this. However, it has another function in that the application program can store away information which will allow it to restart again if it fails. This information will vary between various different types of programs, but it tends to be things like: the root key of the database record currently being processed, the page number of the report which is being output, accumulators giving financial totals and item counts of whatever has been processed, etc. Symbolic checkpointing is the only one which is of relevance in our environment: basic checkpointing is not (and should not be) used in application programs.

As a side effect of both types of call, the current position on all databases is destroyed and must be re-established, by the application, before any further processing can be done. The exception to this for DB2 is the use of SQL DECLARE CURSOR WITH HOLD which maintains position across a COMMIT call.

There are two other IMS facilities which need to be explained before it is possible to show the logic of a typical checkpoint/restartable program.

Firstly, there is a type of file called a GSAM (Generalised Sequential Access Method) database, which is not really a database at all! It is basically a ordinary sequential file (BSAM, or VSAM ESDS) which is accessed through DL/I ISRT (for writing) and GN (for reading) calls.

It might seem rather perverse to give the ability to access sequential files via DL/I calls, but there is a very good reason. This is that on a restart call (explained below!), GSAM files are positioned to the record being processed at the last checkpoint. Also, after a checkpoint call, positioning is not lost, as it is on true databases.

If this facility did not exist, it would be hard to restart a program which used sequential files as it would have to position itself to the right place. This would involve doing things like reading through the input file until the relevant record had been read, which is all rather messy. The situation is even worse for output files, where the program would have to read from the old output file and write them to the new one until correctly positioned.

Secondly, there is a DL/I call XRST (for restart), which must be the very first call which the program issues. This can do one of two things. Either, it will tell the application that it is not a restart and that processing can continue normally. Or, alternatively, it will tell the program that it is a restart and that special action is necessary. In this case DL/I will do some extra things. The information which the program saved at the last checkpoint call before it failed will be returned to the program. The position is restored for GSAM files, so no application work is needed for them. However, for other databases position is not restored and it is the application’s responsibility to restore position as necessary.

The general scheme of things for a batch program which is checkpoint/restartable is as follows:

It issues a XRST call to IMS, which tells it whether a restart is taking place or not.

If necessary, it then re-opens any DB2 cursors needed and re-positions on the various databases. It then resumes processing using the data which was stored away in the checkpoint area.

Otherwise, it will open any GSAM files which are to be used and, if any of them are for output, it will issue an initial CHKP call to IMS.

It then performs the main logic of the program:

Do the Logical Unit of Work (L.U.W.) - the lowest USEFUL function which it performs in a repetitive way, e.g. process an input record.

Issue a CHKP call.

Re-position on the various databases (but not on DB2 cursors declared WITH HOLD)

until there is no more work to be done.

1.3 Database BackoutWhen a database update program abends it is likely to have performed updates since the last checkpoint was taken. These updates must be backed out in order for data integrity to be maintained as the program restarts. In the production environment this occurs as follows:

DB2 automatically backs out table updates to the last COMMIT point. IMS BMP jobs automatically back out DL/1 updates back to the last checkpoint. IMS stand-alone DL/1 jobs do not backout automatically. However production jobs include a job step which performs

this function, using the IMS feature known as DBRC. In fact, DBRC will prevent access to a database which needs backing-out until this step is performed.

-4-

When testing your checkpoint/restart programs, however, you may need to perform a manual batch backout.

1.4 ARCAnother piece of software which is involved in the process of checkpoint restart is ARC (APPLICATION RESTART CONTROL) from BMC Software. ARC provides a wide range of features but at the time of writing its use is limited to replacing the functionality of its predecessor product, BMP Restart. Broadly, these functions are:

To provide a central repository of the checkpoint/restart characteristics of each group of jobs.

To intercept CHKP calls to ensure that these are passed to IMS at an appropriate rate. Typically checkpoints will only be allowed to be honoured once a certain time period has elapsed. This function is known as pacing.

To maintain the restart information for each job, such that jobs can be restarted without manual intervention.

ARC is administered by Information Management and it is their responsibility to ensure that checkpoint pacing is set correctly. However there are certain features of ARC that can assist you in testing your checkpoint/restartable programs and these are noted below. For more information please refer to the following manuals on BookManager bookshelf BMC Software Products for DB2, IMS, CICS, & MVS:

APPLICATION RESTART CONTROL User Guide V 2.1

APPLICATION RESTART CONTROL Reference Manual V 2.1

1.5 Standard for Checkpointing and Restart The following applies to all programs which are to run in the production environment, whether as a regular scheduled job or a ‘one-off’:

Any program which accesses a DB2 or IMS database, whether read or update, must issue regular checkpoint calls. Any program which updates a DB2 or IMS database must also be written to be checkpoint restartable. Any scheduled program whose longest execution is likely to run longer than 20 minutes elapsed, must also be written to

be checkpoint restartable.

Thus a short-running read-only IMS or DB2 program will issue an XRST call and regular CHKP calls, but can use READ/WRITE instructions for sequential files. They should not use GSAM. Such programs will always run 'from the top', whether starting normally or restarting after an abend. The XRST call is necessary to keep IMS happy, but the program should perform the same 'from the top' processing whether the XRST call indicates normal start or restart.

Other IMS or DB2 programs will be either long running, or do updates. These will issue an XRST call and regular CHKP calls, but in addition will be truly restartable. This means they must use GSAM for sequential files, and will be written and tested so that they use a checkpoint save area, if necessary, and do any required repositioning following an XRST call which indicates a restart. Following an abend, they must rerun from the restart point to completion.

Programs which do not access IMS or DB2 databases need not take checkpoints.

An alternative for one-off read-only DL/1 programs which do not require absolute integrity of extracted data that is to use a PSB with PROCOPT=GO. The access for these programs will ignore locks and therefore they run the risk of reading uncommitted data, which will potentially lack referential integrity with other data read during the same run.

2.0 Checkpoint/Restart and Program Structure

2.1 Overview The following are a set of guidelines for coding checkpoint/restartable programs. They are by no means hard and fast rules but they have proved themselves in existing programs. Generally, checkpoint/restart CAN be installed in programs almost invisibly, if a little care is taken. It can also be fairly easy to retrofit it to an existing program, if that was reasonably well written initially. It can however be a blight if misused, and in programs which have been troublesome, one or more of the following guidelines has invariably been broken.

2.2 How to identify the best L.U.W There are two distinct, and counteracting, forces at work here. For conceptual reasons, it is much clearer if the L.U.W. is quite high (in the hierarchy of control breaks of the program). This tends to reflect the fact that as one moves up the structure of a program, the units of work are more generalised, less detailed, and are nearer to its overall function. On the

-5-

other hand, both for program isolation and checkpointing pacing reasons, there is pressure to make the L.U.W as low as possible.

Usually, working out which is the Logical Unit of Work to use for checkpoint/restart purposes is not too hard. Occasionally, you may stumble across a program which has got several possible candidates. This tends to point towards the fact that the program has got too much function and that it should be reduced into two, or more, smaller programs. If it is a new program then this path should be pursued. If it is an older one, into which checkpoint/restart is being installed, you have my sympathies! - it is likely to cause a good deal of trouble.

In most cases however, there does not tend to be much choice. (Indeed in many instances, there may only be one possible L.U.W., in which case it doesn't need to be thought about much more!). This tends to boil down to transaction driven update programs, which can have many input records which to apply to a single database record. An update program which applies unbilled financial transactions to C/M Accounts, would be a typical example.

The rule is that where there is a choice of L.U.W. then go for the lowest. The pacing function of ARC applied externally to the program will then ensure that honoured checkpoints occur at an appropriate rate. Choosing the lower level L.U.W. may well incur more re-positioning overhead after successful checkpoints. However, transaction driven programs usually have to find the particular database record they are interested in, so the re-positioning tends to be 'lost' into this.

2.3 When to issue checkpoints Whether to issue checkpoint calls before or after the L.U.W is really just a matter of style. It can be important under the wrong circumstances, however. The issue can best be examined by looking at two program fragments:

program 1:

Initialise. XRST call. If restart

reposition on database(s) Otherwise

open files, CHKP call. read input file. While input file non-empty

main logic, CHKP call, if checkpoint honoured

reposition on database(s) (if necessary), read next input record.

Write trailers to o/p files. Close files. Terminate.

-6-

program 2:

Initialise. XRST call. If restart

reposition on database(s) Otherwise

open files. While input file non-empty

read next input record, if not End Of File

CHKP call, if checkpoint honoured

reposition on database(s) (if necessary), main logic.

Write trailers to o/p files. Close files. Terminate.

Generally speaking, most people intuitively prefer the first solution, probably without being able to explain exactly why. There are however some who vociferously advocate the second. This causes confusion amongst those trying to design their programs structure, so the air needs clearing a little. The first method is, in almost every instance, preferable to the second. There are two basic reasons for this.

Firstly, the first method uses the well-known structured technique: 'read ahead'. It is a very useful tool, which should not be discarded lightly, unless there are clear advantages in doing so.

Secondly, the first method is much clearer in its approach to how the checkpoint relates to the program logic. That is to say, the contents of the checkpoint area reflect the L.U.W. which has just been completed. The philosophy is that a checkpoint acknowledges total completion of a particular L.U.W. On the other hand, in the second method a checkpoint indicates that the program is about to start processing a particular L.U.W. This does not seem, to most people, a particularly clear way of expressing what checkpoint/restart is all about.

The above arguments are not meant to show that there is never a case for not using the second method. They should however have convinced you that it shouldn't be used very often. Only if you think that there is a strong case should you design a program this way and you should clear it with a senior developer in your area first.

2.4 Where to code checkpoint/restart logic This might sound, on the face of it, somewhat obvious. You have a section, which is performed in the initialisation phase, which does the necessary work for handling restarts or opening GSAM files. There would also be a section, performed from the main control loop, which does the necessary work for checkpointing (including re-positioning after honoured checkpoint calls, if necessary).

It feels like such a natural way to do it, that it is difficult to conceive how else to approach it. However, if checkpoint/restart is being retrofitted to an existing program, then it is quite possible that this way won't easily fit with the current structure. If this is the case then you are strongly recommended to change that, rather than use a different way of implementing checkpoint/restart. In general, where problems have been found in the past with such exercises, it has been with those programs which are not structured very well. Signs of trouble, are things like references to restart flags in sections which don't have anything to do with it. For example, in a section which reads an input file for the rest of the program, it shouldn't need to look at a flag to see whether it should read the file or take it from the checkpoint area (after a restart). Traces of pathological connections between functionally distinct areas of programs are a sure warning that there could be severe problems ahead.

2.5 How to use the checkpoint area itself 1. One does occasionally stumble across programs in which the entire working storage section of the program has been

turned into a huge checkpoint area. In some respects it does rather neatly solve the problem of what to actually put into the checkpoint area - it can simply be forgotten about. However, there will be performance degradation if the whole of working storage is saved with every checkpoint call - so this method must not be used. The ARC feature ‘Automatic Checkpoints’ also takes this approach and hence is not to be used for the same reason.

2. Do not include the current input record in the checkpoint area. This can cause a problem in the situation where Production Support need to strip an input record prior to a restart. If the input record has been included in the checkpoint area then it will be restored on restart and exactly the same problem which caused the first run to fail will recur, despite the record having been stripped.

-7-

3. Do include some information in the checkpoint area that indicates how far processing has got, even if this is not needed for your restart logic. The availability of such information can occasionally be critical to Production Support resolution of problems where a restarting program is failing. Examples would be an input record count or the current key being processed.

4. With only a part of working storage defined as a checkpoint area, there is some disagreement over how the checkpoint areas are used. There is one school which insists that the checkpoint area should contain copies of the various items which need to be kept: these are moved in from their 'proper' locations before a checkpoint call and moved out after a successful restart call. Whilst this does distinguish between the two uses of the items, it does lead to a couple of problems. The minor problem is that there is a slight overhead caused by moving the items into the checkpoint area: this isn't at all significant in performance terms, but it is worth considering. The major problem is that it is extremely easy for the descriptions (PICtures) of the copies in the checkpoint area to get out of step with the real ones (e.g. careless maintenance). Or alternatively, the relevant item may not actually be stored or retrieved, even though it is defined correctly. It is quite easy for one item to get forgotten when writing a large list of MOVE statements, particularly if this is not in the same sequence as the checkpoint area, which quite often happens (unfortunately!). This has proved to be a real headache with some programs, to the point where it can be very hard to show that they are working correctly. The other school of thought, is that items which need to be saved in the checkpoint area should be grouped there, and used directly, regardless of their prefixes (WA-, WT-, etc) which normally dictates their ordering. This does get around the problem rather neatly; I must admit to being of this school myself.

2.6 Initialising items which are stored in the checkpoint area If it is at all possible, it is worth trying to do this statically, using VALUE clauses, rather than dynamically, using MOVEs in the initialisation code. Doing the initialisation dynamically quite often causes two problems.

Firstly, it complicates the restart logic and makes the load module size slightly larger. Also, as a general principle, it is usually considered healthier to give values to variables statically rather than dynamically (although with large arrays and tables this can be somewhat difficult).

Secondly, and more seriously, it can encourage people into making silly errors which cause the program to crash under some circumstances. Take a program which does the following:

1) XRST call, 2) CHKP call, 3) If NOT restart initialise some stored items, 4) main logic, etc.

Because the initialisation is done after the first checkpoint call (which is always honoured remember), the items are stored on the checkpoint database in uninitialised form. After subsequent honoured checkpoints, the items will then have sensible values in them. If the program attempts a restart after that initial checkpoint it will either fail 0C7 (or some such abend), or worse still simply not work properly - usually causing mayhem and pandemonium in other innocent and unsuspecting programs.

2.7 Database Repositioning after XRSTNotwithstanding what has been said thus far, you should be aware that IMS will attempt to automatically re-establish position on DL/1 databases during an XRST call for a restart. The consensus of opinion is that it is best not to rely on this, so your program should explicitly reposition databases based on information saved in the checkpoint area. The programming and run-time overhead is small and you will know exactly where you stand.

For DB2, Appendix A contains some advice provided by Information Management on how to code DECLARE CURSOR statements that will work correctly on a restart.

3.0 Testing a Checkpoint/Restart Program

3.1 Testing ProcedureThe main body of testing should be exactly as normal - the checkpointing aspects of the program should be tested after the main test plan has been completed satisfactorily.

The idea with checkpoint restart testing is to set up two series of runs, one of which is a control set. During the other set of runs, the program is made to fall over at a reasonably well defined points, is restarted and the output from the combined runs should match that from the control run. It is necessary to perform this test several times, stopping the program at various different points during the sequence. The minimum set of acceptable test points are: restarting after the 1st checkpoint, restarting after the last, and restarting after a failure in the main process loop.

-8-

ARC can help with this process. For example, add the following JCL to the job step to force ARC to abend the job with a U1771 abend after 100 honoured checkpoints have been taken:

//ARCSYSIN DD *

TRMAFTERCKP=100

Initially tests should be performed in such a way that checkpoint requests are always honoured. Add the following JCL to the job step to make this happen:

//ARCSYSIN DD *

PACECHKP=N

You should then run the program using high numbers of records (not actual productions volumes, but more than normal program test volumes), with the checkpoint pacing set as per production. This should prove, once and for all, if the program has got any problems with locking up too much of the enqueue pool.

Finally, two general points on testing. Firstly, if a change needs to be made to a program before it can be restarted then mostly this can be done. It may sound like a rather obvious point but you can't change the definition of the checkpoint area! Or at least, not with any expectation of the program actually running for very long after it has been restarted.

Secondly, if you do not abend immediately after a checkpoint then it may be necessary to manually backout DL/1 database changes that occurred in the period between the last checkpoint call and the abend. The job must be logged (using the IEFRDER ddname) to a file which is then input to the backout utility. For more details on batch backout JCL, see BACKOUT on DEVL.STANDARD.JCL. Batch backout is needed if the program contains non-GSAM DLI updates and is not running as BMP. Programs using SPLTDLI or CBLTDLX always need to run a batch backout, although as noted below we should be eliminating the use of these modules.

For programs running as BMP or programs that only update GSAM files no manual backouts are necessary - for BMP programs, IMS handles the backout, and repositioning occurs automatically on GSAM files.

3.2 ARC Pacing ClassThe ARC Pacing class determines the pacing characteristics of the job. The great majority of production jobs are setup to run with pacing class member PACESK. At the time of writing this is defined to honour a checkpoint every 10 seconds elapsed and to return as status code of ‘SK’ where checkpoints are not honoured.

On Testbed the default pacing class is currently PACECLAS, although this is under review. For compatibility with production add the following to your test JCL:

//ARCSYSIN DD *

PACENAME=PACESK

-9-

3.3 Use of ARC Online PanelsARC provides a range of features to allow you to display and control your checkpoint/restart testing. On Testbed enter ‘TSO BMCAES’ on a command line:

Application Enhancement Series V2.1.01 Command ===> Select an option. Then press Enter. Application Enhancement Series (AES) __ 1. AES records 2. BATCH CONTROL FACILITY (BCF) 3. APPLICATION RESTART CONTROL (AR/CTL) AES Common Options 11. Display, Print, Jobcard, Allocation, and Profile Options 12. Messages 13. Security 14. Product authorization 15. Exit PF 1=HELP 2=SPLIT 3=END 4=RETURN 5=RFIND 6=RCHANGE PF 7=UP 8=DOWN 9=SWAP 10=LEFT 11=RIGHT 12=RETRIEVE

Enter option 3:

APPLICATION RESTART CONTROL Command ===> Type or verify the ID of the BMC Consolidated Subsystem (BCSS) to use. Blank the field to request the public BCSS subsystem. BMC Consolidated Subsystem ID . . . BCSS Current BCSS subsystem type . . . : PUBLIC VER 1.1 Select an option. Then press Enter. __ 1. Active jobsteps 7. Reattach options 2. Current shift identifier 8. Dynamic allocation options 3. Checkpoint pacing options 9. Program and data set options 4. Reports 10. Remote VSAM options 5. Processing options 11. AES common utilities 6. Cursor repositioning options 12. Exit Copyright (C) 1994-1998 BMC Software, Inc. as an unpublished licensed work. PF 1=HELP 2=SPLIT 3=END 4=RETURN 5=RFIND 6=RCHANGE PF 7=UP 8=DOWN 9=SWAP 10=LEFT 11=RIGHT 12=RETRIEVE

-10-

Enter option 1:

Select Active Record Type Command ===> BCSID : BCSS Select an option. Then press Enter. _ 1. Restart control records 2. VSAM recovery records PF 1=HELP 2=SPLIT 3=END 4=RETURN 5=RFIND 6=RCHANGE PF 7=UP 8=DOWN 9=SWAP 10=LEFT 11=RIGHT 12=RETRIEVE

Enter option 1:

Limit List of Records Command ===> BCSID : BCSS Record type : Restart control Type information. Then press enter. JOBNAME . . . . . . Y0HG**** JOBSTEP . . . . . . ******** PROCSTEP . . . . . . ******** PROGNAME . . . . . . ******** PSBNAME . . . . . . ******** PF 1=HELP 2=SPLIT 3=END 4=RETURN 5=RFIND 6=RCHANGE PF 7=UP 8=DOWN 9=SWAP 10=LEFT 11=RIGHT 12=RETRIEVE

-11-

Enter your job details:

List Records Command ===> Scroll ===> CSR BCSID : BCSS Record type : Restart control Note: Intervention required for highlighted jobs. Commands: REFresh Type one or more action codes. Then press Enter. D=Delete P=Print S=Display M=Modify C=Display/change restart data set K=Restart data display O=Set manual restart J=Submit restart Act JOBNAME JOBSTEP PROCSTEP PROGNAME PSBNAME Run Start Date/Time Status _ Y0#0OFB LOADTOFF DV1BATCH PBTO00AU FRI 10/16/1998 09:53 ABENDED _ Y0AATR34 STEP010 STEP040 GM03400 GM03400 WED 06/24/1998 09:30 ABENDED _ Y0ABTR34 STEP010 STEP040 GM03400 GM03400 WED 06/24/1998 09:33 ABENDED _ Y0AD47AD NN05000 PS040 NN05000 NNPP5047 THU 05/21/1998 16:38 ABENDED _ Y0AD47V2 NN05000 PS040 NN05000 NNPP5047 THU 09/03/1998 17:30 ABENDED _ Y0AD49A9 NN05000 PS040 NN05000 NNPP5049 TUE 07/21/1998 17:07 ABENDED _ Y0AD53AD NN05000 PS040 NN05000 NNPP5053 WED 10/21/1998 20:12 ABENDED _ Y0AD54AD NN05000 PS040 NN05000 NNPP5054 WED 08/26/1998 14:56 ABENDED _ Y0AE1900 PROCSTEP STEP050 ISB1900 ISB1900 FRI 09/18/1998 16:48 ABENDED _ Y0AKGM34 STEP040 GM03400 GM03400 MON 02/09/1998 12:00 ABENDED PF 1=HELP 2=SPLIT 3=END 4=RETURN 5=RFIND 6=RCHANGE PF 7=UP 8=DOWN 9=SWAP 10=LEFT 11=RIGHT 12=RETRIEVE

From here you can select options to delete a checkpoint to force a cold start on the next test run or to examine the contents of the checkpoint data.

4.0 GSAM Files

4.1 Accessing GSAM files GSAM files must be used for restartable sequential file processing programs.

To access a GSAM file in your program, do the following

1. Code a DBD for each GSAM file.

DBD NAME=xxxx,ACCESS=(GSAM,BSAM),PASSWD=NO

DATASET DD1=Ixxxx,DD2=Oxxxx,RECFM=FB,RECORD=n,SIZE=0

DBDGEN

FINISH

END

where Ixxxx,Oxxxx are the JCL DD names (input and output)

RECORD=n refers to LRECL . For variable-length files IMS will add two bytes to this value for QSAM compatibility as per the note below.

SIZE=0 refers to BLKSIZE. SIZE must be specified as 0 in the DBD to stop DLI calculating it at DBDGEN time.

2. Make an entry in the programs PSB for the GSAM dataset.

PCB TYPE=GSAM,DBDNAME=xxxx,PROCOPT=LS

Always use PROCOPT=LS or GS.

3. Code a PCB mask.

01 DLI-xxxx-PCB.

03 DLI-xxxx-DBD-NAME PIC X(8).

03 DLI-xxxx-SEG-LEVEL PIC XX.

03 DLI-xxxx-STATUS-CODE PIC XX.

03 DLI-xxxx-PROC-OPT-CODE PIC X(4).

03 DLI-xxxx-RESRVED PIC S9(9) COMP.

-12-

03 DLI-xxxx-SEG-NAME PIC X(8).

03 DLI-xxxx-KFB-LENGTH-CNT PIC S9(9) COMP.

03 DLI-xxxx-SENS-SEG-CNT PIC S9(9) COMP.

03 DLI-xxxx-KFB-AREA PIC X(8).

03 DLI-xxxx-UNDEF-LENGTH PIC S9(9) COMP.

4. Use the GN call to read GSAM records sequentially. Use the ISRT call to write new records to the end of the file.

There are several things to note with using GSAM files.

1. It is not necessary to explicitly open or close the file using OPEN and CLSE calls. GSAM implicitly performs these functions. The first call to a GSAM file will open it and, if opened, the file will be closed at the end of the program. Implicit closing also occurs if, on reading sequentially, end-of-file is reached (status GB). Be aware that the next GN call following this will cause the file to be re-opened, and the first record to be retrieved again.

2. GSAM datasets must be pre-allocated. This avoids the need for 2 sets of JCL - one for normal processing which creates the dataset and one for restart processing which adds to an existing dataset. Furthermore, they must be pre-allocated using HUT21, rather than IEFBR14, to avoid creating empty files without end-of-file markers.

If the DDNAME of the file in the pre-allocation step starts 'DD', HUT21 will open and close the file after allocating it. This ensures that the file is empty. So all files in the pre-allocation step must be named 'DD......' e.g. DD1, DD2, etc.

3. You do not need to re-establish position on GSAM files after returning from the checkpoint or after restarting a program - unlike for all other IMS databases. After checkpointing or restarting, repositioning will occur automatically. However, be sure to code a DISP of SHR on all your GSAM files - or else unpredictable results may occur on restart.

4.2 GSAM variable length record files. Define your DBD with RECFM=VB;

DBD NAME=GSIJ,ACCESS=(GSAM,BSAM),PASSWD=NO

DATASET DD1=IGSIJ,DD2=OGSIJ,RECFM=VB,RECORD=352,SIZE=0

Set RECORD=max record/segment size, including 2 byte length field.

In your program (example taken from GS26000):

01 WP-GSAM-AREAS.

03 WP-GSIJ-REC.

05 WP-GSIJ-LEN PIC S9(4) COMP.

05 WP-GSIJ-DATA PIC X(400).

01 GSIJA-SE-FIN-DTL-HDR-REC.

03 GSIJA-FINE-REC-CD PIC XX.

.

01 GSIJB-SE-FIN-ACCT-TRANS-REC.

03 GSIJB-FINE-REC-CD PIC XX.

.

01 GSIJC-SE-FIN-DTL-SUBM-REC.

03 GSIJC-FINE-REC-CD PIC XX.

.

MOVE +352 TO WP-GSIJ-LEN.

MOVE GSIJB-SE-FIN-ACCT-TRANS-REC TO WP-GSIJ-DATA.

CALL 'CBLTDLI' USING DLI-ISRT,

WL-GSIJ-PCB,

WP-GSIJ-REC.

Some points on GSAM vs QSAM variable length record considerations.

Note that the GSIJA- record layout generated from Data Dictionary does not mention the length field. This is so that the layout can be used whether the file is being accessed via GSAM or QSAMIORT. Explanation follows:-

-13-

GSAM uses a 2 byte segment length field. This follows standard DL/1 variable length segment conventions, i.e. 2 byte S9(4) COMP. The length value includes the 2 bytes for the length field. For an example, see WP-GSIJ-REC above.

QSAM uses a 4 byte Record Descriptor Word. This is made up of a 2 byte S9(4) COMP record length and a 2 byte FILLER. See QSAMIORT's description in the Utilities section of Standards for examples. The QSAM record length includes 2 bytes for the length field and 2 bytes for the filler.

A common scenario is a VB dataset output as GSAM in a checkpoint restartable program, and read via QSAMIORT in another program. Have we a compatibility problem between GSAM and QSAMIORT record prefixes? The answer is no - because DL/1 outputs standard QSAM VB records; i.e. DL/1 inserts the FILLER and adds 2 to the segment length before writing the record. Thus the GSAM output program may set record length to 352, but both physically, and to QSAMIORT, the record length will be 354.

If a GSAM or QSAM program is coded like GS26000 (above), then data is moved from an unformatted WORKING STORAGE IO-area to the formatted record layout. This means both the GSAM and the QSAM program can use the same Data Dictionary-generated layout. The different length record prefixes are defined in the unformatted IO-areas.

-14-

5.0 System Design ConsiderationsAll too often, Checkpoint/Restart capability is added as an afterthought, either during program design or, worse still, after program build has been completed. Many of the problems that occur with checkpoint restart could be avoided if sufficient thought was given to these matters at the outset. In other words, one of the key elements of Systems Design process should be to think through the restart implications of a potential failure at every single point in a batch process flow. Here are some guidelines to help with this.

5.1 DriversAll batch processes will be driven in some way. The driver can either be a transaction file (‘hit file’) or a sequential scan of a database. Where you have a choice then use a hit file whenever possible rather than a database scan. This means that you don't need special routines for re-positioning the true databases. There may also be benefits in performance, testing and production support.

You should also avoid having more that one driver (whether the driver is a database or a hit file) in any one program. The merge-update logic does not fit in at all well with checkpoint-restart. Instead, generate a single sequential hit file using conventional merge logic and then use this to update the database in a subsequent step.

5.2 ReportsAvoid the production of reports in checkpoint/restartable programs as it can be difficult to work out how to restart these. If at all possible, write the details required by the report to a GSAM file and then have a small program to print the report in a separate step. The side-benefit here is that we may be able to find a better way to get the data to the customer than a mainframe report, such as transmitting the data to a PC file. We could do this without touching the database program.

5.3 GSAM to GSAMAvoid the situation where an output GSAM file from one step feeds directly to an input GSAM file in another. The reason for this is that as the first program takes checkpoints, the buffers for its GSAM files are flushed resulting in the file containing a number of short blocks. Now consider the situation where the second program has to be restarted. In the normal course of events this will not cause a problem and the second program will re-position on its input GSAM files correctly. However if there has been intervening production support activity (e.g. stripping a record) then the file will get re-blocked and the GSAM positioning for the second program will now be incorrect. To avoid this problem change the job flow to insert an IEBGENER step between the two programs to re-block the data.

5.4 VSAM files The general advice is don't use VSAM files for checkpoint/restart programs unless you absolutely have to! Certainly for newly designed processes DB2 tables should be used in preference to VSAM.

Unfortunately, the facilities provided for checkpointing updates to VSAM files do not mesh in with those provided for IMS databases. For example, if a program fell over at a certain point, the updates to the database would be backed out to a particular point (i.e. the last checkpoint). However, updates to a VSAM file would not be consistent with this: they could be up to a few after this, or (because of purging of unwritten buffers) a few before it. Generally, the problem only exists with KSDSs. GSAM files can be used in place of ESDSs; whilst RRDSs are so rare (at BROC) that they have not yet been used in a checkpoint/restartable program.

There are three ways out of this particular problem, both with their own advantages and disadvantages.

Firstly, the KSDS can be defined as a SHISAM database. This particular database organisation is just a KSDS with IMS wrappings. It only allows root segments, which will simply be the records in the KSDS, as it is viewed elsewhere. IMS looks after data integrity, so that the updates will always be synchronised with those to the 'proper' databases.

Secondly, by brute force it is possible to carve a way through the difficulty. The problem only exists if the file has to be updated across checkpoints (or, by definition across L.U.W.s). (If the job does not update the file then you don't have a problem incidentally!). Synchronisation, of a sort, can be ensured by closing and re-opening the file after every honoured checkpoint call. This would guarantee that no updates before the checkpoint were lost (because doing this causes the unwritten VSAM buffers back to the file). However, it has two obvious disadvantages. It incurs large amounts of system overhead in the opening and closing: this would only be

-15-

acceptable in performance terms if it did not happen very often, say less than a hundred times in a particular run. Also, it does not mean that updates after a successful checkpoint will be backed out, if the program subsequently abends. Accordingly, the application program will have to cater for this: e.g. by doing a REWRITE if a WRITE fails because the record was already present.

Thirdly, if the access pattern of the program to the KSDS is amenable, the problem can simply evaporate. This would happen if the program reads the KSDS at the beginning of a run and updates it again at the end. This is quite frequently the case with control or totals files which are also quite often VSAM. In this case, because updates to the file only happen at one time (in the programs termination phase), they do not cross L.U.W.s and so cannot get out of synch with the database updates.

5.5 Sorts It is an unfortunate fact that sorts and checkpoint/restart don't mix well. It is notionally possible to get around the incompatibilities, but it is almost never worth it. It complicates the logic of the program to the point where it is incredibly tortuous. It is still not possible to checkpoint the program during the sort input/output phases, and it doesn't carry any advantage in saved I/O operations.

This point does sometimes escape the notice of systems designers. It is not unheard of to find a spec for a checkpoint/restartable program, which is also supposed to do some kind of internal sort. The solution is always to split the program into 2 (or occasionally more) smaller ones. Usually, one of them will be a stand-alone (JCL) sort to do the sorting for the checkpoint/restartable one.

-16-

6.0 Checkpoint/Restart with Other Products

6.1 Director

6.1.1 Introduction.

DIRTOBMC is a user exit module that allows DIRECTOR programs to use the ARC checkpoint/restart software. This means that DIRECTOR can be used instead of COBOL when there is a requirement to write a 'oneoff' checkpoint/restartable program.

Using the DIRECTOR language, instead of COBOL, to code and test a checkpoint/restartable program can considerably reduce the amount of programming effort required. However DIRECTOR should only be used for production support and systems development tasks and not for regular scheduled production applications.

You will need to have a reasonable understanding of the DIRECTOR language, checkpoint/restart programming techniques in order to successfully use DIRTOBMC. Examples of checkpoint/restartable DIRECTOR programs can be found in DEVL.STANDARD.JCL members DIRBMC1 and DIRBMC2.

6.1.2 How to use the checkpoint module.To use DIRTOBMC the following fields should be defined at the beginning of your DIRECTOR program.

AREAFUNC LEN=4 4 byte function code AREAPCB LEN=40 IO PCB mask return area AREASAVE LEN=100 Checkpoint save area (any length)

AREAFUNC can be set to XRST or CHKP. The XRST call is used for detecting a normal start or a restart and the CHKP call is used for issuing checkpoints. Positions 11-12 of AREAPCB contain the status-code returned after making a call to DIRTOBMC.

The checkpoint save/return area AREASAVE is for storing data that is required after a restart situation.

6.1.3 The XRST call.The XRST call should be made only once at the beginning of the DIRECTOR program before any processing has taken place and should therefore be coded in the outer most DO/ENDDO process.

An example XRST follows:

AREAFUNC POS=1,C'XRST' UEXIT DIRTOBMC(AREAFUNC,AREAPCB,AREASAVE) | | | | | ---> CHECKPOINT RETURN AREA | ---> IO PCB MASK RETURN AREA ---> FUNCTION IF (AREAPCB(11),EQ,C'RS') perform restart processing ENDIF

The status-code from the XRST call is passed back to your DIRECTOR program in bytes 11-12 of AREAPCB. A status-code of spaces indicates a normal start and a status-code of 'RS' indicates a checkpoint restart.

If an 'RS' status-code is detected then any data saved by a previously issued CHKP call will be returned in AREASAVE. It is then the responsibility of the DIRECTOR program to perform any database repositioning that may be required to successfully restart the program.

Repositioning can be achieved by using the data previously saved in AREASAVE by the last real CHKP call. It is up to you to save any data that may be required for database repositioning. Note that GSAM files are automatically repositioned for you.

-17-

6.1.4 The CHKP call.To make a checkpoint call set the function area AREAFUNC to CHKP and save any data you may require for a restart in AREASAVE (a root segment key value for example). An example CHKP call follows:

AREAFUNC POS=1,C'CHKP' * set function AREASAVE POS=1,KFBA(1,11) * save any restart data

UEXIT DIRTOBMC(AREAFUNC,AREAPCB,AREASAVE) | | | | | ---> CHECKPOINT SAVE AREA | ---> IO PCB MASK ---> FUNCTIONNote that checkpoint ids do not need to be specified as they are generated by DIRTOBMC for you.

It is best to place your CHKP call in a position which will make restarting most easy. The following rules for placing the CHKP call should serve most programs:

1. If your program is driven by an input GSAM file then place the CHKP call just before the GSAM read statement (the GN call).

2. If your program is database driven (ie. you are sequentially processing a database) then place the CHKP call just before reading a root segment.

Actual checkpoint frequency will be controlled by ARC. The status-code from the CHKP call is passed back to your DIRECTOR program in bytes 11-12 of AREAPCB. If using pacing class PACESK then a status-code of 'SK' indicates that checkpoint processing has been skipped and processing can continue as usual. A status code of spaces means that a checkpoint has been taken and the DIRECTOR program must therefore reposition on any databases it is sequentially processing.

6.1.5 GSAM files.Sequential files that require automatic repositioning after a restart situation must be GSAM and therefore declared in the PSB you are using. Files that do not require repositioning can be processed using the DIRECTOR READ and WRITE statements.

A set of ten GSAM DBDS are available for general use located in library DEVT.TESTBED.DBDLIB. They are named DIR1 to DIR10 and can be used to access any sequential datasets required.

An example GSAM file read follows:

DO DBD=DIR1 GN ENDDO

IF STATUS# DOEXIT ENDIFA status-code of 'GB' is returned when end of file is detected.

The format of a GSAM file write is as follows:

DO DBD=DIR2 ISRT ENDDO

IF STATUS# DOEXIT ENDIF

-18-

6.1.6 Testing.The restart code of a checkpoint/restartable DIRECTOR program must be thoroughly tested before it is allowed to run in the production environment. DEVL.STANDARD.JCL members DIRBMC1 and DIRBMC2 provide examples of the JCL required. The following points may also be helpful.

An old style checkpoint database is not required however your PSB must specify COMPAT=YES on the PSBGEN statement.

Any GSAM files must be pre-allocated prior to running the DIRECTOR step for a normal start.

Load library DEVT.TESTBED.LOADLIB which contains DIRTOBMC must be placed on the STEPLIB statement.

The IEFRDER IMS logging DD statement should be present in your run JCL. This file is required for running a database backout.

In order to test that restart is working correctly:

1. Place displays in your DIRECTOR program (using the PRINT just after the CHKP call to show which database record has just been processed.

2. Cause an abend to occur in your DIRECTOR program when a record has been read. You can call module ZCDB029 to cause an abend (see DEVL.STANDARD.JCL(DIRBMC1) for example code).

3. Run a database backout job after the abend. See member DEVL.STANDARD.JCL(DICHKPBO) as an example.

4. Re-run your DIRECTOR program and display which key is returned AREASAVE to prove that you are restarting in the correct place.

6.2 Other ProductsAny product which supports a call interface with standard IBM linkage can in theory be made checkpoint/restartable using ARC Common Calls. Module ASMTARC can be called to request XRST and CHKP functions which are entirely analogous to the standard IMS offerings. You will need to STEPLIB to library IMSVS.BMC.AESLOAD to pick up this module. Details can be found in the ARC User Guide.

A simple test with SELCOPY has been run to establish that this works. Please feed back any further product experience to the Development Centre.

-19-

7.0 Defunct Checkpoint/Restart Components

7.1 BMP Restart BMP Restart is the predecessor product to ARC. At the time of writing production jobs are being converted to run with ARC. By default all Testbed jobs will run with ARC but if ARC is suppressed (by including //AES£EXCL DD DUMMY in the job step JCL) then BMP Restart will operate. The detailed standards for use of BMP Restart are available from the Development Centre, should you happen to need them during this transition period.

7.2 CBLTBMCThis module was developed in response to a bug with GSAM processing in the original version of BMP Restart. It is called instead of the standard module CBLTDLI. Where calls to CBLTBMC are encountered in a program which is undergoing maintenance then you should perform a straight replacement to equivalent calls to CBLTDLI.

7.3 CBLTDLXCBLTDLX was the in-house written checkpoint/restart method in use prior to BMP Restart. This module was called instead of CBLTDLI and checkpoint information was maintained on DL/1 databases. Where CBLTDLX is encountered in a program which is undergoing maintenance then it should be removed and replaced with standard calls to CBLTDLI . Please refer to section Converting COBOL To DLX Modules in the Utilities Replacement Guide for details of how to achieve this.

7.4 SPLTDLIThis is a version of the database split utility which calls CBLTDLX rather than CBLTDLI. Where encountered you can make a straight replacement with the standard module SPLTDLZ. Better still, there is another technique for accessing split databases that is now available which avoids the need for these specialised modules. Please contact the Development Centre for details.

-20-

Appendix A - DB2 Restart Logic

A.1 Simple Key

DB2 DATA Col_A (key) other 37421 … … … 37422 … … … 37423 … … … 37424 … … … 37425 … … … 37426 … … …

SQL CODEDECLARE crs1 CURSOR WITH HOLD FORSELECT …FROM tablesWHERE … AND Col_A > :ws-aORDER BY Col_A

(ws-a = key value saved in checkpoint area)

When the key of a DB2 table consists of just one column the restart logic is very simple to code and is intuitive.

The only thing to bear in mind with any cursor that you want to stay open across logical units of work is that they must have an ORDER BY statement. They should also be defined WITH HOLD so the program doesn’t continually close and re-open the cursor after every checkpoint. It is a significant overhead.

A.2 Compound Key - Wrong!!

DB2 DATA Col_A Col_B 37421 00 37421 01 37421 02 37422 00 37422 01 37422 02

SQL CODESELECT …FROM tablesWHERE … AND Col_A > :ws-aAND Col_B > :ws-bORDER BY Col_A, COL_B

A common mistake in re-positioning logic comes when we have to deal with multi-column keys. The most common error is to code the cursor as above.

The problem comes to light if the program abends after processing, say, the second of the rows in our sample table. In this case, Col_A = 37421 and Col_B = 01

When the program restarts, it would appear to pick up from where it left off and complete with return code of 0. However, the fourth and fifth rows in our table would not have been processed by either the first run or the

-21-

restart. DB2 can give no indication that these rows have been missed as it has returned all the rows the program requested.

A.3 Compound Key - Right

DB2 DATA Col_A Col_B 37421 00 37421 01 37421 02 37422 00 37422 01

SQL CODESELECT …FROM tablesWHERE … AND ((Col_A = :ws-aAND Col_B > :ws-b) OR Col_A > :ws-a)ORDER BY Col_A, COL_B

This is the correct restart logic to use.

It says that we want to process all rows which have a Col_A value greater than the abend value and, in addition, we want to process all rows that have the same Col_A value and a larger Col_B value.

However, even this sometimes causes problems from a performance perspective. DB2’s optimizer, which decides what is the best access path to take, doesn’t realise that the predicates referring to Col_A on either side of the OR are set to the same value. As a result, DB2 sometimes chooses an inefficient route to the data.

A.4 Compound Key - Best

DB2 DATA Col_A Col_B 37421 00 37421 01 37421 02 37422 00 37422 01

SQL CODESELECT …FROM tablesWHERE … AND Col_A >= :ws-aAND ((Col_A = :ws-aAND Col_B > :ws-b )OR Col_A > :ws-a)

In most instances this is the best logic to use:

By adding an additional predicate into the restart logic we are explicitly telling DB2 that every row on the restart will be greater than, or equal to, a particular value. This is usually enough to have the optimizer choose the best access path.

-22-

NB: There are some circumstances when the best access path for a restart of a program is different from the best for a normal run. This is because when a program restarts it will typically process fewer rows and DB2 will always estimate number of rows that will be returned. It then uses that value as a factor to determine the access path to take. As a result, if the query is a join DB2 may choose a less than ideal method of joining the tables (it has three to choose from) or even for simple queries on a single table DB2 may choose to use an index when a table scan is best.

Therefore, when coding restart logic for long running programs on the critical path, or if you have concerns about performance then contact your project team’s DBA. If your project team has not been allocated one then contact Richard Livett, X4211.

-23-