Processing time would depend on many factors (such as number of processors, number of other jobs etc.) and we cannot say that processing time would reduce to n/m times. All we can say is that processing time would reduce considerably.
The idea is to split the number of records in the file by "m", a predefined number that indicates the number of threads that need to be run that can be decided based on the size of the file. It decides the number of records for each thread and gets the relative record number slot for each thread needed. Say there are 100 records and we have 5 threads. Then thread 1 gets rrn 1 to 20, thread 2 gets 21 to 40 and so on. The last job gets the full rrn slot needed for it or a slot smaller than that if the value of the number of records / number of jobs is not a round number.
In a nutshell, we run multiple jobs in parallel in such a way that each job is allocated a unique set of records for processing as opposed to having one job processing all the records. To accomplish this, we wrote 2 RPGLE programs. The first program finds the n/m ratio. It calculates the from RRN and to RRN needed for each thread and updates the LDA with these 2 values and submits the program that does the actual processing. This program reads from the LDA these 2 values of RRN, from RRN and to RRN that is meant for it and processes the file records having RRN in the range.
So as long as the RRNs do not clash, all the threads run fine.
* Get the total number of records (n) in input file.
* Get maximum number of threads/jobs – m
* Get number records to be processed by each job
* Populate local dataarea (LDA) with values of "from RRN" and "To
RRN" (Or range of RRN) that is to be processed by first thread and submit first job.
* Repeat above step for all submitting all the jobs
Program 2: Program that does the actual processing of the Input File
* Retrieve from the LDA the value of fromRRN and toRRN for this Thread
* The current value of RRN,CurRRN is obtained from the file information data structure* Process all records in the file that have RRN between the fromRRN and toRRN
' *Input File, the file that is being processed fInpFile if e disk f InfDs(FileInfo) '* Prototype Definition for QCMDEXEC D QCmdExc pr extpgm('QCMDEXC') D Command 256 const D Length 15p 5 const '* INFDS for retrieving number of records in Input file. DFileInfo DS D $NumRecs 156 159i 0 ,* LDA definition D LDA_DS DS DTAARA(*LDA) DW$fromRRN 10 0 DW$ToRRN 10 0 * DW$Jobs s 4 0 inz(9000) DMaxJobs s 2 0 DCmd s 256 DW$JOB# s 3 0 DRecsPerRun s 10 0 _/free // 20 threads are being used for this example. MaxJobs = 20 ; // Divide the total number of records in the file, which is obtained by // the File information data structure for the file by 20, the number // of threads to get the number of records per thread. RecsPerRun = $NumRecs/MaxJobs ; W$fromRRN = 1 ; W$ToRRN = RecsPerRun ; W$Job# = 1 ; Dow W$Job# <= MaxJobs ; // Write the fromRRN and the toRRN to the LDA Out LDA_DS ; // Submit the job with the job name being Program2_(the job number) // i.e Program2_1 ,Program2_2 etc. Cmd = 'SBMJOB CMD(CALL PGM(Program2)) ' + 'JOB(Program2_'+ %Char(W$JOB#) + ')' ; callp QCMDEXC (Cmd:%len(cmd)) ; W$JOB# = W$JOB# + 1 ; W$FromRRN = W$FromRRN + RecsPerRun ; If W$Job# = MaxJobs ; W$ToRRN = $NumRecs ; Else ; W$ToRRN = W$FromRRN + RecsPerRun ; EndIf ; EndDo ; *InLr = *On ; /End-Free Program 2 fInpFile if e disk f InfDs(FileInfo) f RecNo(CurRRN) D LDA_DS DS DTAARA(*LDA) D W$fromRRN 10 0 D W$ToRRN 10 0 DFileInfo DS D $NumRecs 156 159i 0 D CurRRN 397 400I 0 _/free // Retreive the fromRRN and the toRRN to the LDA In LDA_DS ; // Read from the file the record starting from the corresponding RRN SetLL W$fromRRN InpFIle ; Read InpFile ; Dow CurRRN <= W$ToRRN and Not %Eof(InpFile) ; // Process the input file as needed .... Enddo *inlr = *On /End-Free
This was first published in January 2005