Illustration of multithreading in the iSeries

Multithreading is a general purpose programming technique that reduces the complexity and overhead of concurrent...

programming. Multithreading is the process by which identical processes run in as many threads as needed, accessing the same file in any mode.

Here I illustrate how one can simulate this multithreading concept in reading the same file and processing records from it based on an arbitrary set of criteria in a program running in many threads. For a file having "n" records which is split across "m" jobs the processing time drops by about n/m time fraction.

Processing time would depend on many factors (such as number of processors, number of other jobs etc.) and we cannot say that processing time would reduce to n/m times. All we can say is that processing time would reduce considerably.

The idea is to split the number of records in the file by "m", a predefined number that indicates the number of threads that need to be run that can be decided based on the size of the file. It decides the number of records for each thread and gets the relative record number slot for each thread needed. Say there are 100 records and we have 5 threads. Then thread 1 gets rrn 1 to 20, thread 2 gets 21 to 40 and so on. The last job gets the full rrn slot needed for it or a slot smaller than that if the value of the number of records / number of jobs is not a round number.

In a nutshell, we run multiple jobs in parallel in such a way that each job is allocated a unique set of records for processing as opposed to having one job processing all the records. To accomplish this, we wrote 2 RPGLE programs. The first program finds the n/m ratio. It calculates the from RRN and to RRN needed for each thread and updates the LDA with these 2 values and submits the program that does the actual processing. This program reads from the LDA these 2 values of RRN, from RRN and to RRN that is meant for it and processes the file records having RRN in the range.

So as long as the RRNs do not clash, all the threads run fine.

Program1
* Get the total number of records (n) in input file.
* Get maximum number of threads/jobs – m
* Get number records to be processed by each job
* Populate local dataarea (LDA) with values of "from RRN" and "To
RRN" (Or range of RRN) that is to be processed by first thread and submit first job.
* Repeat above step for all submitting all the jobs

Program 2: Program that does the actual processing of the Input File

* Retrieve from the LDA the value of fromRRN and toRRN for this Thread

* The current value of RRN,CurRRN is obtained from the file information data structure

* Process all records in the file that have RRN between the fromRRN and toRRN

Program 1:

 ' *Input File, the file that is being processed
     fInpFile  if   e             disk     
     f                                     InfDs(FileInfo)

     '* Prototype Definition for QCMDEXEC
     D QCmdExc         pr                  extpgm('QCMDEXC')
     D  Command                     256    const
     D  Length                       15p 5 const
     '* INFDS for retrieving number of records in Input file.
     DFileInfo         DS
     D $NumRecs              156    159i 0
     ,* LDA definition
     D LDA_DS          DS                  DTAARA(*LDA)
     DW$fromRRN                      10  0
     DW$ToRRN                        10  0
     *
     DW$Jobs           s              4  0  inz(9000)
     DMaxJobs          s              2  0
     DCmd              s            256
     DW$JOB#           s              3  0
     DRecsPerRun       s             10  0
          

    _/free

    // 20 threads are being used for this example.
 
           MaxJobs = 20                              ;

    // Divide the total number of records in the file, which is obtained by 
    // the File information data structure for the file by 20, the number 
    // of threads to get the number of records per thread.

           RecsPerRun = $NumRecs/MaxJobs               ;
           W$fromRRN   = 1                             ;
           W$ToRRN = RecsPerRun                        ;
           W$Job# = 1                                  ;
           Dow W$Job# <= MaxJobs                       ;
    // Write the fromRRN and the toRRN to the LDA
           Out LDA_DS    ; 
    // Submit the job with the job name being Program2_(the job number)
    // i.e Program2_1 ,Program2_2 etc.

           Cmd  =   'SBMJOB CMD(CALL PGM(Program2)) ' +
                          'JOB(Program2_'+ %Char(W$JOB#) + ')' ;
           callp  QCMDEXC (Cmd:%len(cmd))              ;
           W$JOB# = W$JOB# + 1                         ;
           W$FromRRN = W$FromRRN + RecsPerRun          ;
           If W$Job# = MaxJobs                         ;
              W$ToRRN = $NumRecs                       ;
           Else                                        ;
           W$ToRRN = W$FromRRN + RecsPerRun            ;
           EndIf                                       ;
           EndDo                                       ;
           *InLr = *On                                 ;
     /End-Free

Program 2 

     fInpFile   if   e             disk    
     f                                     InfDs(FileInfo)
     f                                     RecNo(CurRRN)
        
     D LDA_DS          DS                  DTAARA(*LDA)
     D  W$fromRRN                    10  0
     D  W$ToRRN                      10  0
     DFileInfo         DS
     D $NumRecs              156    159i 0
     D CurRRN                397    400I 0


     _/free
     // Retreive the fromRRN and the toRRN to the LDA

            In LDA_DS   ;

     // Read from the file the record starting from the corresponding RRN
   
             SetLL  W$fromRRN InpFIle               ;
             Read    InpFile                            ;
             Dow  CurRRN  <= W$ToRRN and Not %Eof(InpFile) ;
             
          //             Process the input file as needed ....
             Enddo
            *inlr  =  *On

     /End-Free