Flat File Parser Grouping Example

Purpose

This example shows how to group data from multiple lines into a block which will be passed to a processing program.

Data Format

The data is read from a stream file on the IFS.

This is the data from the echo example.

00012010-12-201234567890123450
00012010-12-201234567890987900

The same data could come in a different format with the data spread over multiple lines with each line containing a line identifier and the data.

1000001
1102010-12-20
2101234567890123
220450
999
1000001
1102010-12-20
2101234567890987
220900
999

or

1000001
1102010-12-20
2101234567890123
220450
2101234567890987
220900
999

Each line type is identified with an identifier (columns 1 - 3). A block of data begins and ends with a "special" line. This line can be fix or contain variable data.

In this case a complete block of data starts with a line with an identifier "100" and ends with a line with an identifier "999". So the starting line format has an image "100####" and the ending line format "999".

Line Formats

	Identifier	Data
Store ID	100	4 digits numeric
Date	110	date ISO format yyyy-MM-dd
Item	210	13 alphanumeric (EAN)
Quantity	220	4 digits numeric
End	999	no extra data

Code

Create a new parser instance

 parser = ffp_create();

Configure the line formats

 lineFormatImage = '100####' + x'00';
 ffp_addLineFormatFromImage(parser : 'store' : %addr(lineFormatImage));

 lineFormatImage = '110####-##-##' + x'00';
 ffp_addLineFormatFromImage(parser : 'date' : %addr(lineFormatImage));

 lineFormatImage = '210@@@@@@@@@@@@@' + x'00';
 ffp_addLineFormatFromImage(parser : 'item' : %addr(lineFormatImage));

 lineFormatImage = '220####' + x'00';
 ffp_addLineFormatFromImage(parser : 'quantity' : %addr(lineFormatImage));

 lineFormatImage = '999' + x'00';
 ffp_addConstantLineFormat(parser : 'end' : %addr(lineFormatImage));

Create line processor

 lineProcessor = ffp_processor_grouping_create(parser);

Configure block start and end

 // define where a block begins and ends
 ffp_processor_grouping_setStartLineFormat(lineProcessor : 
                                           ffp_getLineFormat(parser : 'id'));
 ffp_processor_grouping_setEndLineFormat(lineProcessor : 
                                         ffp_getLineFormat(parser : 'end'));

Set data processing program

 // configure which program will process the grouped data
 ffp_processor_grouping_setExecutor(lineProcessor : 'FFPEX02EX');

Set input provider

 inputProvider = rng_input_stmf_create(%addr(inputPath)); 
 ffp_setInputProvider(parser : inputProvider);

We don't want to parser to skip any invalid lines because we may need them in the block processing and the line processor needs to know if invalid data has been passed at all. So we set a flag to tell the parser to also pass invalid lines to the line processor. The grouping line processor will skip a block if it contains an invalid line. This block will not be passed to the processor. An error message will be sent by the logger and a registered serializer will backup the invalid block of data.

 ffp_setProcessInvalidLines(parser : *on);

Parsing

 // do the actual parsing 
 ffp_parse(parser);
 

Cleanup

 // cleanup (all resources registered at the parser will also be cleaned up)
 ffp_finalize(parser);

Processing Program

The processing program gets the block of data from the line processor in a list data structure (though not data structure as in the RPG language, see Linked List). The program must implement the following prototype:

    D program         PR
    D   logger                        *   const
    D   data                          *   const

Each entry in the list is a data structure in the format of ffp_processor_grouping_line. The data structure contains the associated line format, the raw data and the line number.

    D ffp_processor_grouping_line...
    D                 DS                  qualified
    D   lineFormat                    *
    D   data                     65535A   varying
    D   lineNumber                  10I 0

The following code iterates through the passed data:

    D entry           DS                  likeds(ffp_processor_grouping_line)
    D                                     based(entryPtr)
    D lineFormat      DS                  likeds(ffp_lineFormat) 
    D                                     based(entry.lineFormat)
  
     ...
                     
      list_abortIteration(data);
      entryPtr = list_getNext(data);
      dow (entryPtr <> *null);
  
        // process data here
  
        entryPtr = list_getNext(data);
      enddo;    
  
     ...

Note: If you have defined fields in your configuration, you can extract the data for each field with the procedure ffp_util_getLineFormatFieldData.