Introduction

The purpose of this guide is to provide users with information about batch processing, batch and life cycles within the system, and to demonstrate how to handle and monitor basic operations around the file processing and batch settlement.

Intended Audience

This guide will be useful for merchants and gateway users that want to understand file processing mechanism within the system. In addition, it gives knowledge to administrative users how to manage files at a higher level.

File Processing Aspects

There are two types of transaction processing - realtime and batch. Within the context of file management, realtime involves both realtime/batch transaction cycle and settlement processing.

Let's review described concepts and processes in detail through the terminology below.

Realtime Transaction Cycle

Realtime Transaction Cycle - Submission of a collection of realtime transactions received within a particular period of time, for an associated account. A new RTC is created when all previous cycles close for that account, and a new transaction is received. Realtime transaction cycles are closed with settlement, when pre-authorized transactions are settled and approved for funding.

Batch Transaction Cycle

Batch Transaction Cycle - Submission of a collection of transactions associated with a particular merchant account provided in a single batch file.

Batch

Batch - A collection of transactions that are processed together. Batch is part of batch processing, which is used in two situations:
— Real-time batches - used for the settlement of pre-authorized transactions.
— File batches - used to process together card-not-present transactions (authorization and settlement) of recurring nature, such as bill payments, subscription payments, etc.

Usually, a batch file is split into sub-batches before to comply with the requirements of each specific processor.

Sub-batch

Sub-batch - A sub-group of transactions that a batch contains. Batch transactions are split into sub-batches by the system to meet the requirements of a particular processor for the following reasons:
— Quantity limitation for the transactions included in one batch. If a batch contains a number of transactions that exceeds the fixed limit, it is divided into several sub-batches. The maximum quantity of the transactions included in one file submitted to the processor is defined in transactionSubBatchLimit property of the corresponding Processing Profile. If a sub-batch file contains less transactions than set in the limit, one file (Provider) is generated. In cases when a sub-batch contains a larger number of transactions, two or more files (Providers) are generated.
— The difference in how transactions are processed. For example, a batch may contain both direct debit and payment card transactions. In this case, sub-batches are created for processing to be properly performed within the system.
Currently the Processing Profile transactionSplitPolicy setting allows to split batches by one or more of the following criteria:

• by Industry Type;
• by Transaction Mode (card present/card not present);
• by Terminal Code (only for realtime transaction);
• by OperationType ( sale/credit);
• by Descriptor;
• by Account Type.

Within the context of both realtime and batch processing, there is a concept of submission. A submission may be represented either by a separate transaction processed in real-time or by a batch file. The difference between the realtime and batch cycle types is that real time transactions are grouped into collections and split into sub-batches once the RTC is closed and before the settlement, while batches are split into sub-batches right away.

Settlement

Processing of transactions consists of two phases - authorization and settlement (capture). Settlement is the process that follows authorization, during which previously authorized transactions are settled with the acquirer and approved for subsequent funding. Generally, settlement is done at the end of a business day. Settlement process within the system can be organized in three ways (as defined in Settlement Mode setting):

• Daily – once a day at a fixed time;
• Hourly – multiple times per day within fixed settlement windows;
• On demand – whenever end-of-the-day settlement message is received.

A file is submitted for settlement at the time indicated in merchant's settings. A settlement file consists of pre-authorized/approved realtime transactions. It's important to know that all transactions are grouped in sub-batches, according to the settlement life cycle they are associated with.
You can see a settlement life cycle in the diagram below:

• Sub-batch compilation - data is distributed in the respective sub-batches.
• Transformation – transactional data is translated from the high-level system representation to the low-level processor specific representation.
• Generation – low-level processor transaction representation is converted into a processor specific message format.
• Aggregation* – generated files, collected by the system, are aggregated into one file.
• Upload – file is uploaded to the processor.
• Download – response file is downloaded from the processor.
• Deaggregation* – aggregated file is deaggregated back.
• Parsing – response file is parsed.
• Backward transformation - file is converted from a processor specific message format to the gateway format. When the file is processed on the processor side/in file store, it goes back to the FTP Gates where it is visible whether it was processed or not.

* Aggregation/Deaggregation processes are optional. Whether they occur or not, depends on a processor's requirements and profile settings.

Aggregation

Some processors limit the number of files that can be submitted for processing within one day. To meet this requirement, the aggregation mechanism is used within the gateway.
Aggregation is the process of compiling multiple system-generated files into a single joint file.

The mechanism works as follows:

• Aggregation must be enabled within a provider profile, through which processing is done. To allowing for this, select an aggregation frequency:

• All aggregation processes occur through a system merchant, which is not available in the list of merchants on the user interface. As a rule, it is assigned with ID 1000. This merchant is reserved only for aggregation/deaggregation processes and can't be used for transaction processing.
• If a processor's limit for file submission is exceeded, a response code/message indicating an error is returned.
• You can check the status of an aggregated file on the File Store form under the Monitoring and System perspectives. The file is available under the merchant 1000.

Batch Life Cycle

Within the gateway a submitted file goes through several phases of processing. The transitions between the phases are triggered either my the manual actions of the users or by the system actions (jobs/timers). There are four forms on the user interface which allow to keep track of the files statuses transitions during its life cycle and to perform manual actions applicable at each phase of processing: Files (Clients), Files (Providers), Batches, Sub-batches. All of them are accessible through the Submissions menu on Merchant and System perspectives.

A batch file follows the life cycle defined below:

Submission – a file is uploaded to the server by a merchant. This process can occur either via user interface (UI) or FTP server.

Once submitted, a file is uploaded to the Files (Clients) form. Files (Clients) processes are related to the batch processing within the gateway.
The following phases of the batch life cycle occur:

• Initial parsing – the uploaded submitter's file is being parsed and transactional data is extracted and stored.
• Validation - the data is being verified whether it is relevant for processing. In case if the data is invalid, it is stored in a separate sub-batch which is not submitted for processing.
• Sub-batch compilation - data is collected in one sub-batch.
• Transformation – transactional data is translated from the high-level system representation to the low-level processor specific representation.

Once the file is processed by the gateway, it is submitted to a processor for further processing. On this phase, a file goes to the File (Providers) form. When a batch file is in the File (Providers), it means that it is being processed on a processor's side. At this phase, the file is generated into a batch file. Now, it is visible on the UI on Batches List form.

The following phases of the batch life cycle occur in the Files (Providers):

• Generation – low-level processor transaction representation is converted into a processor specific message format.
• Aggregation* – generated files, collected by the system, are aggregated into one file.
• Upload – file is uploaded to the processor.
• Download – response file is downloaded from the processor.
• Deaggregation* – aggregated file is deaggregated back.

Once a batch file is processed by the processor, it goes back to the Files (Clients). The following phases of the batch life cycle occur in the Files (Clients):

• Parsing – response file is parsed.
• Sub-batch compilation - data is collected in one sub-batch.
• Backward transformation - file is converted from a processor specific message format to the gateway format. When the file is processed on the processor side, it goes back to the Files (Clients) form where it is visible whether it was processed or not.
• Response file aggregation* - response files are aggregated into one file.
• Response generation – response file is prepared for the merchant pick up.
• Response file upload - response file is uploaded on FTP.

The final result is visible on all three forms and processed files are marked as Processed. The sub-batches included in each processed batch and the list of related transactions are displayed on Sub-batches list form.

* These aggregation/deaggregation processes are optional. Whether they occur or not, depends on a processor's requirements and profile settings. ** These deaggregation/aggregation processes occur for cases when a batch file includes transactions for several merchants. For more information about this mechanism, review this note.

File Formats

Within the gateway there are two models of processing the files:
1. Request-Response model - the request file is uploaded by the user or generated by the system and submitted to the processor, and the response file is returned.
2. Response model - the response file is returned to the gateway with no need to submit a request file. For example, Returns files are received from the processor without request.
The detailed information about the types of files processed by the gateway and the requirements to request files formats and names is provided in integration notes.

File Statuses

Once a file is uploaded into the system, a user can identify its current state by checking the status. You can learn how to review file statuses in your gateway using this tutorial for all types of users. The statuses available within the system are the following:
For Files(Clients):

• Pending Upload – request file is waiting for upload to the processor.
• Processing – request file is being generated.
• Pending Parsing – response file is waiting for the parsing.
• Pending Aggregation – request file is waiting for aggregation with another file.
• Pending Deaggregation – response file is waiting for de-aggregation.
• Processed – file is successfully processed.
• Failed – processing of the file failed.

For Files(Providers):

• Pending Detranslation - response file is processed by a processor and is being converted from a processor specific message format to the gateway format.
• Pending Upload – request file is waiting for upload to the processor.
• Pending Download – response file is waiting for download from the processor.
• Pending Aggregation – request file is waiting for aggregation with another file.
• Pending Deaggregation – response file is waiting for de-aggregation.
• Pending Detokenization - response file is waiting for detokenization.
• Pending Detranslation - response file is processed by a processor and is being converted from a processor specific message format to the gateway format.
• Processed – file is successfully processed.

For batches/sub-batches:

• Pending Approval - file is waiting for approval (only for batch transaction cycle)
• Pending Settlement - file is waiting for settlement (equivalent of Pending approval status for realtime transaction cycle). In this case, the realtime transaction cycle is not closed yet. Once the RTC is closed, the collection of transactions contained in the file will be submitted for settlement.
• Account Update - the payment information contained associated with the transactions contained in the file is being updated.
• Pending Due Date - file is waiting for the due date (processing won’t start till the specified due date).
• Pending Translation - file is being converted from the gateway format to the processor's format.
• Processing - file is waiting for the processor’s response.
• Processed - file has been processed on the processor’s side and processor’s response has been retreived.
• Processed (Cancelled) - transactions have beed cancelled by the user. In terms of batch transaction cycle it means that the client declined a pending batch file. In terms of realtime transaction cycle it means that some of the transactions were not settled and, thus, were not included in a sub-batch file.

Batch file status is calculated based on the statuses of sub-batches it is split into. The statuses described above are arranged in the sequence of priority applied when calculating a final batch status. Thus, if at least one sub-batch included in the batch file is in Pending Approval/Pending Settlement status, the batch file will be in this status as well.
A batch file can be in Processed status if none of the sub-batches it contains is in Pending Approval/Pending Settlement, Account Update, Pending Due Date, Pending Translation or Processing statuses. The calculation of Pending Translation, Pending Approval, Pending Settlement, Pending Due Date statuses depends on the Due Date and Batch Review/Settlement Status.

Functionality

During the file processing you may need to apply particular actions with either batch or sub-batch or transactions within your gateway. For these needs, there are a number of administrative operations that allow you to achieve a desirable effect.
You can apply the following actions to a batch (Batches form):

• View batch – allows you to review the details of the batch.
• View totals – allows you to review totals associated with a batch.
• Approve – allows you to approve a pending file if an account is set on manual batch review.
• Decline - allows you to cancel the processing of a pending batch in one of the following ways: Cancel (batch file processing is cancelled and the response file is generated), Cancel without response (response file is not generated).
• Close and Settle Batch – allows to close the realtime transaction cycle and settle transactions included in the batch (only for realtime batches).
• Close Batch – allows to close the realtime transaction cycle (only for realtime batches).
• Settle Batch - allows to send the transactions included in the batch for settlement (only for realtime batches).
• Switch to Manual Settlement - allows to switch to manual settlement mode (only for realtime batches).

You can apply the following actions to a Files (Clients):

• View – allows you to review the details of the file.
• Download request file – allows you to download and review the request file associated with a particular file.
• Download response file – allows you to download and review the response file associated with a particular file.
• Rerun Processing Service – allows you to restart processing of a file with the current status (applicable to the files in the following statuses:Pending Parsing, Processing, Pending Upload, Pending Deaggregation, Pending Aggregation).
• View totals - allows you to review totals associated with a file.

You can apply the following actions to a Files (Providers):

• View – allows you to review the details of the file.
• View totals - allows you to review totals associated with a file.
• Download request file – allows you to download and review request files in the format of a processor (XML message format).
• Download response file – allows you to download and review response files in the format of a processor (XML message format).
• Rerun Processing Service – allows you to restart processing of a file with the current status (applicable to the files in the following statuses: Pending Upload, Pending Download, Pending Detranslation, Pending Aggregation, Pending Deaggregation.
• Regenerate and reupload file – allows you to regenerate a request file and resubmit it to a processor. This actions is active for non-aggregated files in Processed or Pending Download submitted for processing not earlier than 15 days before.

You can apply the following actions to a sub-batch (Sub-batches form):

• View sub-batch - allows you to review details of a sub-batch.
• View totals - allows you to review totals associated with a sub-batch.
• Decline - allows you to cancel the processing of a pending sub-batch in one of the following ways: Cancel (batch file processing is cancelled and the response file is generated), Cancel without response (response file is not generated).
• View original Sub-Batch - allows you to review a duplicate sub-batch file (if present).
• Rerun Processing Service - allows you to reprocess a sub-batch (applicable if the file is in Pending Translation status, and the processing attempt date is in the past).
• Reprocess Sub-Batch – allows you to reprocess a sub-batch. Reprocess Sub-Batch is active for sub-batches in Processed status within 14 days after the sub-batch has been processed. Reprocessing cannot be performed if the sub-batch has been processed through batch proxy emulator.
• Issue credit - allows you to issue credits for transactions included in a particular sub-batch. Once credits are issued, a new sub-batch with refund transaction type will be generated. The Issue credits is active for sub-batches in Processed status within 14 days after the sub-batch has been processed. Credits cannot be issued if the sub-batch has been processed through batch proxy emulator.
• Generate Returns/Declines – allows you to generate returns/declines for transactions included in a particular sub-batch. Generate Returns/Declines is active for sub-batches in Processed status within 14 days after the sub-batch has been processed. Returns/Declines cannot be generated if the sub-batch has been processed through batch proxy emulator.

Use cases

For better understanding of file management process, let’s review use cases that include the most common situations in which several operations with files are involved.

Deleting a transaction

As an example, let’s review the situation when after a batch file has been submitted, several customers requested their transactions to be canceled. In this case, the system allows to delete a transaction included in a particular batch. The ability to delete a transaction is available only for cases when a merchant's processing settings are set for manual batch review and the batch itself is in Pending status.

In this case, the scenario of actions will be as follows:
1. Access the Sub-batch form on Monitoring perspective (for administrative users only).
2. Locate a sub-batch associated with the batch that includes a transaction than needs to be deleted. The search has to be made by the merchant and the date values. 3. When the sub-batch is located, open its details by clicking on View Sub-batch button.
4. Load all transactions included in this sub-batch.
5. Locate the transaction that needs to be deleted and cancel it.
6. The system allows to control how a deleted transaction will be returned to the merchant. It can be returned with X03 response code or in return file. 7. To make sure that transaction was excluded from the batch, review totals of the batch using Batch form.
8. If all steps are followed correctly, approve the batch for further processing.

File processing restart

There are situations when a particular phase of file processing failed, and a user is notified about the failed processes via System Audit email. The system allows to avoid resubmission of a file. An administrative user can restart the failed task without following all processes that had already successfully completed.

In this case, file can be reprocessed in two ways:

• Using user interface (please, review the respective tutorial).
• Using Active MQ form (please, review the respective tutorial).

Duplicated batch file

Let’s review a situation when a batch file was submitted for processing twice and all associated customers were billed two times. In order to resolve this issue, either the Submission date or the Batch ID value must be present.

In this case, the scenario of actions will be as follows:
1. Locate a sub-batch that was submitted the second time by the submission date or batch ID on Monitoring perspective.
2. Click on Actions button and select Issue Credits option.
3. After you have confirmed this action, a new sub-batch will be generated.
3. To check whether refunds were issued, review a respective sub-batch included in the batch and check the status of the transactions. The status must be Refund.

Profile settings issue

Let’s review a situation when file processing got failed and notification that an attempted operation was not configured for a specified account is received. In this case, there is an issue in merchant configuration that should be set up in a proper way.

The scenario of actions will be as follows:
1. Billing settings for a merchant must be verified:
a. Go on Merchant perspective and open Details menu in the top section of the screen.
b. Select Billing menu item and access Settings form.
2. After the settings are set up properly, a failed file should be reprocessed by an administrative user (please, review the respective tutorial).

Batch Emulator

Context There are situations when it is more convenient for a merchant to submit transactions for processing in a batch file, but there is no possibility to provide batch processing to a merchant. This happens, for example, when a batch processor is not supported in the merchant's country or the merchant does not have an account for an existing batch processor, and it must go through a complicated procedure to obtain an account. For such situations, if there is a realtime integration, through which merchant transactions can be processed, it can process batch files through the realtime processor. To submit batch transactions in realtime, you can use a batch emulator.
Definition Batch emulator – an internal mechanism in the gateway that allows merchants to process files in batch mode without integrating with a batch processor. The emulation of batch processing through the emulator is supported for both card and direct debit transactions.

Principle of Work for the Batch Emulator

1. To allow for processing, realtime integration is required. For this integration, card not present transaction must be allowed.
2. When the batch emulator is used, a merchant has two provider profiles set up for transaction processing - a profile for the batch emulator, and a realtime profile through which processing is actually going to be performed.
3. For the file with transactions, the standard batch format is used.
4. Submission of files for processing is typical for batch processing.
5. After the gateway receives the file from the merchant, it submits transactions to the realtime processor for further processing.
6. The speed of transaction processing depends on the number of transactions in the file and the configuration of the batch emulator profile, such as the number of threads (described below).
7. The transaction response is returned to the merchant as typical for batch processing.

Settings for Processing through Batch Emulator

System setup for processing through the batch emulator

To enable processing through the batch emulator, you must have a corresponding job enabled - icharge.processor-upload-file-batch-emulator.
Provider profile setup for processing through the batch emulator

Required settings:

1. Configure a realtime provider profile, through which actual transaction processing is going to be performed (Merchant perspective => Details => Profiles => Provider Profile => [your processor]).
2. Configure a provider profile for the batch emulator (Merchant perspective => Details => Profiles => Provider Profile => cards-batch/emulator or ach/emulator).

To learn how to configure a provider profile, review this tutorial.
Batch emulator configuration specific: Since processing is performed through a realtime processor, transaction processing speed is limited by the number of submitted transactions and configured threads. Processing can be single-threaded or multi-threaded. The threads are configured in the provider profile of the batch emulator.


1) whether processing is limited to single or multiple threads depends on a realtime processor that you are going to use for processing via the batch emulator. For this reason, before configuring this setting, check which mechanism (single or multi-threaded) is supported by your processor.
2) single-threaded processing - one channel is used for realtime transaction processing performed through the batch emulator. Choose this option when:

• when a realtime processor doesn’t support multi-threaded processing;

In case you set up more than one thread for a processor that doesn’t support multi-threaded processing, the batch emulator will attempt to process transactions in multiple threads. However, the realtime processor will process one transaction at a time, while the rest of the transactions will be queued.

• when submitted batch files contain less than 400 transactions (even if you submit a large number of files).

If you are submitting files of small volumes (less than 400 transactions per file), there is no point in setting up multi-thread processing through the batch emulator since it won’t improve the performance.

3) multi-threaded processing – the number of channels used for simultaneous realtime transaction processing performed through the batch emulator is up to 10. Choose this option when:

• when a realtime processor supports multi-threaded processing;
• when submitted batch files contain more than 400 transactions.