Batch API Overview

MX allows partners to upload a batch file containing objects that will be processed by the MX Platform. Partners must send properly formatted and encrypted CSV files to MX via SFTP. If needed, MX will send a response file for each batch file submitted.

File Format

To create users, members, accounts, and transactions via Batch system, partners must provide batch files and (if required) manifest files in a standardized format.

• UTF-8 encoded CSV file.
• Use field quoting when necessary.
• Header row must be included in each file to specify which fields are being provided. Details about the fields expected in each file can be found below.
• The line ending, separator character, and quote character may be customized depending on the partner's needs.

Initial SFTP setup

The initial setup for accessing MX integration and production SFTP servers is as follows:

1. Partners must provide IP addresses and domain name to be whitelisted.
2. MX will securely deliver a username and password for both integration and production, as well as the port to be used for upload.
3. MX will provide its public PGP keys for both integration and production. All batch files must be encrypted with the environment-specific MX public PGP key.
4. If partners require response files, they must provide the following to MX:
   • The SFTP URL and login information for their test and production servers.
   • The IP addresses used by their SFTP server.
   • Their public PGP key; if needed, partners can provide separate keys for their test and production servers.

Sending files to MX through SFTP

The following steps detail what is needed for a client to send batch files to MX’s SFTP server.

1. The partner will prepare and encrypt all batch files using the environment-specific MX public PGP key.
2. The partner will log into the MX SFTP server.
3. Once logged in, the partner must navigate to the files folder. From there, the partner will have access to all SFTP commands to upload the files.
4. If the partner requires response files, MX will encrypt the files using the partner's public PGP key and upload them to the partner's SFTP server.

SFTP hostname and port

The hostnames for each of the MX environments are listed below:

• MX Integration Environment: int-sftp.mx.com
• MX Production Environment: sftp.mx.com

MX uses a non-standard port for connections to the SFTP servers. The port number will be provided with the credentials.

Identifiers

Users, members, accounts, and transactions all use identifiers provided by the partner system. This makes it easy for you to later retrieve any of your data from other MX APIs because your system will know the identifier rather than having to store our identifier.

Because these identifiers might later be used as part of a URL when calling an MX API, you should specify the identifier using letters, numbers, dashes, and underscores only. Identifiers should be less than 1,024 bytes in length. Many partners choose to use 128-bit UUIDs or 64-bit sequential integers.

Date and times

Dates and times Many of the resources in the Batch API contain a pair of fields that end in _at and _on. You may choose to send either the _at or _on field according to what is easiest for your system. You should not provide dates in both formats.

• _at fields accept a Unix datetime.
• _on fields accept ISO 8601 dates without a timestamp, i.e., YYYY-MM-DD. The month and day should be zero padded.

File naming and size

Files sent to Batch should use this naming convention: PREFIX.TYPE.DATE.SUFFIX.csv.pgp. The .pgp extension is necessary due to the file being PGP encrypted.

All of the filename sections are required in order for Batch to recognize the file as valid.

An example of a correct file name follows:

_10

Bank.User.2016-12-31.Split001.csv.pgp

_10

BackfillProject.Member.2017-01-20.Split002.csv.pgp

_10

Credit.Account.2017-02-01.FixPreviousLoad01.csv.pgp

_10

MyBank.Transaction.2017-02-01.Complete.csv.pgp

An example of a batch file group follows:

_10

PreloadGroup.User.2017-10-31.batch001.csv.pgp

_10

PreloadGroup.Member.2017-10-31.batch001.csv.pgp

_10

PreloadGroup.Account.2017-10-31.batch001.csv.pgp

_10

PreloadGroup.Transaction.2017-10-31.batch001.csv.pgp

Prefix

The PREFIX is used to group files together into a batch.

For instance, a file called Bank.Transaction.20170202.Split001.csv.pgp will wait for the user, member. and account files prefixed with Bank to complete before it starts processing.

A file called Backfill.Transaction.20170202.Split001.csv.pgp will not wait for the Bank.* files to complete before it starts processing. Independent data sets should be given separate PREFIX values to make sure they can process in parallel.

Datasets that depend on one another should share the same PREFIX to make sure MX processes them in order, i.e., make sure accounts are processed before trying to process transactions.

Batch uses PREFIX.*.DATE to define the file grouping and the order of execution of the files. In other words, BANK.User.2019-06-26 will not start processing if BANK.User.2019-06-25 is still unprocessed.

Suffix

The SUFFIX should be used to identify unique groups of files. For example, if multiple batch files of transactions are being uploaded on the same day, the SUFFIX can be incremented with each new upload (e.g. batch001, batch002, etc.)

Type

The TYPE will be one of: User, Member, Account, Transaction.

Date

The DATE should be an ISO 8601 date with a four-digit year and zero-padded months and days. The format can be either YYYY-MM-DD or YYYYMMDD.

The date should reflect the date that the batch files are uploaded to MX. The date should be the same for all files in a file group.

File size

MX strongly recommends files containing one million records or fewer. This best optimizes Batch performance. Larger datasets should be split up to accommodate this recommendation.

The maximum supported size of a batch file is 3 million records; datasets containing more than 3 million records must be split into smaller files.

Upload order

Objects created in the MX Platform must follow the MX Architecture hierarchy. This means that batch files should be grouped appropriately and uploaded together to avoid errors due to order of processing. In other words, transactions can not be created before the user has been created.

MX requires manifest files for new integrations. The manifest ensures that all files within a batch will be received before processing and that MX will process them in the correct order.

For partners not using manifest files, care must be taken to ensure the files are processed in the correct order. All files that are part of the same upload grouping must have the same PREFIX, DATE, and SUFFIX.

Files should also be uploaded in the following order within each grouping to ensure the correct order of processing:

1. User files
2. Member files
3. Account files
4. Transaction files

This allows partners to create hierarchical objects in the correct order within one batch group. For example, an account can be created in the accounts batch file and then transactions for that account can be created in the transactions batch file.

Manifest files

Manifest files will be required for all new integrations. The manifest file specifies all the files that will be included in the batch and it will guarantee correct processing and data integrity. Manifests contain a list of filenames with optional checksums for each file in the batch.

The manifest file should be sent first; MX will begin processing the batch only when all listed files have been received.

Manifest file naming

Bank.Manifest.2016-12-31.csv.pgp

Manifests follow the same naming pattern as given above. Use the PREFIX and DATE values being used for the files in the batch, use Manifest as the TYPE, and omit the SUFFIX.

Manifest file contents

Manifests must be in CSV format and encrypted as specified above. Two columns are currently supported.

Field	Description
filename	The name of the files included in the batch.
checksum	The SHA256 checksum of the unencrypted file.

An example of manifest file contents follows:

_10

filename, checksum

_10

Bank.User.2020-04-13.Split1112.csv.pgp,389438d0f857041757a754e696375dc2b8089f2423ccbe5e3ff2a0e60ac1544f

_10

Bank.Member.2020-04-13.Split1112.csv.pgp,89fbc47675603079be7f9feca7625d8ac38802ac3e2c492edeed04a327db8015

_10

Bank.Acct.2020-04-13.Split1112.csv.pgp,5474f58d52395952268b10342b45dff780a0039636aba3a21a0b0ad5775d062d

_10

Bank.Tran.2020-04-13.Split1112.csv.pgp,b423960ce42eadd02ab36f3ba9313d5ac2d533ba3a4d2266d15a2df43d46e0c1