Batch API Overview

MX allows you to upload batch files of records representing held account data to be processed by MX’s Platform. These batch files must be properly formatted and encrypted CSV files, and are transmitted to MX via SFTP. Additionally, MX can send a response file for each batch file submitted.

File Format

Use the MX Batch API to create users, members, accounts, and transactions. These batch files must follow this standardized format:

• UTF-8 encoded CSV file.
• Use field quoting when necessary.
• Header row must be included in each file to specify which fields are being provided. Details about the fields expected in each file can be found below.
• The line ending, separator character, and quote character may be customized depending on your needs.

Initial SFTP Setup

The initial setup for accessing MX integration and production SFTP servers is as follows:

1. Provide both source and destination IP addresses to be whitelisted and a domain name where possible.
2. MX will securely deliver a username and password for both integration and production environments, and the port to be used for upload.
3. (Optional) MX will provide its public PGP keys for both integration and production environments. All batch files must be encrypted with the environment-specific MX public PGP key.
4. (Optional) If you require response files, provide the following to MX:
   • The SFTP URL and login information for your test and production servers.
   • The destination IP addresses used by your SFTP server.
   • (Optional) Your public PGP key. If needed, you can provide separate keys for your test and production servers.

Sending Files to MX through SFTP

The following steps detail what's needed for you to send batch files to MX’s SFTP server.

1. Prepare each file by ensuring proper formatting, file-naming requirements, and optionally encrypting all batch files using the environment-specific MX public PGP key.
2. Log into the MX SFTP server.
3. Once logged in, navigate to the appropriate directory for uploading batch files. The default location is the files folder. From there, you can access all SFTP commands to upload the files.
4. If you require response files, MX will upload them to the your SFTP server. If you opted to have response files encrypted, they'll be encrypted using your provided PGP key.

SFTP Hostname and Port

The MX SFTP hostnames for each of the MX environments are as follows:

• MX Integration Environment: int-sftp.mx.com
• MX Production Environment: sftp.mx.com

MX uses a non-standard port for connections to the SFTP servers. This port number will be provided with the credentials.

Identifiers

Users, members, accounts, and transactions all use identifiers provided by your system. This allows you to later retrieve any of your data from other MX APIs because your system will know the identifier rather than having to store our identifier.

Because these identifiers might later be used as part of a URL when calling an MX API, you should specify the identifier using letters, numbers, dashes, and underscores only. Identifiers should be less than 1,024 bytes in length. Many partners choose to use 128-bit UUIDs or 64-bit sequential integers.

Date and Times

Many of the resources in the Batch API contain a pair of fields that end in _at and _on. You may choose to send either the _at or _on field according to what's easiest for your system. Don't provide dates in both formats.

• _at fields accept a Unix datetime.
• _on fields accept ISO 8601 dates without a timestamp, for example, YYYY-MM-DD. The month and day should be zero padded.

File Naming and Size

Files sent to Batch should use this naming convention: PREFIX.TYPE.DATE.SUFFIX.csv.pgp. The .pgp extension is only necessary when files are being PGP encrypted.

All of the filename sections are required in order for Batch to recognize the file as valid.

Examples of correct file names are as follows:

_10

Bank.User.2016-12-31.Split001.csv.pgp

_10

BackfillProject.Member.2017-01-20.Split002.csv.pgp

_10

Credit.Account.2017-02-01.FixPreviousLoad01.csv.pgp

_10

MyBank.Transaction.2017-02-01.Complete.csv.pgp

An example of properly named batch files that are part of the same batch file group follows:

_10

PreloadGroup.User.2017-10-31.batch001.csv.pgp

_10

PreloadGroup.Member.2017-10-31.batch001.csv.pgp

_10

PreloadGroup.Account.2017-10-31.batch001.csv.pgp

_10

PreloadGroup.Transaction.2017-10-31.batch001.csv.pgp

Prefix

The PREFIX is used to identify files as being part of the same batch.

Grouping files into a batch using the prefix ensures that the files are processed in order according to MX Architecture heirarchy (for example, users → members → accounts → transactions). If different prefixes are used then MX may attempt to process a transactions file in parallel with the file containing the users, members, and accounts that are the parent objects of those transactions, resulting in errors for the transactions.

For instance, a transactions file called Bank.Transaction.20170202.Split001.csv.pgp will wait for the user, member, and account files prefixed with Bank to complete before it starts processing.

In contrast, independent data sets should be given separate PREFIX values to make sure they can process in parallel. A file called Backfill.Transaction.20170202.Split001.csv.pgp will not wait for the Bank prefixed files to complete before it starts processing.

Type

The TYPE indicates what type of object records the file contains and must be one of: User, Member, Account, Transaction.

The TYPE must be exactly as described above, in the singular form. Files that provide the type in plural form will not be recognized by the Batch system.

Date

The DATE should be an ISO 8601 date with a four-digit year and zero-padded months and days. The format can be either YYYY-MM-DD or YYYYMMDD.

The date should reflect the date that the batch files are uploaded to MX and does not need to reflect the date of the data contained in the file. The date should be the same for all files in a file group.

Suffix

The SUFFIX should be used to identify unique groups of files. For example, if multiple batch files of transactions are being uploaded on the same day, the SUFFIX can be incremented with each new upload (e.g. batch001, batch002, etc.)

File Size

MX strongly recommends files containing one million records or fewer. This best optimizes Batch performance. Larger datasets should be split up to accommodate this recommendation.

The maximum supported size of a batch file is 3 million records; datasets containing more than 3 million records must be split into smaller files.

Upload Order

Objects created in the MX Platform must follow the MX Architecture hierarchy. Group batch files appropriately and upload them together to avoid errors due to order of processing. In other words, transactions can't be created before the user has been created.

MX requires manifest files for new integrations. The manifest ensures that all files within a batch will be received before processing and that MX will process them in the correct order.

If you're not using manifest files, you must ensure the files are processed in the correct order. All files that are part of the same upload grouping must have the same PREFIX, DATE, and SUFFIX.

Upload files in the following order within each grouping to ensure the correct order of processing:

1. User files
2. Member files
3. Account files
4. Transaction files

This allows you to create hierarchical objects in the correct order within one batch group. For example, an account can be created in the accounts batch file and then transactions for that account can be created in the transactions batch file.

Manifest Files

Manifest files will be required for all new integrations. The manifest file specifies all the files that will be included in the batch and it will guarantee correct processing and data integrity. Manifests contain a list of filenames with optional checksums for each file in the batch.

Send the manifest file first; MX will begin processing the batch only when all listed files have been received.

Manifest File Naming

Bank.Manifest.2016-12-31.csv.pgp

Manifests follow the same naming pattern as given above. Use the PREFIX and DATE values being used for the files in the batch, use Manifest as the TYPE, and omit the SUFFIX.

Manifest File Contents

Manifests must be in CSV format and encrypted as specified above. Two columns are currently supported.

Field	Description	Required?
filename	The name of the files included in the batch.	Yes
checksum	The SHA256 checksum of the unencrypted file.	No

An example of manifest file contents follows:

_10

filename, checksum

_10

Bank.User.2020-04-13.Split1112.csv.pgp,389438d0f857041757a754e696375dc2b8089f2423ccbe5e3ff2a0e60ac1544f

_10

Bank.Member.2020-04-13.Split1112.csv.pgp,89fbc47675603079be7f9feca7625d8ac38802ac3e2c492edeed04a327db8015

_10

Bank.Acct.2020-04-13.Split1112.csv.pgp,5474f58d52395952268b10342b45dff780a0039636aba3a21a0b0ad5775d062d

_10

Bank.Tran.2020-04-13.Split1112.csv.pgp,b423960ce42eadd02ab36f3ba9313d5ac2d533ba3a4d2266d15a2df43d46e0c1