Check Files
Ensure your batch files are correctly formatted before submitting them to MX for processing by using our Batch File Checker:- Download the Batch File Checker, unzip it, and open csv-checker/index.html.
- Follow the instructions presented on the webpage.
File Format
Batch files for creating users, members, accounts, and transactions must adhere to the following standardized format:- Encoding: UTF-8.
- Format: CSV.
- Quoting: Use field quoting when necessary.
- Header Row: Required to specify the fields you’re providing. For which fields are required/optional for each file, see File Contents.
- Customization: Line ending, separator, and quote characters are customizable.
Initial SFTP Setup
To set up access to MX’s integration and production SFTP servers:- Provide source and destination IP addresses (and domain name, if applicable) for whitelisting.
- MX will securely provide usernames and passwords for both integration and production environments, and port numbers.
- (Optional) MX will provide public PGP keys for integration and production environments. All batch files must be encrypted with the environment-specific MX public PGP key.
- (Optional) If you require response files, provide MX with:
- SFTP URL and login information for your test and production servers.
- Destination IP addresses for your SFTP server.
- (Optional) Your public PGP key.
Sending Files via SFTP
To send batch files to MX’s SFTP server:- Prepare each file, ensuring proper formatting, file-naming requirements, and optional encryption with the environment-specific MX public PGP key.
- Log into the MX SFTP server.
- Navigate to the upload directory (default:
filesfolder). - Upload files using SFTP commands.
- If requested, MX will upload encrypted response files (using your provided PGP key, if applicable) to your SFTP server.
SFTP Hostname and Port
The MX SFTP hostnames for each MX environment are as follows:- MX Integration Environment:
int-sftp.mx.com - MX Production Environment:
sftp.mx.com
PGP Keys
These keys are used to encrypt files. Use the key that matches the target environment:Integration Environment Public Key
Production Environment Public Key
Identifiers
Users, members, accounts, and transactions use identifiers provided by your system. This enables retrieval via other MX APIs using your system’s identifiers. We recommend identifiers consist of letters, numbers, dashes, and underscores, and are less than 1,024 bytes. Common choices include 128-bit UUIDs or 64-bit sequential integers.Date and Times
Batch API resources often include_at and _on fields. Provide dates using either format, but not both:
_at: Unix datetime._on: ISO 8601 dates (for example, YYYY-MM-DD). Zero pad the month and day.
File Naming and Size
Batch files must follow a fixed naming convention. Each batch filename will have four portions plus the filename extensions. A manifest filename has three portions plus the filename extensions. Files that are not named correctly will not be processed. Filename Format:- Batch files:
PREFIX.TYPE.DATE.SUFFIX.csv.pgp - Manifest file:
PREFIX.Manifest.DATE.csv.pgp
Batch File Naming Convention
Full Format:
PREFIX.TYPE.DATE.SUFFIX.csv.pgp- Manifest:
PREFIX.Manifest.DATE.csv.pgp
Bank.User.2016-12-31.Split001.csv.pgpBackfillProject.Member.2017-01-20.Split002.csv.pgpCredit.Account.2017-02-01.Part01of01.csv.pgpMyBank.Transaction.2017-02-01.Complete.csv.pgp
Batch Definition
A batch is a group of batch files that are intended to be processed together. The batch is identified by the combination of thePREFIX and DATE fields in the filenames. Every file in the batch must have the same PREFIX and DATE values, and the combination of those values must be unique across all batches for the client. If multiple batches are uploaded with the same DATE value, they must have different PREFIX values to be considered separate batches.
Example filenames for a batch file group containing one each of user, member, and account files, two transaction files, and the manifest file:
Prefix
ThePREFIX groups files from the same DATE into a batch, ensuring sequential processing according to the MX Architecture hierarchy (for example, users → members → accounts → transactions).
For example, a transactions file named Bank.Transaction.20170202.Split001.csv.pgp will not be processed until the user, member, and account files prefixed with Bank and having date 2017-02-02 have completed processing.
In contrast, independent data sets for a particular DATE can be given a different PREFIX value. MX will consider that set of files a different batch because of the different PREFIX. A batch using the same 2017-02-02 date as in the example above, but using Backfill as its prefix, won’t wait for the batch with Bank as its prefix to complete before it starts processing. The batches will be processed in parallel.
Type
TheTYPE indicates the object type and must be one of: User, Member, Account, Transaction, or Manifest. Use the singular form only.
Date
TheDATE must be an ISO 8601 date with a four-digit year and zero-padded month and day (YYYY-MM-DD or YYYYMMDD). Note that this date represents the data itself, not necessarily the date of transmission. While it often coincides with the upload date for daily processing, it should reflect the historical date when sending backfill data. This DATE must be consistent for all files within the batch.
Suffix
TheSUFFIX differentiates unique files of the same type within a batch. For example, if multiple Transaction files are being included in the batch, the SUFFIX allows them to be uniquely named. A recommended practice is to use a SUFFIX ending with a number and incrementing the number for each file of that type (part001, part002, and so on). A file of type Manifest does not require a suffix.
File Size
- Recommended: 1 million records or fewer for optimal performance.
- Maximum: 3 million records. Larger datasets must be split.
Upload Order
Objects must be created following the MX Architecture hierarchy. Group batches appropriately and upload them in the correct order to avoid errors. For example, transactions can’t be created before the user, member, and account records have been created. Early integrations that did not use manifest files had to ensure files were uploaded in strict hierarchy order to ensure files in the same batch were processed in the correct order. Manifest files (required for all new integrations) ensure all files for a batch are received before processing starts, and ensure they are processed in the correct order. When doing an initial data load, it is sometimes easier for a client to use separate batches to create each object in the MX hierarchy rather than trying to create them all in a single batch. In such a case, the batches must be uploaded in strict MX hierarchical order to ensure parent objects are created before child objects that attach to them. The order is:- User files
- Member files
- Account files
- Transaction files
Manifest Files
Manifest files are required for all new integrations. The manifest lists all files that MX should expect to receive for that batch and allows optional checksums for each file, guaranteeing correct processing and data integrity. It is a best practice to send the manifest file first. MX will begin processing the batch only when all listed files are received, and no files will be processed that aren’t listed in the batch manifest.Manifest File Naming
Manifests follow the same naming pattern:PREFIX.TYPE.DATE.csv.pgp. Use the PREFIX and DATE values for the batch, set TYPE to Manifest, and omit the SUFFIX.
Example: Bank.Manifest.2016-12-31.csv.pgp
Manifest File Contents
Manifests must be in CSV format. The filenames must be listed exactly as they will be sent to MX, including filename extensions. If the batch is encrypted, each batch file must be PGP encrypted and the manifest file itself must also be PGP encrypted and include the.csv.pgp extensions.
Two columns are supported:
An example of manifest file contents follows.