> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mx.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Using the Reporting API

> Best practices for consuming and staging Reporting API data in your system.

The MX Reporting API enables you to track changes for all data held on the MX platform for your clients without having to read this data individually for each user. This is done by providing daily change files which indicate how objects have changed throughout the day.

This guide provides best practices on consuming and staging the data in your system.

## Consuming Daily Reporting Files

Objects on the MX platform are organized in a hierarchy. This means that your systems must consume daily reporting files in a particular order so that the objects are created, updated, and deleted in the proper order in your data store/warehouse.

First, you must consume all files with **create** actions in this order:

1. Users
2. Members
3. Accounts
4. Transactions
5. Holdings
6. Categories
7. Tags
8. Taggings
9. Goals
10. Budgets
11. Notification Profiles
12. Beat
13. Beat Feedback
14. Devices
15. Analytics Events
16. Analytics Page Views
17. Analytics Screen Views
18. Analytics Timed Events
19. Insights

Second, you must consume all **update** files in the **same order** as above.

Third, you must consume all **delete** files in the **reverse order**. This ensures transactions are deleted on your systems before the account they belong to is deleted, and so forth.

## Generating Sample Data for the Integrations Environment

Files in the Reporting API are generated from system events which represent user activity and aggregated account and transactional data. This presents a challenge in the integrations environment because there are no users creating activity on the system.

If you need a more robust set of test data, add your own by creating internal test users that add accounts and use the system for a few days to generate log events.

The following steps describe this process:

1. Create test users in your integration client using whichever MX API you use for this purpose, e.g., the Platform API or MDX v5 Real Time.
2. Generate `master_widget` URLs for those users with the appropriate API, e.g., the Platform API the the SSO API.
3. Copy the URL from the `master_widget` response and paste it in a browser window.
4. Have your test user(s) use the system. Some recommended actions are:
   * Add savings, checking, loans, and investment accounts;
   * Categorize transactions;
   * Add tags to transactions;
   * Create custom categories;
   * Create and update goals and budgets;
5. The next day, new files containing the event logs of the actions performed will become available to download via the [*download daily files*](/api-reference/more-apis/reporting/reference/daily-files/download-daily-files).

## Byte Serving for Large Files

Avro files can become very large (multiple gigabytes) which can result in partial downloads. This can be resolved by using byte serving, which allows you to request data in a set of ranged chunks that can later be assembled into the full raw Avro file response.

### Step 1: Check the Size of Available Files

Use the [list daily files](/api-reference/more-apis/reporting/reference/daily-files/list-daily-files) endpoint to get a list of all the files that are available from the last 7 days. This list will include the size of each file. All files larger than 1GB should be downloaded using byte serving.

```shell theme={null}
curl -X GET https://int-logs.moneydesktop.com/download/{client_id} \
  -H 'Accept: application/vnd.mx.logs.v1+json' \
  -H 'MD-API-KEY: {api_key}'
```

### Step 2: Download File Segments Using the curl Command

The command line tool `curl` can be used to download HTTP ranges by specifying the `-r` or `--range` option. This example shows a scenario where the Avro file is larger than 1GB. The first `curl` command specifies the range for the first gigabyte (`0-1073741823`) and the second command specifies the range for the rest of the data (`1073741824-`).

```shell theme={null}
# Download the first part of the file

curl -X GET -r 0-1073741823 https://int-logs.moneydesktop.com/download/{client_id}/2019-10-07/transactions/created -o 20191007-transactions-created.avro.part1 \
  -H 'Accept: application/vnd.mx.logs.v1+avro' \
  -H 'MD-API-KEY: {api_key}'

# Download the second part of the file

curl -X GET -r 1073741824- https://int-logs.moneydesktop.com/download/{client_id}/2019-10-07/transactions/created -o 20191007-transactions-created.avro.part2 \
  -H 'Accept: application/vnd.mx.logs.v1+avro' \
  -H 'MD-API-KEY: {api_key}'
```

### Step 3: Assemble the Segments Back into a Single File

The next step is to assemble the two file partials into a single file. The `cat` command `cat input1 input2 > output` can be used for this purpose. In this case `input1` and `input2` are the file segments downloaded in the previous step. The output file will be a complete Avro file.

```shell theme={null}
cat 20191007-transactions-created.avro.part1 20191007-transactions-created.avro.part2 > 20191007-transactions-created.avro
```

## Parsing Avro Files

Avro files are built in a way that they can be parsed and serialized easily into other formats. [Avro's documentation](http://avro.apache.org/docs/current/index.html) provides guidance on parsing these files using different methods.

<Note>
  Decimal numbers may be represented using exponential notation. Your implementation should accept decimals in exponential notation to avoid conversion errors.
</Note>

Below we show how to read an Avro file into a Ruby script and parse the output to JSON and CSV. There is an Avro gem available from rubygems.org which we use to parse the Avro file.

**Avro to JSON**

```
require 'avro'
require 'json'

json_array = []
avro_file_path = "some_avro_file.avro"

Avro::DataFile.open(avro_file_path, "r") do |reader|
    reader.each do |row|
        json_array << row.to_json
    end
end
```

**Avro to CSV**

```
require 'avro'
require 'csv'

result_file_path = "some_csv_file.csv"
avro_file_path = "some_avro_file.avro"

Avro::DataFile.open(avro_file_path, "r") do |reader|
    CSV.open(result_file_path, "a+") do |csv|
        reader.each_with_index do |row, index|
            csv << row.keys if index == 0
            csv << row.values
        end
    end
end
```