Setting Up Data Exports

MyDataHelps Designer can automatically export participant survey results and sensor data from your project so you can perform detailed analysis on the data, or archive it.

Understanding Exports

This section covers several of the basic concepts needed to understand and use MyDataHelps Designer's data exports.

Data Storage Locations

MyDataHelps Designer uploads exported data to its secure, encrypted data storage system. You can access past exports through the Export History project tab (left pane) and download them for archive or analysis.

CareEvolution-approved projects can optionally configure MyDataHelps Designer to also upload a copy of the data to your own private Secure FTP server using SFTP or Amazon S3 Bucket (contact us if interested). You are responsible for setting up and maintaining the SFTP server and/or S3 bucket.

Data exports may contain Protected Health Information (PHI), Personally Identifiable Information (PII), and/or other health-related sensitive information, and must be transferred/stored in accordance with your organization’s security policies in order to ensure participant privacy and data security.

Incremental Exports

MyDataHelps Designer's data export feature performs only incremental exports, where the export date is based on when the data was synced, not when the data occurred (i.e. yesterday’s daily export contains all data received by MyDataHelps yesterday). Your first export will contain all project data so far, but subsequent exports will contain only new or updated data.

When a participant first connects their device to MyDataHelps, historical data may be pulled, which may result in a larger file size. See Collecting Sensor Data for the specific historical data lookback ranges for each device data type.

Additionally, you may notice that the incremental export range differs from the configured interval (i.e. daily, weekly) when you first turn on exports or update the settings. This could be due to a variety of reasons, such as:

1. Turning on incremental exports after the project has started collecting data, resulting in larger exports to catch up from project start to the current date.
2. Changing the UTC offset, which could make a daily export longer than a day since incremental exports cannot be less than one day.
3. Changing the interval type or day. For example, if on the 26th of the month you changed a daily export to a weekly export on the 27th, the first weekly export could just be a few days.

We advise enabling data export before you start collecting data, to cut down on the size of your first data export. If you are ever in need of a full data backup, please contact CareEvolution's support team.

Export Scheduling

You can configure your project to perform daily or weekly incremental exports. For weekly exports, you may also select the day of week when the export will occur.

Data exports will begin at the next scheduled interval. For example, if you set up your project to export data weekly on Monday, and it's currently Tuesday, your next export will not occur until the next Monday.

Export Configuration and Data Formats

MyDataHelps Designer can export data in either Comma-Separated Value (CSV) or JSON format.

• CSV format is easy to import into common spreadsheet applications.
• JSON format is well-suited for programmatic analysis.

You can select which files are included in your data export, and what format each file will use. See Export File Type and Format Summary for detailed information about the data files, including example data.

Using Exported Data

Most projects use the exports for data analysis. Export files contain detailed participant information, survey results, sensor data, and more—everything you need to dive deep into analytics for your projects. 

To analyze your data, you will need to download the exported data files to another computer. If you are using SFTP upload, your data export files will be uploaded to your private server. You can also download data exports from the project screen.

1. Go to your project and select "Export History" tab.
2. Select a specific export.
3. Click the download button to save the export file to your local PC.

The downloaded export file is a ZIP archive containing each of the data files (in CSV or JSON format) you selected in your export settings.  The "Manifest" file lists all files included in the export. See Export File Type and Format Summary for detailed information about the data files, including example data.

You have the option to filter your exports, selecting the "Failed" toggle to show failed exports (e.g., expired SFTP credentials) or the "Custom" toggle to show custom exports (typically run by CareEvolution upon request).

Setting Up Data Exports

Data exports may contain Protected Health Information (PHI), Personally Identifiable Information (PII), and/or other health-related sensitive information, and must be transferred/stored in accordance with your organization’s security policies in order to ensure participant privacy and data security.

To set up data exports:

1. Go to your project and select the "Export Settings" tab.
2. Select your export settings, described below.
3. Click "Save" to save your settings. (You will need to scroll down to see the button.)

Available options include:

Upload to SFTP Server

If SFTP upload is enabled, you will need to enter information about your SFTP server so that MyDataHelps Designer can upload export files.

Please check with your organization's IT Support for assistance setting up SFTP.

To set up SFTP upload:

1. Check "Upload to SFTP Server."
2. Enter all the required server information. See below for details.
3. Click "Test SFTP Upload" to perform a test upload to the server.
4. Check the specified directory on your SFTP server to ensure the upload was successful.
5. Save export settings.

When setting up SFTP uploads, you can choose from several authentication methods:

• Password
• User Key
• Password and User Key

The "User Key" option allows you to connect to your SFTP server using a private SSH key, which is more secure than a password.

We recommend that you use the OpenSSH ssh-keygen command to generate your SSH keys. The private key should be generated in the PEM format, which you can check by verifying that the key file starts with "----BEGIN RSA PRIVATE KEY----". Older versions of ssh-keygen will default to the PEM format, but current versions require that you provide the "- m PEM" argument.

For an additional layer of security, you can enable both a password and a user key with an optional passphrase.

Whatever authentication setting you select must match the configuration on your SFTP server.

You may also enter the public key fingerprint (in MD5 format) so that MyDataHelps Designer can verify your server identity before transferring files.

In addition to the security options, you will need to configure the server host name, port, and a directory where the files will be stored.

Upload to Amazon S3 Bucket

In order for MyDataHelps Designer to upload the export files to your own S3 bucket, you will need to do the following:

1. Check "Upload to Amazon S3 Bucket."
2. Enter the S3 Bucket Name.
3. Enter the S3 Bucket System Region (e.g., us-east-2).
4. Enter an optional S3 Folder Prefix to upload the exports to.
5. Enter the S3 Storage Class you would like us to use (e.g., Standard, Standard Infrequent Access, etc.). See list of storage classes here.
6. If the bucket is encrypted with a KMS key, specify the KMS Key ARN as well, otherwise we will encrypt the data with SSE_S3.
7. Click "Populate Role ARN" and our system will create a role for your project in our AWS account for exporting data to your S3 bucket. The role ARN will look something like: arn:aws:iam::123456789012:role/EES3-RK-AAAAAA-Project-Name
8. Click "Test S3 Upload" to test that all settings are correct.
9. Save export settings.

Once the Role ARN has been generated and your S3 export settings have been saved, add the following to your S3 bucket's policy:

{
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/EES3-RK-AAAAAA-Project-Name"
    },
    "Action": [
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:ListBucket",
        "s3:PutObject",
        "s3:PutObjectAcl"
    ],
    "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
    ]
}

Additionally, if your bucket is encrypted with a KMS key, you will need to add the following to the KMS key policy:

{
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/EES3-RK-AAAAAA-Project-Name"
},
    "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
],
    "Resource": "your-kms-key-arn"
}

Sync Export Database to Amazon S3 Bucket

Instead of (or in addition to) transferring each incremental export to an Amazon S3 bucket, you have the option to sync the export database to an Amazon S3 bucket as parquet files. The export database is what the Export Explorer uses to run SQL queries across your project's export data or to make external connections to your data analysis applications.

In order for MyDataHelps Designer to sync the export database to your S3 bucket, you can mostly follow the instructions to Upload to Amazon S3 Bucket, but for the Sync Export Database to Amazon S3 Bucket setting.

You'll use a bucket policy similar to:

{
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/EES3-RK-AAAAAA-Project-Name"
    },
    "Action": [
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:ListBucket",
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:DeleteObject",
        "s3:ListBucketMultipartUploads",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts",
    ],
    "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
    ]
}

In addition to your KMS key policy, you have the option to provide a SNS Topic ARN, which can notify you when export synchronization starts and stops. If specified, the SNS Topic policy must be similar to:

{
   "Sid": "MyDataHelpsPublish",
   "Effect": "Allow",
   "Principal": {
      "AWS": "arn:aws:iam::621233246578:role/EES3-RK-10B79640-General-Research-Study"
   },
   "Action": "SNS:Publish",
   "Resource": "your-topic-arn"
}

Troubleshooting Errors

When making updates to the Data Export, it is essential that you click the green "Save" button at the bottom of the page before testing or exiting. This should always be your first step in troubleshooting. See the scenarios below for additional troubleshooting tips:

1. "Key 'OpenSSH' is not supported." This error appears at the top of the Data Export screen and indicates that the SFTP User Key needs to be updated to the default key generation format, as described in the previous section.
   
2. "Upload Error: An established connection was aborted by server."  This error will appear on the attempted data export within the Export History tab and indicates that your network may be rejecting the connection attempt.  In this case, you should confirm with your organization's IT whether you can connect to the SFTP server from outside your network with a standard SFTP client. If possible, whitelist the following IP address: 52.200.215.173.