MyDataHelps Designer can automatically export participant survey results and sensor data from your project so you can perform detailed analysis on the data, or archive it.
Understanding Exports
This section covers several of the basic concepts needed to understand and use MyDataHelps Designer's data exports.
Data Storage Locations
MyDataHelps Designer uploads exported data to its secure, encrypted data storage system. You can access past exports through the Export History project tab (left pane) and download them for archive or analysis.
CareEvolution-approved projects can optionally configure MyDataHelps Designer to also upload a copy of the data to your own private Secure FTP server using SFTP or Amazon S3 Bucket (contact us if interested). You are responsible for setting up and maintaining the SFTP server and/or S3 bucket.
Data exports may contain Protected Health Information (PHI), Personally Identifiable Information (PII), and/or other health-related sensitive information, and must be transferred/stored in accordance with your organization’s security policies in order to ensure participant privacy and data security.
Incremental Exports
MyDataHelps Designer's data export feature performs only incremental exports, where the export date is based on when the data was synced, not when the data occurred (i.e. yesterday’s daily export contains all data received by MyDataHelps yesterday). Your first export will contain all project data so far, but subsequent exports will contain only new or updated data.
When a participant first connects their device to MyDataHelps, historical data may be pulled, which may result in a larger file size. See Collecting Sensor Data for the specific historical data lookback ranges for each device data type.
Additionally, you may notice that the incremental export range differs from the configured interval (i.e. daily, weekly) when you first turn on exports or update the settings. This could be due to a variety of reasons, such as:
- Turning on incremental exports after the project has started collecting data, resulting in larger exports to catch up from project start to the current date.
- Changing the UTC offset, which could make a daily export longer than a day since incremental exports cannot be less than one day.
- Changing the interval type or day. For example, if on the 26th of the month you changed a daily export to a weekly export on the 27th, the first weekly export could just be a few days.
We advise enabling data export before you start collecting data, to cut down on the size of your first data export. If you are ever in need of a full data backup, please contact CareEvolution's support team.
Export Scheduling
You can configure your project to perform daily or weekly incremental exports. For weekly exports, you may also select the day of week when the export will occur.
Data exports will begin at the next scheduled interval. For example, if you set up your project to export data weekly on Monday, and it's currently Tuesday, your next export will not occur until the next Monday.
Export Configuration and Data Formats
MyDataHelps Designer can export data in either Comma-Separated Value (CSV) or JSON format.
- CSV format is easy to import into common spreadsheet applications.
- JSON format is well-suited for programmatic analysis.
You can select which files are included in your data export, and what format each file will use. See Export File Type and Format Summary for detailed information about the data files, including example data.
Using Exported Data
Most projects use the exports for data analysis. Export files contain detailed participant information, survey results, sensor data, and more—everything you need to dive deep into analytics for your projects.
To analyze your data, you will need to download the exported data files to another computer. If you are using SFTP upload, your data export files will be uploaded to your private server. You can also download data exports from the project screen.
- Go to your project and select "Export History" tab.
- Select a specific export.
- Click the download button to save the export file to your local PC.
The downloaded export file is a ZIP archive containing each of the data files (in CSV or JSON format) you selected in your export settings. The "Manifest" file lists all files included in the export. See Export File Type and Format Summary for detailed information about the data files, including example data.
Setting Up Data Exports
Data exports may contain Protected Health Information (PHI), Personally Identifiable Information (PII), and/or other health-related sensitive information, and must be transferred/stored in accordance with your organization’s security policies in order to ensure participant privacy and data security.
To set up data exports:
- Go to your project and select the "Export Settings" tab.
- Select your export settings, described below.
- Click "Save" to save your settings. (You will need to scroll down to see the button.)
Available options include:
| Setting | Function | Examples |
|---|---|---|
| Incremental Exports |
The interval when incremental exports will be generated. See Incremental Exports for more information. Note: Selecting "None" will disable exports entirely. |
None, Daily, Weekly |
| UTC Offset |
By default, exports are performed at midnight UTC on the selected interval (daily/weekly). You can select a different time of day by changing the UTC Offset setting. Note: MyDataHelps Designer will never export less than 24 hours worth of data. If you're using daily exports, changing the UTC Offset may push your next export back a day. |
00:00 (midnight) UTC |
| Upload to SFTP Server | Whether the data exports should also be uploaded to your own Amazon S3 bucket. (Available to CareEvolution-approved projects only.) | Yes/No |
| SFTP Settings | SFTP server information. See below for details. | ---- |
| Upload to Amazon S3 Bucket | Whether the data exports should also be uploaded to your private SFTP server. (Available to CareEvolution-approved projects only.) | Yes/No |
| SFTP Settings | Amazon S3 bucket information. See below for details. | ---- |
| Allow Data File Uploads | Allow auxiliary data files to be uploaded directly to your export location. | Yes / No |
| Export Configuration |
Select which data files you want to include in your data exports. See Export Configuration and File Formats for more information. |
---- |
Upload to SFTP Server
If SFTP upload is enabled, you will need to enter information about your SFTP server so that MyDataHelps Designer can upload export files.
Please check with your organization's IT Support for assistance setting up SFTP.
To set up SFTP upload:
- Check "Upload to SFTP Server."
- Enter all the required server information. See below for details.
- Click "Test Upload Settings" to perform a test upload to the server.
- Check the specified directory on your SFTP server to ensure the upload was successful.
When setting up SFTP uploads, you can choose from several authentication methods:
- Password
- User Key
- Password and User Key
The "User Key" option allows you to connect to your SFTP server using a private SSH key, which is more secure than a password.
We recommend that you use the OpenSSH ssh-keygen command to generate your SSH keys. The private key should be generated in the PEM format, which you can check by verifying that the key file starts with "----BEGIN RSA PRIVATE KEY----". Older versions of ssh-keygen will default to the PEM format, but current versions require that you provide the "- m PEM" argument.
For an additional layer of security, you can enable both a password and a user key with an optional passphrase.
Whatever authentication setting you select must match the configuration on your SFTP server.
You may also enter the public key fingerprint (in MD5 format) so that MyDataHelps Designer can verify your server identity before transferring files.
In addition to the security options, you will need to configure the server host name, port, and a directory where the files will be stored.
Upload to Amazon S3 Bucket
In order for MyDataHelps designer to upload the export files to your own S3 bucket, you will need to do the following:
- Check "Upload to Amazon S3 Bucket."
- Enter the S3 Bucket Name.
- Enter the S3 Bucket System Region (e.g., us-east-2).
- Enter an optional S3 Folder Prefix to upload the exports to.
- Enter the S3 Storage Class you would like us to use (e.g., Standard, Standard Infrequent Access, etc.). See list of storage classes here.
- If the bucket is encrypted with a KMS key, specify the KMS Key ARN as well, otherwise we will encrypt the data with
SSE_S3.
After you save your settings, our application is going to create a role for your project in our AWS account for exporting data to your S3 bucket. The role ARN will look something like: arn:aws:iam::123456789012:role/EES3-RK-AAAAAA-Project-Name
Once the Role ARN has been generated, you will need to add the following to your S3 bucket's policy:
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:role/EES3-RK-AAAAAA-Project-Name"
},
"Action": [
"s3:GetObject",
"s3:GetObjectAcl",
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::your-bucket-name",
"arn:aws:s3:::your-bucket-name/*"
]
}
Additionally, if your bucket is encrypted with a KMS key, you will need to add the following to the KMS key policy:
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:role/EES3-RK-AAAAAA-Project-Name"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "your-kms-key-arn"
}
Troubleshooting Errors
When making updates to the Data Export, it is essential that you click the green "Save" button at the bottom of the page before testing or exiting. This should always be your first step in troubleshooting. See the scenarios below for additional troubleshooting tips:
- "Key 'OpenSSH' is not supported." This error appears at the top of the Data Export screen and indicates that the SFTP User Key needs to be updated to the default key generation format, as described in the previous section.
- "Upload Error: An established connection was aborted by server." This error will appear on the attempted data export within the Export History tab and indicates that your network may be rejecting the connection attempt. In this case, you should confirm with your organization's IT whether you can connect to the SFTP server from outside your network with a standard SFTP client. If possible, whitelist the following IP address: 52.200.215.173.