Raw Data Introduction
The raw data summary view shows detailed information about the raw data already registered for the measurements within your experiment. Additionally, it enables you to generate the URLs necessary to start the command line based raw data download.
Process
- Create a personal access token (PAT)
- Navigate to the raw data summary view
- Acquire download URLs for the registered measurement raw data.
- Start the measurement raw data download measurement and get a coffee
Personal access token
Before you can begin with downloading any data via HTTPS, you need to tell the download server who you are. We enforce a token-based authentication (personal access token: PAT) so you are not required to expose your password to anyone or to any system.
multiple tokens
You can create as many tokens as you like, however consider them as a secret.
Generate a token
First of all, navigate to your PAT overview page in your profile overview (top-right corner)
You should now be able to see the PAT token overview page:
Personal access tokens have a life-time, which can be set based on your requirements. Your token expires automatically, there is nothing you have to do manually.
Token accessibility
A generated token will be only visible in its raw notation once! Make sure to store it safely in your local password manager, you will not be able to access the token value again.
Manage tokens
You can see your created tokens in the overview, however since they are instantly encrypted after generation, you cannot access the actual token text anymore. We encourage you to use meaningful descriptions for your tokens, so you remember for what purpose you have created them.
Token security
If you are unsure, if your PAT got exposed or shared with untrusted parties, delete them right away. You can create new, safe tokens at any time.
Raw Data Navigation
To navigate to the raw data summary view start by navigating to the project summary view of your project of interest.
From there navigate into the experiment summary view of interest.
From within the experiment summary you can navigate into the raw data summary view.
To do so, click on the "Download Raw Data" tab within the experiment navigation bar on the top.
This will take you to the raw data summary view
Raw Data URL Generation
To acquire the URLS necessary for the measurement data download the following steps have to be taken:
- Navigate to the raw data summary view
- Select the measurements for which the raw data URLs should be generated.
- Press the "Download URL List" button to download a text file containing the URLs of the selected measurements
Download Raw Data
To download the raw data for your measurements, basic command line/terminal knowledge is required. First start by opening the command line in your operating system of choice.
Open command line on Mac
Open the terminal via the spotlight search function by clicking on the magnifying glass on the top right or via pressing the command and spacebar key on your keyboard. Input "terminal" into the text field and select the terminal application from the suggestions.
Open command line on Windows
Click on the magnifying glass in the toolbar at the bottom of the screen. Input "terminal" into the search field and select the terminal application from the suggestions.
Download data via command line
We show how to run the download via the two popular command line clients cURL and GNU Wget. Of course, you can use any other software that supports HTTP.
Warning
The data will be downloaded into the directory in which the command is run. Please ensure that it's the correct directory and there is enough space before running the download
The commands contain two placeholders, where you have to provide your custom input:
- ACCESS_TOKEN: your generated personal access token
- MEASUREMENT_URL: the URL associated with the data of the measurement
curl -OJ -H "Authorization: Bearer <ACCESS_TOKEN>" <MEASUREMENT_URL>
wget --content-disposition --trust-server-names --header "Authorization: Bearer <ACCESS_TOKEN>" <MEASUREMENT_URL>
For example to download the raw data associated with measurement MSQ7645002AL-182987406699583 the commands would look as follows:
curl --parallel --fail -OJ -H "Authorization: Bearer v71E00f750Z78oBW4SKs90Vrd39h98eG" https://download.qbic.uni-tuebingen.de/measurements/MSQ7645002AL-182987406699583
wget --content-disposition --trust-server-names --header "Authorization: Bearer v71E00f750Z78oBW4SKs90Vrd39h98eG" https://download.qbic.uni-tuebingen.de/measurements/MSQ7645002AL-182987406699583
To download multiple measurements using one command, you can provide a text file containing one measurement URL per line (as in the one you can download in the raw data summary!).
<MEASUREMENT_URL_1>
<MEASUREMENT_URL_2>
The corresponding commands need to be adapted accordingly.
curl --remote-name-all -OJ -H "Authorization: Bearer <ACCESS_TOKEN>" $(cat <file-with-urls>)
wget --content-disposition --trust-server-names --header "Authorization: Bearer <ACCESS_TOKEN>" -i <file-with-urls>
Warning
To ensure character validity in the text file, please format it in the UTF-8 format
The easiest way to generate such a text file is to generate it in the raw data view.
This will download the text file containing the urls for the selected measurements
For example to download the raw data for all measurements within the text file download_urls.txt the command would look as follows:
curl --remote-name-all -OJ -H "Authorization: Bearer v71E00f750Z78oBW4SKs90Vrd39h98eG" $(cat /Your/awesome/path/download_urls.txt)
wget --content-disposition --trust-server-names --header "Authorization: Bearer v71E00f750Z78oBW4SKs90Vrd39h98eG" -i /Your/awesome/path/download_urls.txt