Download data¶

ANC datasets are stored as Git repositories in the ANC GitLab.

Downloading a dataset to a local system is done by cloning the dataset repository from its ANC GitLab project. A clone is a local version of the dataset repository. It remains linked to the remote repository, so changes can be made locally and, if you have write permissions, pushed back to the repository. See our guide on how to work locally.

To work with ANC dataset repositories, Git LFS must be installed. Git LFS allows large files to be tracked by Git without storing their full contents directly in the Git repository. Instead, Git stores small pointer files, while the actual large file contents are stored separately in LFS storage.

Git LFS also makes it possible to clone a dataset repository without immediately downloading all large files. This is useful for large datasets, or when only a subset of the data, such as one subject, is needed.

Install Git and Git LFS¶

Git and Git LFS may already be installed on your system, especially if you are using a Linux operating system. You can check this by running the following commands in your terminal and verifying that version numbers are returned.

git --version
git lfs --version

If Git or Git LFS is not installed on your system, follow the installation instructions for Git and Git LFS.

Set up Git LFS¶

Open the command line interface and enter the following command:

git lfs install --skip-smudge

Using the --skip-smudge flag ensures that LFS files are not downloaded automatically during cloning or checkout. Instead, large files appear as small pointer files until their contents are explicitly pulled.

We recommend installing Git LFS this way, regardless of whether you plan to download the large files immediately. This helps prevent accidentally downloading an entire dataset.

Get the clone link¶

To clone a dataset repository, first open the dataset project in the ANC GitLab. How to find relevant dataset projects is described in Find relevant datasets.

In the dataset project, click the Code button and copy one of the available clone links:

• Clone with SSH, if your SSH key is configured in GitLab.
• Clone with HTTPS, if you want to authenticate with GitLab credentials.

The copied link is used in the git clone command below.

Cloning a repository without LFS files¶

You can clone a dataset repository without downloading the contents of the LFS files. In this case, LFS files appear as representative pointer files in your local clone. These pointer files are small text files that refer to the actual large files in LFS storage.

The pointer files can be moved, renamed, or deleted like regular files. Such operations affect the corresponding files in the dataset repository. However, the actual contents of the large files are not available locally until they are pulled with Git LFS.

Open the command line interface, navigate to the directory where the repository should be stored, and enter the following command:

git clone <yourrepositorylink>

Getting LFS files¶

If the contents of the LFS files are required, for example for analysis, they can be pulled after cloning. It is possible to pull all LFS files in a dataset, or only a subset.

Cloning the entire repository with all LFS files¶

Open the command line interface, navigate to the directory where the repository should be stored, and enter the following command:

GIT_LFS_SKIP_SMUDGE=0 git clone <yourrepositorylink>

By setting GIT_LFS_SKIP_SMUDGE=0, Git LFS downloads the contents of the LFS files during cloning instead of leaving them as pointer files.

Pulling specific LFS files¶

It is also possible to download only specific LFS files by using filenames or filename patterns.

Open the command line interface and navigate to the cloned dataset repository. Then enter the following command with the relative paths to the files:

git lfs pull --include="file1,file2"

This is a resource-saving option if only a small fraction of the dataset is needed.

You can also use a file pattern. For example, to download all runs of the Stroop task within the first session of subject sub-01, use:

git lfs pull --include="sub-01/ses-01/func/sub-01_ses-01_task-stroop*"

To download all files for one subject, use a subject-level pattern:

git lfs pull --include="sub-01/**"

Removing downloaded LFS files from local storage¶

After downloading selected LFS files, for example one subject, you may later want to free local disk space again. This can be done by replacing the downloaded LFS file contents with pointer files and then pruning the local Git LFS cache.

Run the following commands from inside the cloned dataset repository:

git read-tree HEAD
GIT_LFS_SKIP_SMUDGE=1 git checkout -f HEAD
git lfs prune --force

The first two commands replace locally checked-out LFS file contents with pointer files. The final command removes locally cached LFS objects where possible.