Migrating files to LFS

This guide shows how to migrate files that were (accidentally) not tracked by Git-LFS.

Follow this guide to inspect the ARC project storage in the DataHUB

You can easily check the usage quota used by your ARC by navigating to your ARC in the DataHUB and clicking on “Project Storage” in the right sidebar (1).

In terms of usage quota, two numbers are particularly important: LFS (1) and Repository (2) The majority of quota should fall into “LFS”, while the size of “Repository” should be kept small. If too many files are in “Repository”, this slows down the communication with and processing of your ARC.

Tip

As a rule of thumb, the Repository size should be kept below 1GB.

Contact a Data Steward

If you are new to Git or the command line,
please reach out to a data steward for On-ARC-Support.

Requirements

Section titled Requirements

• At least Maintainer level permission for the ARC being modified
• Git and Git-LFS installed locally
• Enough disk space to download a fresh copy of the ARC

Before you start

Section titled Before you start

Backup

This process rewrites history and can cause loss of data or issues with collaborators if not handled properly.

The selected large files in the (local) ARC are replaced by lightweight pointers. We recommend performing this on a freshly downloaded (“cloned”) ARC instead of the original copy on a user’s machine. For the migration, it suffices to download the ARC without already “properly LFS-tracked” large files, if the ARC is then synced back to the same remote (URL) in the DataHUB as shown below.

Migrating files to LFS

Section titled Migrating files to LFS

1. In the DataHUB, allow “force push” for the ARC

   Details

   1. Open your ARC in the DataHUB
   2. Via the sidebar navigate to Settings —> Repository
   3. Open the section “Protected branches”
   4. Activate the button Allowed to force push

2. Download a fresh copy of your ARC

   Details

   via ARC Commander

   arc get -n -r <ARC URL>

   via Git

   GIT_LFS_SKIP_SMUDGE=1 git clone --filter=blob:none <ARC URL>

3. Migrate to LFS

   specific file types

   You can use a pattern for a certain file type (here: .tiff and .fastq.gz) with

   git lfs migrate import --everything --include="*.tiff,*fastq.gz"

   specific files

   You can migrate all files in a certain folder (here runs/run01/run_results/) with

   git lfs migrate import --everything --include="runs/run01/run_results/*"

   or a single file with

   git lfs migrate import --everything --include="path/to/large-file.zip"

   by size threshold

   You can also migrate all files larger than a threshold (here 1 MB) with

   git lfs migrate import --everything --above="1 MB"

   Git-LFS docs

   For more details about git lfs migrate check out the Git-LFS docs.

4. Force push the changes to the DataHUB

   git push --all --force

5. (Recommended:) Deactivate allowing force push (see step 1)

6. (Recommended:) Run Housekeeping in the DataHUB

   Details

   1. Open your ARC in the DataHUB
   2. Via the sidebar navigate to Settings —> General
   3. Open the section “Advanced”
   4. Click the button Run housekeeping
   5. Click the button Prune unreachable objects

Collaborators

If the ARC is used by collaborators, coordinate the transition. A fresh download is the easiest way to have a clean ARC.

Merge and pull LFS files

If the migration was not performed on a fresh copy and the user does not want to download a fresh copy of the ARC:

1. Re-set tracking information for the current (e.g. main) branch via git branch --set-upstream-to=origin/main main
2. Setup merge for divergent branches via git config pull.rebase false
3. Locally merge the changes via git pull origin main --allow-unrelated-histories
4. (Optional) Download the large files now migrated to LFS via git lfs pull --include "*"