How to Handle Large Files with Git?

Git can face performance issues when handling large files because it is designed for versioning small to medium-sized files, including source code. For large files, we can use several methods to manage them effectively.

1. Using Git LFS (Large File Storage)

Git LFS is an open-source Git extension developed by GitHub for handling large files and binary files. It works by storing the content of large files on an LFS server, while only storing pointers to these large files in the Git repository. This avoids excessive local storage usage due to large files and improves the speed of cloning and pulling repositories.

Usage Steps:

1. Install Git LFS: Use the command git lfs install.
2. Select file types to track with LFS: git lfs track "*.psd" (to track all Photoshop files).
3. Commit the updated .gitattributes file.
4. Add and commit large files to the repository: git add file.psd and git commit -m "Add large file".
5. Push to the remote repository: git push origin main.

2. Optimizing the .gitignore File

For files that do not need version control, such as dependency packages and compiled outputs, add them to the .gitignore file. This reduces repository size and speeds up operations. For example, for Java projects, add target/, or for Node.js projects, add node_modules/.

3. Using Separated Storage

For certain projects, it may not be necessary to store all large files in the Git repository. For example, store datasets and user-uploaded files in external storage services (such as Amazon S3), and store links or access methods to these resources in the repository.

4. Regularly Cleaning the Repository

Use the git gc (garbage collection) command to optimize repository performance. Additionally, use git prune and git reflog expire to clean unnecessary objects and reference logs, freeing up space.

5. Shallow Cloning the Repository

If you only need the latest version, use shallow cloning to reduce the amount of data downloaded: git clone --depth 1 <repository-url>.

Example

In a previous project, we used numerous video files and images. To manage these large files, we introduced Git LFS. First, install and configure LFS using git lfs install, then use git lfs track to specify the file types to track. This significantly improved our repository management efficiency and sped up cloning and pulling operations.

By using these methods, we can effectively manage large files in Git while maintaining good performance and efficiency.