Domain 6CI/CD & GitOpsINTERMEDIATE

Git Infrastructure, Architecture, Operations, and Troubleshooting Guide

A comprehensive guide to Git architecture, CLI workflows, index and object mechanics, remote management, troubleshooting, and production best practices for Linux system administrators and SREs.

Domain: Domain 6: CI/CD, GitOps & Registry•Updated: 2026-11-03
#linux#sysadmin#git#devops#sre

1. Overview

Git is a distributed version control system (DVCS) designed to track changes in source code and configuration files over time. Unlike legacy centralized version control systems (such as Subversion or CVS) that rely on a central server to store file history, Git provides every working copy with a full repository containing the complete commit history. Git stores project history as a directed acyclic graph (DAG) of immutable commits—snapshots of the filesystem tree paired with metadata such as author, timestamp, commit message, and parent commit hashes.

In modern Linux system administration, Site Reliability Engineering (SRE), and DevOps environments, Git serves as the foundational storage and state-tracking engine. Production teams use Git not only for application code, but also for managing Infrastructure as Code (IaC) via Terraform, configuration management code via Ansible or Puppet, and Kubernetes declarative manifests. Continuous Integration and Continuous Deployment (CI/CD) pipelines trigger automatically on branch updates or tag pushes, pulling tagged releases or commit SHAs to execute automated testing, image builds, and production deployments.

+-----------------------------------------------------------------------+
|                         Git Production Ecosystem                      |
|                                                                       |
|  +------------------+    Git Push / PR    +------------------------+  |
|  | Developer Workstation| ------------------> | Remote Git Host        |  |
|  | (Local DAG & Code)   | <------------------ | (GitHub/GitLab/Gitea)  |  |
|  +------------------+     Git Fetch       +------------------------+  |
|                                                       |               |
|                                               Webhook / Pipeline      |
|                                                       v               |
|                                           +------------------------+  |
|                                           | CI/CD Runner           |  |
|                                           | (Pulls Tag / SHA)      |  |
|                                           +------------------------+  |
|                                                       |               |
|                                                    Deploy             |
|                                                       v               |
|                                           +------------------------+  |
|                                           | Production Servers     |  |
|                                           | (K8s, Virtual Machines)|  |
|                                           +------------------------+  |
+-----------------------------------------------------------------------+

Advantages and Limitations

Feature Domain Git (Distributed VCS) Centralized VCS (SVN, Perforce)
Architecture Fully distributed; local repository has full history. Centralized; clients hold working copies without history.
Network Dependency Operations (commit, diff, log, branch) are local and offline. Requires constant connection to central server for history operations.
Branching & Merging Lightweight branch pointers; fast creation, switching, and merging. Heavyweight directory copying; merging is often complex and slow.
Cryptographic Integrity Directed Acyclic Graph anchored by SHA checksums for every object. File revision numbers without built-in graph integrity validation.
Large Binary Storage Requires extensions (e.g., Git LFS) for massive binaries. Natively handles large binary assets better via file locking.

Simple Practical Example

To illustrate Git’s core workflow: if a system administrator updates an Nginx site configuration, Git tracks the state change across three distinct areas—the Working Tree, the Staging Area (Index), and the Local Repository.

# Edit Nginx configuration file in the working tree
vim /etc/nginx/sites-available/app.conf

# Inspect working tree status
git status

# Stage the file to record it in the index for the next commit
git add /etc/nginx/sites-available/app.conf

# Commit snapshot to local repository DAG
git commit -m "feat(nginx): enable HTTP/2 and adjust worker connections"

# Sync changes to remote repository host
git push origin main

2. Architecture and Mental Model

To effectively operate and troubleshoot Git, system administrators must understand how Git manages data internally. Git does not record changes as file-based deltas or diffs. Instead, Git treats data as a stream of snapshots of a small filesystem. Every time you commit state, Git records a snapshot of all files at that moment and stores a reference to that snapshot. To remain efficient, if files have not changed, Git does not store the file again; it stores a reference link to the previously stored identical file.

Internal Storage Abstractions (Git Objects)

All history lives within the .git/objects directory, stored as content-addressable storage hashed using SHA-1 (or SHA-256 in newer versions). There are four primary low-level object types in Git:

  1. Blob (Binary Large Object): Stores file data only. A blob contains raw file contents without file metadata, file names, directory structures, or permissions.
  2. Tree: Represents a directory. A tree object contains a list of entries mapping file modes, object types (blob or sub-tree), file names, and their corresponding SHA hashes.
  3. Commit: Points to a top-level Tree object representing the state of the repository at a specific point in time. It includes metadata: author, committer, timestamp, commit message, and zero or more parent commit SHAs.
  4. Annotated Tag: Points to a specific commit SHA, storing tagger metadata, message, and optional GPG cryptographic signatures.
+--------------------------------------------------------------------------+
|                       Git Commit Architecture Graph                      |
|                                                                          |
|   +------------------------------------------------------------------+   |
|   | Commit Object (SHA: a1b2c3d...)                                  |   |
|   | Author: SysAdmin <admin@example.com>                             |   |
|   | Message: "Deploy firewall rules"                                 |   |
|   | Parent: e5f6g7h...                                               |   |
|   | Tree: 9x8y7z6... --------------------------------------------+   |
|   +----------------------------------------------------------|-------+   |
|                                                              v           |
|                                                 +--------------------+   |
|                                                 | Tree Object        |   |
|                                                 | (Root Directory)   |   |
|                                                 +--------------------+   |
|                                                   |                |     |
|                                100644 blob 4f3a.. |                | 040000 tree 8b2c..
|                                (e.g. main.tf)     v                v     |
|                              +-----------------------+   +---------------+
|                              | Blob Object           |   | Sub-Tree      |
|                              | (Raw file contents)   |   | (scripts/)    |
|                              +-----------------------+   +---------------+
+--------------------------------------------------------------------------+

Data Flow Across the Three States

Git manages files across three primary areas on your local machine:

  • Working Tree: The physical directory on disk where files are checked out and edited.
  • Staging Area (Index): A single binary file (.git/index) that prepares and structures the contents of the next commit snapshot.
  • Local Repository (.git/ directory): The object database containing all historical commits, trees, blobs, and reference pointers.
graph LR
    subgraph Local Workstation
        WT[Working Tree<br/>Files on Disk] -->|git add| SA[Staging Area<br/>Index File]
        SA -->|git commit| LR[Local Repository<br/>.git/objects & refs]
        LR -->|git checkout / switch| WT
    end
    subgraph Remote Host
        RM[Remote Repository<br/>origin / upstream]
    end
    LR -->|git push| RM
    RM -->|git fetch| LR
    RM -->|git pull| WT

3. Core Concepts and Terminology

Understanding Git’s core concepts prevents operational mistakes on shared production repositories.

  • Repository: The project directory containing application code alongside the .git hidden folder storing all historical metadata.
  • Working Tree: The local filesystem view of the repository files available for active modification.
  • Staging Area (Index): The intermediate snapshot file (.git/index) holding changes queued for inclusion in the next commit.
  • Commit: An immutable snapshot of the project state referenced by a unique 40-character SHA hash.
  • Branch: A lightweight, movable text pointer referencing a specific commit SHA (e.g., main, feature/x). Switching branches updates the working tree to match the commit referenced by the branch tip.
  • HEAD: A reference pointer (.git/HEAD) tracking the currently checked-out branch or commit.
  • Remote: A named network URL pointing to a shared repository location (e.g., origin pointing to GitHub/GitLab) used for synchronization.
  • Fast-Forward Merge: An operation where the destination branch tip is directly upstream of the incoming branch, allowing Git to simply advance the branch pointer without creating a merge commit.
  • Three-Way Merge: A merge operation used when histories have diverged, combining changes from two branch tips and their common ancestor commit, creating a distinct merge commit with two parent SHAs.
  • Rebase: The process of replaying a sequence of local commits on top of another base commit, producing a linear commit history with new commit SHAs.
  • Detached HEAD: A state where HEAD points directly to an explicit commit SHA or tag rather than a local named branch pointer.

4. Installation and Version Management

Installing Git on Major Linux Distributions

Git is available via standard package repositories across all major Linux distributions.

Debian / Ubuntu / Linux Mint

sudo apt-get update
sudo apt-get install -y git

RHEL / CentOS / Rocky Linux / AlmaLinux / Fedora

sudo dnf install -y git

Arch Linux

sudo pacman -S --noconfirm git

Verifying Installation and Version Requirements

System administrators should verify the installed version to ensure modern CLI commands (such as git switch and git restore, introduced in Git version 2.23+) are supported.

git --version

System Directory Locations

  • Binary Executable: /usr/bin/git
  • Core Executables: /usr/libexec/git-core/ or /usr/lib/git-core/
  • System-Wide Configuration: /etc/gitconfig
  • Global User Configuration: ~/.gitconfig or ~/.config/git/config
  • Repository-Specific Configuration: .git/config

5. Configuration Files and Directory Structure

Internal Layout of the .git Directory

All historical state, references, and local metadata are stored inside the .git directory at the root of the repository. Deleting or corrupting this directory destroys local history.

File / Directory Purpose Important Details
.git/config Repository-level configuration file. Contains remote URLs, branch tracking rules, and local overrides.
.git/HEAD Pointer to the active branch or commit. Text file containing ref: refs/heads/main or a raw commit SHA.
.git/index Staging area (Index) binary cache file. Tracks file path mappings, modification times, and blob SHAs.
.git/objects/ Content-addressable object store. Subdirectories 00 to ff contain compressed blobs, trees, and commits. Also holds .pack and .idx packfiles.
.git/refs/ Storage for reference pointers. Subdirectories include refs/heads/ (local branches), refs/remotes/ (remote branches), and refs/tags/ (tags).
.git/hooks/ Client-side and server-side automation scripts. Contains executable sample scripts (e.g., pre-commit, post-receive) triggered on events.
.git/logs/ Reference transaction history (Reflog). Stores chronological audit logs of HEAD and reference movements (.git/logs/HEAD).
.git/info/exclude Repository-level pattern exclusion file. Functions identically to .gitignore but is not committed or shared with remotes.
.git/COMMIT_EDITMSG Temporary commit message cache. Holds the commit message draft of the most recent commit operation.

Configuration Precedence and Scopes

Git resolves configuration settings across four cascading scopes, where narrower scopes override wider ones:

  1. System Scope (/etc/gitconfig): Applies to all users and repositories on the OS instance. Configure using git config --system.
  2. Global Scope (~/.gitconfig): Applies to all repositories owned by the current user account. Configure using git config --global.
  3. Local Scope (.git/config): Applies strictly to the active repository. Configure using git config --local.
  4. Worktree Scope (.git/config.worktree): Applies to a specific linked working tree in multi-worktree setups.

To inspect effective settings and verify file origins, run:

git config --list --show-origin

Essential Initial Configuration Examples

# Set global author identity
git config --global user.name "SysAdmin Operator"
git config --global user.email "sysadmin@example.com"

# Set default primary branch name for new repositories
git config --global init.defaultBranch main

# Set default pull behavior to rebase to maintain linear history
git config --global pull.rebase true

# Configure SSH key-based signing (Git 2.34+)
git config --global commit.gpgSign true
git config --global gpg.format ssh
git config --global user.signingkey "~/.ssh/id_ed25519.pub"

6. Service Management and Common Operations

Git operates primarily as a command-line client interacting directly with local repositories. However, remote synchronization relies on underlying network protocols and system daemons:

  • SSH Protocol (sshd.service): Secure Shell handles authentication via SSH key pairs. SSH transport operates over TCP port 22.
  • HTTPS Protocol (nginx / apache2 / Git Forge Services): Web authentication using Personal Access Tokens (PATs) or OAuth over TCP port 443.
  • Git Daemon Protocol (git-daemon.service): Unauthenticated, plain-text read-only service operating over TCP port 9418 (legacy/internal mirrors).

Service Verification and Health Check Commands

# Verify SSH transport connectivity to host
ssh -T git@github.com

# Verify local repository integrity
git fsck --full

# Check current working directory status
git status

# Display configured remotes and their network protocols
git remote -v

7. Common Administrative Tasks and Workflows

Workflow 1: Initializing and Provisioning a Local Repository

# Create project directory and navigate into it
mkdir -p /opt/infrastructure/terraform-k8s
cd /opt/infrastructure/terraform-k8s

# Initialize new Git repository
git init

# Create initial baseline configuration
echo "# Terraform Infrastructure Repository" > README.md
echo "*.tfstate" > .gitignore
echo "*.tfvars" >> .gitignore

# Stage and commit baseline files
git add README.md .gitignore
git commit -m "chore: initialize terraform repository"

# Link to remote central host and push main branch
git remote add origin git@github.com:org/terraform-k8s.git
git branch -M main
git push -u origin main

Workflow 2: Feature Branching and Interactive Patch Staging

# Ensure local main branch is up to date
git checkout main
git pull --rebase

# Create and switch to isolated feature branch
git switch -c feature/add-ingress-controller

# Make updates across multiple files...
# Selectively stage changes line-by-line or hunk-by-hunk
git add -p main.tf

# Verify staged changes against index
git diff --staged

# Commit changes
git commit -m "feat(ingress): configure Nginx ingress controller module"

# Push feature branch to remote and track upstream
git push -u origin feature/add-ingress-controller

Workflow 3: Synchronizing Remote Changes via Rebase

# Fetch latest remote references without modifying working tree
git fetch origin

# Rebase local feature branch onto updated remote main branch
git rebase origin/main

# Push updated feature branch (using safe force-with-lease)
git push --force-with-lease origin feature/add-ingress-controller

Workflow 4: Release Tagging and Deployment Baseline

# Create an annotated release tag with a message
git tag -a v1.4.0 -m "Production release v1.4.0: Updated Kubernetes clusters"

# Push specific tag to remote origin
git push origin v1.4.0

# Deploy pipelines check out the explicit release tag SHA
git checkout tags/v1.4.0

8. CLI Command Reference and Cheatsheet

1. Setup and Clone

Command Purpose Important Flags / Arguments Example Explanation
git config Get and set repository or global options. --global, --local, --list, --show-origin git config --global user.name "Admin" Configures global commit author identity.
git init Create an empty Git repository or reinitialize an existing one. --initial-branch=<name> git init --initial-branch=main Initializes .git directory structure.
git clone Clone a repository into a new directory. --depth <n>, --branch <name>, --recurse-submodules git clone --depth 1 git@github.com:org/repo.git Performs shallow clone fetching only the latest commit to save bandwidth/space.

2. Daily Workflow

Command Purpose Important Flags / Arguments Example Explanation
git status Show working tree status. -s (short format), -b (show branch) git status -sb Summarizes changed, staged, and untracked files.
git add Add file contents to the staging area (index). -A (all), -u (tracked only), -p (patch) git add -p script.sh Interactive patch staging allows selecting specific hunks.
git commit Record changes to the repository. -m (message), --amend (modify last), -v (verbose) git commit -m "fix: resolve memory leak" Creates an immutable commit snapshot from staged index.
git diff Show changes between commits, commit and working tree, etc. --staged / --cached git diff --staged Displays diff between staging area (index) and HEAD.
git pull Fetch from and integrate with another repository or a local branch. --rebase, --ff-only git pull --rebase origin main Fetches remote changes and replays local commits on top.
git push Update remote refs along with associated objects. -u (set upstream), --force-with-lease git push -u origin feature/x Uploads local branch commits to remote repository.

3. Branching and Merging

Command Purpose Important Flags / Arguments Example Explanation
git branch List, create, or delete branches. -a (all), -d (delete merged), -D (force delete), -vv git branch -vv Lists local branches with upstream tracking details.
git switch Switch branches (Git 2.23+). -c (create and switch) git switch -c fix/bug-1 Safely switches working tree to target branch.
git checkout Switch branches or restore working tree files. -b (create and switch) git checkout -b feature/y Legacy command for branch switching and file restoration.
git merge Join two or more development histories together. --no-ff, --abort git merge feature/y Integrates named branch history into current branch.
git rebase Reapply commits on top of another base tip. -i (interactive), --continue, --abort git rebase origin/main Re-chains commits to create a linear timeline.

4. Remote Management

Command Purpose Important Flags / Arguments Example Explanation
git remote Manage set of tracked repositories. -v (verbose), add, remove, set-url git remote -v Lists configured remote names and fetch/push URLs.
git fetch Download objects and refs from another repository. --prune (remove deleted remote refs), --all git fetch origin --prune Updates local remote-tracking branches (origin/main) without modifying working tree.

5. Undo and Maintenance

Command Purpose Important Flags / Arguments Example Explanation
git restore Restore working tree files (Git 2.23+). --staged (unstage) git restore --staged app.py Discards changes in index or working tree.
git reset Reset current HEAD to specified state. --soft, --mixed, --hard git reset --soft HEAD~1 Undoes commits; --soft keeps changes staged, --hard discards changes.
git revert Revert existing commits by creating new commits. --no-edit git revert a1b2c3d Safely undoes changes on shared public branches without rewriting history.
git stash Stash changes in a dirty working directory. pop, apply, list, drop git stash pop Shelves working tree modifications temporarily.
git reflog Manage reference log information. show git reflog Displays audit log of local HEAD movements for commit recovery.

WARNING: git reset --hard and git push --force destroy uncommitted changes and rewrite public history. Always prefer git push --force-with-lease on private feature branches, and use git revert for shared branches.


9. Logs, Monitoring and Diagnostics

Historical Inspection Commands

# View compact one-line log history for the last 10 commits
git log --oneline -10

# Display graphic representation of all branches, commits, and tags
git log --graph --oneline --all --decorate

# Display commit diff details for a specific commit hash
git show a1b2c3d

# Trace line-by-line file authorship to audit code modifications
git blame /etc/haproxy/haproxy.cfg

System Health and Diagnostic Workflows

When inspecting issues on a workstation or CI server, follow a structured workflow:

# Step 1: Verify current state and active branch
git status
git branch -vv

# Step 2: Check remote connectivity and tracking settings
git remote -v
ssh -T git@github.com

# Step 3: Inspect reflog to view historical movements of HEAD
git reflog -n 15

# Step 4: Audit repository database integrity
git fsck --full

10. Troubleshooting Guide and Common Failure Scenarios

Scenario 1: Merge Conflicts During Merge or Rebase

Symptoms

git merge or git rebase halts with error outputs: CONFLICT (content): Merge conflict in main.tf Automatic merge failed; fix conflicts and then commit the result. Files contain conflict markers (<<<<<<<, =======, >>>>>>>).

<<<<<<< HEAD
max_connections = 1024
=======
max_connections = 2048
>>>>>>> feature/tune-db

Possible Causes

Concurrent edits modified the exact same lines of code across divergent branches.

Investigation and Diagnosis

  1. Run git status to identify unmerged file paths marked as both modified.
  2. Run git diff to view conflict blocks across the working tree.

Root Cause Identification

Conflicting modifications exist on the same line boundaries between HEAD and the incoming commit.

Resolution

  1. Open conflicting files and resolve markers manually, or use a diff tool (git mergetool).
  2. Stage resolved files:
git add main.tf
  1. Complete the operation:
  • For merge conflicts: git commit -m "fix: resolve merge conflicts"
  • For rebase conflicts: git rebase --continue
  1. Alternatively, to abort the operation completely:
git merge --abort
# OR
git rebase --abort

Verification

Run git status to verify the working tree is clean and unmerged paths are cleared.

Prevention

Synchronize feature branches regularly with git pull --rebase origin main to resolve smaller conflicts early.


Scenario 2: Push Rejected (Non-Fast-Forward)

Symptoms

git push fails with rejection output: error: failed to push some refs to 'git@github.com:org/repo.git' hint: Updates were rejected because the remote contains work that you do not have locally.

Possible Causes

Another engineer pushed commits to the remote branch, or local branch history was rewritten via rebase.

Investigation and Diagnosis

  1. Fetch remote changes without merging:
git fetch origin
  1. Compare local and remote branch tips:
git log --oneline HEAD..origin/main

Root Cause Identification

The remote reference point (origin/main) contains commit SHAs missing from the local working tree.

Resolution

  1. Rebase local commits on top of incoming remote changes:
git pull --rebase origin main
  1. Fix any merge conflicts if prompted, then push:
git push origin main
  1. If force-pushing rewritten local history on a private feature branch, use safe lease validation:
git push --force-with-lease origin feature/my-branch

CRITICAL: Never execute force push (git push --force) on shared public branches like main.

Verification

Run git status to verify branch is up-to-date with origin/main.

Prevention

Always run git fetch or git pull --rebase before initiating pushes to shared remote branches.


Scenario 3: Authentication Failed (HTTPS or SSH Protocol)

Symptoms

Push or fetch fails with terminal output: Permission denied (publickey). fatal: Could not read from remote repository. OR fatal: Authentication failed for '[https://github.com/](https://github.com/)...'

Possible Causes

Expired account password / Personal Access Token (PAT), missing SSH private key, SSH key not loaded into ssh-agent, changed host keys, or mismatched HTTPS/SSH remote URL schemes.

Investigation and Diagnosis

  1. Verify configured remote protocol URL format:
git remote -v
  1. Test SSH connectivity explicitly:
ssh -Tv git@github.com
  1. Check loaded SSH keys in the current session agent:
ssh-add -l

Root Cause Identification

Missing identity keys in ssh-agent or invalid personal token credentials for HTTPS authentication.

Resolution

  • For SSH Authentication: Start the agent and add your SSH key:
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

If the remote URL is set to HTTPS, switch it to SSH:

git remote set-url origin git@github.com:org/repo.git
  • For HTTPS Authentication: Replace account passwords with a Personal Access Token (PAT) containing required repo scopes.

Verification

Run ssh -T git@github.com or git fetch origin to ensure clean connection without errors.

Prevention

Use SSH keys with passphrase protection, configure system credential helpers, and enforce short-lived PAT tokens.


Scenario 4: Detached HEAD State

Symptoms

Terminal displays: You are in 'detached HEAD' state. Commands warn that commits made will not be attached to any branch pointer.

Possible Causes

Checked out a specific commit hash, remote tracking reference, or tag directly instead of a local branch name.

Investigation and Diagnosis

  1. Inspect HEAD pointer status:
git status
git branch
  1. Observe output showing * (HEAD detached at a1b2c3d).

Root Cause Identification

.git/HEAD contains a direct commit hash (a1b2c3d...) instead of a symbolic reference (refs/heads/main).

Resolution

  1. To keep new work done in detached state, create a new branch containing the commits:
git switch -c fix-detached-work
# OR
git checkout -b fix-detached-work
  1. To discard detached changes and return to main:
git switch main
  1. If detached commits were lost after switching away, recover them via reflog:
git reflog
git checkout -b recovered-branch HEAD@{1}

Verification

Run git branch to confirm an asterisk (*) points to a valid named branch.

Prevention

Avoid active development on bare tags or SHAs; always create a named branch for local changes.


Scenario 5: Stale Lock Files (fatal: index.lock / another git process)

Symptoms

Commands fail instantly with: fatal: Unable to create '/path/to/.git/index.lock': File exists.

Possible Causes

An interrupted or crashed Git process left behind a lock file, concurrent execution of IDE background fetches, or NFS lock latency.

Investigation and Diagnosis

  1. Check for running Git processes:
ps aux | grep git
  1. Check file lock creation time:
ls -la .git/index.lock

Root Cause Identification

A residual .git/index.lock file exists while no active Git process is running.

Resolution

  1. Confirm no active Git commands or background tasks are executing.
  2. Remove stale lock file manually:
rm -f .git/index.lock

Verification

Run git status to confirm normal repository operation.

Prevention

Avoid abruptly killing Git processes (SIGKILL/Ctrl+C during write operations) and resolve underlying NFS storage delays.


Scenario 6: Large File Commit / Push Rejection Over Host Limit

Symptoms

git push fails with rejection message from remote host: GH001: Large files detected. File binaries/db_dump.sql is 250 MB; this exceeds GitHub's file size limit of 100 MB.

Possible Causes

Large binary blobs, disk dumps, or logs were staged and committed directly into the repository history.

Investigation and Diagnosis

Find large objects stored in recent commits:

git rev-list --objects --all | grep $(git verify-pack -v .git/objects/pack/*.idx | sort -k3 -n | tail -5 | awk '{print $1}')

Root Cause Identification

A commit in local history references a blob object larger than host platform policy constraints.

Resolution

  • If commit is local and unpushed: Reset commit while preserving modifications, remove binary, update .gitignore, and re-commit:
git reset --soft HEAD~1
rm binaries/db_dump.sql
echo "binaries/" >> .gitignore
git add .gitignore
git commit -m "fix: remove large binary dump"
  • If already committed across local history: Purge binary from commit history using git filter-repo or BFG Repo Cleaner:
git filter-repo --invert-paths --path binaries/db_dump.sql

Alternatively, track large files via Git LFS before staging:

git lfs install
git lfs track "*.sql"

Verification

Push branch changes to origin and confirm the object size validation checks pass.

Prevention

Maintain complete .gitignore files and use pre-commit hooks to block staging files exceeding size limits.


Scenario 7: Corrupted or Missing Object Files

Symptoms

Git operations halt with fatal output: error: object file .git/objects/4f/3a... is empty error: fatal: loose object 4f3a... (stored in .git/objects/4f/3a...) is corrupt

Possible Causes

Disk space exhausted during git write operations, ungraceful server shutdowns, or storage media filesystem corruption.

Investigation and Diagnosis

Run repository diagnostic check:

git fsck --full

Root Cause Identification

Empty (zero-byte) loose object files exist inside .git/objects/.

Resolution

  1. Backup uncommitted working tree edits:
git diff > ~/uncommitted_changes.patch
  1. Move broken repository directory to backup:
cd ..
mv repo repo_broken
  1. Fresh clone repository from remote origin:
git clone git@github.com:org/repo.git
cd repo
  1. Apply saved patch to restore unpushed work:
git apply ~/uncommitted_changes.patch

Verification

Run git fsck --full to verify complete object graph integrity.

Prevention

Ensure पर्याप्त storage space on disk and maintain remote backups.


Scenario 8: Accidental Commit Performed Directly on Main Branch

Symptoms

git status reports local main branch is ahead of origin/main by N commits, but changes belong on a separate feature branch.

Possible Causes

Executing git commit directly on main without creating a feature branch first.

Investigation and Diagnosis

Inspect commit log:

git log --oneline -3

Root Cause Identification

Unpushed local commits are attached to refs/heads/main.

Resolution

  • If commits are local and UNPUSHED:
  1. Create new feature branch pointer at current HEAD:
git branch feature/my-work
  1. Hard reset local main back to match upstream remote origin:
git reset --hard origin/main
  1. Switch to new feature branch:
git switch feature/my-work
  • If commits were ALREADY PUSHED to shared main: Revert commits on main using new inverse commits to avoid breaking shared history, then cherry-pick to feature branch:
git checkout main
git revert <commit-sha>
git checkout -b feature/my-work
git cherry-pick <commit-sha>

Verification

Run git log on main to confirm it matches origin/main.

Prevention

Configure remote branch protection rules blocking direct pushes to main.


11. Security and Best Practices

  • Credential Management: Never commit private SSH keys, cloud provider API tokens, database passwords, or TLS certificates into Git repositories. Add sensitive patterns to .gitignore and use secret scanning tools (gitleaks) in CI pipelines.
  • File System Permissions: Enforce standard Linux permissions on .git/: directory mode 0700 and configuration mode 0600.
  • Transport Protocol Security: Always prefer SSH (git@host:org/repo.git) or HTTPS with short-lived Personal Access Tokens (PATs) over unauthenticated git daemon protocols.
  • Commit Authenticity Verification: Sign commits using GPG or SSH key pairs (git commit -S) to guarantee author identity.
  • Branch Protection Rules: Restrict push privileges on production branches (main, release/*) on remote hosts, requiring Pull Request reviews and status checks.

12. Performance and Resource Optimisation

  • Garbage Collection and Compression: Run git gc --prune=now periodically to compress loose objects into packed files (.pack) and clear unreachable dangling objects.
  • Shallow Clones in CI/CD: Use git clone --depth 1 in automated deployment pipelines to skip downloading historical commit objects, reducing network transfer times and disk footprint.
  • Large File Handling: Use Git LFS (git lfs) for repositories containing binary media or large database fixtures to prevent object store bloat.
  • Sparse Checkouts: In monolithic repositories (monorepos), use git sparse-checkout to populate only required subdirectories into the working tree.

13. Backup, Recovery and Maintenance

Backup Strategies

  • Local Mirror Clones: Maintain automated server-side bare backups using mirror clones:
git clone --mirror git@github.com:org/repo.git /backup/git/repo.git
  • Object Store Recovery with Reflog: If a branch is accidentally deleted or reset hard, use git reflog to recover the lost commit SHA:
git reflog
# Identify target commit SHA (e.g., e5f6g7h)
git checkout -b recovered-branch e5f6g7h

Maintenance Checklist

  • Execute git fsck --full quarterly to audit object graph integrity.
  • Prune stale remote tracking branches: git fetch origin --prune.

14. Automation and Integration

Integrating Git with CI/CD and Configuration Management

System administrators and SREs integrate Git with deployment runners (Ansible, Jenkins, GitLab CI, GitHub Actions):

# Example CI/CD Runner Step: Deploy Infrastructure via Ansible
deploy_job:
  stage: deploy
  script:
    - git checkout tags/$CI_COMMIT_TAG
    - ansible-playbook -i production site.yml

Git Pre-Commit Hook for Linting and Security Scans

Create an executable hook script at .git/hooks/pre-commit:

#!/bin/bash
# Pre-commit hook to prevent committing unencrypted secrets

if grep -qE "(AWS_SECRET_ACCESS_KEY|PRIVATE KEY)" $(git diff --cached --name-only); then
    echo "ERROR: Unencrypted secrets detected in staged files. Commit aborted."
    exit 1
fi

15. Practical Labs and Exercises

Lab 1: Staging Selective Patch Hunks

  • Objective: Stage specific file modifications while leaving other lines in the working tree.
  • Prerequisites: Git installed; clean working repository.
  • Instructions:
  1. Create a file config.txt containing 5 lines of text and commit it.
  2. Edit lines 1 and 5 in config.txt.
  3. Execute git add -p config.txt.
  4. Select y to stage the first hunk and n to reject the second hunk.
  • Validation: Run git diff (shows unstaged hunk) and git diff --staged (shows staged hunk).

Lab 2: Recovering a Lost Commit After Hard Reset

  • Objective: Use git reflog to recover a commit deleted via git reset --hard.
  • Prerequisites: Existing local Git repository.
  • Instructions:
  1. Create and commit a new file: echo "Critical Data" > data.txt && git add data.txt && git commit -m "feat: add critical data".
  2. Note the commit SHA hash.
  3. Execute hard reset to discard commit: git reset --hard HEAD~1.
  4. Verify data.txt is missing from the working tree.
  5. Run git reflog to locate the commit hash prior to the reset.
  6. Restore commit to a new branch: git checkout -b recovery-branch <SHA>.
  • Validation: Inspect working tree on recovery-branch to verify data.txt is restored.

16. Quick Reference and Cheat Sheet

Important Configuration Files

File / Path Purpose Scope
/etc/gitconfig System-wide configuration All users / repos on system
~/.gitconfig User-global configuration. User repositories.
.git/config Local repository configuration. Target repository only.

Important Log and Ref Locations

File Location Purpose
.git/HEAD Current active branch/commit pointer.
.git/logs/HEAD Reflog transaction audit log.
.git/index Staging area cache file.

Common Administrative Commands

Task Command
Check status git status -sb
Selective staging git add -p <file>
Rebase pull git pull --rebase origin main
Linear log view git log --graph --oneline --all
Reflog audit git reflog

Troubleshooting Quick Reference

Symptom Initial Diagnostic Command Primary Fix Command
Merge Conflict git status Resolve markers, git add, git commit
Push Rejected git log HEAD..origin/main git pull --rebase origin main
Auth Failed ssh -Tv git@github.com ssh-add ~/.ssh/id_ed25519
Detached HEAD git status git switch -c <branch-name>
Stale Lock ls -la .git/index.lock rm -f .git/index.lock

17. Further Reading and References

  1. Pro Git Book (Official Reference): https://git-scm.com/book
  2. Official Git CLI Manual and Documentation: https://git-scm.com/docs
  3. Learn Git Branching (Interactive Visualization Tool): https://learngitbranching.js.org
  4. Atlassian Git Workflows & Tutorials: https://www.atlassian.com/git/tutorials