Domain 6CI/CD & GitOpsINTERMEDIATE

Enterprise Container Registry Administration: Architectural Deep Dive, Operational Runbook, and Troubleshooting Guide for Harbor

A comprehensive operational handbook and technical guide to deploying, managing, securing, and troubleshooting CNCF Harbor in enterprise Linux and Kubernetes production environments.

Domain: Domain 6: CI/CD, GitOps & Registry•Updated: 2026-11-03
#linux#sysadmin#harbor#container-registry#docker#kubernetes#security

1. Overview

CNCF Harbor is an open-source, enterprise-class cloud-native container image and OCI artifact registry. Modern software engineering relies heavily on container images (Docker/OCI) and Helm deployment packages. While public registries like Docker Hub provide access to upstream software, enterprise production environments require strict governance, auditability, fine-grained access controls, automated vulnerability scanning, and multi-region data replication. Harbor addresses these demands by serving as a private, self-hosted registry control plane that sits between developers, CI/CD pipelines, and production Kubernetes clusters.

What Harbor Does in Production

In production environments, Kubernetes clusters, continuous integration (CI) engines (such as GitLab CI, GitHub Actions, or Jenkins), and developer workstations pull container images directly from Harbor instead of public internet registries. This internal routing delivers three core capabilities:

  1. Supply Chain Security & Governance: Enforces access policies so only authorized identities can push images, while automatically scanning artifacts for Common Vulnerabilities and Exposures (CVEs) to block high-risk deployments.
  2. Bandwidth Optimization & Air-Gapped Operation: Acts as a proxy cache for upstream registries (e.g., Docker Hub, Quay, ECR), reducing egress costs, mitigating external rate limits, and supplying software to isolated, air-gapped environments.
  3. High Availability & Disaster Recovery: Replicates container artifacts across multiple cloud providers, regional data centers, or edge locations to guarantee operational continuity.

Harbor vs. Plain Docker Registry

A standard Docker Distribution registry (registry:2) is a minimalist storage engine: it accepts OCI blobs and manifests over HTTP, storing them on local disk or object storage. It lacks native authentication management, a user interface, access control lists, vulnerability scanners, or artifact retention policies. Harbor wraps standard OCI registry capabilities inside an enterprise management framework.

Feature Plain Docker Registry (registry:2) CNCF Harbor
Core Protocol OCI / Docker Registry HTTP API v2 OCI / Docker Registry HTTP API v2
User Interface None (CLI / Third-party extensions only) Native Web UI (Angular/Clarity Portal)
Multi-Tenancy Single flat storage namespace Projects (Namespaces) with isolated RBAC
Authentication & RBAC Basic htpasswd / static token server Local DB, OIDC (Keycloak/Okta), LDAP/AD, Robot Accounts
Vulnerability Scanning None Integrated Trivy scanner with CVE policy enforcement
Artifact Signatures None Cosign / Notary integration for content trust
Replication & HA External sync tools required Built-in async multi-registry replication engine
Storage Management Manual CLI garbage collection Scheduled UI/API Garbage Collection & Tag Retention
OCI Helm Support Basic OCI storage Native OCI Helm chart management and inspection

Production Practical Example

Consider a microservices backend deployment where a developer pushes a new API service image:

# 1. Authenticate against private Harbor registry
docker login harbor.example.com -u developer1 -p SecretPass123

# 2. Tag local build to match Harbor project and repository path
docker tag backend-api:v2.1.0 harbor.example.com/finance-team/backend-api:v2.1.0

# 3. Push artifact to Harbor
docker push harbor.example.com/finance-team/backend-api:v2.1.0

Upon push, Harbor automatically stores the image layers under the finance-team project, triggers an automated Trivy security scan, evaluates whether any detected CVEs violate security policies, and asynchronously replicates the validated image to a secondary disaster recovery registry in another region.


2. Architecture and Mental Model

Understanding Harbor’s internal architecture is vital for diagnosing operational issues, identifying performance bottlenecks, and maintaining high availability. Harbor utilizes a microservices architecture where each component handles a dedicated domain responsibility.

flowchart TD
    Client[Client / Docker CLI / Kubelet / Helm] -->|Port 443 HTTPS| Proxy[NGINX Reverse Proxy / Ingress]
    
    Proxy -->|REST API / Web UI| Portal[Harbor Portal UI]
    Proxy -->|API Routing / Auth Authz| Core[Harbor Core API]
    Proxy -->|OCI Distribution API v2| Registry[Harbor Registry Engine]
    
    Core -->|Read/Write Metadata| Postgres[(PostgreSQL DB)]
    Core -->|Cache / Session State| Redis[(Redis Key-Value)]
    Core -->|Enqueue Tasks| JobService[Harbor Jobservice]
    
    Registry -->|Auth Verification| Core
    Registry -->|Read/Write Blobs| Storage[Blob Storage: S3 / Azure / Local Disk]
    
    JobService -->|Scan Tasks| Scanner[Trivy Scanner Adapter]
    JobService -->|Replication Jobs| ExtReg[External Registries: ECR / Docker Hub / Remote Harbor]
    JobService -->|Read/Write Job State| Postgres

Core Architecture Components

  1. Harbor Portal: A web-based management user interface built with Angular and Clarity. It communicates exclusively with the Harbor Core REST API to render project dashboards, scan reports, user assignments, and system logs.
  2. Harbor Core: The central control plane microservice. It exposes the RESTful API (/api/v2.0), manages user authentication (Local, LDAP, OIDC), evaluates role-based access control (RBAC), generates bearer tokens for container runtime requests, enforces system configurations, and orchestrates actions across sub-services.
  3. Harbor Registry (Distribution): Based on the open-source Docker Distribution engine. It implements the standard Docker Registry HTTP API v2 protocol for receiving layer blobs, handling manifest pushes/pulls, and managing content storage. It offloads token validation back to Harbor Core.
  4. Harbor Jobservice: An asynchronous job queue engine. It handles long-running background worker tasks, including image replication to remote registries, execution of vulnerability scans, execution of storage garbage collection, and outbound webhook dispatching.
  5. PostgreSQL Database: The persistent metadata repository. It stores non-blob information: user accounts, project configurations, RBAC permissions, audit logs, artifact manifests, tag metadata, vulnerability reports, and replication rules.
  6. Redis Cache: High-performance, key-value storage used for temporary session storage, API caching, and managing task queues for the Jobservice engine.
  7. Trivy Scanner Adapter: The default security scanning engine integrated into Harbor. It analyzes container filesystem layers and language-specific dependency manifests (Go, Python, Java, Node.js) against CVE databases.
  8. Storage Backend: The storage layer where physical OCI layer blobs and manifests reside. Harbor supports local POSIX file systems, Network File System (NFS), or cloud object storage services (AWS S3, Google Cloud Storage, Azure Blob Storage).

End-to-End Image Push Lifecycle

  1. Authentication: The client executes docker login harbor.example.com. Harbor Core verifies credentials against PostgreSQL (or OIDC/LDAP) and issues an authenticated JWT token.
  2. Layer Upload Initialization: The client initiates a push: docker push [harbor.example.com/project/app:tag](https://harbor.example.com/project/app:tag). The client sends an HTTP POST request to /v2/project/app/blobs/uploads/.
  3. Authorization Check: Harbor Registry intercepts the request, calls Harbor Core to validate bearer token permissions for project/app (requiring Developer role or higher).
  4. Blob Streaming: Upon successful authorization, the client streams raw layer files to Harbor Registry, which writes them directly to configured blob storage.
  5. Manifest Commit: After layer blobs are uploaded, the client posts the image manifest. Harbor Core records the manifest metadata, digest hash, and tag mapping in PostgreSQL.
  6. Async Event Trigger: Harbor Core emits a POST_SCAN and REPLICATION event. Jobservice receives these events and enqueues scan jobs for Trivy and replication tasks for any configured replication rules.

3. Core Concepts and Terminology

Mastering Harbor requires a clear understanding of its organizational hierarchy and administrative abstractions.

+-----------------------------------------------------------------------------------+
| HARBOR REGISTRY INSTANCE (harbor.example.com)                                     |
|                                                                                   |
|  +-----------------------------------------------------------------------------+  |
|  | PROJECT: "finance-team" (Access Level: Private | Storage Quotas | Policies)   |  |
|  |                                                                             |  |
|  |   +---------------------------------------------------------------------+   |  |
|  |   | REPOSITORY: "finance-team/api-service"                              |   |  |
|  |   |                                                                     |   |  |
|  |   |   +-------------------------------------------------------------+   |  |
|  |   |   | ARTIFACT (Digest: sha256:9f8a7b...)                         |   |  |
|  |   |   |  |- Tags: [ "v1.0.0", "stable", "latest" ]                  |   |  |
|  |   |   |  |- OCI Manifest & Config                                   |   |  |
|  |   |   |  |- Layers: [ LayerA, LayerB, LayerC ]                      |   |  |
|  |   |   |  |- Trivy Scan Report: 0 Critical, 2 Medium                 |   |  |
|  |   |   +-------------------------------------------------------------+   |  |
|  |   +---------------------------------------------------------------------+   |  |
|  +-----------------------------------------------------------------------------+  |
+-----------------------------------------------------------------------------------+

Key Concepts

  • Project: The top-level multi-tenancy logical boundary. Every artifact in Harbor belongs to a project. Projects govern member RBAC roles, storage quotas, public/private visibility, CVE deployment policies, and tag retention rules.
  • Repository: A named collection of related container images or artifacts under a project (e.g., finance-team/api-service).
  • Artifact: A distinct content unit stored within a repository. An artifact consists of a manifest, configuration schema, and one or more layer blobs. Artifacts can be Docker images, OCI container images, or Helm chart packages.
  • Tag: A human-readable pointer referencing a specific immutable artifact digest (e.g., v1.2.3 or latest). Re-assigning a tag changes which digest it references.
  • Digest: A cryptographic SHA-256 hash representing the exact content of an artifact manifest (e.g., sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855). Pulling by digest guarantees exact content immutability regardless of tag reassignment.
  • Robot Account: A system-to-system authentication identity created specifically for automated CI/CD runners, Kubernetes imagePullSecrets, or scripts. Robot accounts use generated credentials (format: robot$project+name) and scoped, short- or long-lived JWT tokens with fine-grained access rights.
  • Replication Rule: An automated job rule set up to synchronize artifacts bidirectionally between the local Harbor registry and remote registries (e.g., another Harbor instance, AWS ECR, Google Artifact Registry, Docker Hub).
  • Proxy Cache: A specialized project mode configured to mirror a public upstream registry (e.g., Docker Hub). When a client pulls an upstream image through a proxy cache project, Harbor transparently downloads, caches, and scans the external image locally.
  • Garbage Collection (GC): A manual or scheduled background process that purges unreferenced layer blobs from underlying physical storage after image tags or manifests are deleted.

Glossary of Operational Terms

Term Definition
Immutable Tag A project policy setting that prevents pushed tags from being overwritten by subsequent builds.
CVE Allowlist A configured list of specific CVE identifiers (e.g., CVE-2023-12345) granted temporary exceptions to bypass blocking rules.
Cosign / Notary Artifact signing mechanisms used to cryptographically verify image provenance and integrity prior to deployment.
OCI Distribution Spec The open standard governing container registry HTTP APIs, blob storage structures, and artifact manifests.

4. Installation and Version Management

Harbor can be deployed using two primary operational patterns: Docker Compose (ideal for single-node setups, testing, and edge deployments) or Helm on Kubernetes (ideal for enterprise high-availability production clusters).

System Prerequisites

  • Hardware Minimums: 2 vCPUs, 4 GB RAM, 40 GB storage space.
  • Production Recommended: 4+ vCPUs, 8+ GB RAM, dedicated SSD/NVMe or Cloud Object Storage.
  • Operating System: Linux Kernel 4.19+ (RHEL 8/9, Ubuntu 20.04/22.04/24.04, Rocky Linux 9).
  • Software Dependencies: Docker Engine v20.10.0+ and Docker Compose v2.0.0+ (for Compose installs); Kubernetes v1.24+ and Helm v3.8+ (for K8s installs).
  • Network Prerequisites: Valid FQDN (e.g., harbor.example.com), accessible ports 80 (HTTP) and 443 (HTTPS), and a valid TLS certificate issued by a trusted CA or internal PKI.

Method A: Single-Node Installation via Docker Compose

  1. Download the Installer Package:
# Download the offline installer package (includes container images)
cd /tmp
wget https://github.com/goharbor/harbor/releases/download/v2.10.0/harbor-offline-installer-v2.10.0.tgz

# Extract to system deployment path
tar -xvf harbor-offline-installer-v2.10.0.tgz -C /usr/local/
cd /usr/local/harbor
  1. Configure Harbor Environment:
# Copy template configuration
cp harbor.yml.tmpl harbor.yml

# Edit core parameters in harbor.yml
vim harbor.yml

Key changes in harbor.yml:

hostname: harbor.example.com
http:
  port: 80
https:
  port: 443
  certificate: /etc/harbor/certs/harbor.crt
  private_key: /etc/harbor/certs/harbor.key
harbor_admin_password: StrongProductionPassword123!
database:
  password: StrongDBPassword123!
  max_idle_conns: 50
  max_open_conns: 100
data_volume: /var/lib/harbor/data
  1. Execute Installation Script:
# Prepare container configurations and systemd service generation
./prepare

# Run installation script enabling Trivy security scanner
./install.sh --with-trivy

Method B: High-Availability Installation via Helm on Kubernetes

For production Kubernetes deployments, use the official Harbor Helm chart:

# 1. Add Harbor Helm repository
helm repo add harbor https://helm.goharbor.io
helm repo update

# 2. Create target namespace
kubectl create namespace harbor

# 3. Install Harbor with custom values
helm install harbor harbor/harbor \
  --namespace harbor \
  --set expose.type=ingress \
  --set expose.ingress.hosts.core=harbor.example.com \
  --set externalURL=https://harbor.example.com \
  --set persistence.persistentVolumeClaim.registry.size=500Gi \
  --set trivy.enabled=true \
  --set database.type=external \
  --set database.external.host=postgres-ha.internal \
  --set redis.type=external \
  --set redis.external.addr=redis-ha.internal:6379

Verifying Installation and Version

# Inspect Docker Compose service health
cd /usr/local/harbor && docker compose ps

# Inspect Kubernetes pod status
kubectl get pods -n harbor -o wide

# Query Harbor API system information endpoint
curl -sk https://harbor.example.com/api/v2.0/systeminfo | jq .

Expected API Output excerpt:

{
  "harbor_version": "v2.10.0",
  "self_registration": false,
  "has_ca_root": true,
  "auth_mode": "db_auth"
}

5. Configuration Files and Directory Structure

Configuration File and Directory Map

Path / Directory Component / Purpose Important Details
/usr/local/harbor/harbor.yml Master Configuration File Defines hostname, TLS certificates, database credentials, storage backend, and ports.
/usr/local/harbor/prepare Config Generator Script Parses harbor.yml and renders Docker Compose configuration templates.
/usr/local/harbor/docker-compose.yml Runtime Container Map Generated docker-compose manifest mapping microservices, volumes, networks, and environment variables.
/var/lib/harbor/data/ Default Storage Volume Contains persistent data directories: registry/ (blobs), database/ (Postgres), redis/.
/var/log/harbor/ System Log Directory Host log path containing microservice logs (core.log, jobservice.log, registry.log).
/etc/docker/certs.d/ Docker Engine Trust Store Host path for trusting Harbor CA certificates (/etc/docker/certs.d/[harbor.example.com/ca.crt](https://harbor.example.com/ca.crt)).

Core harbor.yml Directives Reference

# Master Hostname definition (Required)
hostname: harbor.example.com

# Transport Layer Security Configuration
https:
  port: 443
  certificate: /etc/harbor/certs/harbor.crt
  private_key: /etc/harbor/certs/harbor.key

# Root Admin Credentials (Required for initial setup)
harbor_admin_password: ProductionAdminPassword123!

# Physical Storage Engine Backend Configuration
storage_service:
  filesystem:
    maxthreads: 100
  # Alternative S3 Cloud Storage Backend Configuration:
  # s3:
  #   accesskey: AWS_ACCESS_KEY_ID
  #   secretkey: AWS_SECRET_ACCESS_KEY
  #   region: us-east-1
  #   bucket: my-company-harbor-registry-blobs
  #   encrypt: true

# Logging Configuration and Log Rotation Rules
log:
  level: info
  local:
    rotate_count: 50
    rotate_size: 200M
    location: /var/log/harbor

# Internal or External PostgreSQL Database Configuration
database:
  password: DB_Super_Secret_Password
  max_idle_conns: 50
  max_open_conns: 200

# Integrated Trivy Vulnerability Scanner Settings
trivy:
  vuln_type: os,library
  severity: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
  ignore_unfixed: false

Applying Configuration Changes

Configuration changes modified in harbor.yml do not take effect automatically. You must run the prepare script to regenerate internal component configurations and restart the service stack:

cd /usr/local/harbor

# 1. Regenerate runtime service configurations
./prepare

# 2. Restart Harbor container microservices
docker compose down
docker compose up -d

6. Service Management and Common Operations

Service Lifecycle Management (Docker Compose Deployments)

# Navigate to deployment directory
cd /usr/local/harbor

# Start all Harbor microservices
docker compose up -d

# Stop all Harbor services without removing containers
docker compose stop

# Start stopped services
docker compose start

# Restart all services (graceful bounce)
docker compose restart

# Gracefully stop and remove containers and internal networks
docker compose down

# Check live running container status and health checks
docker compose ps

Service Lifecycle Management (Kubernetes Helm Deployments)

# Inspect all Harbor components in namespace
kubectl get all -n harbor

# Restart Harbor Core API deployment
kubectl rollout restart deployment/harbor-core -n harbor

# Restart Harbor Jobservice deployment
kubectl rollout restart deployment/harbor-jobservice -n harbor

# Scale Jobservice workers horizontally during heavy workload pipelines
kubectl scale deployment/harbor-jobservice -n harbor --replicas=5

Health Checks and Operational Diagnostics

# Check Harbor System Health API endpoint
curl -sk https://harbor.example.com/api/v2.0/health | jq .

# Verify HTTP NGINX Reverse Proxy response headers
curl -Iv https://harbor.example.com/v2/

# Verify listening ports on host
ss -tulpn | grep -E ':80|:443'

Expected /api/v2.0/health JSON response:

{
  "status": "healthy",
  "components": [
    { "name": "database", "status": "healthy" },
    { "name": "redis", "status": "healthy" },
    { "name": "portal", "status": "healthy" },
    { "name": "core", "status": "healthy" },
    { "name": "registry", "status": "healthy" },
    { "name": "jobservice", "status": "healthy" },
    { "name": "trivy", "status": "healthy" }
  ]
}

7. Common Administrative Tasks and Workflows

Task 1: Setting up a Project with RBAC & Member Assignment

Objective: Create a private project named payment-team and assign developer permissions.

  1. Create Project via REST API:
curl -sk -X POST "https://harbor.example.com/api/v2.0/projects" \
  -u "admin:ProductionAdminPassword123!" \
  -H "Content-Type: application/json" \
  -d '{
    "project_name": "payment-team",
    "public": false,
    "metadata": {
      "public": "false",
      "auto_scan": "true",
      "prevent_vul": "true",
      "severity": "high"
    }
  }'
  1. Add User to Project with Developer Role:
# Role IDs: 1=ProjectAdmin, 2=Developer, 3=Guest, 4=Maintainer
curl -sk -X POST "https://harbor.example.com/api/v2.0/projects/payment-team/members" \
  -u "admin:ProductionAdminPassword123!" \
  -H "Content-Type: application/json" \
  -d '{
    "role_id": 2,
    "member_user": {
      "username": "jdoe"
    }
  }'

Task 2: Creating a Robot Account for CI/CD Pipelines

Objective: Provision a scoped identity for Jenkins to push images to the payment-team project.

  1. Create Robot Account via API:
curl -sk -X POST "https://harbor.example.com/api/v2.0/robots" \
  -u "admin:ProductionAdminPassword123!" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "jenkins-builder",
    "description": "Scoped CI push account for Payment Team",
    "duration": 365,
    "level": "project",
    "permissions": [
      {
        "resource": "repository",
        "action": "push",
        "kind": "project",
        "namespace": "payment-team"
      },
      {
        "resource": "repository",
        "action": "pull",
        "kind": "project",
        "namespace": "payment-team"
      }
    ]
  }'
  1. Test CI Pipeline Login:
# Capture the generated secret from the API output (e.g., SecretToken12345)
ROBOT_USER="robot\$payment-team+jenkins-builder"
ROBOT_TOKEN="SecretToken12345"

docker login harbor.example.com -u "$ROBOT_USER" -p "$ROBOT_TOKEN"

Task 3: Configuring Vulnerability Scanning & Deployment Blocking

Objective: Configure Harbor to automatically scan images on push and block pulls if High or Critical CVEs are detected.

  1. Enable Vulnerability Prevention via Web UI:
  • Navigate to Projects $\rightarrow$ payment-team $\rightarrow$ Configuration.
  • Enable Automatically scan images on push.
  • Enable Prevent vulnerable images from running.
  • Set Vulnerability Severity Threshold to High.
  1. Validate Enforcement: If a pushed image contains High or Critical vulnerabilities, container runtime pull requests (docker pull or kubelet) will be rejected with an authorization error: vulnerability policy blocking image.

Task 4: Configuring a Proxy Cache for Docker Hub

Objective: Mitigate Docker Hub rate limits and speed up local image pulls by creating a pull-through cache.

  1. Create Registry Endpoint:
  • Navigate to Administration $\rightarrow$ Registries $\rightarrow$ New Endpoint.
  • Provider: Docker Hub.
  • Name: docker-hub-upstream.
  1. Create Proxy Cache Project:
  • Navigate to Projects $\rightarrow$ New Project.
  • Project Name: dockerhub-proxy.
  • Enable Proxy Cache toggle and select docker-hub-upstream.
  1. Pull Upstream Image Through Harbor:
# Pull official redis image transparently through Harbor cache
docker pull harbor.example.com/dockerhub-proxy/library/redis:alpine

Task 5: Managing Storage Garbage Collection & Tag Retention

Objective: Free physical disk space by scheduling regular garbage collection and defining retention rules.

flowchart LR
    A[Step 1: Tag Deletion / Retention Policy Execution] -->|Marks Tag Unreferenced| B[PostgreSQL Metadata Updated]
    B -->|Blobs remain on disk| C[Step 2: Admin Triggers Storage Garbage Collection]
    C -->|Jobservice Scans Blobs| D[Unreferenced Blobs Purged from Physical Storage / S3]
  1. Set Tag Retention Policy:
  • Project $\rightarrow$ payment-team $\rightarrow$ Tag Retention.
  • Rule: Retain the last 10 tags for repositories matching *, matching branches main-*.
  1. Run Storage Garbage Collection via API:
# Trigger an immediate dry-run or full Garbage Collection job
curl -sk -X POST "https://harbor.example.com/api/v2.0/system/gc/schedule" \
  -u "admin:ProductionAdminPassword123!" \
  -H "Content-Type: application/json" \
  -d '{
    "schedule": {
      "type": "Manual"
    },
    "parameters": {
      "delete_untagged": true
    }
  }'

Task 6: Setting up Registry Replication for Disaster Recovery

Objective: Synchronize artifacts from production Harbor to a secondary remote registry.

  1. Define Destination Registry:
  • Administration $\rightarrow$ Registries $\rightarrow$ Add Endpoint (e.g., target remote Harbor instance).
  1. Create Replication Rule:
  • Administration $\rightarrow$ Replications $\rightarrow$ New Rule.
  • Mode: Push-based.
  • Source resource filter: payment-team/*.
  • Trigger: Event-based (replicate immediately upon image push).

Task 7: Pushing & Pulling Helm Charts as OCI Artifacts

Objective: Use Harbor as a enterprise Helm chart repository.

# 1. Login to Harbor Helm OCI registry
helm registry login harbor.example.com -u "admin" -p "ProductionAdminPassword123!"

# 2. Package a local Helm chart
helm package ./my-microservice-chart

# 3. Push Helm chart package to Harbor project
helm push my-microservice-chart-1.2.0.tgz oci://harbor.example.com/payment-team

# 4. Pull and install Helm chart directly from Harbor
helm install my-release oci://harbor.example.com/payment-team/my-microservice-chart --version 1.2.0

8. CLI Command Reference and Cheatsheet

Container Engine & Helm CLI Commands

Command Purpose Important Flags / Arguments Practical Example Explanation
docker login Authenticate client to Harbor -u (username), -p (password/token) docker login harbor.example.com -u 'robot$proj+ci' -p 'TOKEN' Authenticates user or robot account against Harbor Core.
docker tag Format local image to match Harbor repository path Source image, target FQDN path docker tag app:1.0 [harbor.example.com/finance/app:1.0](https://harbor.example.com/finance/app:1.0) Maps standard image to target registry, project, and repository schema.
docker push Upload image layers and manifest to Harbor Target repository path docker push [harbor.example.com/finance/app:1.0](https://harbor.example.com/finance/app:1.0) Streams layer blobs to Harbor Registry and records manifest in Postgres.
docker pull Download image from Harbor Target repository path / digest docker pull [harbor.example.com/finance/app@sha256](https://harbor.example.com/finance/app@sha256):... Fetches image layers by tag or immutable digest SHA.
helm push Upload Helm chart as OCI artifact Package path, OCI URL helm push chart.tgz oci://[harbor.example.com/finance](https://harbor.example.com/finance) Stores Helm charts directly in standard Harbor projects.

Harbor REST API Commands (curl)

Command Target Purpose Flags Practical Example Explanation
/api/v2.0/health Check core service health -sk (silent, ignore untrusted TLS) curl -sk [https://harbor.example.com/api/v2.0/health](https://harbor.example.com/api/v2.0/health) Evaluates DB, Redis, Core, Registry, and Trivy status.
/api/v2.0/projects List all projects -u (credentials), -X GET curl -sk -u admin:PASS [https://harbor.example.com/api/v2.0/projects](https://harbor.example.com/api/v2.0/projects) Returns JSON array of projects and visibility settings.
/api/v2.0/repositories List project repositories Path parameters curl -sk -u admin:PASS [https://harbor.example.com/api/v2.0/projects/finance/repositories](https://harbor.example.com/api/v2.0/projects/finance/repositories) Returns repositories contained within specific project.
/api/v2.0/users Manage system users -X GET / POST curl -sk -u admin:PASS [https://harbor.example.com/api/v2.0/users](https://harbor.example.com/api/v2.0/users) Queries local database user accounts.
/api/v2.0/systeminfo Get system version & configuration -X GET curl -sk [https://harbor.example.com/api/v2.0/systeminfo](https://harbor.example.com/api/v2.0/systeminfo) Displays version number, auth mode, and TLS status.

System & Diagnostic CLI Commands

Command Purpose Useful Flags Practical Example Explanation
docker compose logs Tail Harbor microservice logs --tail=50, -f (follow), service name docker compose logs --tail=100 -f core registry jobservice Streams microservice container logs to stdout.
kubectl logs Inspect Harbor logs on Kubernetes -n harbor, -l (label selector) kubectl logs -n harbor -l component=core --tail=50 Aggregates logs across Kubernetes microservice pods.
crictl pull Test image pull directly from K8s worker node Node runtime command crictl pull [harbor.example.com/finance/app:1.0](https://harbor.example.com/finance/app:1.0) Verifies node-level networking and container runtime pull capabilities.
kubectl create secret Provision K8s Harbor pull secret docker-registry, parameters kubectl create secret docker-registry harbor-reg --docker-server=... Generates Kubernetes authentication secret for pod deployment specs.

9. Logs, Monitoring and Diagnostics

Log File Locations and Microservice Log Targets

In Docker Compose deployments, system logs are written to /var/log/harbor/ on the host machine:

  • /var/log/harbor/core.log: API requests, authentication, authorization, and project management.
  • /var/log/harbor/registry.log: Docker Distribution OCI layer uploads, blob streaming, and manifest reads.
  • /var/log/harbor/jobservice.log: Asynchronous execution logs for Trivy scans, replications, and garbage collection.
  • /var/log/harbor/trivy-adapter.log: Vulnerability database updates and artifact scan execution.
  • /var/log/harbor/proxy.log: NGINX reverse proxy access and error logs.

Live Diagnostic Log Workflows

# Tail all microservice logs simultaneously
cd /usr/local/harbor
docker compose logs -f --tail=50

# Filter Core API logs for authentication errors
tail -f /var/log/harbor/core.log | grep -i -E "error|unauthorized|401|403"

# Filter Jobservice logs for failed replication tasks
grep -i "failed" /var/log/harbor/jobservice.log

Illustrative Log Entries and Interpretation

1. Failed Authentication Log (core.log)

2026-11-03T08:14:22Z [ERROR] [/common/api/base.go:75]: user robot$payment-team+ci-runner authentication failed: invalid secret token

Interpretation: A CI pipeline attempted to login with an invalid or expired robot account token.

2. Storage Quota Exceeded (registry.log)

2026-11-03T09:30:11Z [ERROR] [/registry/handlers/images.go:120]: failed to write blob sha256:4a8b...: storage quota of project payment-team reached limit 100GB

Interpretation: The project quota enforced in Harbor Core has been breached. The push fails at layer upload.


10. Troubleshooting Guide and Common Failure Scenarios

Problem 1: docker login fails / unauthorized

Symptoms

Executing docker login harbor.example.com returns Error response from daemon: Get "[https://harbor.example.com/v2/](https://harbor.example.com/v2/)": unauthorized: incorrect username or password.

Possible Causes

  1. Invalid user credentials or expired robot account token.
  2. Incorrect robot account username syntax (missing robot$ prefix or incorrect project scope).
  3. Untrusted self-signed TLS certificate or missing root CA on client machine.
  4. OIDC/LDAP identity provider sync failure.

Investigation and Diagnosis

# Step 1: Verify username formatting for robot accounts
# Correct format: robot$PROJECT_NAME+ROBOT_NAME
echo "Testing format for robot account..."

# Step 2: Test API login using curl to isolate Docker CLI issues from API failure
curl -i -u "admin:ProductionAdminPassword123!" https://harbor.example.com/api/v2.0/users/current

# Step 3: Check Core container logs during login attempt
docker compose logs --tail=50 core | grep -i "auth"

Root Cause Identification

If curl returns 200 OK but Docker CLI fails, the issue is client TLS certificate trust. If curl returns 401 Unauthorized, the credentials or identity provider mapping is invalid.

Resolution

  • For Robot Accounts: Ensure username includes full escape character in shell: robot\$payment-team+jenkins-builder.
  • For TLS Issues: Copy Harbor’s CA certificate to client Docker trust directory:
mkdir -p /etc/docker/certs.d/harbor.example.com/
cp harbor-ca.crt /etc/docker/certs.d/harbor.example.com/ca.crt
systemctl restart docker

Verification

Execute docker login harbor.example.com and ensure WARNING! Your password will be stored unencrypted... Login Succeeded is returned.

Prevention

Automate CA certificate distribution to developer workstations and Kubernetes nodes via Ansible or Cloud-Init.


Problem 2: denied: requested access to the resource is denied

Symptoms

docker push [harbor.example.com/payment-team/backend:v1](https://harbor.example.com/payment-team/backend:v1) fails with denied: requested access to the resource is denied.

Possible Causes

  1. Pushing user/robot account lacks Developer or Maintainer role on project payment-team.
  2. Non-existent project name specified in target push path.
  3. Image tag is locked by an Immutable Tag policy.
  4. Prevent Vulnerable Images policy is blocking deployment/push due to unresolved high CVEs.

Investigation and Diagnosis

# Step 1: Check project existence and membership via API
curl -sk -u admin:PASS https://harbor.example.com/api/v2.0/projects/payment-team/members

# Step 2: Inspect Harbor Core logs for authorization denials
docker compose logs --tail=50 core | grep -i "denied"

Root Cause Identification

Inspect log entries. If core.log records user lacks push permission, membership is missing. If log shows tag is immutable, tag overwrite was attempted.

Resolution

  • Assign Developer role or higher to the user/robot account in project settings.
  • If tag is immutable, increment image version tag (e.g., v1.0.1 instead of overwriting v1.0.0).

Verification

Re-run docker push with corrected target path or updated permissions.


Problem 3: ImagePullBackOff in Kubernetes

Symptoms

Kubernetes pods fail to start, reporting ImagePullBackOff or ErrImagePull status. kubectl describe pod shows: Failed to pull image "[harbor.example.com/payment-team/api:v1](https://harbor.example.com/payment-team/api:v1)": rpc error: code = Unauthenticated.

Possible Causes

  1. Missing or invalid imagePullSecrets configuration in Kubernetes Pod spec.
  2. Secret contains invalid/expired robot credentials.
  3. Worker node cannot resolve Harbor FQDN or blocked by network firewall.
  4. Harbor registry enforced CVE security policy blocking image pull.

Investigation and Diagnosis

# Step 1: Inspect detailed Kubernetes Pod event stream
kubectl describe pod <pod-name> -n <namespace>

# Step 2: Test pull capability directly from Kubernetes worker node using container runtime CLI
crictl pull harbor.example.com/payment-team/api:v1

Root Cause Identification

If node-level crictl pull works with credentials but Pod fails, imagePullSecrets is missing or improperly referenced in service account / pod spec.

Resolution

  1. Create valid docker-registry secret in target namespace:
kubectl create secret docker-registry harbor-secret \
  --docker-server=harbor.example.com \
  --docker-username='robot$payment-team+k8s-pull' \
  --docker-password='ROBOT_TOKEN_SECRET' \
  -n target-namespace
  1. Add secret reference to Pod specification:
spec:
  imagePullSecrets:
    - name: harbor-secret
  containers:
    - name: api
      image: harbor.example.com/payment-team/api:v1

Problem 4: Push Succeeds but UI Shows No Image

Symptoms

docker push outputs Digest: sha256:... Status: Pushed, but the Harbor web UI shows 0 repositories inside project.

Possible Causes

  1. Image pushed to wrong project path (e.g., [harbor.example.com/backend:v1](https://harbor.example.com/backend:v1) instead of [harbor.example.com/my-project/backend:v1](https://harbor.example.com/my-project/backend:v1)).
  2. Database write latency or Redis cache synchronization failure.
  3. Asynchronous event broker issue between Registry and Core.

Resolution & Verification

Verify full image path matches REGISTRY/PROJECT/REPOSITORY:TAG schema. Inspect registry.log and core.log to confirm database sync.


Problem 5: Disk Full / Storage Growing Continuously

Symptoms

Host disk usage reaches 100% or cloud object storage costs balloon. Deleting repositories via Web UI does not release physical storage space.

Possible Causes

  1. Harbor tag deletion only removes PostgreSQL metadata references; physical layer blobs remain stored on disk until Garbage Collection is executed.
  2. High build frequency creating untagged (orphaned) manifest blobs.

Resolution

# Step 1: Execute manual Garbage Collection via Harbor Web UI or API
# Navigate to Administration -> Clean up -> Garbage Collection -> Run Now

# Step 2: Alternatively, trigger GC via API with untagged delete enabled
curl -sk -X POST "https://harbor.example.com/api/v2.0/system/gc/schedule" \
  -u admin:ProductionAdminPassword123! \
  -H "Content-Type: application/json" \
  -d '{"schedule":{"type":"Manual"},"parameters":{"delete_untagged":true}}'

Problem 6: Vulnerability Scan Stuck or Failed

Symptoms

Pushed images remain stuck in Scanning... state indefinitely, or UI shows Scan Failed status.

Possible Causes

  1. Trivy scanner container service crashed or unreachable by Jobservice.
  2. Outbound internet blocked in air-gapped environment preventing Trivy vulnerability DB updates.
  3. Jobservice task worker queue timeout on massive images.

Investigation and Resolution

# Check Trivy microservice logs
docker compose logs --tail=50 trivy adapter

# For air-gapped installs, download offline Trivy database bundle and mount into Trivy container

Problem 7: 502 / 503 Bad Gateway from Harbor UI or API

Symptoms

Web browser displays 502 Bad Gateway or 503 Service Unavailable when accessing Harbor portal.

Possible Causes

  1. Harbor Core API service down or failing health checks.
  2. Core cannot establish connection to PostgreSQL database or Redis cache.
  3. Database connection pool exhaustion.

Investigation and Diagnosis

# Step 1: Inspect container status
cd /usr/local/harbor && docker compose ps

# Step 2: Inspect Core and Postgres logs for connection errors
docker compose logs --tail=50 core database

Resolution

If database connection exhaustion is observed, restart Harbor stack and increase max_idle_conns and max_open_conns in harbor.yml.


11. Security and Best Practices

System Security Baseline Checklist

  1. Change Default Credentials: Change initial admin password (Harbor12345) immediately post-installation.
  2. Enforce HTTPS / TLS: Disable plain HTTP access. Force TLS 1.3/1.2 for all API endpoints and registry traffic.
  3. Principle of Least Privilege:
  • Do not grant global admin rights to developers.
  • Assign project roles (Developer, Guest, Maintainer) granularly.
  • Use dedicated scoped Robot Accounts for automated CI/CD push/pull operations.
  1. Vulnerability Gatekeeping: Enable automated scanning on push (auto_scan=true) and block deployment of images containing Critical or High severity CVEs.
  2. Content Trust & Image Signing: Integrate Cosign or Notary to sign container images within CI/CD pipelines. Configure Harbor project policies to reject unsigned image pulls.
  3. Integrate Identity Providers: Replace local authentication (db_auth) with corporate Single Sign-On (SSO) using OIDC (Keycloak, Okta, Azure AD) or LDAP/Active Directory.

12. Performance and Resource Optimisation

Performance Tuning Matrix

Target Subsystem Bottleneck Symptom Configuration Directive / Action Recommended Tuning Value
Harbor Core API HTTP 504 Gateway Timeouts during heavy CI builds Increase database pool in harbor.yml max_open_conns: 200, max_idle_conns: 50
OCI Registry Engine Slow layer streaming during concurrent pushes Tune storage max threads in harbor.yml filesystem.maxthreads: 100
Jobservice Workers Slow background replication / scanning queues Scale Jobservice replicas / concurrency workers Increase worker count to 10+
Trivy CVE Scanner High CPU utilization during peak push hours Adjust scan concurrency and memory limit Set Trivy RAM limits to minimum 2-4 GB
PostgreSQL DB High disk I/O and query latency Dedicated NVMe storage; tune shared_buffers shared_buffers = 25% Total RAM

13. Backup, Recovery and Maintenance

Backup Strategy

A complete Harbor backup requires capturing three distinct components:

  1. System Configuration: harbor.yml, custom certificates, and SSL keys.
  2. Database Metadata: PostgreSQL dump containing users, projects, policies, audit logs, and artifact manifests.
  3. Layer Blobs: Physical registry data directory (/var/lib/harbor/data/registry) or cloud S3 bucket contents.

Operational Backup Procedure

#!/bin/bash
# Harbor Automated Backup Script
BACKUP_DIR="/backups/harbor/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

# 1. Backup Configuration Files
cp /usr/local/harbor/harbor.yml "$BACKUP_DIR/"
cp -r /etc/harbor/certs "$BACKUP_DIR/"

# 2. Backup PostgreSQL Database
docker exec -t harbor-db pg_dump -U postgres -d registry > "$BACKUP_DIR/harbor_db.sql"

# 3. Backup Filesystem Blobs (if local filesystem storage is used)
tar -czf "$BACKUP_DIR/registry_blobs.tar.gz" /var/lib/harbor/data/registry

echo "Harbor Backup completed successfully to $BACKUP_DIR"

Restoration Procedure

# 1. Stop Harbor Microservices
cd /usr/local/harbor && docker compose down

# 2. Restore Configuration Files
cp /backups/harbor/latest/harbor.yml /usr/local/harbor/

# 3. Restore Blob Data
tar -xzf /backups/harbor/latest/registry_blobs.tar.gz -C /

# 4. Start Database Service and Restore PostgreSQL SQL Dump
docker compose up -d database
sleep 10
cat /backups/harbor/latest/harbor_db.sql | docker exec -i harbor-db psql -U postgres -d registry

# 5. Start Complete Harbor Stack
docker compose up -d

14. Automation and Integration

GitHub Actions CI/CD Integration Pipeline Example

name: Build, Scan, and Push to Harbor

on:
  push:
    branches: [ "main" ]

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3

      - name: Log in to Private Harbor Registry
        uses: docker/login-action@v2
        with:
          registry: harbor.example.com
          username: ${{ secrets.HARBOR_ROBOT_USER }}
          password: ${{ secrets.HARBOR_ROBOT_TOKEN }}

      - name: Build Docker Image
        run: |
          docker build -t harbor.example.com/payment-team/api-service:${{ github.sha }} .
          docker tag harbor.example.com/payment-team/api-service:${{ github.sha }} harbor.example.com/payment-team/api-service:latest

      - name: Push Image to Harbor
        run: |
          docker push harbor.example.com/payment-team/api-service:${{ github.sha }}
          docker push harbor.example.com/payment-team/api-service:latest

15. Practical Labs and Exercises

Lab 1 (Beginner): Deploying Single-Node Harbor & Executing Image Lifecycle

  • Objective: Install Harbor via Docker Compose, create a public project, and perform push/pull operations.
  • Setup: Clean Linux virtual machine with Docker and Docker Compose installed.
  • Instructions:
  1. Download and extract Harbor offline installer.
  2. Configure harbor.yml with host IP/FQDN and HTTP mode (dev testing).
  3. Run ./install.sh.
  4. Log in to UI, create project demo-project.
  5. Authenticate via Docker CLI, tag alpine:latest to hostname/demo-project/alpine:v1, and push.
  • Validation: Confirm image manifest and layers appear in Harbor Web UI.

Lab 2 (Intermediate): Securing Kubernetes Pulls with Scoped Robot Accounts

  • Objective: Configure zero-trust Kubernetes image pulling using scoped robot accounts.
  • Setup: Running Harbor instance and Kubernetes cluster.
  • Instructions:
  1. Create a private project prod-apps in Harbor.
  2. Create a robot account k8s-puller with pull-only permissions on prod-apps.
  3. Create a Kubernetes docker-registry secret using the robot credentials.
  4. Deploy a Pod referencing the secret in imagePullSecrets.
  • Validation: Confirm pod transitions to Running state without authentication errors.

Lab 3 (Troubleshooting): Simulating and Resolving Storage Bloat Incident

  • Objective: Diagnose a simulated 100% disk capacity event caused by deleted image tags.
  • Setup: Running Harbor instance with local volume storage.
  • Instructions:
  1. Push 5 large images (1GB+ each) to a test project.
  2. Delete all tags from the Web UI.
  3. Run df -h /var/lib/harbor/data and observe that physical disk space is not freed.
  4. Execute Harbor Garbage Collection job with delete_untagged=true.
  • Validation: Re-run df -h and confirm host storage is reclaimed.

16. Quick Reference and Cheat Sheet

Important Configuration Files

File Path Purpose
Master Config /usr/local/harbor/harbor.yml Central parameters (FQDN, TLS, DB, Storage).
Compose Spec /usr/local/harbor/docker-compose.yml Container runtime specifications.
Prepare Script /usr/local/harbor/prepare Template rendering utility.

Important Log Locations

Service Log Path
Harbor Core /var/log/harbor/core.log
Docker Registry /var/log/harbor/registry.log
Jobservice /var/log/harbor/jobservice.log
Trivy Scanner /var/log/harbor/trivy-adapter.log

Common Operational Commands

Objective Command
Check Health curl -sk [https://harbor.example.com/api/v2.0/health](https://harbor.example.com/api/v2.0/health)
Start Harbor cd /usr/local/harbor && docker compose up -d
Stop Harbor cd /usr/local/harbor && docker compose stop
Tail Service Logs docker compose logs -f --tail=50 core registry jobservice
K8s Pull Secret kubectl create secret docker-registry harbor-reg --docker-server=...

Troubleshooting Quick Reference

Symptom Initial Diagnostic Command Primary Fix
docker login 401 curl -i -u user:pass https://harbor/api/v2.0/users/current Verify credentials or update Docker CA cert.
Push Denied `docker compose logs core grep denied`
ImagePullBackOff kubectl describe pod <pod-name> Provision imagePullSecrets with robot token.
Disk Full after Deletes df -h /var/lib/harbor/data Run Garbage Collection via UI/API.
502 Bad Gateway docker compose ps Check DB/Redis status; restart Core service.

Important Ports and Protocols

Port Protocol Service Description
80 TCP HTTP NGINX Reverse Proxy (Redirects to HTTPS).
443 TCP HTTPS Web UI, REST API, and OCI Registry v2 Endpoint.
5432 TCP PostgreSQL Internal/External Database Port.
6379 TCP Redis Key-Value Cache and Job Queue Engine.

17. Further Reading and References

  1. Official Harbor Documentation: https://goharbor.io/docs
  2. Harbor Installation & Configuration Guide: https://goharbor.io/docs/install-config/
  3. Harbor Administration Guide: https://goharbor.io/docs/administration/
  4. OCI Distribution Specification: https://github.com/opencontainers/distribution-spec
  5. CNCF Harbor GitHub Repository: https://github.com/goharbor/harbor