Domain 10Linux & SREINTERMEDIATE

Secure Shell (SSH) Administration: Architecture, Hardening, and Troubleshooting Guide

A comprehensive operational guide to OpenSSH architecture, public-key authentication, server configuration, terminal multiplexing with tmux/screen, CLI workflows, logging, and systematic troubleshooting for SREs and SysAdmins.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-03
#linux#sysadmin#ssh#security#troubleshooting

1. Overview

Secure Shell (SSH) is a cryptographically secured network protocol designed to provide safe remote login, command execution, and file transfer services over untrusted networks. It serves as the primary administration protocol for Linux infrastructure, replacing legacy, unencrypted management utilities such as Telnet, RSH, RLOGIN, and plain FTP, which transmitted user credentials and session payloads in plaintext across the wire.

In a standard OpenSSH architecture:

  • Client Utility (ssh): Initiates outbound connections, manages local private keys, handles host verification, and authenticates against remote endpoints.
  • Server Daemon (sshd): Runs continuously as a system service, listens for incoming TCP connections (defaulting to port 22), authenticates client requests, and spawns session environments.
  • Companion Tools: Includes scp and sftp for file transfers, ssh-keygen for key pair creation, ssh-agent for managing decrypted keys in memory, and rsync for bandwidth-efficient file synchronization over SSH tunnels.

Primary Use Cases

  1. Interactive Remote Administration: Secure shell access for system configuration, monitoring, and interactive maintenance.
  2. Automated Batch Execution: Non-interactive command execution for scripts, deployment pipelines, and remote agent runners.
  3. Secure File Operations: Encrypted file copy and directory synchronization across systems.
  4. Network Tunneling & Port Forwarding: Forwarding local or remote TCP ports to bypass strict firewall boundaries or encrypt non-native protocols.
  5. Long-Running Remote Agent Workloads: Running multi-hour builds, test suites, and autonomous local AI coding agent loops via terminal multiplexers over SSH sessions.

Advantages and Limitations

  • Advantages: End-to-end payload encryption, strong key-based authentication, integrity protection against tampering, strict host key identity verification, and extensive integration with operational tooling.
  • Limitations: High dependence on proper file ownership and permission structures, susceptibility to credential brute-force when password authentication is exposed, and potential configuration lockouts if configuration files are modified without syntax testing.

2. Architecture and Mental Model

To effectively diagnose SSH connection failures, a Linux Administrator or Site Reliability Engineer (SRE) must understand the exact sequence of events that occurs when executing ssh user@host.

sequenceDiagram
    autonumber
    participant Client as SSH Client (ssh)
    participant Server as SSH Daemon (sshd)

    Note over Client,Server: Phase 1: Transport Layer Setup
    Client->>Server: TCP SYN (Port 22 / Custom Port)
    Server-->>Client: TCP SYN-ACK
    Client->>Server: TCP ACK

    Note over Client,Server: Phase 2: Protocol Banner Exchange
    Client->>Server: SSH Version Banner (SSH-2.0-OpenSSH...)
    Server-->>Client: SSH Version Banner (SSH-2.0-OpenSSH...)

    Note over Client,Server: Phase 3: Key Exchange (KEX) & Session Cipher
    Client<->>Server: KEXINIT (Algorithm Negotiation: ECDH/Diffie-Hellman)
    Note over Client,Server: Shared Session Key Established (Traffic Encrypted From Here)

    Note over Client,Server: Phase 4: Host Authentication
    Server->>Client: Server Host Key & Signature
    Client->>Client: Verify Host Key in ~/.ssh/known_hosts

    Note over Client,Server: Phase 5: User Authentication
    Client->>Server: Auth Request (Public Key / Password)
    Server->>Server: Verify Key in ~/.ssh/authorized_keys or DB
    Server-->>Client: Authentication Success

    Note over Client,Server: Phase 6: Session & Channel Multiplexing
    Server->>Server: Spawn User Shell or Execute Remote Command
    Client<->>Server: Interactive Shell / Subsystem Payload Exchange

Connection Workflow Breakdown

  1. TCP Connection: The client initiates a standard TCP 3-way handshake on port 22 (or a custom port specified via -p) to establish transport connectivity.
  2. Protocol Banner Exchange: Both endpoints exchange version strings (e.g., SSH-2.0-OpenSSH_8.9p1). This verifies protocol compatibility.
  3. Key Exchange (KEX): Diffie-Hellman or Elliptic Curve Diffie-Hellman (ECDH) key exchange algorithms are negotiated. A shared, ephemeral session key is derived to encrypt all subsequent communication. Payload encryption begins immediately after this phase.
  4. Host Authentication: The server presents its host public key and a signature. The client validates this key against its local ~/.ssh/known_hosts store to prevent Man-in-the-Middle (MITM) attacks.
  5. User Authentication: The client attempts to prove its identity using configured methods (Public Key, Password, Keyboard-Interactive). In production environments, public key authentication is standard. The server validates the presented signature against public keys stored in ~/.ssh/authorized_keys.
  6. Session and Channel Allocation: Upon successful authentication, sshd allocates a session channel, requests a pseudo-terminal (PTY) if interactive, and spawns the user’s default login shell or runs the requested remote command.

3. Core Concepts and Terminology

Public-Key Cryptography

SSH public-key authentication relies on asymmetric cryptography using a key pair:

  • Private Key (e.g., ~/.ssh/id_ed25519): Stored securely on the local client machine with restricted permissions (0600). It must never be shared or exposed.
  • Public Key (e.g., ~/.ssh/id_ed25519.pub): Placed on target servers inside the destination user’s ~/.ssh/authorized_keys file.

Key Types: Ed25519 vs. RSA

  • Ed25519: Elliptic curve signature scheme (EdDSA). Offers superior security, high performance, and short key lengths. It is the modern default and recommendation for Linux environments.
  • RSA: Legacy public-key algorithm. Requires key lengths of at least 3072 or 4096 bits to remain secure. Use Ed25519 over RSA unless legacy operating systems dictate RSA usage.

SSH Agent and Agent Forwarding

  • ssh-agent: A background process that holds decrypted private keys in memory so users do not need to re-enter passphrases on every SSH connection.
  • Agent Forwarding (ForwardAgent yes): Allows remote servers to leverage your local ssh-agent to hop between internal servers without copying private keys to intermediate nodes. Security Note: Use agent forwarding sparingly; compromised intermediate root users can hijack access to the forwarded agent socket.

Bastion Hosts and ProxyJump

A bastion host (or jump host) acts as a single, hardened entry point into a private network segment. Modern OpenSSH uses ProxyJump (-J or ProxyJump in ~/.ssh/config) to establish an encrypted tunnel through the bastion to the internal host, eliminating the need for agent forwarding or placing private keys on the bastion.

Persistent Remote Workflows: tmux and screen

When an SSH connection breaks due to Wi-Fi drops, laptop sleep, or network timeouts, any process running directly inside that interactive SSH shell receives a SIGHUP signal and terminates. Terminal multiplexers such as tmux and screen run as persistent daemons on the remote server. They decouple terminal sessions from the active SSH connection:

  • screen: Traditional, ubiquitous multiplexer available across legacy Linux systems (Ctrl-a d to detach, screen -r to reattach).
  • tmux: Modern multiplexer featuring split panes, multiple windows, robust session management, and scriptability (Ctrl-b d to detach, tmux attach to reattach).
Client Laptop                Jump Host (Bastion)               Target Production Host
+------------------+         +------------------+             +-----------------------+
|  ssh -J bastion  | ------> |  sshd (Port 22)  |  (Tunnel)   |  sshd (Port 22)       |
|  user@internal   |         |  No Shell Needed | ----------> |  Spawns tmux session  |
+------------------+         +------------------+             +-----------------------+

4. Installation and Version Management

Prerequisites and Package Installation

OpenSSH consists of client software (openssh-client) and server software (openssh-server). Installation varies by distribution:

Debian / Ubuntu Systems

sudo apt update
sudo apt install -y openssh-client openssh-server

RHEL / CentOS / Rocky Linux / AlmaLinux

sudo dnf install -y openssh-clients openssh-server

Systemd Service Unit Differences

A critical distinction across Linux distribution families is the systemd service name:

  • Debian/Ubuntu: The SSH daemon unit is named ssh.
  • RHEL/CentOS/Fedora: The SSH daemon unit is named sshd.
Distribution Family Client Package Server Package Systemd Service Name Auth Log File
Debian / Ubuntu openssh-client openssh-server ssh /var/log/auth.log
RHEL / CentOS / Rocky openssh-clients openssh-server sshd /var/log/secure

Inspecting Installed Versions

To verify client and server binary versions:

# Check client version
ssh -V

# Check server daemon binary version
sshd -V

5. Configuration Files and Directory Structure

Configuration File Locations

File / Directory Purpose Mandatory Permissions Important Details
~/.ssh/ User SSH directory 0700 (drwx------) Stores user keys, configuration, and host identity stores.
~/.ssh/id_ed25519 Private key file 0600 (-rw-------) Private identity key. Must never be world/group readable.
~/.ssh/id_ed25519.pub Public key file 0644 (-rw-r--r--) Paired public key to be installed on remote servers.
~/.ssh/authorized_keys Remote authorization store 0600 (-rw-------) List of client public keys allowed to log into this user account.
~/.ssh/known_hosts Trusted host store 0644 (-rw-r--r--) Database of public keys presented by trusted servers.
~/.ssh/config Client configuration 0600 (-rw-------) Client host aliases, jump host rules, and connection parameters.
/etc/ssh/sshd_config Server daemon config 0644 (-rw-r--r--) Master system configuration for incoming SSH connections.
/etc/ssh/sshd_config.d/ Server drop-in directory 0755 (drwxr-xr-x) Directory for modular server configuration files on modern distros.
/etc/ssh/ssh_host_*_key Server identity host keys 0600 (-rw-------) Server-side private host keys used to authenticate server identity.

Core Server Directives (/etc/ssh/sshd_config)

  • Port 22: Defines the listening TCP port.
  • PermitRootLogin no or prohibit-password: Controls whether root can log in via password or at all.
  • PasswordAuthentication no: Disables plaintext password logins, enforcing key-based access.
  • PubkeyAuthentication yes: Enables public key authentication.
  • AllowUsers deploy admin: Restricts access exclusively to explicit user accounts.
  • AllowGroups devops: Restricts access to users belonging to specific Linux groups.
  • AuthorizedKeysFile .ssh/authorized_keys: Path to user public key definitions.

Validating Configuration Changes

Before reloading or restarting sshd, always test the syntax of /etc/ssh/sshd_config to avoid locking out administrative sessions:

sudo sshd -t

If sshd -t returns no output, the configuration syntax is valid.


6. Service Management and Common Operations

Service Control Commands

On Debian / Ubuntu:

# Check service status
systemctl status ssh

# Reload service (applies sshd_config without dropping active connections)
sudo sshd -t && sudo systemctl reload ssh

# Restart service
sudo systemctl restart ssh

On RHEL / CentOS / Rocky Linux:

# Check service status
systemctl status sshd

# Reload service
sudo sshd -t && sudo systemctl reload sshd

# Restart service
sudo systemctl restart sshd

Verifying Listening Sockets

To verify that sshd is active and listening on the designated port:

# Using ss utility
sudo ss -tlnp | grep ssh

# Alternatively, check specifically for port 22
sudo ss -tlnp | grep :22

Expected output:

LISTEN 0      128          0.0.0.0:22        0.0.0.0:*    users:(("sshd",pid=1042,fd=3))
LISTEN 0      128             [::]:22           [::]:*    users:(("sshd",pid=1042,fd=4))

7. Common Administrative Tasks and Workflows

Workflow 1: Generating Ed25519 Key Pairs

Generate an Ed25519 key pair with a descriptive comment:

ssh-keygen -t ed25519 -C "sysadmin@company.com"

When prompted, provide a strong passphrase to encrypt the private key on disk.

Workflow 2: Copying Public Keys to Remote Target

Transfer the generated public key to a remote server using ssh-copy-id:

ssh-copy-id -i ~/.ssh/id_ed25519.pub user@192.168.1.50

This automatically creates ~/.ssh/ and appends the key to ~/.ssh/authorized_keys with proper file permissions on the target server.

Workflow 3: Hardening SSH Daemon Configuration

  1. Backup existing configuration:
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
  1. Edit /etc/ssh/sshd_config to include security directives:
Port 22
PermitRootLogin prohibit-password
PasswordAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
AllowGroups sysadmin
  1. Test syntax and reload:
sudo sshd -t && sudo systemctl reload ssh # (or sshd on RHEL)

Workflow 4: Configuring Client Profiles (~/.ssh/config)

Simplify complex connections by defining shortcuts in ~/.ssh/config:

# Global defaults
Host *
    ServerAliveInterval 60
    ServerAliveCountMax 3
    AddKeysToAgent yes

# Bastion / Jump Host definition
Host bastion
    HostName jump.example.com
    User deploy
    Port 22
    IdentityFile ~/.ssh/deploy_ed25519

# Private Target Hosts routed via Jump Host
Host app-*
    User ubuntu
    ProxyJump bastion
    IdentityFile ~/.ssh/deploy_ed25519

With this configuration, connecting to an internal node is simplified to:

ssh app-01

Workflow 5: Protecting Long Workloads with tmux

  1. SSH into the remote node:
ssh user@host
  1. Start a named tmux session:
tmux new -s agent-build
  1. Execute long-running builds, test suites, or autonomous AI agent scripts.
  2. Safely detach from the session by pressing Ctrl-b then d.
  3. Reconnect later after network interruption or disconnection:
tmux attach -t agent-build

8. CLI Command Reference and Cheatsheet

Category Command Important Flags / Options Example Operational Purpose
Connecting ssh -p (port)


-i (identity file)


-J (jump host) | ssh -p 2222 -i ~/.ssh/deploy_key user@host | Connect to remote SSH server on specified port using explicit identity key. | | Connecting | ssh | Remote Command Execution | ssh user@host 'uptime' | Execute command on remote host non-interactively and return stdout/stderr. | | Connecting | ssh | -J | ssh -J bastion user@internal | Tunnel connection to internal host through bastion. | | Port Forwarding | ssh | -L (Local Forward)


-N (No Shell) | ssh -N -L 8080:localhost:80 user@host | Bind local port 8080 to remote server’s local port 80. | | Port Forwarding | ssh | -R (Remote Forward) | ssh -N -R 9000:localhost:3000 user@host | Expose local port 3000 on remote server’s port 9000. | | Key Management | ssh-keygen | -t (type)


-C (comment)


-b (bits) | ssh-keygen -t ed25519 -C "you@host" | Generate an Ed25519 asymmetric SSH key pair. | | Key Management | ssh-copy-id | -i (identity pub key) | ssh-copy-id -i ~/.ssh/id_ed25519.pub user@host | Install local public key into remote ~/.ssh/authorized_keys. | | Key Management | ssh-keygen | -lf (fingerprint) | ssh-keygen -lf ~/.ssh/id_ed25519.pub | Display key fingerprint and length. | | Key Management | ssh-keygen | -R (remove host) | ssh-keygen -R host.example.com | Remove modified or stale host key entries from ~/.ssh/known_hosts. | | Agent Operations | ssh-add | Default | ssh-add ~/.ssh/id_ed25519 | Load private key into running ssh-agent memory. | | Agent Operations | ssh-add | -l (list) | ssh-add -l | List fingerprints of all keys loaded in active agent. | | File Transfer | scp | -P (port)


-r (recursive) | scp -P 2222 -r ./dir user@host:/var/www/ | Securely copy local directory to remote destination. | | File Transfer | sftp | Default | sftp user@host | Open interactive secure file transfer session. | | File Transfer | rsync | -e ssh | rsync -avz -e ssh src/ user@host:dest/ | Synchronize files incrementally over SSH transport. | | Diagnostics | ssh | -v, -vv, -vvv | ssh -vvv user@host | Enable maximum client verbosity for debugging connection failures. | | Server Admin | sshd | -t (test syntax) | sudo sshd -t | Test /etc/ssh/sshd_config syntax for errors prior to service reload. |


9. Logs, Monitoring and Diagnostics

Log File Locations and Systemd Integration

  • Debian / Ubuntu: System authentication logs are written to /var/log/auth.log.
  • RHEL / CentOS / Rocky: System authentication logs are written to /var/log/secure.
  • Systemd Journal: Accessible via journalctl on modern systemd-based Linux systems:
# Debian/Ubuntu systemd log tail
sudo journalctl -u ssh -n 50 --no-pager

# RHEL/CentOS systemd log tail
sudo journalctl -u sshd -n 50 --no-pager

Real-Time Diagnostics

1. Client-Side Verbose Debugging

When connections fail, execute ssh with increased verbosity:

ssh -vvv -i ~/.ssh/key user@host

Look for:

  • KEX algorithm negotiation errors (debug1: kex: algorithm...).
  • Key file offering sequence (debug1: Offering public key:...).
  • Specific server rejection notices (debug1: Authentications that can continue:...).

2. Server-Side Authentication Monitoring

Monitor authorization events on the target host while attempting a connection:

# Debian / Ubuntu
sudo tail -f /var/log/auth.log | grep sshd

# RHEL / CentOS
sudo tail -f /var/log/secure | grep sshd

Illustrative Log Output Analysis

Bad Permissions Warning
Authentication refused: bad ownership or modes for directory /home/deploy/.ssh

Meaning: The target user’s ~/.ssh directory has group/world-writable permissions (e.g., 0777). OpenSSH strictly rejects public keys when directories are insecure.

Public Key Rejection
Failed publickey for invalid user admin from 192.168.1.100 port 54322 ssh2

Meaning: The presented public key does not match any entry in ~/.ssh/authorized_keys, or the user account does not exist.


10. Troubleshooting Guide and Common Failure Scenarios

Problem: Permission denied (publickey)

Symptoms

Connection attempt immediately fails with output:

user@host: Permission denied (publickey).

Possible Causes

  1. Target user’s ~/.ssh/authorized_keys file does not contain the client’s public key.
  2. Insecure directory or file permissions on ~/.ssh or authorized_keys.
  3. Client SSH agent or CLI is offering the wrong key identity.
  4. Target home directory is group- or world-writable.

Investigation and Diagnosis

  1. Execute client connection with verbose debugging:
ssh -vvv user@host

Check which key identities are offered during authentication. 2. Inspect server-side authentication logs:

sudo grep sshd /var/log/auth.log   # Debian/Ubuntu
sudo grep sshd /var/log/secure     # RHEL/CentOS
  1. Check permissions on the target server directory structure.

Resolution

Fix file and directory ownership and permissions on the remote server:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 600 ~/.ssh/id_ed25519
chown -R user:user ~/.ssh

Ensure the public key (~/.ssh/id_ed25519.pub) string is correctly copied as a single line inside ~/.ssh/authorized_keys.

Verification

Attempt connection explicitly specifying the key file:

ssh -i ~/.ssh/id_ed25519 user@host

Prevention

Use ssh-copy-id to install public keys; it enforces proper permissions automatically.


Problem: Connection refused

Symptoms

Client attempt terminates immediately:

ssh: connect to host 192.168.1.50 port 22: Connection refused

Possible Causes

  1. The sshd daemon is not running on the target server.
  2. sshd is listening on a non-standard port instead of port 22.
  3. Local host firewall (iptables/nftables/ufw) is actively rejecting packets on port 22.

Investigation and Diagnosis

  1. Verify if sshd service is active on the server:
systemctl status ssh     # Debian/Ubuntu
systemctl status sshd    # RHEL/CentOS
  1. Verify listening ports on the server:
sudo ss -tlnp | grep ssh

Resolution

  1. If the service is stopped, start it:
sudo systemctl start ssh   # or sshd on RHEL
  1. If listening on a custom port (e.g., 2222), connect using -p:
ssh -p 2222 user@host

Verification

Re-run ss -tlnp | grep ssh and test TCP connectivity using nc -zv host 22.


Problem: Connection timed out

Symptoms

Command hangs for several seconds before exiting:

ssh: connect to host 192.168.1.50 port 22: Connection timed out

Possible Causes

  1. Network path issue (incorrect IP address or routing failure).
  2. Firewall dropping packets silently (AWS Security Group, cloud network ACL, or ufw/firewalld).
  3. Target host is powered off or unreachable.

Investigation and Diagnosis

Unlike “Connection refused”, a timeout indicates packet drop.

  1. Test network-layer connection:
nc -zv 192.168.1.50 22
  1. Trace path routing:
traceroute 192.168.1.50

Resolution

Update security group rules or local firewall (e.g., ufw allow 22/tcp or cloud network security groups) to permit incoming traffic on port 22 from your source IP.


Problem: Host key verification failed

Symptoms

Client output warns of potential security risk:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Host key verification failed.

Possible Causes

  1. Target server was rebuilt or reinstalled, generating new host keys.
  2. Server IP address was reassigned to a different instance.
  3. Potential Man-In-The-Middle (MITM) attack.

Investigation and Diagnosis

Verify out-of-band (via cloud console or IPMI) whether the host was recently re-provisioned.

Resolution

If the host key change was legitimate, remove the obsolete host key entry from ~/.ssh/known_hosts:

ssh-keygen -R host.example.com

Verification

Reconnect to the server and verify the new fingerprint against your out-of-band server records:

ssh user@host.example.com

Problem: Too many authentication failures

Symptoms

Disconnect during handshake phase:

Received disconnect from 192.168.1.50 port 22:2: Too many authentication failures

Possible Causes

The client’s ssh-agent contains numerous loaded keys. The SSH client offers every loaded key sequentially until sshd hits its MaxAuthTries limit (default 6) and drops the connection.

Resolution

Force SSH to use only the identity key specified on the command line or in configuration:

ssh -o IdentitiesOnly=yes -i ~/.ssh/right_key user@host

Alternatively, configure IdentitiesOnly yes in ~/.ssh/config.


Problem: Locked out after editing sshd_config

Symptoms

Administrative users cannot establish new SSH sessions after altering /etc/ssh/sshd_config.

Possible Causes

Syntax error or invalid directive in /etc/ssh/sshd_config prevented sshd from restarting or caused it to reject valid credentials.

Investigation and Resolution

  1. Do not close your existing SSH terminal! Keep existing sessions open while testing.
  2. If completely locked out, gain out-of-band management access (Cloud Serial Console, IPMI, hypervisor console, or VNC).
  3. Execute syntax check to pinpoint error:
sudo sshd -t
  1. Fix syntax errors in /etc/ssh/sshd_config and reload sshd service.

11. Security and Best Practices

  1. Disable Root Password Authentication: Force system administrators to log in using non-root user accounts with public keys, then escalate privileges using sudo.
PermitRootLogin prohibit-password
  1. Disable Password Authentication: Eliminate brute-force attacks by requiring public-key authentication.
PasswordAuthentication no
  1. Prefer Modern Key Algorithms: Use Ed25519 public keys over legacy RSA keys.
  2. Restrict Authorized Key Capabilities: Enforce key options inside ~/.ssh/authorized_keys for automated scripts or service accounts:
from="192.168.1.0/24",command="/usr/bin/backup.sh",no-pty ed25519 AAAAC3NzaC1lZDI1NTE5... backup-key
  1. Restrict Permitted Users: Limit login access using explicit allow-lists:
AllowUsers deploy admin
AllowGroups sysadmin devops
  1. Use Jump Hosts Instead of Agent Forwarding: Use ProxyJump (-J) rather than ForwardAgent yes to avoid exposing agent sockets to compromised jump servers.

12. Performance and Resource Optimisation

Connection Multiplexing (ControlMaster)

Reusing an existing TCP socket for subsequent SSH connections to the same host eliminates handshake overhead and speeds up automation tools: Add the following to ~/.ssh/config:

Host *
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h:%p
    ControlPersist 10m

Create the socket directory:

mkdir -p ~/.ssh/sockets && chmod 700 ~/.ssh/sockets

Keeping Connections Alive

Prevent firewalls and NAT routers from dropping idle SSH sessions due to state timeouts:

  • Client side (~/.ssh/config):
ServerAliveInterval 60
ServerAliveCountMax 3
  • Server side (/etc/ssh/sshd_config):
ClientAliveInterval 60
ClientAliveCountMax 3

13. Backup, Recovery and Maintenance

Key Configurations to Backup

  • Server State: /etc/ssh/sshd_config, /etc/ssh/sshd_config.d/, and server host keys /etc/ssh/ssh_host_*.
  • User State: ~/.ssh/authorized_keys and ~/.ssh/config.

Backup Command

sudo tar -czvf ssh_server_backup.tar.gz /etc/ssh/sshd_config /etc/ssh/ssh_host_*

Emergency Recovery Access

Always maintain secondary access mechanisms for critical servers:

  • Ensure out-of-band access (Cloud Serial Console, IPMI, hypervisor serial console) is functional prior to applying major configuration changes.

14. Automation and Integration

Ansible Integration

Ansible relies on SSH as its default control transport. Configure ansible.cfg to leverage native SSH optimizations:

[defaults]
remote_user = deploy
private_key_file = ~/.ssh/id_ed25519
host_key_checking = True

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=15m

CI/CD Pipeline Execution

In automated pipelines (e.g., GitHub Actions or Jenkins), pass non-interactive flags when executing commands or syncing files:

ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new user@deploy.example.com 'docker pull myapp:latest && docker restart myapp'

15. Practical Labs and Exercises

Lab 1: Setting Up Ed25519 Keys and Copying to Remote Server

  • Objective: Generate an Ed25519 key pair and deploy it to a remote Linux instance.
  • Instructions:
  1. Generate key pair on local system:
ssh-keygen -t ed25519 -C "lab1-user"
  1. Copy key to remote server:
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@<remote-ip>
  1. Verify key-based login without password prompt.

Lab 2: Safe Hardening of sshd_config

  • Objective: Disable password authentication securely.
  • Instructions:
  1. Open two separate terminal windows on your local machine.
  2. In Terminal 1, log into the server and edit /etc/ssh/sshd_config, setting PasswordAuthentication no.
  3. Validate configuration syntax:
sudo sshd -t
  1. Reload service:
sudo systemctl reload ssh   # (or sshd)
  1. In Terminal 2 (without closing Terminal 1!), test a new connection:
ssh user@<remote-ip>

Lab 3: Terminal Session Persistence with tmux

  • Objective: Run long-running jobs that survive SSH session disconnects.
  • Instructions:
  1. SSH into remote host.
  2. Launch a named tmux session:
tmux new -s persistent_task
  1. Start a long-running process (e.g., ping -i 2 127.0.0.1 or a build script).
  2. Detach from session using Ctrl-b then d.
  3. Disconnect SSH connection.
  4. Re-establish SSH connection and attach to session:
tmux attach -t persistent_task

Lab 4: Diagnosing Permissions Failures

  • Objective: Troubleshoot and fix a broken Permission denied (publickey) state.
  • Instructions:
  1. Break target user permissions intentionally on remote host:
chmod 777 ~/.ssh
  1. Attempt SSH connection from client and observe error.
  2. Fix permissions using correct file modes:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
  1. Verify successful login.

16. Quick Reference and Cheat Sheet

Important Configuration Files

File / Directory Purpose
~/.ssh/config Client SSH connection aliases and options.
~/.ssh/authorized_keys Remote server public key store for user account.
/etc/ssh/sshd_config Main SSH daemon server configuration.
/etc/ssh/sshd_config.d/ Modular server drop-in configuration directory.

Important Log Locations

Distribution Log Path / Command
Debian / Ubuntu /var/log/auth.log or journalctl -u ssh
RHEL / CentOS / Rocky /var/log/secure or journalctl -u sshd

Common Commands

Task Command
Generate Ed25519 Key ssh-keygen -t ed25519 -C "user@host"
Copy Key to Server ssh-copy-id -i ~/.ssh/id_ed25519.pub user@host
Test Configuration sudo sshd -t
Connect via ProxyJump ssh -J user@bastion user@internal
Verbose Connection Debug ssh -vvv user@host

Troubleshooting Quick Reference

Symptom Initial Investigation
Permission denied (publickey) Check ~/.ssh permissions (700), authorized_keys (600), and inspect auth logs.
Connection refused Verify systemctl status ssh/sshd and check listening port via ss -tlnp.
Connection timed out Check network routing and test port access with nc -zv host 22.
Host key verification failed Remove stale host key using ssh-keygen -R host after verifying identity out-of-band.

Important Ports and Protocols

Port Protocol Purpose
22 TCP Default OpenSSH Server Listening Port.

17. Further Reading and References

  1. OpenSSH Client Manual (ssh(1)): https://man7.org/linux/man-pages/man1/ssh.1.html
  2. OpenSSH Daemon Configuration Manual (sshd_config(5)): https://man7.org/linux/man-pages/man5/sshd_config.5.html
  3. OpenSSH Key Management Manual (ssh-keygen(1)): https://man7.org/linux/man-pages/man1/ssh-keygen.1.html
  4. Official OpenSSH Documentation Index: https://www.openssh.com/manual.html