1. Overview
Secure Shell (SSH) is a cryptographically secured network protocol designed to provide safe remote login, command execution, and file transfer services over untrusted networks. It serves as the primary administration protocol for Linux infrastructure, replacing legacy, unencrypted management utilities such as Telnet, RSH, RLOGIN, and plain FTP, which transmitted user credentials and session payloads in plaintext across the wire.
In a standard OpenSSH architecture:
- Client Utility (
ssh): Initiates outbound connections, manages local private keys, handles host verification, and authenticates against remote endpoints. - Server Daemon (
sshd): Runs continuously as a system service, listens for incoming TCP connections (defaulting to port 22), authenticates client requests, and spawns session environments. - Companion Tools: Includes
scpandsftpfor file transfers,ssh-keygenfor key pair creation,ssh-agentfor managing decrypted keys in memory, andrsyncfor bandwidth-efficient file synchronization over SSH tunnels.
Primary Use Cases
- Interactive Remote Administration: Secure shell access for system configuration, monitoring, and interactive maintenance.
- Automated Batch Execution: Non-interactive command execution for scripts, deployment pipelines, and remote agent runners.
- Secure File Operations: Encrypted file copy and directory synchronization across systems.
- Network Tunneling & Port Forwarding: Forwarding local or remote TCP ports to bypass strict firewall boundaries or encrypt non-native protocols.
- Long-Running Remote Agent Workloads: Running multi-hour builds, test suites, and autonomous local AI coding agent loops via terminal multiplexers over SSH sessions.
Advantages and Limitations
- Advantages: End-to-end payload encryption, strong key-based authentication, integrity protection against tampering, strict host key identity verification, and extensive integration with operational tooling.
- Limitations: High dependence on proper file ownership and permission structures, susceptibility to credential brute-force when password authentication is exposed, and potential configuration lockouts if configuration files are modified without syntax testing.
2. Architecture and Mental Model
To effectively diagnose SSH connection failures, a Linux Administrator or Site Reliability Engineer (SRE) must understand the exact sequence of events that occurs when executing ssh user@host.
sequenceDiagram
autonumber
participant Client as SSH Client (ssh)
participant Server as SSH Daemon (sshd)
Note over Client,Server: Phase 1: Transport Layer Setup
Client->>Server: TCP SYN (Port 22 / Custom Port)
Server-->>Client: TCP SYN-ACK
Client->>Server: TCP ACK
Note over Client,Server: Phase 2: Protocol Banner Exchange
Client->>Server: SSH Version Banner (SSH-2.0-OpenSSH...)
Server-->>Client: SSH Version Banner (SSH-2.0-OpenSSH...)
Note over Client,Server: Phase 3: Key Exchange (KEX) & Session Cipher
Client<->>Server: KEXINIT (Algorithm Negotiation: ECDH/Diffie-Hellman)
Note over Client,Server: Shared Session Key Established (Traffic Encrypted From Here)
Note over Client,Server: Phase 4: Host Authentication
Server->>Client: Server Host Key & Signature
Client->>Client: Verify Host Key in ~/.ssh/known_hosts
Note over Client,Server: Phase 5: User Authentication
Client->>Server: Auth Request (Public Key / Password)
Server->>Server: Verify Key in ~/.ssh/authorized_keys or DB
Server-->>Client: Authentication Success
Note over Client,Server: Phase 6: Session & Channel Multiplexing
Server->>Server: Spawn User Shell or Execute Remote Command
Client<->>Server: Interactive Shell / Subsystem Payload Exchange
Connection Workflow Breakdown
- TCP Connection: The client initiates a standard TCP 3-way handshake on port 22 (or a custom port specified via
-p) to establish transport connectivity. - Protocol Banner Exchange: Both endpoints exchange version strings (e.g.,
SSH-2.0-OpenSSH_8.9p1). This verifies protocol compatibility. - Key Exchange (KEX): Diffie-Hellman or Elliptic Curve Diffie-Hellman (ECDH) key exchange algorithms are negotiated. A shared, ephemeral session key is derived to encrypt all subsequent communication. Payload encryption begins immediately after this phase.
- Host Authentication: The server presents its host public key and a signature. The client validates this key against its local
~/.ssh/known_hostsstore to prevent Man-in-the-Middle (MITM) attacks. - User Authentication: The client attempts to prove its identity using configured methods (Public Key, Password, Keyboard-Interactive). In production environments, public key authentication is standard. The server validates the presented signature against public keys stored in
~/.ssh/authorized_keys. - Session and Channel Allocation: Upon successful authentication,
sshdallocates a session channel, requests a pseudo-terminal (PTY) if interactive, and spawns the user’s default login shell or runs the requested remote command.
3. Core Concepts and Terminology
Public-Key Cryptography
SSH public-key authentication relies on asymmetric cryptography using a key pair:
- Private Key (e.g.,
~/.ssh/id_ed25519): Stored securely on the local client machine with restricted permissions (0600). It must never be shared or exposed. - Public Key (e.g.,
~/.ssh/id_ed25519.pub): Placed on target servers inside the destination user’s~/.ssh/authorized_keysfile.
Key Types: Ed25519 vs. RSA
- Ed25519: Elliptic curve signature scheme (EdDSA). Offers superior security, high performance, and short key lengths. It is the modern default and recommendation for Linux environments.
- RSA: Legacy public-key algorithm. Requires key lengths of at least 3072 or 4096 bits to remain secure. Use Ed25519 over RSA unless legacy operating systems dictate RSA usage.
SSH Agent and Agent Forwarding
ssh-agent: A background process that holds decrypted private keys in memory so users do not need to re-enter passphrases on every SSH connection.- Agent Forwarding (
ForwardAgent yes): Allows remote servers to leverage your localssh-agentto hop between internal servers without copying private keys to intermediate nodes. Security Note: Use agent forwarding sparingly; compromised intermediate root users can hijack access to the forwarded agent socket.
Bastion Hosts and ProxyJump
A bastion host (or jump host) acts as a single, hardened entry point into a private network segment. Modern OpenSSH uses ProxyJump (-J or ProxyJump in ~/.ssh/config) to establish an encrypted tunnel through the bastion to the internal host, eliminating the need for agent forwarding or placing private keys on the bastion.
Persistent Remote Workflows: tmux and screen
When an SSH connection breaks due to Wi-Fi drops, laptop sleep, or network timeouts, any process running directly inside that interactive SSH shell receives a SIGHUP signal and terminates.
Terminal multiplexers such as tmux and screen run as persistent daemons on the remote server. They decouple terminal sessions from the active SSH connection:
screen: Traditional, ubiquitous multiplexer available across legacy Linux systems (Ctrl-a dto detach,screen -rto reattach).tmux: Modern multiplexer featuring split panes, multiple windows, robust session management, and scriptability (Ctrl-b dto detach,tmux attachto reattach).
Client Laptop Jump Host (Bastion) Target Production Host
+------------------+ +------------------+ +-----------------------+
| ssh -J bastion | ------> | sshd (Port 22) | (Tunnel) | sshd (Port 22) |
| user@internal | | No Shell Needed | ----------> | Spawns tmux session |
+------------------+ +------------------+ +-----------------------+
4. Installation and Version Management
Prerequisites and Package Installation
OpenSSH consists of client software (openssh-client) and server software (openssh-server). Installation varies by distribution:
Debian / Ubuntu Systems
sudo apt update
sudo apt install -y openssh-client openssh-server
RHEL / CentOS / Rocky Linux / AlmaLinux
sudo dnf install -y openssh-clients openssh-server
Systemd Service Unit Differences
A critical distinction across Linux distribution families is the systemd service name:
- Debian/Ubuntu: The SSH daemon unit is named
ssh. - RHEL/CentOS/Fedora: The SSH daemon unit is named
sshd.
| Distribution Family | Client Package | Server Package | Systemd Service Name | Auth Log File |
|---|---|---|---|---|
| Debian / Ubuntu | openssh-client |
openssh-server |
ssh |
/var/log/auth.log |
| RHEL / CentOS / Rocky | openssh-clients |
openssh-server |
sshd |
/var/log/secure |
Inspecting Installed Versions
To verify client and server binary versions:
# Check client version
ssh -V
# Check server daemon binary version
sshd -V
5. Configuration Files and Directory Structure
Configuration File Locations
| File / Directory | Purpose | Mandatory Permissions | Important Details |
|---|---|---|---|
~/.ssh/ |
User SSH directory | 0700 (drwx------) |
Stores user keys, configuration, and host identity stores. |
~/.ssh/id_ed25519 |
Private key file | 0600 (-rw-------) |
Private identity key. Must never be world/group readable. |
~/.ssh/id_ed25519.pub |
Public key file | 0644 (-rw-r--r--) |
Paired public key to be installed on remote servers. |
~/.ssh/authorized_keys |
Remote authorization store | 0600 (-rw-------) |
List of client public keys allowed to log into this user account. |
~/.ssh/known_hosts |
Trusted host store | 0644 (-rw-r--r--) |
Database of public keys presented by trusted servers. |
~/.ssh/config |
Client configuration | 0600 (-rw-------) |
Client host aliases, jump host rules, and connection parameters. |
/etc/ssh/sshd_config |
Server daemon config | 0644 (-rw-r--r--) |
Master system configuration for incoming SSH connections. |
/etc/ssh/sshd_config.d/ |
Server drop-in directory | 0755 (drwxr-xr-x) |
Directory for modular server configuration files on modern distros. |
/etc/ssh/ssh_host_*_key |
Server identity host keys | 0600 (-rw-------) |
Server-side private host keys used to authenticate server identity. |
Core Server Directives (/etc/ssh/sshd_config)
Port 22: Defines the listening TCP port.PermitRootLogin noorprohibit-password: Controls whether root can log in via password or at all.PasswordAuthentication no: Disables plaintext password logins, enforcing key-based access.PubkeyAuthentication yes: Enables public key authentication.AllowUsers deploy admin: Restricts access exclusively to explicit user accounts.AllowGroups devops: Restricts access to users belonging to specific Linux groups.AuthorizedKeysFile .ssh/authorized_keys: Path to user public key definitions.
Validating Configuration Changes
Before reloading or restarting sshd, always test the syntax of /etc/ssh/sshd_config to avoid locking out administrative sessions:
sudo sshd -t
If sshd -t returns no output, the configuration syntax is valid.
6. Service Management and Common Operations
Service Control Commands
On Debian / Ubuntu:
# Check service status
systemctl status ssh
# Reload service (applies sshd_config without dropping active connections)
sudo sshd -t && sudo systemctl reload ssh
# Restart service
sudo systemctl restart ssh
On RHEL / CentOS / Rocky Linux:
# Check service status
systemctl status sshd
# Reload service
sudo sshd -t && sudo systemctl reload sshd
# Restart service
sudo systemctl restart sshd
Verifying Listening Sockets
To verify that sshd is active and listening on the designated port:
# Using ss utility
sudo ss -tlnp | grep ssh
# Alternatively, check specifically for port 22
sudo ss -tlnp | grep :22
Expected output:
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1042,fd=3))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=1042,fd=4))
7. Common Administrative Tasks and Workflows
Workflow 1: Generating Ed25519 Key Pairs
Generate an Ed25519 key pair with a descriptive comment:
ssh-keygen -t ed25519 -C "sysadmin@company.com"
When prompted, provide a strong passphrase to encrypt the private key on disk.
Workflow 2: Copying Public Keys to Remote Target
Transfer the generated public key to a remote server using ssh-copy-id:
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@192.168.1.50
This automatically creates ~/.ssh/ and appends the key to ~/.ssh/authorized_keys with proper file permissions on the target server.
Workflow 3: Hardening SSH Daemon Configuration
- Backup existing configuration:
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
- Edit
/etc/ssh/sshd_configto include security directives:
Port 22
PermitRootLogin prohibit-password
PasswordAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
AllowGroups sysadmin
- Test syntax and reload:
sudo sshd -t && sudo systemctl reload ssh # (or sshd on RHEL)
Workflow 4: Configuring Client Profiles (~/.ssh/config)
Simplify complex connections by defining shortcuts in ~/.ssh/config:
# Global defaults
Host *
ServerAliveInterval 60
ServerAliveCountMax 3
AddKeysToAgent yes
# Bastion / Jump Host definition
Host bastion
HostName jump.example.com
User deploy
Port 22
IdentityFile ~/.ssh/deploy_ed25519
# Private Target Hosts routed via Jump Host
Host app-*
User ubuntu
ProxyJump bastion
IdentityFile ~/.ssh/deploy_ed25519
With this configuration, connecting to an internal node is simplified to:
ssh app-01
Workflow 5: Protecting Long Workloads with tmux
- SSH into the remote node:
ssh user@host
- Start a named
tmuxsession:
tmux new -s agent-build
- Execute long-running builds, test suites, or autonomous AI agent scripts.
- Safely detach from the session by pressing
Ctrl-bthend. - Reconnect later after network interruption or disconnection:
tmux attach -t agent-build
8. CLI Command Reference and Cheatsheet
| Category | Command | Important Flags / Options | Example | Operational Purpose |
|---|---|---|---|---|
| Connecting | ssh |
-p (port) |
-i (identity file)
-J (jump host) | ssh -p 2222 -i ~/.ssh/deploy_key user@host | Connect to remote SSH server on specified port using explicit identity key. |
| Connecting | ssh | Remote Command Execution | ssh user@host 'uptime' | Execute command on remote host non-interactively and return stdout/stderr. |
| Connecting | ssh | -J | ssh -J bastion user@internal | Tunnel connection to internal host through bastion. |
| Port Forwarding | ssh | -L (Local Forward)
-N (No Shell) | ssh -N -L 8080:localhost:80 user@host | Bind local port 8080 to remote server’s local port 80. |
| Port Forwarding | ssh | -R (Remote Forward) | ssh -N -R 9000:localhost:3000 user@host | Expose local port 3000 on remote server’s port 9000. |
| Key Management | ssh-keygen | -t (type)
-C (comment)
-b (bits) | ssh-keygen -t ed25519 -C "you@host" | Generate an Ed25519 asymmetric SSH key pair. |
| Key Management | ssh-copy-id | -i (identity pub key) | ssh-copy-id -i ~/.ssh/id_ed25519.pub user@host | Install local public key into remote ~/.ssh/authorized_keys. |
| Key Management | ssh-keygen | -lf (fingerprint) | ssh-keygen -lf ~/.ssh/id_ed25519.pub | Display key fingerprint and length. |
| Key Management | ssh-keygen | -R (remove host) | ssh-keygen -R host.example.com | Remove modified or stale host key entries from ~/.ssh/known_hosts. |
| Agent Operations | ssh-add | Default | ssh-add ~/.ssh/id_ed25519 | Load private key into running ssh-agent memory. |
| Agent Operations | ssh-add | -l (list) | ssh-add -l | List fingerprints of all keys loaded in active agent. |
| File Transfer | scp | -P (port)
-r (recursive) | scp -P 2222 -r ./dir user@host:/var/www/ | Securely copy local directory to remote destination. |
| File Transfer | sftp | Default | sftp user@host | Open interactive secure file transfer session. |
| File Transfer | rsync | -e ssh | rsync -avz -e ssh src/ user@host:dest/ | Synchronize files incrementally over SSH transport. |
| Diagnostics | ssh | -v, -vv, -vvv | ssh -vvv user@host | Enable maximum client verbosity for debugging connection failures. |
| Server Admin | sshd | -t (test syntax) | sudo sshd -t | Test /etc/ssh/sshd_config syntax for errors prior to service reload. |
9. Logs, Monitoring and Diagnostics
Log File Locations and Systemd Integration
- Debian / Ubuntu: System authentication logs are written to
/var/log/auth.log. - RHEL / CentOS / Rocky: System authentication logs are written to
/var/log/secure. - Systemd Journal: Accessible via
journalctlon modern systemd-based Linux systems:
# Debian/Ubuntu systemd log tail
sudo journalctl -u ssh -n 50 --no-pager
# RHEL/CentOS systemd log tail
sudo journalctl -u sshd -n 50 --no-pager
Real-Time Diagnostics
1. Client-Side Verbose Debugging
When connections fail, execute ssh with increased verbosity:
ssh -vvv -i ~/.ssh/key user@host
Look for:
- KEX algorithm negotiation errors (
debug1: kex: algorithm...). - Key file offering sequence (
debug1: Offering public key:...). - Specific server rejection notices (
debug1: Authentications that can continue:...).
2. Server-Side Authentication Monitoring
Monitor authorization events on the target host while attempting a connection:
# Debian / Ubuntu
sudo tail -f /var/log/auth.log | grep sshd
# RHEL / CentOS
sudo tail -f /var/log/secure | grep sshd
Illustrative Log Output Analysis
Bad Permissions Warning
Authentication refused: bad ownership or modes for directory /home/deploy/.ssh
Meaning: The target user’s ~/.ssh directory has group/world-writable permissions (e.g., 0777). OpenSSH strictly rejects public keys when directories are insecure.
Public Key Rejection
Failed publickey for invalid user admin from 192.168.1.100 port 54322 ssh2
Meaning: The presented public key does not match any entry in ~/.ssh/authorized_keys, or the user account does not exist.
10. Troubleshooting Guide and Common Failure Scenarios
Problem: Permission denied (publickey)
Symptoms
Connection attempt immediately fails with output:
user@host: Permission denied (publickey).
Possible Causes
- Target user’s
~/.ssh/authorized_keysfile does not contain the client’s public key. - Insecure directory or file permissions on
~/.sshorauthorized_keys. - Client SSH agent or CLI is offering the wrong key identity.
- Target home directory is group- or world-writable.
Investigation and Diagnosis
- Execute client connection with verbose debugging:
ssh -vvv user@host
Check which key identities are offered during authentication. 2. Inspect server-side authentication logs:
sudo grep sshd /var/log/auth.log # Debian/Ubuntu
sudo grep sshd /var/log/secure # RHEL/CentOS
- Check permissions on the target server directory structure.
Resolution
Fix file and directory ownership and permissions on the remote server:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 600 ~/.ssh/id_ed25519
chown -R user:user ~/.ssh
Ensure the public key (~/.ssh/id_ed25519.pub) string is correctly copied as a single line inside ~/.ssh/authorized_keys.
Verification
Attempt connection explicitly specifying the key file:
ssh -i ~/.ssh/id_ed25519 user@host
Prevention
Use ssh-copy-id to install public keys; it enforces proper permissions automatically.
Problem: Connection refused
Symptoms
Client attempt terminates immediately:
ssh: connect to host 192.168.1.50 port 22: Connection refused
Possible Causes
- The
sshddaemon is not running on the target server. sshdis listening on a non-standard port instead of port 22.- Local host firewall (iptables/nftables/ufw) is actively rejecting packets on port 22.
Investigation and Diagnosis
- Verify if
sshdservice is active on the server:
systemctl status ssh # Debian/Ubuntu
systemctl status sshd # RHEL/CentOS
- Verify listening ports on the server:
sudo ss -tlnp | grep ssh
Resolution
- If the service is stopped, start it:
sudo systemctl start ssh # or sshd on RHEL
- If listening on a custom port (e.g., 2222), connect using
-p:
ssh -p 2222 user@host
Verification
Re-run ss -tlnp | grep ssh and test TCP connectivity using nc -zv host 22.
Problem: Connection timed out
Symptoms
Command hangs for several seconds before exiting:
ssh: connect to host 192.168.1.50 port 22: Connection timed out
Possible Causes
- Network path issue (incorrect IP address or routing failure).
- Firewall dropping packets silently (AWS Security Group, cloud network ACL, or ufw/firewalld).
- Target host is powered off or unreachable.
Investigation and Diagnosis
Unlike “Connection refused”, a timeout indicates packet drop.
- Test network-layer connection:
nc -zv 192.168.1.50 22
- Trace path routing:
traceroute 192.168.1.50
Resolution
Update security group rules or local firewall (e.g., ufw allow 22/tcp or cloud network security groups) to permit incoming traffic on port 22 from your source IP.
Problem: Host key verification failed
Symptoms
Client output warns of potential security risk:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Host key verification failed.
Possible Causes
- Target server was rebuilt or reinstalled, generating new host keys.
- Server IP address was reassigned to a different instance.
- Potential Man-In-The-Middle (MITM) attack.
Investigation and Diagnosis
Verify out-of-band (via cloud console or IPMI) whether the host was recently re-provisioned.
Resolution
If the host key change was legitimate, remove the obsolete host key entry from ~/.ssh/known_hosts:
ssh-keygen -R host.example.com
Verification
Reconnect to the server and verify the new fingerprint against your out-of-band server records:
ssh user@host.example.com
Problem: Too many authentication failures
Symptoms
Disconnect during handshake phase:
Received disconnect from 192.168.1.50 port 22:2: Too many authentication failures
Possible Causes
The client’s ssh-agent contains numerous loaded keys. The SSH client offers every loaded key sequentially until sshd hits its MaxAuthTries limit (default 6) and drops the connection.
Resolution
Force SSH to use only the identity key specified on the command line or in configuration:
ssh -o IdentitiesOnly=yes -i ~/.ssh/right_key user@host
Alternatively, configure IdentitiesOnly yes in ~/.ssh/config.
Problem: Locked out after editing sshd_config
Symptoms
Administrative users cannot establish new SSH sessions after altering /etc/ssh/sshd_config.
Possible Causes
Syntax error or invalid directive in /etc/ssh/sshd_config prevented sshd from restarting or caused it to reject valid credentials.
Investigation and Resolution
- Do not close your existing SSH terminal! Keep existing sessions open while testing.
- If completely locked out, gain out-of-band management access (Cloud Serial Console, IPMI, hypervisor console, or VNC).
- Execute syntax check to pinpoint error:
sudo sshd -t
- Fix syntax errors in
/etc/ssh/sshd_configand reloadsshdservice.
11. Security and Best Practices
- Disable Root Password Authentication: Force system administrators to log in using non-root user accounts with public keys, then escalate privileges using
sudo.
PermitRootLogin prohibit-password
- Disable Password Authentication: Eliminate brute-force attacks by requiring public-key authentication.
PasswordAuthentication no
- Prefer Modern Key Algorithms: Use Ed25519 public keys over legacy RSA keys.
- Restrict Authorized Key Capabilities: Enforce key options inside
~/.ssh/authorized_keysfor automated scripts or service accounts:
from="192.168.1.0/24",command="/usr/bin/backup.sh",no-pty ed25519 AAAAC3NzaC1lZDI1NTE5... backup-key
- Restrict Permitted Users: Limit login access using explicit allow-lists:
AllowUsers deploy admin
AllowGroups sysadmin devops
- Use Jump Hosts Instead of Agent Forwarding: Use
ProxyJump(-J) rather thanForwardAgent yesto avoid exposing agent sockets to compromised jump servers.
12. Performance and Resource Optimisation
Connection Multiplexing (ControlMaster)
Reusing an existing TCP socket for subsequent SSH connections to the same host eliminates handshake overhead and speeds up automation tools:
Add the following to ~/.ssh/config:
Host *
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h:%p
ControlPersist 10m
Create the socket directory:
mkdir -p ~/.ssh/sockets && chmod 700 ~/.ssh/sockets
Keeping Connections Alive
Prevent firewalls and NAT routers from dropping idle SSH sessions due to state timeouts:
- Client side (
~/.ssh/config):
ServerAliveInterval 60
ServerAliveCountMax 3
- Server side (
/etc/ssh/sshd_config):
ClientAliveInterval 60
ClientAliveCountMax 3
13. Backup, Recovery and Maintenance
Key Configurations to Backup
- Server State:
/etc/ssh/sshd_config,/etc/ssh/sshd_config.d/, and server host keys/etc/ssh/ssh_host_*. - User State:
~/.ssh/authorized_keysand~/.ssh/config.
Backup Command
sudo tar -czvf ssh_server_backup.tar.gz /etc/ssh/sshd_config /etc/ssh/ssh_host_*
Emergency Recovery Access
Always maintain secondary access mechanisms for critical servers:
- Ensure out-of-band access (Cloud Serial Console, IPMI, hypervisor serial console) is functional prior to applying major configuration changes.
14. Automation and Integration
Ansible Integration
Ansible relies on SSH as its default control transport. Configure ansible.cfg to leverage native SSH optimizations:
[defaults]
remote_user = deploy
private_key_file = ~/.ssh/id_ed25519
host_key_checking = True
[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=15m
CI/CD Pipeline Execution
In automated pipelines (e.g., GitHub Actions or Jenkins), pass non-interactive flags when executing commands or syncing files:
ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new user@deploy.example.com 'docker pull myapp:latest && docker restart myapp'
15. Practical Labs and Exercises
Lab 1: Setting Up Ed25519 Keys and Copying to Remote Server
- Objective: Generate an Ed25519 key pair and deploy it to a remote Linux instance.
- Instructions:
- Generate key pair on local system:
ssh-keygen -t ed25519 -C "lab1-user"
- Copy key to remote server:
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@<remote-ip>
- Verify key-based login without password prompt.
Lab 2: Safe Hardening of sshd_config
- Objective: Disable password authentication securely.
- Instructions:
- Open two separate terminal windows on your local machine.
- In Terminal 1, log into the server and edit
/etc/ssh/sshd_config, settingPasswordAuthentication no. - Validate configuration syntax:
sudo sshd -t
- Reload service:
sudo systemctl reload ssh # (or sshd)
- In Terminal 2 (without closing Terminal 1!), test a new connection:
ssh user@<remote-ip>
Lab 3: Terminal Session Persistence with tmux
- Objective: Run long-running jobs that survive SSH session disconnects.
- Instructions:
- SSH into remote host.
- Launch a named
tmuxsession:
tmux new -s persistent_task
- Start a long-running process (e.g.,
ping -i 2 127.0.0.1or a build script). - Detach from session using
Ctrl-bthend. - Disconnect SSH connection.
- Re-establish SSH connection and attach to session:
tmux attach -t persistent_task
Lab 4: Diagnosing Permissions Failures
- Objective: Troubleshoot and fix a broken
Permission denied (publickey)state. - Instructions:
- Break target user permissions intentionally on remote host:
chmod 777 ~/.ssh
- Attempt SSH connection from client and observe error.
- Fix permissions using correct file modes:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
- Verify successful login.
16. Quick Reference and Cheat Sheet
Important Configuration Files
| File / Directory | Purpose |
|---|---|
~/.ssh/config |
Client SSH connection aliases and options. |
~/.ssh/authorized_keys |
Remote server public key store for user account. |
/etc/ssh/sshd_config |
Main SSH daemon server configuration. |
/etc/ssh/sshd_config.d/ |
Modular server drop-in configuration directory. |
Important Log Locations
| Distribution | Log Path / Command |
|---|---|
| Debian / Ubuntu | /var/log/auth.log or journalctl -u ssh |
| RHEL / CentOS / Rocky | /var/log/secure or journalctl -u sshd |
Common Commands
| Task | Command |
|---|---|
| Generate Ed25519 Key | ssh-keygen -t ed25519 -C "user@host" |
| Copy Key to Server | ssh-copy-id -i ~/.ssh/id_ed25519.pub user@host |
| Test Configuration | sudo sshd -t |
| Connect via ProxyJump | ssh -J user@bastion user@internal |
| Verbose Connection Debug | ssh -vvv user@host |
Troubleshooting Quick Reference
| Symptom | Initial Investigation |
|---|---|
Permission denied (publickey) |
Check ~/.ssh permissions (700), authorized_keys (600), and inspect auth logs. |
Connection refused |
Verify systemctl status ssh/sshd and check listening port via ss -tlnp. |
Connection timed out |
Check network routing and test port access with nc -zv host 22. |
Host key verification failed |
Remove stale host key using ssh-keygen -R host after verifying identity out-of-band. |
Important Ports and Protocols
| Port | Protocol | Purpose |
|---|---|---|
| 22 | TCP | Default OpenSSH Server Listening Port. |
17. Further Reading and References
- OpenSSH Client Manual (
ssh(1)): https://man7.org/linux/man-pages/man1/ssh.1.html - OpenSSH Daemon Configuration Manual (
sshd_config(5)): https://man7.org/linux/man-pages/man5/sshd_config.5.html - OpenSSH Key Management Manual (
ssh-keygen(1)): https://man7.org/linux/man-pages/man1/ssh-keygen.1.html - Official OpenSSH Documentation Index: https://www.openssh.com/manual.html