Domain 3Systems & IaCCORE

(Istanbul) Identifying and Deleting Orphaned AWS EBS Volumes via AWS CLI

A practical guide to detecting and safely deleting unattached (orphaned) AWS EBS volumes using AWS CLI queries, JMESPath filtering, and Unix command piping.

Domain: Domain 3: Systems & IaC•Updated: 2026-11-11
#AWS CLI#EBS#FinOps#Storage Cleanup#Cloud Ops#JMESPath

1. Introduction

In cloud environments, unattached Elastic Block Store (EBS) volumes—often referred to as “orphaned” or “zombie” volumes—are one of the most common causes of silent cost leaks. When an Amazon EC2 instance is terminated without configured automatic volume deletion, or when secondary storage volumes are manually detached during maintenance and forgotten, AWS continues to bill for the provisioned block storage capacity regardless of whether any compute instance is actively reading or writing to it.

For Cloud Engineers, Systems Administrators, and Site Reliability Engineers (SREs), implementing effective Financial Operations (FinOps) hygiene is just as critical as maintaining service uptime. Cleaning up orphaned infrastructure requires precision: operating at scale requires automated CLI commands and scripts, but accidental deletion of attached boot or data drives can cause immediate outage or data loss.

This article details a complete, step-by-step walkthrough of identifying and safely purging orphaned EBS volumes in an AWS environment (simulated via the Floci emulator in the SadServers “Istanbul” scenario) using the AWS CLI, JMESPath querying, and standard Linux text-processing pipelines.


2. Problem Description and Environment

Original Problem Statement

Scenario: “Istanbul”: Delete unattached AWS EBS volumes Finance Findings: Finance identified orphaned EBS volumes accruing unnecessary storage costs in the AWS account. Objective: Find every EBS volume that is not attached to an EC2 instance and delete it. Constraints:

  • Do not delete volumes that are attached (in-use).
  • Do not stop or terminate any running EC2 instances.
  • The AWS CLI is configured to interact with a local endpoint (Floci emulator).
  • Verification script located at /home/admin/agent/check.sh.

Environment Details

  • Environment: Linux terminal with AWS CLI preconfigured against local Floci endpoint.
  • Target AWS Subsystem: Amazon EC2 / Elastic Block Store (EBS) API.
  • User Privileges: Standard user with root/sudo access (admin@i-05882b9baca7c3742).
  • Validation Standard: Execution of /home/admin/agent/check.sh returning OK.

3. Initial Investigation and Hypotheses

When approaching cloud storage remediation, jumping straight to deletion commands without inspecting state runs significant risk. A systematic investigation begins by establishing a complete resource baseline.

Key Questions to Answer Before Action

  1. How many total EBS volumes exist in the target region/account?
  2. What are their respective operational states (available vs. in-use)?
  3. For in-use volumes, which EC2 instances are they attached to, and what mount devices do they occupy?
  4. What is the total storage capacity (in GiB) currently orphaned?

Investigation Hypothesis Matrix

Hypothesis Supporting Logic Risk if Unverified Verification Step
Volumes in available state are fully detached. AWS EC2 API marks any volume not linked to an instance block device mapping as available. Low risk, but must confirm non-attachment in response metadata. Inspect Attachments list in describe-volumes output.
Volumes in in-use state belong to active workloads. in-use volumes are attached to running or stopped EC2 instances. High risk—deleting an attached volume can crash applications or corrupt filesystems. Filter API results to strictly exclude in-use states.

4. Troubleshooting & Remediation Process

Step 1: Inventory All Account EBS Volumes

Objective

Retrieve a full inventory of all EBS volumes in the account, capturing Volume ID, Size, Attachment Status, and Instance Bindings in a human-readable table format.

Command

aws ec2 describe-volumes --query "Volumes[*].{ID:VolumeId, State:State, Size:Size, Attached:Attachments}" --output table

Command Explanation

  • aws ec2 describe-volumes: Calls the EC2 API DescribeVolumes endpoint to retrieve metadata for all volumes accessible by the configured credentials.
  • --query "Volumes[*].{ID:VolumeId, State:State, Size:Size, Attached:Attachments}": Applies client-side JMESPath filtering to extract only the four relevant key-value pairs into custom named attributes (ID, State, Size, Attached).
  • --output table: Formats the JSON API payload into a structured ASCII table.

Output and Analysis

---------------------------------------------------------------
|                        DescribeVolumes                      |
+---------------+---------------------------------------------+
|  ID           |  vol-4f4f9629f9fcc594b                      |
|  Size         |  100                                        |
|  State        |  available                                  |
+---------------+---------------------------------------------+
|                        DescribeVolumes                      |
+---------------+---------------------------------------------+
|  ID           |  vol-fa6767ceaf58cc090                      |
|  Size         |  8                                          |
|  State        |  in-use                                     |
+---------------+---------------------------------------------+
||                          Attached                         ||
|+----------------------+------------------------------------+|
||  AttachTime          |  2026-09-27T19:45:03.149000+00:00  ||
||  DeleteOnTermination |  True                              ||
||  Device              |  /dev/xvda                         ||
||  InstanceId          |  i-1973d0bb4c69e4b85               ||
||  State               |  attached                          ||
||  VolumeId            |  vol-fa6767ceaf58cc090             ||
|+----------------------+------------------------------------+|
|                        DescribeVolumes                      |
+---------------+---------------------------------------------+
|  ID           |  vol-a9c8805a079e22c2c                      |
|  Size         |  8                                          |
|  State        |  in-use                                     |
+---------------+---------------------------------------------+
||                          Attached                         ||
|+----------------------+------------------------------------+|
||  AttachTime          |  2026-09-27T19:45:04.190000+00:00  ||
||  DeleteOnTermination |  True                              ||
||  Device              |  /dev/xvda                         ||
||  InstanceId          |  i-c8e5a468839883897               ||
||  State               |  attached                          ||
||  VolumeId            |  vol-a9c8805a079e22c2c             ||
|+----------------------+------------------------------------+|
|                        DescribeVolumes                      |
+---------------+---------------------------------------------+
|  ID           |  vol-b04df433874905862                      |
|  Size         |  50                                         |
|  State        |  available                                  |
+---------------+---------------------------------------------+
|                        DescribeVolumes                      |
+---------------+---------------------------------------------+
|  ID           |  vol-c7492185598b40da5                      |
|  Size         |  8                                          |
|  State        |  available                                  |
+---------------+---------------------------------------------+
|                        DescribeVolumes                      |
+---------------+---------------------------------------------+
|  ID           |  vol-9d8fee9aa212dc531                      |
|  Size         |  20                                         |
|  State        |  in-use                                     |
+---------------+---------------------------------------------+
||                          Attached                         ||
|+----------------------+------------------------------------+|
||  AttachTime          |  2026-09-27T19:45:21.621000+00:00  ||
||  DeleteOnTermination |  False                             ||
||  Device              |  /dev/sdf                          ||
||  InstanceId          |  i-1973d0bb4c69e4b85               ||
||  State               |  attached                          ||
||  VolumeId            |  vol-9d8fee9aa212dc531             ||
|+----------------------+------------------------------------+|

Analysis Findings:

  1. Exactly 6 volumes exist in the environment.
  2. 3 volumes are attached (in-use):
  • vol-fa6767ceaf58cc090 (8 GiB, attached to i-1973d0bb4c69e4b85 on /dev/xvda)
  • vol-a9c8805a079e22c2c (8 GiB, attached to i-c8e5a468839883897 on /dev/xvda)
  • vol-9d8fee9aa212dc531 (20 GiB, attached to i-1973d0bb4c69e4b85 on /dev/sdf)
  1. 3 volumes are unattached (available):
  • vol-4f4f9629f9fcc594b (100 GiB)
  • vol-b04df433874905862 (50 GiB)
  • vol-c7492185598b40da5 (8 GiB)

Decision

Target the 3 identified available volumes (vol-4f4f9629f9fcc594b, vol-b04df433874905862, vol-c7492185598b40da5) for deletion using an automated API query and pipeline.


Step 2: Construct and Execute Deletion Pipeline

Objective

Extract only the IDs of unattached volumes using API-level filtering, format the IDs as plain text, and pass them to aws ec2 delete-volume via xargs.

Command

aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query "Volumes[*].VolumeId" \
  --output text | xargs -n 1 aws ec2 delete-volume --volume-id

Command Explanation

  • --filters Name=status,Values=available: Server-side API filter instructing the EC2 service to return only volume objects whose status attribute equals available.
  • --query "Volumes[*].VolumeId": JMESPath projection extracting a flat list of Volume ID strings.
  • --output text: Outputs raw string identifiers separated by tabs/newlines without JSON quotes or array syntax.
  • |: Standard Unix pipe redirecting the text stream of volume IDs to the input of xargs.
  • xargs -n 1 aws ec2 delete-volume --volume-id: Reads one argument at a time (-n 1) from stdout and executes aws ec2 delete-volume --volume-id <VolumeID>.

Output and Analysis

The command completed silently without returning error messages, indicating all three HTTP DELETE requests succeeded.

Decision

Proceed to verification phase to confirm that no unattached volumes remain and that in-use volumes were unaffected.


5. Root Cause Analysis

FinOps Failure Mechanism

In AWS, storage capacity and compute capacity are decoupled. An EBS volume exists independently of EC2 instances within an Availability Zone.

When an EC2 instance is terminated:

  1. Root Block Device Behavior: If DeleteOnTermination is set to true (default for root volumes), AWS automatically deletes the root EBS volume upon instance termination.
  2. Secondary Device Behavior: If DeleteOnTermination is set to false (often the case for non-root data volumes attached at device nodes like /dev/sdf or /dev/sdb), the volume remains intact in the available state after instance termination.
  3. Manual Detachment: If an administrator manually detaches a volume via API or Console and does not explicitly delete it, the volume remains in available state indefinitely.
+-----------------------------------------------------------------------------------+
|                                 EBS Storage State                                 |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  +------------------------+                    +-------------------------------+  |
|  | EC2 Instance Attached  |                    |  Unattached / Orphaned Volume |  |
|  | State: "in-use"        |                    |  State: "available"           |  |
|  +-----------+------------+                    +---------------+---------------+  |
|              |                                                 |                  |
|              | Detach / Instance Termination                   | Delete Volume    |
|              | (DeleteOnTermination=false)                     | API Request      |
|              v                                                 v                  |
|  +------------------------+                    +-------------------------------+  |
|  | Orphaned EBS Volume    |------------------->| Volume Permanently Purged     |  |
|  | (Billing accures!)     |   Cleanup Pipeline | (Costs halted)                |  |
|  +------------------------+                    +-------------------------------+  |
|                                                                                   |
+-----------------------------------------------------------------------------------+

In this scenario, three volumes totaling 158 GiB remained in the available state. Without automated lifecycle rules or scheduled FinOps purges, these volumes would continue to accrue monthly AWS storage fees (e.g., ~$0.08–$0.10 per GB-month for standard gp3 volumes).


6. Solution and Remediation

The remediation executed on the system consisted of the following sequence:

# Execute server-side query and delete pipeline
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query "Volumes[*].VolumeId" \
  --output text | xargs -n 1 aws ec2 delete-volume --volume-id

Affected Resources Summary

  • vol-4f4f9629f9fcc594b (100 GiB): Deleted
  • vol-b04df433874905862 (50 GiB): Deleted
  • vol-c7492185598b40da5 (8 GiB): Deleted

7. Verification and Validation

Verification requires confirming both positive and negative conditions:

  1. Positive Validation: All unattached (available) volumes have been purged.
  2. Negative Validation: All attached (in-use) volumes remain unharmed and functional.
  3. Scenario Validation: Execution of the scenario verification runner.

Step 1: Re-query Storage Posture

aws ec2 describe-volumes --query "Volumes[*].{ID:VolumeId, State:State}" --output table

Output:

-------------------------------------
|          DescribeVolumes          |
+------------------------+----------+
|           ID           |  State   |
+------------------------+----------+
|  vol-fa6767ceaf58cc090 |  in-use  |
|  vol-a9c8805a079e22c2c |  in-use  |
|  vol-9d8fee9aa212dc531 |  in-use  |
+------------------------+----------+

Result: Exactly 3 volumes remain in the account, all of which are in in-use state. Zero available volumes remain.

Step 2: Run Scenario Check Script

agent/check.sh

Output:

OK

Result: Challenge test harness confirms complete and accurate solution.


8. Alternative Troubleshooting & Automation Approaches

While a command pipeline combining aws ec2 describe-volumes and xargs works well for an interactive shell, different environments call for alternative toolsets.

Approach 1: AWS CLI Native Iteration (Bash for Loop)

If xargs is undesirable or complex formatting is needed, standard shell loops provide explicit control.

VOLUMES=$(aws ec2 describe-volumes --filters Name=status,Values=available --query "Volumes[*].VolumeId" --output text)

for VOL in $VOLUMES; do
  echo "Deleting volume: $VOL"
  aws ec2 delete-volume --volume-id "$VOL"
done

Trade-offs:

  • Advantages: Highly readable, easy to inject logging or conditional logic (echo, dry-run prompts).
  • Disadvantages: Slower execution due to sequential subshell loop processing.

Approach 2: Programmatic Automation via Python (boto3)

For enterprise cloud management, Python scripts running inside AWS Lambda functions or CI/CD runners offer robust error handling and safety features.

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')
    
    # Retrieve all unattached volumes
    response = ec2.describe_volumes(
        Filters=[{'Name': 'status', 'Values': ['available']}]
    )
    
    deleted_volumes = []
    for vol in response.get('Volumes', []):
        vol_id = vol['VolumeId']
        size = vol['Size']
        print(f"Purging unattached volume {vol_id} ({size} GiB)")
        
        # Delete volume
        ec2.delete_volume(VolumeId=vol_id)
        deleted_volumes.append(vol_id)
        
    return {
        'statusCode': 200,
        'deleted_volumes': deleted_volumes
    }

Trade-offs:

  • Advantages: Can be triggered on a cron schedule (Amazon EventBridge), supports structured logging, retry logic, and Slack/Teams alerting.
  • Disadvantages: Requires maintaining code, IAM execution roles, and deployment infrastructure.

Approach Comparison Table

Approach Ideal Use Case Speed Safety & Control Overhead
AWS CLI Pipeline (xargs) Interactive ad-hoc cleanup, terminal work. Very Fast High (if query is validated first) Minimal
Bash for Loop Shell scripts requiring inline logging/prompts. Fast High (supports explicit confirmation steps) Low
Python boto3 Script Automated scheduled maintenance (Lambda). Fast Maximum (full exception handling & safety checks) Moderate
Terraform / IaC Managing declarative state environments. N/A High (detects drift via terraform plan) High

9. AWS CLI & Linux Command Deep Dive

Command 1: aws ec2 describe-volumes

Purpose

Queries the AWS EC2 API endpoint to fetch detailed metadata regarding EBS volumes in the targeted region.

Syntax & Flags

aws ec2 describe-volumes [options]
Flag / Option Type Purpose & Troubleshooting Context
--filters List Applies server-side filtering evaluated directly by the AWS API before returning data.
--query String Applies client-side JMESPath expression to structure, format, or extract specific fields.
--output String Controls output formatting style (json, text, table, yaml).
--volume-ids List Restricts output to specific volume ID strings (e.g., --volume-ids vol-12345).
--max-items Integer Specifies total number of items to return for pagination handling.

Useful --filters Examples for SREs:

  • Filter by status: --filters Name=status,Values=available
  • Filter by volume type: --filters Name=volume-type,Values=gp3
  • Filter by tag: --filters Name=tag:Environment,Values=production

Command 2: aws ec2 delete-volume

Purpose

Permanently deletes a specified EBS volume. The volume must be in the available state.

Syntax & Flags

aws ec2 delete-volume --volume-id <vol-id> [options]
Flag / Option Type Purpose & Troubleshooting Context
--volume-id String Required. The ID of the EBS volume to delete.
--dry-run Flag Simulates the API request to check for proper IAM permissions without deleting the volume.

Command 3: xargs

Purpose

A standard Unix utility that builds and executes command lines from standard input (stdin).

Syntax & Flags

command | xargs [options] [utility [argument ...]]
Option Purpose Example
-n <max-args> Limit the number of arguments passed to each command invocation. xargs -n 1 aws ec2 delete-volume --volume-id
-P <max-procs> Run up to max-procs processes in parallel (concurrent execution). xargs -n 1 -P 4 aws ec2 delete-volume --volume-id
-I <replace-str> Replaces occurrences of replace-str with input items. xargs -I {} aws ec2 delete-volume --volume-id {}
-t / --verbose Prints the full command line to stderr before executing it. Useful for dry-running pipelines safely.

10. Underlying Concepts and Architecture

1. Amazon EBS Volume Lifecycle States

Understanding state transitions in block storage prevents invalid API calls:

                  +-------------------+
                  |   creating        |
                  +---------+---------+
                            |
                            v
+------------------+  Attach   +--------------------+
|  in-use          |<----------|  available         |
|  (Attached to    |---------->|  (Orphaned / Ready |
|   EC2 Instance)  |  Detach   |   for Attachment)  |
+------------------+           +---------+----------+
                                         |
                                         | Delete Volume
                                         v
                               +--------------------+
                               |  deleting / deleted|
                               +--------------------+
  • creating: Provisioning volume blocks in the AWS storage cluster.
  • available: Fully provisioned and unattached to any EC2 instance. Billed per GB-month.
  • in-use: Attached to an EC2 instance via Xen/KVM hypervisor block device attachment.
  • deleting: Deletion request received; underlying block references are being scrubbed.

2. Server-Side Filtering (--filters) vs. Client-Side Querying (--query)

Understanding where data filtering occurs is vital when working with large cloud deployments:

  • Server-Side Filtering (--filters):

  • Processed by the AWS API endpoint.

  • Reduces network transfer payload by returning only matching records.

  • Faster execution when querying accounts with thousands of resources.

  • Client-Side Querying (--query):

  • Processed locally by the AWS CLI binary using JMESPath.

  • Evaluated after the raw response is returned over the network.

  • Used for formatting, array projection, and extracting exact strings for Unix pipelines.


11. Real-World Sysadmin & SRE Applications

Production Detection & Observability

In enterprise production accounts, relying on manual CLI runs is insufficient. SRE teams use automated governance patterns:

  1. AWS Cost Explorer & Anomaly Detection: Configure alerts when storage category spending exceeds historical baselines.
  2. AWS Config Rules: Deploy the managed rule ec2-volume-inuse-check to continuously evaluate whether EBS volumes are attached to EC2 instances. Non-compliant resources trigger CloudWatch events.
  3. Automated Janitor Lambdas: Implement automated routines that tag unattached volumes with OrphanedDate=YYYY-MM-DD. If a volume remains unattached for 7 days, the routine automatically takes a final EBS snapshot and purges the volume.

Runbook Strategy: Safety Measures Prior to Bulk Purging

Before deleting unattached storage in a production environment:

  1. Take Safety Snapshots: Always create an EBS snapshot (aws ec2 create-snapshot) prior to volume deletion in production.
  2. Verify Tag Metadata: Check for tags such as DoNotDelete or BackupTarget.
  3. Use Dry-Run Verification: Append --dry-run to CLI calls to ensure IAM permissions and parameters are valid without triggering state changes.

12. Further Learning & Experiments

Safe Local Experiments

To practice AWS CLI filtering without risking real AWS resources or incurring costs, use LocalStack or Floci in a local container:

  1. Create Test Volumes:
aws ec2 create-volume --availability-zone us-east-1a --size 10 --volume-type gp3
aws ec2 create-volume --availability-zone us-east-1a --size 5 --volume-type gp2
  1. Experiment with Complex JMESPath Filtering: Extract volume IDs for all gp3 volumes that are available:
aws ec2 describe-volumes \
  --query "Volumes[?State=='available' && VolumeType=='gp3'].VolumeId" \
  --output text
  1. Test Safety Flags: Execute a delete command with --dry-run to inspect CLI authorization checks:
aws ec2 delete-volume --volume-id vol-1234567890abcdef0 --dry-run

Official References & Documentation


13. Key Takeaways

  1. Cloud FinOps Responsibility: Unattached (available) EBS volumes accumulate billing charges indefinitely until explicitly deleted or cleaned up by lifecycle policies.
  2. Distinguish available vs. in-use: Always verify state and attachment fields prior to issuing deletion calls. in-use volumes belong to running/stopped instances and must be preserved.
  3. Leverage Server-Side Filtering: Combine --filters Name=status,Values=available with client-side --query projections for efficient, high-performance CLI pipelines.
  4. Unix Pipeline Efficiency: Pipelining aws ec2 describe-volumes ... --output text into xargs -n 1 aws ec2 delete-volume --volume-id enables precise, single-command remediation.
  5. Production Safety First: In live enterprise environments, combine automated purges with pre-deletion snapshot backups and grace-period tagging strategies.

14. Quick Troubleshooting Reference / Runbook

# ==============================================================================
# EBS ORPHANED VOLUME PURGE RUNBOOK
# ==============================================================================

# 1. Audit all volumes in current region (Inspect state & attachments)
aws ec2 describe-volumes \
  --query "Volumes[*].{ID:VolumeId, State:State, Size:Size, Attached:Attachments}" \
  --output table

# 2. Count total orphaned storage capacity (GiB)
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query "sum(Volumes[*].Size)" \
  --output text

# 3. List IDs of unattached volumes strictly
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query "Volumes[*].VolumeId" \
  --output text

# 4. Dry-run safety test prior to deletion
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query "Volumes[*].VolumeId" \
  --output text | xargs -n 1 -I {} aws ec2 delete-volume --volume-id {} --dry-run

# 5. Execute purge of all unattached volumes
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query "Volumes[*].VolumeId" \
  --output text | xargs -n 1 aws ec2 delete-volume --volume-id

# 6. Verify remaining volumes are strictly in-use
aws ec2 describe-volumes \
  --query "Volumes[*].{ID:VolumeId, State:State}" \
  --output table