1. Introduction
In cloud environments, unattached Elastic Block Store (EBS) volumes—often referred to as “orphaned” or “zombie” volumes—are one of the most common causes of silent cost leaks. When an Amazon EC2 instance is terminated without configured automatic volume deletion, or when secondary storage volumes are manually detached during maintenance and forgotten, AWS continues to bill for the provisioned block storage capacity regardless of whether any compute instance is actively reading or writing to it.
For Cloud Engineers, Systems Administrators, and Site Reliability Engineers (SREs), implementing effective Financial Operations (FinOps) hygiene is just as critical as maintaining service uptime. Cleaning up orphaned infrastructure requires precision: operating at scale requires automated CLI commands and scripts, but accidental deletion of attached boot or data drives can cause immediate outage or data loss.
This article details a complete, step-by-step walkthrough of identifying and safely purging orphaned EBS volumes in an AWS environment (simulated via the Floci emulator in the SadServers “Istanbul” scenario) using the AWS CLI, JMESPath querying, and standard Linux text-processing pipelines.
2. Problem Description and Environment
Original Problem Statement
Scenario: “Istanbul”: Delete unattached AWS EBS volumes Finance Findings: Finance identified orphaned EBS volumes accruing unnecessary storage costs in the AWS account. Objective: Find every EBS volume that is not attached to an EC2 instance and delete it. Constraints:
- Do not delete volumes that are attached (
in-use).- Do not stop or terminate any running EC2 instances.
- The AWS CLI is configured to interact with a local endpoint (Floci emulator).
- Verification script located at
/home/admin/agent/check.sh.
Environment Details
- Environment: Linux terminal with AWS CLI preconfigured against local Floci endpoint.
- Target AWS Subsystem: Amazon EC2 / Elastic Block Store (EBS) API.
- User Privileges: Standard user with root/sudo access (
admin@i-05882b9baca7c3742). - Validation Standard: Execution of
/home/admin/agent/check.shreturningOK.
3. Initial Investigation and Hypotheses
When approaching cloud storage remediation, jumping straight to deletion commands without inspecting state runs significant risk. A systematic investigation begins by establishing a complete resource baseline.
Key Questions to Answer Before Action
- How many total EBS volumes exist in the target region/account?
- What are their respective operational states (
availablevs.in-use)? - For
in-usevolumes, which EC2 instances are they attached to, and what mount devices do they occupy? - What is the total storage capacity (in GiB) currently orphaned?
Investigation Hypothesis Matrix
| Hypothesis | Supporting Logic | Risk if Unverified | Verification Step |
|---|---|---|---|
Volumes in available state are fully detached. |
AWS EC2 API marks any volume not linked to an instance block device mapping as available. |
Low risk, but must confirm non-attachment in response metadata. | Inspect Attachments list in describe-volumes output. |
Volumes in in-use state belong to active workloads. |
in-use volumes are attached to running or stopped EC2 instances. |
High risk—deleting an attached volume can crash applications or corrupt filesystems. | Filter API results to strictly exclude in-use states. |
4. Troubleshooting & Remediation Process
Step 1: Inventory All Account EBS Volumes
Objective
Retrieve a full inventory of all EBS volumes in the account, capturing Volume ID, Size, Attachment Status, and Instance Bindings in a human-readable table format.
Command
aws ec2 describe-volumes --query "Volumes[*].{ID:VolumeId, State:State, Size:Size, Attached:Attachments}" --output table
Command Explanation
aws ec2 describe-volumes: Calls the EC2 APIDescribeVolumesendpoint to retrieve metadata for all volumes accessible by the configured credentials.--query "Volumes[*].{ID:VolumeId, State:State, Size:Size, Attached:Attachments}": Applies client-side JMESPath filtering to extract only the four relevant key-value pairs into custom named attributes (ID,State,Size,Attached).--output table: Formats the JSON API payload into a structured ASCII table.
Output and Analysis
---------------------------------------------------------------
| DescribeVolumes |
+---------------+---------------------------------------------+
| ID | vol-4f4f9629f9fcc594b |
| Size | 100 |
| State | available |
+---------------+---------------------------------------------+
| DescribeVolumes |
+---------------+---------------------------------------------+
| ID | vol-fa6767ceaf58cc090 |
| Size | 8 |
| State | in-use |
+---------------+---------------------------------------------+
|| Attached ||
|+----------------------+------------------------------------+|
|| AttachTime | 2026-09-27T19:45:03.149000+00:00 ||
|| DeleteOnTermination | True ||
|| Device | /dev/xvda ||
|| InstanceId | i-1973d0bb4c69e4b85 ||
|| State | attached ||
|| VolumeId | vol-fa6767ceaf58cc090 ||
|+----------------------+------------------------------------+|
| DescribeVolumes |
+---------------+---------------------------------------------+
| ID | vol-a9c8805a079e22c2c |
| Size | 8 |
| State | in-use |
+---------------+---------------------------------------------+
|| Attached ||
|+----------------------+------------------------------------+|
|| AttachTime | 2026-09-27T19:45:04.190000+00:00 ||
|| DeleteOnTermination | True ||
|| Device | /dev/xvda ||
|| InstanceId | i-c8e5a468839883897 ||
|| State | attached ||
|| VolumeId | vol-a9c8805a079e22c2c ||
|+----------------------+------------------------------------+|
| DescribeVolumes |
+---------------+---------------------------------------------+
| ID | vol-b04df433874905862 |
| Size | 50 |
| State | available |
+---------------+---------------------------------------------+
| DescribeVolumes |
+---------------+---------------------------------------------+
| ID | vol-c7492185598b40da5 |
| Size | 8 |
| State | available |
+---------------+---------------------------------------------+
| DescribeVolumes |
+---------------+---------------------------------------------+
| ID | vol-9d8fee9aa212dc531 |
| Size | 20 |
| State | in-use |
+---------------+---------------------------------------------+
|| Attached ||
|+----------------------+------------------------------------+|
|| AttachTime | 2026-09-27T19:45:21.621000+00:00 ||
|| DeleteOnTermination | False ||
|| Device | /dev/sdf ||
|| InstanceId | i-1973d0bb4c69e4b85 ||
|| State | attached ||
|| VolumeId | vol-9d8fee9aa212dc531 ||
|+----------------------+------------------------------------+|
Analysis Findings:
- Exactly 6 volumes exist in the environment.
- 3 volumes are attached (
in-use):
vol-fa6767ceaf58cc090(8 GiB, attached toi-1973d0bb4c69e4b85on/dev/xvda)vol-a9c8805a079e22c2c(8 GiB, attached toi-c8e5a468839883897on/dev/xvda)vol-9d8fee9aa212dc531(20 GiB, attached toi-1973d0bb4c69e4b85on/dev/sdf)
- 3 volumes are unattached (
available):
vol-4f4f9629f9fcc594b(100 GiB)vol-b04df433874905862(50 GiB)vol-c7492185598b40da5(8 GiB)
Decision
Target the 3 identified available volumes (vol-4f4f9629f9fcc594b, vol-b04df433874905862, vol-c7492185598b40da5) for deletion using an automated API query and pipeline.
Step 2: Construct and Execute Deletion Pipeline
Objective
Extract only the IDs of unattached volumes using API-level filtering, format the IDs as plain text, and pass them to aws ec2 delete-volume via xargs.
Command
aws ec2 describe-volumes \
--filters Name=status,Values=available \
--query "Volumes[*].VolumeId" \
--output text | xargs -n 1 aws ec2 delete-volume --volume-id
Command Explanation
--filters Name=status,Values=available: Server-side API filter instructing the EC2 service to return only volume objects whose status attribute equalsavailable.--query "Volumes[*].VolumeId": JMESPath projection extracting a flat list of Volume ID strings.--output text: Outputs raw string identifiers separated by tabs/newlines without JSON quotes or array syntax.|: Standard Unix pipe redirecting the text stream of volume IDs to the input ofxargs.xargs -n 1 aws ec2 delete-volume --volume-id: Reads one argument at a time (-n 1) from stdout and executesaws ec2 delete-volume --volume-id <VolumeID>.
Output and Analysis
The command completed silently without returning error messages, indicating all three HTTP DELETE requests succeeded.
Decision
Proceed to verification phase to confirm that no unattached volumes remain and that in-use volumes were unaffected.
5. Root Cause Analysis
FinOps Failure Mechanism
In AWS, storage capacity and compute capacity are decoupled. An EBS volume exists independently of EC2 instances within an Availability Zone.
When an EC2 instance is terminated:
- Root Block Device Behavior: If
DeleteOnTerminationis set totrue(default for root volumes), AWS automatically deletes the root EBS volume upon instance termination. - Secondary Device Behavior: If
DeleteOnTerminationis set tofalse(often the case for non-root data volumes attached at device nodes like/dev/sdfor/dev/sdb), the volume remains intact in theavailablestate after instance termination. - Manual Detachment: If an administrator manually detaches a volume via API or Console and does not explicitly delete it, the volume remains in
availablestate indefinitely.
+-----------------------------------------------------------------------------------+
| EBS Storage State |
+-----------------------------------------------------------------------------------+
| |
| +------------------------+ +-------------------------------+ |
| | EC2 Instance Attached | | Unattached / Orphaned Volume | |
| | State: "in-use" | | State: "available" | |
| +-----------+------------+ +---------------+---------------+ |
| | | |
| | Detach / Instance Termination | Delete Volume |
| | (DeleteOnTermination=false) | API Request |
| v v |
| +------------------------+ +-------------------------------+ |
| | Orphaned EBS Volume |------------------->| Volume Permanently Purged | |
| | (Billing accures!) | Cleanup Pipeline | (Costs halted) | |
| +------------------------+ +-------------------------------+ |
| |
+-----------------------------------------------------------------------------------+
In this scenario, three volumes totaling 158 GiB remained in the available state. Without automated lifecycle rules or scheduled FinOps purges, these volumes would continue to accrue monthly AWS storage fees (e.g., ~$0.08–$0.10 per GB-month for standard gp3 volumes).
6. Solution and Remediation
The remediation executed on the system consisted of the following sequence:
# Execute server-side query and delete pipeline
aws ec2 describe-volumes \
--filters Name=status,Values=available \
--query "Volumes[*].VolumeId" \
--output text | xargs -n 1 aws ec2 delete-volume --volume-id
Affected Resources Summary
vol-4f4f9629f9fcc594b(100 GiB): Deletedvol-b04df433874905862(50 GiB): Deletedvol-c7492185598b40da5(8 GiB): Deleted
7. Verification and Validation
Verification requires confirming both positive and negative conditions:
- Positive Validation: All unattached (
available) volumes have been purged. - Negative Validation: All attached (
in-use) volumes remain unharmed and functional. - Scenario Validation: Execution of the scenario verification runner.
Step 1: Re-query Storage Posture
aws ec2 describe-volumes --query "Volumes[*].{ID:VolumeId, State:State}" --output table
Output:
-------------------------------------
| DescribeVolumes |
+------------------------+----------+
| ID | State |
+------------------------+----------+
| vol-fa6767ceaf58cc090 | in-use |
| vol-a9c8805a079e22c2c | in-use |
| vol-9d8fee9aa212dc531 | in-use |
+------------------------+----------+
Result: Exactly 3 volumes remain in the account, all of which are in in-use state. Zero available volumes remain.
Step 2: Run Scenario Check Script
agent/check.sh
Output:
OK
Result: Challenge test harness confirms complete and accurate solution.
8. Alternative Troubleshooting & Automation Approaches
While a command pipeline combining aws ec2 describe-volumes and xargs works well for an interactive shell, different environments call for alternative toolsets.
Approach 1: AWS CLI Native Iteration (Bash for Loop)
If xargs is undesirable or complex formatting is needed, standard shell loops provide explicit control.
VOLUMES=$(aws ec2 describe-volumes --filters Name=status,Values=available --query "Volumes[*].VolumeId" --output text)
for VOL in $VOLUMES; do
echo "Deleting volume: $VOL"
aws ec2 delete-volume --volume-id "$VOL"
done
Trade-offs:
- Advantages: Highly readable, easy to inject logging or conditional logic (
echo, dry-run prompts). - Disadvantages: Slower execution due to sequential subshell loop processing.
Approach 2: Programmatic Automation via Python (boto3)
For enterprise cloud management, Python scripts running inside AWS Lambda functions or CI/CD runners offer robust error handling and safety features.
import boto3
def lambda_handler(event, context):
ec2 = boto3.client('ec2')
# Retrieve all unattached volumes
response = ec2.describe_volumes(
Filters=[{'Name': 'status', 'Values': ['available']}]
)
deleted_volumes = []
for vol in response.get('Volumes', []):
vol_id = vol['VolumeId']
size = vol['Size']
print(f"Purging unattached volume {vol_id} ({size} GiB)")
# Delete volume
ec2.delete_volume(VolumeId=vol_id)
deleted_volumes.append(vol_id)
return {
'statusCode': 200,
'deleted_volumes': deleted_volumes
}
Trade-offs:
- Advantages: Can be triggered on a cron schedule (Amazon EventBridge), supports structured logging, retry logic, and Slack/Teams alerting.
- Disadvantages: Requires maintaining code, IAM execution roles, and deployment infrastructure.
Approach Comparison Table
| Approach | Ideal Use Case | Speed | Safety & Control | Overhead |
|---|---|---|---|---|
AWS CLI Pipeline (xargs) |
Interactive ad-hoc cleanup, terminal work. | Very Fast | High (if query is validated first) | Minimal |
Bash for Loop |
Shell scripts requiring inline logging/prompts. | Fast | High (supports explicit confirmation steps) | Low |
Python boto3 Script |
Automated scheduled maintenance (Lambda). | Fast | Maximum (full exception handling & safety checks) | Moderate |
| Terraform / IaC | Managing declarative state environments. | N/A | High (detects drift via terraform plan) |
High |
9. AWS CLI & Linux Command Deep Dive
Command 1: aws ec2 describe-volumes
Purpose
Queries the AWS EC2 API endpoint to fetch detailed metadata regarding EBS volumes in the targeted region.
Syntax & Flags
aws ec2 describe-volumes [options]
| Flag / Option | Type | Purpose & Troubleshooting Context |
|---|---|---|
--filters |
List | Applies server-side filtering evaluated directly by the AWS API before returning data. |
--query |
String | Applies client-side JMESPath expression to structure, format, or extract specific fields. |
--output |
String | Controls output formatting style (json, text, table, yaml). |
--volume-ids |
List | Restricts output to specific volume ID strings (e.g., --volume-ids vol-12345). |
--max-items |
Integer | Specifies total number of items to return for pagination handling. |
Useful --filters Examples for SREs:
- Filter by status:
--filters Name=status,Values=available - Filter by volume type:
--filters Name=volume-type,Values=gp3 - Filter by tag:
--filters Name=tag:Environment,Values=production
Command 2: aws ec2 delete-volume
Purpose
Permanently deletes a specified EBS volume. The volume must be in the available state.
Syntax & Flags
aws ec2 delete-volume --volume-id <vol-id> [options]
| Flag / Option | Type | Purpose & Troubleshooting Context |
|---|---|---|
--volume-id |
String | Required. The ID of the EBS volume to delete. |
--dry-run |
Flag | Simulates the API request to check for proper IAM permissions without deleting the volume. |
Command 3: xargs
Purpose
A standard Unix utility that builds and executes command lines from standard input (stdin).
Syntax & Flags
command | xargs [options] [utility [argument ...]]
| Option | Purpose | Example |
|---|---|---|
-n <max-args> |
Limit the number of arguments passed to each command invocation. | xargs -n 1 aws ec2 delete-volume --volume-id |
-P <max-procs> |
Run up to max-procs processes in parallel (concurrent execution). |
xargs -n 1 -P 4 aws ec2 delete-volume --volume-id |
-I <replace-str> |
Replaces occurrences of replace-str with input items. |
xargs -I {} aws ec2 delete-volume --volume-id {} |
-t / --verbose |
Prints the full command line to stderr before executing it. | Useful for dry-running pipelines safely. |
10. Underlying Concepts and Architecture
1. Amazon EBS Volume Lifecycle States
Understanding state transitions in block storage prevents invalid API calls:
+-------------------+
| creating |
+---------+---------+
|
v
+------------------+ Attach +--------------------+
| in-use |<----------| available |
| (Attached to |---------->| (Orphaned / Ready |
| EC2 Instance) | Detach | for Attachment) |
+------------------+ +---------+----------+
|
| Delete Volume
v
+--------------------+
| deleting / deleted|
+--------------------+
creating: Provisioning volume blocks in the AWS storage cluster.available: Fully provisioned and unattached to any EC2 instance. Billed per GB-month.in-use: Attached to an EC2 instance via Xen/KVM hypervisor block device attachment.deleting: Deletion request received; underlying block references are being scrubbed.
2. Server-Side Filtering (--filters) vs. Client-Side Querying (--query)
Understanding where data filtering occurs is vital when working with large cloud deployments:
-
Server-Side Filtering (
--filters): -
Processed by the AWS API endpoint.
-
Reduces network transfer payload by returning only matching records.
-
Faster execution when querying accounts with thousands of resources.
-
Client-Side Querying (
--query): -
Processed locally by the AWS CLI binary using JMESPath.
-
Evaluated after the raw response is returned over the network.
-
Used for formatting, array projection, and extracting exact strings for Unix pipelines.
11. Real-World Sysadmin & SRE Applications
Production Detection & Observability
In enterprise production accounts, relying on manual CLI runs is insufficient. SRE teams use automated governance patterns:
- AWS Cost Explorer & Anomaly Detection: Configure alerts when storage category spending exceeds historical baselines.
- AWS Config Rules:
Deploy the managed rule
ec2-volume-inuse-checkto continuously evaluate whether EBS volumes are attached to EC2 instances. Non-compliant resources trigger CloudWatch events. - Automated Janitor Lambdas:
Implement automated routines that tag unattached volumes with
OrphanedDate=YYYY-MM-DD. If a volume remains unattached for 7 days, the routine automatically takes a final EBS snapshot and purges the volume.
Runbook Strategy: Safety Measures Prior to Bulk Purging
Before deleting unattached storage in a production environment:
- Take Safety Snapshots: Always create an EBS snapshot (
aws ec2 create-snapshot) prior to volume deletion in production. - Verify Tag Metadata: Check for tags such as
DoNotDeleteorBackupTarget. - Use Dry-Run Verification: Append
--dry-runto CLI calls to ensure IAM permissions and parameters are valid without triggering state changes.
12. Further Learning & Experiments
Safe Local Experiments
To practice AWS CLI filtering without risking real AWS resources or incurring costs, use LocalStack or Floci in a local container:
- Create Test Volumes:
aws ec2 create-volume --availability-zone us-east-1a --size 10 --volume-type gp3
aws ec2 create-volume --availability-zone us-east-1a --size 5 --volume-type gp2
- Experiment with Complex JMESPath Filtering:
Extract volume IDs for all
gp3volumes that areavailable:
aws ec2 describe-volumes \
--query "Volumes[?State=='available' && VolumeType=='gp3'].VolumeId" \
--output text
- Test Safety Flags:
Execute a delete command with
--dry-runto inspect CLI authorization checks:
aws ec2 delete-volume --volume-id vol-1234567890abcdef0 --dry-run
Official References & Documentation
- AWS CLI EC2 Command Reference: describe-volumes
- AWS CLI EC2 Command Reference: delete-volume
- JMESPath Specification for AWS CLI Querying
13. Key Takeaways
- Cloud FinOps Responsibility: Unattached (
available) EBS volumes accumulate billing charges indefinitely until explicitly deleted or cleaned up by lifecycle policies. - Distinguish
availablevs.in-use: Always verify state and attachment fields prior to issuing deletion calls.in-usevolumes belong to running/stopped instances and must be preserved. - Leverage Server-Side Filtering: Combine
--filters Name=status,Values=availablewith client-side--queryprojections for efficient, high-performance CLI pipelines. - Unix Pipeline Efficiency: Pipelining
aws ec2 describe-volumes ... --output textintoxargs -n 1 aws ec2 delete-volume --volume-idenables precise, single-command remediation. - Production Safety First: In live enterprise environments, combine automated purges with pre-deletion snapshot backups and grace-period tagging strategies.
14. Quick Troubleshooting Reference / Runbook
# ==============================================================================
# EBS ORPHANED VOLUME PURGE RUNBOOK
# ==============================================================================
# 1. Audit all volumes in current region (Inspect state & attachments)
aws ec2 describe-volumes \
--query "Volumes[*].{ID:VolumeId, State:State, Size:Size, Attached:Attachments}" \
--output table
# 2. Count total orphaned storage capacity (GiB)
aws ec2 describe-volumes \
--filters Name=status,Values=available \
--query "sum(Volumes[*].Size)" \
--output text
# 3. List IDs of unattached volumes strictly
aws ec2 describe-volumes \
--filters Name=status,Values=available \
--query "Volumes[*].VolumeId" \
--output text
# 4. Dry-run safety test prior to deletion
aws ec2 describe-volumes \
--filters Name=status,Values=available \
--query "Volumes[*].VolumeId" \
--output text | xargs -n 1 -I {} aws ec2 delete-volume --volume-id {} --dry-run
# 5. Execute purge of all unattached volumes
aws ec2 describe-volumes \
--filters Name=status,Values=available \
--query "Volumes[*].VolumeId" \
--output text | xargs -n 1 aws ec2 delete-volume --volume-id
# 6. Verify remaining volumes are strictly in-use
aws ec2 describe-volumes \
--query "Volumes[*].{ID:VolumeId, State:State}" \
--output table