1. Introduction
Managing long-term object storage efficiently is a primary responsibility in Financial Operations (FinOps) and Site Reliability Engineering (SRE). Unmanaged database dumps and log archives accumulating in Amazon Simple Storage Service (S3) can generate significant unnecessary costs if left in high-cost storage tiers (STANDARD) indefinitely.
While AWS provides native S3 Lifecycle Policies to transition or expire objects based on creation dates (LastModified) or prefix paths, real-world operational environments frequently present legacy schemas where the logical backup date is embedded directly within the S3 object key path (e.g., backups/YYYY-MM-DD/db.dump). When historical data is restored or migrated into S3, system metadata timestamps like LastModified reset to the migration date, making native LastModified-based lifecycle policies inaccurate.
In this guide, we review a practical troubleshooting and remediation exercise: parsing absolute date strings from S3 object keys, implementing strict retention tiers, deleting expired data, and transitioning historical archives to GLACIER using the AWS CLI and native Bash scripting.
2. Problem Description and Environment
The operational task originated from the “Auckland”: S3 delete & archive old backups scenario on SadServers.
Scenario & Constraints
- Bucket Name:
s3://auckland-backups. - Key Naming Convention: Database dumps are stored using an absolute date prefix:
backups/YYYY-MM-DD/.... - Environment Context: Execution takes place on an administrative control host connected to Floci, a local AWS emulator. The AWS CLI is pre-configured and authenticated.
- Validation Standard: Solution verification is executed via
/home/admin/agent/check.sh.
Target Retention Policy Rules
The retention logic must strictly parse the date embedded in the object key path and ignore the S3 LastModified timestamp:
- **Before
2023-01-01**(< 2023-01-01): Permanently delete objects. - **From
2023-01-01through2024-12-31**: Retain objects, but transition storage class toGLACIER(orDEEP_ARCHIVE). - **On or after
2025-01-01**(>= 2025-01-01): Leave storage class asSTANDARD. - Keys without the
backups/YYYY-MM-DD/prefix: Leave untouched.
3. Initial Investigation and Hypotheses
Prior to executing modifying operations against storage infrastructure, a structured diagnostic inventory must be performed to categorize keys and verify path formats.
Diagnostic Plan
- Retrieve a complete list of objects from
s3://auckland-backupsincluding key names and current storage classes. - Validate whether path strings strictly follow the expected
backups/YYYY-MM-DD/format. - Identify non-conforming or non-dated keys to ensure script rules bypass them safely.
Object Categorization Matrix
| Object Key Pattern | Target Date Range | Expected Action | Storage Class Outcome |
|---|---|---|---|
backups/YYYY-MM-DD/... |
$< 2023-01-01$ | Delete object | Object Removed |
backups/YYYY-MM-DD/... |
$2023-01-01 \le \text{Date} \le 2024-12-31$ | Transition tier | GLACIER |
backups/YYYY-MM-DD/... |
$\ge 2025-01-01$ | No action | Retain STANDARD |
| Non-matching path | N/A | Bypass | Retain STANDARD |
4. Troubleshooting Process
Step 1: Object Inventory and Schema Inspection
Objective
Enumerate all objects stored in s3://auckland-backups and inspect their keys and current storage classes.
Command
aws s3api list-objects-v2 --bucket auckland-backups --query 'Contents[].{Key:Key, StorageClass:StorageClass}' --output table
Command Explanation
aws s3api: Low-level CLI interface mapping directly to raw S3 REST API calls.list-objects-v2: Standard S3 API operation used to enumerate up to 1,000 objects per response.--bucket auckland-backups: Specifies the target S3 bucket name.--query 'Contents[].{Key:Key, StorageClass:StorageClass}': Client-side JMESPath expression extracting only theKeyandStorageClassattributes.--output table: Formats JSON response data into an easily readable ASCII table.
Output and Analysis
------------------------------------------------
| ListObjectsV2 |
+-----------------------------+----------------+
| Key | StorageClass |
+-----------------------------+----------------+
| backups/2020-06-15/db.dump | STANDARD |
| backups/2021-03-01/db.dump | STANDARD |
| backups/2022-12-31/db.dump | STANDARD |
| backups/2023-01-01/db.dump | STANDARD |
| backups/2023-07-15/db.dump | STANDARD |
| backups/2024-12-31/db.dump | STANDARD |
| backups/2025-01-01/db.dump | STANDARD |
| backups/2026-03-15/db.dump | STANDARD |
| backups/README.txt | STANDARD |
| current/latest.dat | STANDARD |
+-----------------------------+----------------+
- 10 total objects exist in the bucket.
- 8 objects follow the
backups/YYYY-MM-DD/...directory structure. - 2 objects (
backups/README.txtandcurrent/latest.dat) do not contain date subdirectories and must remain untouched. - All existing objects are currently assigned to
STANDARDstorage class.
Decision
Develop a shell script (run.sh) that reads all keys from the bucket, parses date components via regular expressions, and applies aws s3api delete-object or aws s3api copy-object based on date boundaries.
5. Root Cause Analysis
The operational issue is caused by historical accumulation of backup objects without lifecycle management rules in place.
[ s3://auckland-backups ]
│
┌────────────────┼────────────────┐
▼ ▼ ▼
< 2023-01-01 2023-01-01 to >= 2025-01-01
(Expired) 2024-12-31 (Active)
│ (Cold Archive) │
▼ │ ▼
DELETE ▼ Keep STANDARD
Copy In-Place
StorageClass: GLACIER
Key Technical Mechanisms
- S3 Storage Class Transitioning: S3 objects cannot have their storage class modified via direct attribute update APIs. In S3 architecture, changing an object’s storage class requires copying the object onto itself (
copy-object) with a modified--storage-classflag. - ISO 8601 Date Format Properties: Because date strings in the path use ISO 8601 standard (
YYYY-MM-DD), lexicographical string comparison in Bash (<,>) matches chronological comparison.
6. Solution and Remediation
To automate the lifecycle management, the shell script run.sh was written and executed.
Applied Solution Script (run.sh)
#!/bin/bash
set -euo pipefail
BUCKET="auckland-backups"
REGEX='^backups/([0-9]{4}-[0-9]{2}-[0-9]{2})/'
aws s3api list-objects-v2 --bucket "$BUCKET" --query 'Contents[].Key' --output text | tr '\t' '\n' | while read -r key; do
if [[ "$key" =~ $REGEX ]]; then
date_str="${BASH_REMATCH[1]}"
if [[ "$date_str" < "2023-01-01" ]]; then
echo "Deleting: $key"
aws s3api delete-object --bucket "$BUCKET" --key "$key"
elif [[ "$date_str" > "2023-01-01" || "$date_str" == "2023-01-01" ]] && [[ "$date_str" < "2024-12-31" || "$date_str" == "2024-12-31" ]]; then
echo "Archiving to GLACIER: $key"
aws s3api copy-object \
--bucket "$BUCKET" \
--copy-source "$BUCKET/$key" \
--key "$key" \
--storage-class GLACIER \
--metadata-directive COPY > /dev/null
else
echo "Keeping STANDARD: $key"
fi
else
echo "Untouched: $key"
fi
done
Script Logic Breakdown
- Safety Flags (
set -euo pipefail): Ensures script termination on unhandled error, prevents execution with uninitialized variables, and preserves pipeline failures. - Regex Capture (
^backups/([0-9]{4}-[0-9]{2}-[0-9]{2})/): Matches standard date prefixes and capturesYYYY-MM-DDinto index 1 of theBASH_REMATCHarray. - Data Pipeline (
tr '\t' '\n'): Converts tab-separated output fromaws s3api list-objects-v2 --output textinto newline-separated keys suitable forwhile readprocessing. - Lexicographical Comparison:
$date_str < "2023-01-01"triggers deletion viaaws s3api delete-object.- Range check
2023-01-01 <= date_str <= 2024-12-31executescopy-objectwith--storage-class GLACIERand--metadata-directive COPY. - All other conditions bypass modifying calls.
7. Verification and Validation
1. Script Execution Output
Executing bash run.sh produced the following log output:
Deleting: backups/2020-06-15/db.dump
Deleting: backups/2021-03-01/db.dump
Deleting: backups/2022-12-31/db.dump
Archiving to GLACIER: backups/2023-01-01/db.dump
Archiving to GLACIER: backups/2023-07-15/db.dump
Archiving to GLACIER: backups/2024-12-31/db.dump
Keeping STANDARD: backups/2025-01-01/db.dump
Keeping STANDARD: backups/2026-03-15/db.dump
Untouched: backups/README.txt
Untouched: current/latest.dat
2. Post-Execution Bucket State Check
Re-querying the bucket via aws s3api list-objects-v2:
aws s3api list-objects-v2 --bucket auckland-backups --query 'Contents[].{Key:Key, StorageClass:StorageClass}' --output table
------------------------------------------------
| ListObjectsV2 |
+-----------------------------+----------------+
| Key | StorageClass |
+-----------------------------+----------------+
| backups/2023-01-01/db.dump | GLACIER |
| backups/2023-07-15/db.dump | GLACIER |
| backups/2024-12-31/db.dump | GLACIER |
| backups/2025-01-01/db.dump | STANDARD |
| backups/2026-03-15/db.dump | STANDARD |
| backups/README.txt | STANDARD |
| current/latest.dat | STANDARD |
+-----------------------------+----------------+
3. Verification Script Execution
Executing the scenario verification check /home/admin/agent/check.sh returned:
OK
8. Alternative Troubleshooting Approaches
Approach A: Python (AWS SDK for Python / boto3)
In large-scale production environments containing millions of keys, piping AWS CLI output through shell subshells can hit memory limits or suffer performance degradation. A native Python script using boto3 handles pagination and JSON responses cleanly.
import boto3
import re
BUCKET = "auckland-backups"
REGEX = re.compile(r"^backups/(\d{4}-\d{2}-\d{2})/")
s3 = boto3.client('s3')
paginator = s3.get_paginator('list_objects_v2')
for page in paginator.paginate(Bucket=BUCKET):
for obj in page.get('Contents', []):
key = obj['Key']
match = REGEX.match(key)
if not match:
print(f"[SKIP] {key}")
continue
date_str = match.group(1)
if date_str < "2023-01-01":
print(f"[DELETE] {key}")
s3.delete_object(Bucket=BUCKET, Key=key)
elif "2023-01-01" <= date_str <= "2024-12-31":
print(f"[GLACIER] {key}")
s3.copy_object(
Bucket=BUCKET,
Key=key,
CopySource={'Bucket': BUCKET, 'Key': key},
StorageClass='GLACIER',
MetadataDirective='COPY'
)
else:
print(f"[KEEP STANDARD] {key}")
Approach B: High-Level AWS CLI (aws s3 cp / aws s3 rm)
The high-level aws s3 CLI commands simplify syntactic overhead compared to aws s3api:
# Deletion via high-level interface
aws s3 rm s3://auckland-backups/backups/2020-06-15/db.dump
# Storage class transition via high-level interface
aws s3 cp s3://auckland-backups/backups/2023-01-01/db.dump s3://auckland-backups/backups/2023-01-01/db.dump --storage-class GLACIER
Method Comparison Matrix
| Approach | Portability | Performance | Code Complexity | API Control |
|---|---|---|---|---|
Bash + AWS CLI (s3api) |
High (Default shell) | Moderate | Low | Full API access |
Python (boto3) |
High (Requires Python/boto3) | High (Native Pagination) | Moderate | Full API access |
High-level AWS CLI (aws s3) |
High | Moderate | Minimal | Limited parameter tuning |
9. Linux Command Deep Dive
Command: aws s3api list-objects-v2
Purpose
Lists objects stored in an S3 bucket with low-level pagination and JMESPath projection options.
General Syntax
aws s3api list-objects-v2 --bucket <bucket-name> [options]
Commonly Used Options
| Command Variant / Flag | Purpose |
|---|---|
--bucket <name> |
Specifies target bucket name. |
--prefix <path> |
Limits response to keys matching prefix string. |
--max-items <integer> |
Sets total number of items returned across API paginations. |
--query <JMESPath> |
Client-side filtering/formatting of returned JSON structure. |
| `–output text | table |
Command: aws s3api copy-object
Purpose
Copies an object from a source S3 location to a destination key within the same or another bucket. Used for in-place storage class modification.
General Syntax
aws s3api copy-object --bucket <dest-bucket> --copy-source <src-bucket/src-key> --key <dest-key> --storage-class <CLASS> --metadata-directive COPY
Key Parameters
--copy-source: URI specifying source bucket and key path (Bucket/Key).--metadata-directive COPY: Retains existing user metadata without re-writing.--storage-class: Assigns target storage tier (GLACIER,DEEP_ARCHIVE,INTELLIGENT_TIERING,STANDARD_IA).
Command: [[ "$var" =~ $regex ]] (Bash Regex Expression)
Purpose
Evaluates a regular expression test natively inside Bash 3.0+ double-bracket expressions.
Usage Example
REGEX='^backups/([0-9]{4}-[0-9]{2}-[0-9]{2})/'
if [[ "backups/2023-01-01/db.dump" =~ $REGEX ]]; then
echo "Matched Year-Month-Day: ${BASH_REMATCH[1]}"
fi
Notes & Best Practices
- Store regular expressions in shell variables to avoid shell escaping pitfalls across different Bash sub-versions.
- Extracted capture groups are automatically stored in array
${BASH_REMATCH[1]}through${BASH_REMATCH[N]}.
10. Underlying Linux Concepts and Architecture
ISO 8601 Lexicographical Sorting
ISO 8601 representation (YYYY-MM-DD) maintains identical ordering under ASCII string comparison and chronological time evaluation.
For any two valid ISO date strings $D_1$ and $D_2$:
$$\text{StringCompare}(D_1, D_2) \equiv \text{TemporalCompare}(D_1, D_2)$$
This property allows shell scripts to safely perform date range validation ([[ "$date_str" < "2023-01-01" ]]) without invoking external binaries like date -d.
S3 Object Storage Immutability
Objects in S3 are immutable bitstreams. Modifying metadata or changing storage class attributes requires creating a copy of the object entry with the updated metadata fields set.
11. Real-World Sysadmin and SRE Applications
1. Application-Level Date Parsing vs. Native Lifecycle Rules
S3 Lifecycle Rules operate exclusively on object creation time (LastModified) or prefix paths. When managing legacy backups, historical data migrations, or external dump ingestions:
- Relying on
LastModifiedcan lead to premature expiration or unexpected retention costs. - Custom automation scripts or scheduled AWS Lambda functions must be deployed when lifecycle policies depend on date metadata embedded within object keys.
2. FinOps Cold Tier Management
Storage class costs vary significantly:
STANDARD: $\approx $0.023$ per GB/month.GLACIER Flexible Retrieval: $\approx $0.004$ per GB/month ($>80%$ cost reduction).DEEP_ARCHIVE: $\approx $0.00099$ per GB/month ($>95%$ cost reduction).
Implementing date-based tiering ensures long-term cold archives are stored cost-effectively.
12. Further Learning and Experiments
Hands-on Experiments
- LocalStack/MinIO Testing Environment:
- Deploy a local S3 emulation container using Docker (
docker run -p 4566:4566 localstack/localstack). - Populate test buckets with multi-tier mock keys and execute dry-run test scripts.
- Regex Subcapture Exploration:
- Test regex subcaptures in Bash with complex nested path strings (e.g.,
app/region/YYYY/MM/DD/dump.sql.gz).
Documentation References
- AWS CLI Command Reference: s3api
- Amazon S3 Storage Classes Guide
- GNU Bash Manual: Conditional Expressions
13. Key Takeaways
- S3 In-Place Storage Class Transition: To change an existing object’s storage class via
s3api, executecopy-objectpointing--copy-sourceto the object’s current path. - ISO 8601 String Comparison: Date strings in
YYYY-MM-DDformat can be compared chronologically using standard Bash string comparison operators (<,>). - Bash Regular Expression Captures: Use
[[ "$str" =~ $regex ]]and reference array${BASH_REMATCH[1]}to extract substring path components natively. - Defensive Shell Scripting: Include
set -euo pipefailto ensure scripts exit immediately if an API call or subshell command fails.
14. Quick Troubleshooting Reference / Runbook
# 1. Inspect bucket keys and storage classes
aws s3api list-objects-v2 --bucket <BUCKET> --query 'Contents[].{Key:Key, StorageClass:StorageClass}' --output table
# 2. Delete specific legacy object key
aws s3api delete-object --bucket <BUCKET> --key <KEY>
# 3. Perform in-place storage class transition to GLACIER
aws s3api copy-object \
--bucket <BUCKET> \
--copy-source <BUCKET>/<KEY> \
--key <KEY> \
--storage-class GLACIER \
--metadata-directive COPY
# 4. Verify updated object storage class state
aws s3api list-objects-v2 --bucket <BUCKET> --prefix <KEY_PREFIX> --query 'Contents[].{Key:Key, StorageClass:StorageClass}'