Domain 3Systems & IaCCORE

(Auckland) Automating AWS S3 Lifecycle Transitions and Deletions via Date-Based Key Parsing

A deep dive into enforcing custom, date-in-path S3 retention policies using AWS CLI, Bash regular expressions, and S3 in-place object storage class transitions.

Domain: Domain 3: Systems & IaC•Updated: 2026-11-11
#AWS#AWS-CLI#S3#FinOps#Bash#Lifecycle-Management

1. Introduction

Managing long-term object storage efficiently is a primary responsibility in Financial Operations (FinOps) and Site Reliability Engineering (SRE). Unmanaged database dumps and log archives accumulating in Amazon Simple Storage Service (S3) can generate significant unnecessary costs if left in high-cost storage tiers (STANDARD) indefinitely.

While AWS provides native S3 Lifecycle Policies to transition or expire objects based on creation dates (LastModified) or prefix paths, real-world operational environments frequently present legacy schemas where the logical backup date is embedded directly within the S3 object key path (e.g., backups/YYYY-MM-DD/db.dump). When historical data is restored or migrated into S3, system metadata timestamps like LastModified reset to the migration date, making native LastModified-based lifecycle policies inaccurate.

In this guide, we review a practical troubleshooting and remediation exercise: parsing absolute date strings from S3 object keys, implementing strict retention tiers, deleting expired data, and transitioning historical archives to GLACIER using the AWS CLI and native Bash scripting.


2. Problem Description and Environment

The operational task originated from the “Auckland”: S3 delete & archive old backups scenario on SadServers.

Scenario & Constraints

  • Bucket Name: s3://auckland-backups.
  • Key Naming Convention: Database dumps are stored using an absolute date prefix: backups/YYYY-MM-DD/....
  • Environment Context: Execution takes place on an administrative control host connected to Floci, a local AWS emulator. The AWS CLI is pre-configured and authenticated.
  • Validation Standard: Solution verification is executed via /home/admin/agent/check.sh.

Target Retention Policy Rules

The retention logic must strictly parse the date embedded in the object key path and ignore the S3 LastModified timestamp:

  1. **Before 2023-01-01** (< 2023-01-01): Permanently delete objects.
  2. **From 2023-01-01 through 2024-12-31**: Retain objects, but transition storage class to GLACIER (or DEEP_ARCHIVE).
  3. **On or after 2025-01-01** (>= 2025-01-01): Leave storage class as STANDARD.
  4. Keys without the backups/YYYY-MM-DD/ prefix: Leave untouched.

3. Initial Investigation and Hypotheses

Prior to executing modifying operations against storage infrastructure, a structured diagnostic inventory must be performed to categorize keys and verify path formats.

Diagnostic Plan

  1. Retrieve a complete list of objects from s3://auckland-backups including key names and current storage classes.
  2. Validate whether path strings strictly follow the expected backups/YYYY-MM-DD/ format.
  3. Identify non-conforming or non-dated keys to ensure script rules bypass them safely.

Object Categorization Matrix

Object Key Pattern Target Date Range Expected Action Storage Class Outcome
backups/YYYY-MM-DD/... $< 2023-01-01$ Delete object Object Removed
backups/YYYY-MM-DD/... $2023-01-01 \le \text{Date} \le 2024-12-31$ Transition tier GLACIER
backups/YYYY-MM-DD/... $\ge 2025-01-01$ No action Retain STANDARD
Non-matching path N/A Bypass Retain STANDARD

4. Troubleshooting Process

Step 1: Object Inventory and Schema Inspection

Objective

Enumerate all objects stored in s3://auckland-backups and inspect their keys and current storage classes.

Command

aws s3api list-objects-v2 --bucket auckland-backups --query 'Contents[].{Key:Key, StorageClass:StorageClass}' --output table

Command Explanation

  • aws s3api: Low-level CLI interface mapping directly to raw S3 REST API calls.
  • list-objects-v2: Standard S3 API operation used to enumerate up to 1,000 objects per response.
  • --bucket auckland-backups: Specifies the target S3 bucket name.
  • --query 'Contents[].{Key:Key, StorageClass:StorageClass}': Client-side JMESPath expression extracting only the Key and StorageClass attributes.
  • --output table: Formats JSON response data into an easily readable ASCII table.

Output and Analysis

------------------------------------------------
|                 ListObjectsV2                |
+-----------------------------+----------------+
|             Key             | StorageClass   |
+-----------------------------+----------------+
|  backups/2020-06-15/db.dump |  STANDARD      |
|  backups/2021-03-01/db.dump |  STANDARD      |
|  backups/2022-12-31/db.dump |  STANDARD      |
|  backups/2023-01-01/db.dump |  STANDARD      |
|  backups/2023-07-15/db.dump |  STANDARD      |
|  backups/2024-12-31/db.dump |  STANDARD      |
|  backups/2025-01-01/db.dump |  STANDARD      |
|  backups/2026-03-15/db.dump |  STANDARD      |
|  backups/README.txt         |  STANDARD      |
|  current/latest.dat         |  STANDARD      |
+-----------------------------+----------------+
  • 10 total objects exist in the bucket.
  • 8 objects follow the backups/YYYY-MM-DD/... directory structure.
  • 2 objects (backups/README.txt and current/latest.dat) do not contain date subdirectories and must remain untouched.
  • All existing objects are currently assigned to STANDARD storage class.

Decision

Develop a shell script (run.sh) that reads all keys from the bucket, parses date components via regular expressions, and applies aws s3api delete-object or aws s3api copy-object based on date boundaries.


5. Root Cause Analysis

The operational issue is caused by historical accumulation of backup objects without lifecycle management rules in place.

       [ s3://auckland-backups ]
                   │
  ┌────────────────┼────────────────┐
  ▼                ▼                ▼
< 2023-01-01   2023-01-01 to   >= 2025-01-01
 (Expired)      2024-12-31      (Active)
  │            (Cold Archive)       │
  ▼                │                ▼
DELETE             ▼           Keep STANDARD
              Copy In-Place
           StorageClass: GLACIER

Key Technical Mechanisms

  1. S3 Storage Class Transitioning: S3 objects cannot have their storage class modified via direct attribute update APIs. In S3 architecture, changing an object’s storage class requires copying the object onto itself (copy-object) with a modified --storage-class flag.
  2. ISO 8601 Date Format Properties: Because date strings in the path use ISO 8601 standard (YYYY-MM-DD), lexicographical string comparison in Bash (<, >) matches chronological comparison.

6. Solution and Remediation

To automate the lifecycle management, the shell script run.sh was written and executed.

Applied Solution Script (run.sh)

#!/bin/bash
set -euo pipefail

BUCKET="auckland-backups"
REGEX='^backups/([0-9]{4}-[0-9]{2}-[0-9]{2})/'

aws s3api list-objects-v2 --bucket "$BUCKET" --query 'Contents[].Key' --output text | tr '\t' '\n' | while read -r key; do
    if [[ "$key" =~ $REGEX ]]; then
        date_str="${BASH_REMATCH[1]}"
        
        if [[ "$date_str" < "2023-01-01" ]]; then
            echo "Deleting: $key"
            aws s3api delete-object --bucket "$BUCKET" --key "$key"
        elif [[ "$date_str" > "2023-01-01" || "$date_str" == "2023-01-01" ]] && [[ "$date_str" < "2024-12-31" || "$date_str" == "2024-12-31" ]]; then
            echo "Archiving to GLACIER: $key"
            aws s3api copy-object \
                --bucket "$BUCKET" \
                --copy-source "$BUCKET/$key" \
                --key "$key" \
                --storage-class GLACIER \
                --metadata-directive COPY > /dev/null
        else
            echo "Keeping STANDARD: $key"
        fi
    else
        echo "Untouched: $key"
    fi
done

Script Logic Breakdown

  1. Safety Flags (set -euo pipefail): Ensures script termination on unhandled error, prevents execution with uninitialized variables, and preserves pipeline failures.
  2. Regex Capture (^backups/([0-9]{4}-[0-9]{2}-[0-9]{2})/): Matches standard date prefixes and captures YYYY-MM-DD into index 1 of the BASH_REMATCH array.
  3. Data Pipeline (tr '\t' '\n'): Converts tab-separated output from aws s3api list-objects-v2 --output text into newline-separated keys suitable for while read processing.
  4. Lexicographical Comparison:
  • $date_str < "2023-01-01" triggers deletion via aws s3api delete-object.
  • Range check 2023-01-01 <= date_str <= 2024-12-31 executes copy-object with --storage-class GLACIER and --metadata-directive COPY.
  • All other conditions bypass modifying calls.

7. Verification and Validation

1. Script Execution Output

Executing bash run.sh produced the following log output:

Deleting: backups/2020-06-15/db.dump
Deleting: backups/2021-03-01/db.dump
Deleting: backups/2022-12-31/db.dump
Archiving to GLACIER: backups/2023-01-01/db.dump
Archiving to GLACIER: backups/2023-07-15/db.dump
Archiving to GLACIER: backups/2024-12-31/db.dump
Keeping STANDARD: backups/2025-01-01/db.dump
Keeping STANDARD: backups/2026-03-15/db.dump
Untouched: backups/README.txt
Untouched: current/latest.dat

2. Post-Execution Bucket State Check

Re-querying the bucket via aws s3api list-objects-v2:

aws s3api list-objects-v2 --bucket auckland-backups --query 'Contents[].{Key:Key, StorageClass:StorageClass}' --output table
------------------------------------------------
|                 ListObjectsV2                |
+-----------------------------+----------------+
|             Key             | StorageClass   |
+-----------------------------+----------------+
|  backups/2023-01-01/db.dump |  GLACIER       |
|  backups/2023-07-15/db.dump |  GLACIER       |
|  backups/2024-12-31/db.dump |  GLACIER       |
|  backups/2025-01-01/db.dump |  STANDARD      |
|  backups/2026-03-15/db.dump |  STANDARD      |
|  backups/README.txt         |  STANDARD      |
|  current/latest.dat         |  STANDARD      |
+-----------------------------+----------------+

3. Verification Script Execution

Executing the scenario verification check /home/admin/agent/check.sh returned:

OK

8. Alternative Troubleshooting Approaches

Approach A: Python (AWS SDK for Python / boto3)

In large-scale production environments containing millions of keys, piping AWS CLI output through shell subshells can hit memory limits or suffer performance degradation. A native Python script using boto3 handles pagination and JSON responses cleanly.

import boto3
import re

BUCKET = "auckland-backups"
REGEX = re.compile(r"^backups/(\d{4}-\d{2}-\d{2})/")

s3 = boto3.client('s3')
paginator = s3.get_paginator('list_objects_v2')

for page in paginator.paginate(Bucket=BUCKET):
    for obj in page.get('Contents', []):
        key = obj['Key']
        match = REGEX.match(key)
        if not match:
            print(f"[SKIP] {key}")
            continue
            
        date_str = match.group(1)
        
        if date_str < "2023-01-01":
            print(f"[DELETE] {key}")
            s3.delete_object(Bucket=BUCKET, Key=key)
        elif "2023-01-01" <= date_str <= "2024-12-31":
            print(f"[GLACIER] {key}")
            s3.copy_object(
                Bucket=BUCKET,
                Key=key,
                CopySource={'Bucket': BUCKET, 'Key': key},
                StorageClass='GLACIER',
                MetadataDirective='COPY'
            )
        else:
            print(f"[KEEP STANDARD] {key}")

Approach B: High-Level AWS CLI (aws s3 cp / aws s3 rm)

The high-level aws s3 CLI commands simplify syntactic overhead compared to aws s3api:

# Deletion via high-level interface
aws s3 rm s3://auckland-backups/backups/2020-06-15/db.dump

# Storage class transition via high-level interface
aws s3 cp s3://auckland-backups/backups/2023-01-01/db.dump s3://auckland-backups/backups/2023-01-01/db.dump --storage-class GLACIER

Method Comparison Matrix

Approach Portability Performance Code Complexity API Control
Bash + AWS CLI (s3api) High (Default shell) Moderate Low Full API access
Python (boto3) High (Requires Python/boto3) High (Native Pagination) Moderate Full API access
High-level AWS CLI (aws s3) High Moderate Minimal Limited parameter tuning

9. Linux Command Deep Dive

Command: aws s3api list-objects-v2

Purpose

Lists objects stored in an S3 bucket with low-level pagination and JMESPath projection options.

General Syntax

aws s3api list-objects-v2 --bucket <bucket-name> [options]

Commonly Used Options

Command Variant / Flag Purpose
--bucket <name> Specifies target bucket name.
--prefix <path> Limits response to keys matching prefix string.
--max-items <integer> Sets total number of items returned across API paginations.
--query <JMESPath> Client-side filtering/formatting of returned JSON structure.
`–output text table

Command: aws s3api copy-object

Purpose

Copies an object from a source S3 location to a destination key within the same or another bucket. Used for in-place storage class modification.

General Syntax

aws s3api copy-object --bucket <dest-bucket> --copy-source <src-bucket/src-key> --key <dest-key> --storage-class <CLASS> --metadata-directive COPY

Key Parameters

  • --copy-source: URI specifying source bucket and key path (Bucket/Key).
  • --metadata-directive COPY: Retains existing user metadata without re-writing.
  • --storage-class: Assigns target storage tier (GLACIER, DEEP_ARCHIVE, INTELLIGENT_TIERING, STANDARD_IA).

Command: [[ "$var" =~ $regex ]] (Bash Regex Expression)

Purpose

Evaluates a regular expression test natively inside Bash 3.0+ double-bracket expressions.

Usage Example

REGEX='^backups/([0-9]{4}-[0-9]{2}-[0-9]{2})/'
if [[ "backups/2023-01-01/db.dump" =~ $REGEX ]]; then
    echo "Matched Year-Month-Day: ${BASH_REMATCH[1]}"
fi

Notes & Best Practices

  • Store regular expressions in shell variables to avoid shell escaping pitfalls across different Bash sub-versions.
  • Extracted capture groups are automatically stored in array ${BASH_REMATCH[1]} through ${BASH_REMATCH[N]}.

10. Underlying Linux Concepts and Architecture

ISO 8601 Lexicographical Sorting

ISO 8601 representation (YYYY-MM-DD) maintains identical ordering under ASCII string comparison and chronological time evaluation.

For any two valid ISO date strings $D_1$ and $D_2$:

$$\text{StringCompare}(D_1, D_2) \equiv \text{TemporalCompare}(D_1, D_2)$$

This property allows shell scripts to safely perform date range validation ([[ "$date_str" < "2023-01-01" ]]) without invoking external binaries like date -d.

S3 Object Storage Immutability

Objects in S3 are immutable bitstreams. Modifying metadata or changing storage class attributes requires creating a copy of the object entry with the updated metadata fields set.


11. Real-World Sysadmin and SRE Applications

1. Application-Level Date Parsing vs. Native Lifecycle Rules

S3 Lifecycle Rules operate exclusively on object creation time (LastModified) or prefix paths. When managing legacy backups, historical data migrations, or external dump ingestions:

  • Relying on LastModified can lead to premature expiration or unexpected retention costs.
  • Custom automation scripts or scheduled AWS Lambda functions must be deployed when lifecycle policies depend on date metadata embedded within object keys.

2. FinOps Cold Tier Management

Storage class costs vary significantly:

  • STANDARD: $\approx $0.023$ per GB/month.
  • GLACIER Flexible Retrieval: $\approx $0.004$ per GB/month ($>80%$ cost reduction).
  • DEEP_ARCHIVE: $\approx $0.00099$ per GB/month ($>95%$ cost reduction).

Implementing date-based tiering ensures long-term cold archives are stored cost-effectively.


12. Further Learning and Experiments

Hands-on Experiments

  1. LocalStack/MinIO Testing Environment:
  • Deploy a local S3 emulation container using Docker (docker run -p 4566:4566 localstack/localstack).
  • Populate test buckets with multi-tier mock keys and execute dry-run test scripts.
  1. Regex Subcapture Exploration:
  • Test regex subcaptures in Bash with complex nested path strings (e.g., app/region/YYYY/MM/DD/dump.sql.gz).

Documentation References


13. Key Takeaways

  1. S3 In-Place Storage Class Transition: To change an existing object’s storage class via s3api, execute copy-object pointing --copy-source to the object’s current path.
  2. ISO 8601 String Comparison: Date strings in YYYY-MM-DD format can be compared chronologically using standard Bash string comparison operators (<, >).
  3. Bash Regular Expression Captures: Use [[ "$str" =~ $regex ]] and reference array ${BASH_REMATCH[1]} to extract substring path components natively.
  4. Defensive Shell Scripting: Include set -euo pipefail to ensure scripts exit immediately if an API call or subshell command fails.

14. Quick Troubleshooting Reference / Runbook

# 1. Inspect bucket keys and storage classes
aws s3api list-objects-v2 --bucket <BUCKET> --query 'Contents[].{Key:Key, StorageClass:StorageClass}' --output table

# 2. Delete specific legacy object key
aws s3api delete-object --bucket <BUCKET> --key <KEY>

# 3. Perform in-place storage class transition to GLACIER
aws s3api copy-object \
    --bucket <BUCKET> \
    --copy-source <BUCKET>/<KEY> \
    --key <KEY> \
    --storage-class GLACIER \
    --metadata-directive COPY

# 4. Verify updated object storage class state
aws s3api list-objects-v2 --bucket <BUCKET> --prefix <KEY_PREFIX> --query 'Contents[].{Key:Key, StorageClass:StorageClass}'