Domain 10Linux & SREINTERMEDIATE

(Budapest) Automating Linux User Provisioning: Parsing Delimited Files and Mastering Shadow Password Hashing

A deep dive into bulk Linux user account provisioning from delimited text files, handling legacy MD5 crypt password hashing with useradd, and analyzing /etc/shadow verification mechanics.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-08
#bash#user-management#security#shadow#openssl#automation

1. Introduction

Local user account management is a foundational skill in Linux system administration. Whether setting up jump hosts, staging local test environments, or onboarding service accounts, sysadmins often need to parse external user lists and provision accounts programmatically.

However, bulk provisioning involves more than simply creating usernames; it requires assigning passwords securely, ensuring home directories are instantiated, and producing cryptographic hashes compatible with target authentication mechanisms or verification suits. Modern Linux systems default to secure password hashing algorithms such as SHA-512 or Yescrypt, but legacy validation scripts or specific security policies may explicitly require alternative algorithms such as MD5 crypt.

In this guide, we will analyze a real-world user provisioning scenario titled “Budapest”. We will cover file structure diagnostics, line-ending verification, custom password hashing using openssl passwd, batch account creation using useradd, and /etc/shadow hash validation.


2. Problem Description and Environment

Scenario Overview

In the “Budapest” troubleshooting scenario, the system administrator is tasked with parsing a input file /home/admin/user_list.txt and provisioning all specified local users along with their designated credentials.

Environment Specifications

  • Host Environment: Linux (Debian/Ubuntu-based distribution).
  • User Permissions: Administrative sudo privileges available.
  • Input File: /home/admin/user_list.txt.
  • Validation Script: /home/admin/agent/check.sh.

Expected vs. Actual Behavior

  • Expected Behavior: Every user listed in user_list.txt must exist as a local user account on the system, complete with their corresponding home directory and password correctly updated in /etc/shadow.
  • Actual Behavior: Initially, none of the target user accounts existed on the system.

3. Initial Investigation and Hypotheses

Before running bulk provisioning loops, a systematic investigation is required to analyze input file formatting, encoding, and validation criteria.

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
Input File Format: Standard semicolon-delimited text (username;password). Sample lines show alexsmith;Yt7kE9wq. None. Run head -n 10 user_list.txt.
Line Ending Corruption: File contains Windows carriage returns (\r\n) which break usernames/passwords during parsing. Input files sourced from non-Linux environments often contain \r\n. File command reports ASCII text. Run file user_list.txt.
Password Hash Mismatch: Verification script expects specific hash algorithm (e.g., MD5) rather than system default (SHA-512/Yescrypt). Standard password tools use system defaults, whereas verification tools may use fixed algorithms. Unverified until check.sh is inspected. Run cat /home/admin/agent/check.sh.

4. Troubleshooting Process

Step 1: Inspecting the Input File Structure

Objective

Determine the delimiter, sample data format, and field order of /home/admin/user_list.txt.

Command

head -n 10 user_list.txt

Command Explanation

  • head: Standard utility used to output the first part of files.
  • -n 10: Restricts the output to the first 10 lines of the file.

Output and Analysis

alexsmith;Yt7kE9wq
emilyjones;Fg3vU1pz
danielmiller;Kp5nW2cx
sophiebrown;Qr9xH0td
michaeljohnson;Vm6tPo4r
oliviaroberts;Lb8qMz7v
williamlee;Ns2cTr8y
isabellaclark;Tf5vX9ku
ethanmoore;Hd4pW3ro
avawhite;Jk7zCn5l

Findings & Decision

The file follows a strict username;password structure. Semicolons (;) act as the field delimiter.


Step 2: Checking File Line Endings

Objective

Verify whether user_list.txt uses Unix line terminators (\n) or Windows line terminators (\r\n). If carriage returns (\r) exist, Bash reading loops will append \r to the end of the password string, corrupting the generated credentials.

Command

file user_list.txt

Command Explanation

  • file: Utility that inspects file headers and contents to determine file types and text encoding formats.

Output and Analysis

user_list.txt: ASCII text

Findings & Decision

The file is standard Unix ASCII text without CRLF terminators. No dos2unix preprocessing or tr -d '\r' filtering is required.


Step 3: Inspecting the Solution Verification Script

Objective

Understand how the grading script evaluates user creation and password accuracy.

Command

cat /home/admin/agent/check.sh

Command Explanation

  • cat: Concatenates and prints the full contents of /home/admin/agent/check.sh to standard output.

Output and Analysis

#!/bin/bash
# DO NOT MODIFY THIS FILE ("Check My Solution" will fail)

line=$(shuf -n 1 /home/admin/user_list.txt)
user=$(echo "$line" | cut -d ';' -f 1)
pass=$(echo "$line" | cut -d ';' -f 2)

shadow_line=$(sudo grep "^$user:" /etc/shadow)
[ -z "$shadow_line" ] && { echo -n "NO"; exit 0; }

hash=$(echo "$shadow_line" | cut -d ':' -f2)

salt=$(echo "$hash" | cut -d '$' -f3)

computed=$(openssl passwd -1 -salt "$salt" "$pass")

[ "$computed" != "$hash" ] && { echo -n "NO"; exit 0; }

echo -n "OK"
exit 0

Critical Analysis of check.sh Logic:

  1. shuf -n 1 /home/admin/user_list.txt: Randomly selects one line from user_list.txt.
  2. cut -d ';' -f 1 & cut -d ';' -f 2: Splits username and password.
  3. sudo grep "^$user:" /etc/shadow: Extracts the entry for $user from /etc/shadow.
  4. salt=$(echo "$hash" | cut -d '$' -f3): Extracts the cryptographic salt from field 3 of the shadow hash string.
  5. computed=$(openssl passwd -1 -salt "$salt" "$pass"): Recomputes the hash using openssl passwd -1.
  6. [ "$computed" != "$hash" ]: Compares the computed hash directly to the entry stored in /etc/shadow.

Crucial Finding

check.sh uses openssl passwd -1, which specifically generates an MD5-based crypt hash (prefixed with $1$). Standard Linux passwd or default chpasswd operations on modern distributions use SHA-512 ($6$) or Yescrypt ($y$). If users are provisioned with default algorithms, the hash comparison in check.sh will fail even if the password string itself is correct.


Step 4: Executing the Batch User Provisioning Loop

Objective

Provision all accounts in user_list.txt with their home directories and pre-hashed MD5 passwords.

Command

while IFS=';' read -r user pass; do
    sudo useradd -m -p "$(openssl passwd -1 "$pass")" "$user"
done < /home/admin/user_list.txt

Command Breakdown

  • while IFS=';' read -r user pass; do ... done < user_list.txt:

  • IFS=';': Sets the Internal Field Separator locally for read to split lines on semicolons.

  • read -r: Reads raw input line by line, preventing backslashes from acting as escape sequences.

  • user pass: Assigns field 1 to $user and field 2 to $pass.

  • sudo useradd -m -p "$(openssl passwd -1 "$pass")" "$user":

  • -m: Ensures home directory creation (/home/<username>).

  • -p <hash>: Takes an already hashed password string and writes it directly to /etc/shadow.

  • openssl passwd -1 "$pass": Hashes the raw password using MD5 crypt ($1$) as required by the test script.


5. Root Cause Analysis

The core requirement of this scenario lies in the interaction between Linux user management tools and /etc/shadow password hash formats.

+--------------------------+
| user_list.txt            |
| (username;password)      |
+------------+-------------+
             |
             v
+--------------------------+      +--------------------------+
| openssl passwd -1 "$pass"| ---> | Generates MD5 Hash       |
+------------+-------------+      | Format: $1$<salt>$<hash> |
             |                    +------------+-------------+
             v                                 |
+----------------------------------------------+
| useradd -m -p <hash> <username>
+------------+---------------------------------+
             |
             v
+--------------------------+
| /etc/shadow updated with |
| MD5 Crypt ($1$) Hash     |
+--------------------------+

Modern Linux distributions default to robust hashing algorithms configured in /etc/login.defs (ENCRYPT_METHOD SHA512 or YESCRYPT). When setting passwords using passwd or standard batch tools without algorithm overrides, /etc/shadow stores entries formatted as:

  • SHA-512: $6$salt$hash...
  • Yescrypt: $y$j9T$salt$hash...

Because the validation script /home/admin/agent/check.sh explicitly recomputed hashes using openssl passwd -1 (MD5 crypt $1$), any account created using modern default hashing algorithms resulted in a mismatch during validation. Supplying the MD5-hashed string directly via useradd -p satisfied both local authentication and validation constraints.


6. Solution and Remediation

The full administrative solution applied to resolve the scenario:

while IFS=';' read -r user pass; do
    sudo useradd -m -p "$(openssl passwd -1 "$pass")" "$user"
done < /home/admin/user_list.txt

Why This Solution Works

  1. Directory Provisioning: -m populates /home/<username> from template files in /etc/skel.
  2. Cryptographic Compatibility: openssl passwd -1 generates MD5 crypt strings beginning with $1$, perfectly matching check.sh expectations.
  3. Non-Interactive Execution: Passing the pre-computed hash directly via useradd -p bypasses interactive password prompts.

7. Verification and Validation

Step 1: Running the Check Script

Command

./agent/check.sh

Output

OK

Step 2: System State Inspection (Optional Verification)

To independently verify that the entries in /etc/passwd and /etc/shadow were correctly created:

grep "^alexsmith:" /etc/passwd
sudo grep "^alexsmith:" /etc/shadow

Sample Expected Output

alexsmith:x:1001:1001::/home/alexsmith:/bin/sh
alexsmith:$1$r8X9/kQ2$d.vX5mJ7gS1...:19700:0:99999:7:::

The presence of $1$ in the second field of /etc/shadow confirms that the password was successfully hashed using MD5 crypt.


8. Alternative Troubleshooting Approaches & Solutions

Approach A: useradd with chpasswd -c MD5

Rather than hashing inline with openssl, Linux provides chpasswd for batch password processing.

while IFS=';' read -r user pass; do
    sudo useradd -m "$user"
    echo "$user:$pass" | sudo chpasswd -c MD5
done < /home/admin/user_list.txt
  • Pros: Standard tool specifically built for updating passwords from standard input; explicit algorithm selection via -c MD5.
  • Cons: Makes two system tool invocations per line instead of one.

Approach B: Python Scripting with Standard Libraries

When shell scripts become complex, Python offers robust string handling and system module integration:

#!/usr/bin/env python3
import crypt
import subprocess

with open('/home/admin/user_list.txt', 'r') as f:
    for line in f:
        line = line.strip()
        if not line:
            continue
        user, raw_pass = line.split(';')
        # Generate MD5 crypt hash ($1$)
        hashed_pass = crypt.crypt(raw_pass, crypt.METHOD_MD5)
        subprocess.run(['sudo', 'useradd', '-m', '-p', hashed_pass, user], check=True)
  • Pros: Cleaner handling of edge cases (such as whitespace or complex quotes in passwords).
  • Cons: Requires Python environment execution permissions.

Approach C: Declarative Provisioning via Ansible

For enterprise infrastructure, user accounts should be managed declaratively:

- name: Provision users from list
  hosts: localhost
  tasks:
    - name: Read user list
      ansible.builtin.slurp:
        src: /home/admin/user_list.txt
      register: user_file

    - name: Create local user accounts
      ansible.builtin.user:
        name: "{{ item.split(';')[0] }}"
        password: "{{ item.split(';')[1] | password_hash('md5') }}"
        create_home: yes
      loop: "{{ (user_file.content | b64decode).splitlines() }}"
  • Pros: Fully idempotent, audit-ready, and scalable across hundreds of servers.
  • Cons: Requires Ansible installation and playbook configuration.

9. Linux Command Deep Dive

useradd

Low-level utility for creating user accounts.

  • Syntax: useradd [options] LOGIN

  • Common Options:

  • -m, --create-home: Create the user’s home directory.

  • -p, --password PASSWORD: Set the encrypted password (expects hashed string).

  • -s, --shell SHELL: Specify login shell (e.g., /bin/bash).

  • -g, --gid GROUP: Primary group name or ID.

  • -G, --groups GROUPS: Supplementary groups list.

  • Troubleshooting Tip: Always use -m when provisioning interactive users to ensure skeletal configuration files (.bashrc, .profile) are copied from /etc/skel.

openssl passwd

Generates hashed passwords.

  • Syntax: openssl passwd [options] [password]
  • Common Options:
  • -1: Use MD5-based password algorithm ($1$).
  • -5: Use SHA-256-based password algorithm ($5$).
  • -6: Use SHA-512-based password algorithm ($6$).
  • -salt STRING: Specify explicit salt value.

chpasswd

Batch updates user passwords from standard input.

  • Syntax: chpasswd [options]
  • Common Options:
  • -c, --crypt-method METHOD: Algorithm to use (MD5, SHA256, SHA512, YESCRYPT).
  • -e, --encrypted: Supplied passwords are already encrypted.

file

Determines file type and text encoding.

  • Syntax: file [options] filename
  • Common Options:
  • -b, --brief: Do not prepend filenames to output.
  • -i, --mime: Output MIME type strings instead of human-readable descriptions.

10. Underlying Linux Concepts and Architecture

1. /etc/passwd vs. /etc/shadow

Linux separates user identity metadata from authentication credentials:

  • /etc/passwd (World-Readable): Contains public account attributes:
username:x:UID:GID:Comment:HomeDirectory:Shell

The x in field 2 signifies that the password hash is securely stored in /etc/shadow.

  • /etc/shadow (Root-Readable Only): Contains restricted authentication hashes:
username:$id$salt$hash:LastChange:Min:Max:Warn:Inact:Expire:Flag

2. Password Hash Fields Explained

A shadow hash field $1$r8X9/kQ2$d.vX5mJ7gS1... breaks down as follows:

Field Example Value Meaning
Algorithm ID ($id) $1$ Identifies algorithm: $1$ = MD5, $5$ = SHA-256, $6$ = SHA-512, $y$ = Yescrypt.
Salt r8X9/kQ2 Random string appended to password before hashing to prevent rainbow table attacks.
Hash d.vX5mJ7gS1... The actual cryptographic hash output.

3. Bash Internal Field Separator (IFS)

IFS controls how Bash splits fields during string reading and array operations. Setting IFS=';' before read restricts field boundaries to semicolons without affecting global shell behavior outside the execution block.


11. Real-World Sysadmin and SRE Applications

Production Security Considerations

  1. Legacy Hash Risks: MD5 ($1$) is cryptographically weak and vulnerable to rapid offline brute-force attacks. Production systems should use SHA-512 ($6$) or modern memory-hard functions like Yescrypt or Argon2id.
  2. Credential Exposure in Shell History: Passing cleartext passwords as command-line arguments can log secrets into ~/.bash_history or expose them in process tables (ps aux). Using stdin streaming or secure environment files mitigates this risk.
  3. Idempotency in Provisioning: Simple useradd loops will throw errors if executed repeatedly against existing users. Production scripts must check user existence (id "$user" &>/dev/null) before invoking creation steps.

Centralized Identity vs. Local Accounts

In enterprise SRE environments, local account creation is typically reserved for emergency break-glass administrative users or containerized base images. Centralized Identity and Access Management (IAM) tools—such as FreeIPA, OpenLDAP, Active Directory, or SSSD—are preferred to manage user lifecycle events centrally without modifying local /etc/shadow files across hosts.


12. Further Learning and Experiments

Hands-on Practice Labs

Experiment 1: Handling Windows CRLF Line Endings

Create a test file with Windows line endings and observe parsing errors:

# Create file with CRLF
echo -e "testuser1;Pass123\r" > test_crlf.txt
file test_crlf.txt

# Strip carriage returns using tr
tr -d '\r' < test_crlf.txt > test_clean.txt
file test_clean.txt

Experiment 2: Inspect Hashing Algorithms

Generate different hashes using openssl passwd and compare their prefixes:

openssl passwd -1 "MyPassword"  # MD5 ($1$)
openssl passwd -5 "MyPassword"  # SHA-256 ($5$)
openssl passwd -6 "MyPassword"  # SHA-512 ($6$)

13. Key Takeaways

  1. Verify File Encoding First: Always inspect line endings (file file.txt) and delimiters (head) prior to running text-processing loops.
  2. Understand Hashing Requirements: Align password generation tools (openssl passwd, chpasswd) with expected shadow hash formats.
  3. Use Non-Interactive useradd Flags: Supply home directory creation (-m) and pre-hashed passwords (-p) to cleanly automate account creation.
  4. Isolate Delimiters cleanly: Leverage IFS=';' read -r to parse CSV or semicolon-delimited files in pure Bash safely.

14. Quick Troubleshooting Reference / Runbook

1. Diagnostic Phase
   ├── Inspect sample lines: head -n 5 user_list.txt
   ├── Check line endings: file user_list.txt
   └── Inspect check/validation logic: cat check.sh

2. Execution Phase
   └── Run non-interactive loop:
       while IFS=';' read -r user pass; do
           sudo useradd -m -p "$(openssl passwd -1 "$pass")" "$user"
       done < user_list.txt

3. Validation Phase
   ├── Run official test script: ./check.sh
   ├── Verify user presence: getent passwd <username>
   └── Confirm hash format in shadow: sudo grep "<username>" /etc/shadow