Domain 10Linux & SREINTERMEDIATE

(Verona) Troubleshooting Apache Portal Migration: Resolving 403 Forbidden Root and Disabled .htaccess Redirects

A comprehensive technical guide to diagnosing and fixing a failed Apache web portal migration, resolving root 403 Forbidden directory errors, enabling .htaccess overrides, and validating automated test requirements.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-09
#Apache#Linux#SRE#Troubleshooting#Web Server#HTTP#Redirects

1. Introduction

Migrating web applications between servers is a routine task for Linux Systems Administrators and Site Reliability Engineers (SREs). However, migration cutovers often expose configuration mismatches between old and new environments. Common post-migration failures include altered directory structures, missing index pages, unreadable control files, and restrictive web server directives.

This article details a systematic troubleshooting exercise focused on an internal Apache web portal migrated to a new Linux host. We analyze how to diagnose two distinct HTTP failure modes: an HTTP 403 Forbidden error on the site’s root URL and an HTTP 404 Not Found error on a legacy bookmark path intended to redirect to a status endpoint.

Through this guide, you will learn:

  • How to use low-risk diagnostic tools (curl, apache2ctl, ls, cat) to isolate web server issues.
  • The internal interaction between Apache directory indexes, options, and permission directives.
  • How Apache’s AllowOverride directive controls .htaccess processing.
  • Best practices for applying configuration fixes, validating service health, and writing production-grade VirtualHost rules.

2. Problem Description and Environment

An internal Apache portal was migrated to a target Debian host running Apache 2.4.67. The web portal’s document root was relocated to /var/www/portal.

Scenario Overview

  • Scenario Name: “Verona”: Apache Portal Won’t Open
  • Difficulty: Medium
  • Access Method: Command-line shell as user admin with sudo privileges

Expected vs. Actual Behaviour

Endpoint Expected Behaviour Actual Observed Behaviour
http://localhost/ Serves the main portal homepage (200 OK). Fails to serve the page; returns HTTP 403 Forbidden.
http://localhost/reports Redirects to /status/ (302/301), returning HTTP 200 OK with "SadServers - Status OK". Fails to redirect; returns HTTP 404 Not Found.
http://localhost/status/ Serves status page (200 OK) displaying "SadServers - Status OK". Serves status page directly without issue (200 OK).

Environment Details

  • Operating System: Debian Linux
  • Web Server: Apache/2.4.67 (Debian)
  • Document Root: /var/www/portal
  • Active VirtualHost Config: /etc/apache2/sites-enabled/portal.conf
  • Validation Harness: /home/admin/agent/check.sh

3. Initial Investigation and Hypotheses

Before modifying server configuration or adding files, a systematic SRE approach requires gathering evidence to formulate testable hypotheses.

System Diagnostic Plan

  1. HTTP Inspection: Execute verbose client requests to capture exact status codes, headers, and redirection paths.
  2. Filesystem Audit: Check directory listings, permissions, ownership, and structural contents in /var/www/portal.
  3. Configuration Audit: Inspect Apache VirtualHost definitions and .htaccess configuration files.
  4. Syntax & Runtime Audit: Verify global Apache configuration syntax and active VirtualHost bindings.

Initial Hypotheses Matrix

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
H1: Missing Index Page / Disabled Indexing Root path returns 403 Forbidden. Apache defaults to denying directory browsing if no index file exists and Options -Indexes is set. Permissions on /var/www/portal permit access. Run ls -la /var/www/portal to verify index file existence.
H2: File/Directory Ownership Restrictions Apache’s www-data user lacks execution (+x) permissions on path components. Directory ownership is www-data:www-data (755). Inspect file system permissions across /var/www/portal.
H3: Disabled .htaccess Overrides /reports returns 404 Not Found. RedirectMatch rule in .htaccess is ignored if AllowOverride None is configured. .htaccess file exists on disk. Inspect /etc/apache2/sites-enabled/portal.conf for AllowOverride values.
H4: Missing Apache Rewrite/Alias Module Redirect rule fails to execute. Simple RedirectMatch directives use mod_alias, which is enabled by default in standard Debian Apache builds. Run apache2ctl -M if .htaccess configuration checks pass.

4. Troubleshooting Process

Step 1: HTTP Response Inspection

Objective

Determine the exact HTTP status codes and headers returned by the portal root and the legacy /reports endpoint.

Commands Executed

curl -Iv http://localhost/
curl -Iv http://localhost/reports

Command Explanation

  • curl: Command-line tool for transferring data with URLs.
  • -I (--head): Sends an HTTP HEAD request to fetch headers only without downloading the message body.
  • -v (--verbose): Outputs details regarding DNS resolution, TCP handshake, request headers, and response headers.

Output & Analysis

admin@i-0786b0f0e548dc52b:~$ curl -Iv http://localhost/
* Trying [::1]:80...
* Connected to localhost (::1) port 80
> HEAD / HTTP/1.1
> Host: localhost
> User-Agent: curl/8.14.1
> Accept: */*

< HTTP/1.1 403 Forbidden
< Date: Mon, 09 Nov 2026 05:10:59 GMT
< Server: Apache/2.4.67 (Debian)
< Content-Type: text/html; charset=iso-8859-1

admin@i-0786b0f0e548dc52b:~$ curl -Iv http://localhost/reports
* Trying [::1]:80...
* Connected to localhost (::1) port 80
> HEAD /reports HTTP/1.1
> Host: localhost
> User-Agent: curl/8.14.1
> Accept: */*

< HTTP/1.1 404 Not Found
< Date: Mon, 09 Nov 2026 05:11:12 GMT
< Server: Apache/2.4.67 (Debian)
< Content-Type: text/html; charset=iso-8859-1

Findings:

  1. http://localhost/ explicitly returns 403 Forbidden.
  2. http://localhost/reports explicitly returns 404 Not Found.

Decision:

Investigate the DocumentRoot filesystem to explain the root 403 Forbidden error, and check Apache VirtualHost configs for /reports mapping.


Step 2: Document Root Filesystem Audit

Objective

Inspect directory contents, hidden control files (.htaccess), ownership, and permissions inside /var/www/portal.

Command Executed

ls -la /var/www/portal

Command Explanation

  • ls: Lists directory contents.
  • -l: Uses long listing format displaying permissions, owner, group, file size, and timestamp.
  • -a: Includes hidden files (entries starting with .).

Output & Analysis

admin@i-0786b0f0e548dc52b:~$ ls -la /var/www/portal
total 16
drwxr-xr-x 3 www-data www-data 4096 Jun 16 16:07 .
drwxr-xr-x 4 root     root     4096 Jun 16 16:07 ..
-rw-r--r-- 1 www-data www-data  135 Jun 16 16:07 .htaccess
drwxr-xr-x 2 www-data www-data 4096 Jun 16 16:07 status

Findings:

  1. Directory ownership (www-data:www-data) and permissions (755) are correct.
  2. Missing Index File: There is no index.html, index.php, or similar default document in /var/www/portal.
  3. .htaccess exists in the root directory.
  4. A /status directory exists.

Decision:

Check the Apache configuration to verify how directory requests without an index file are handled and check active VirtualHost settings.


Step 3: VirtualHost Topology and Syntax Audit

Objective

Identify active Apache VirtualHosts, main configuration files, and syntax validity.

Commands Executed

sudo apache2ctl -S
sudo apache2ctl configtest

Command Explanation

  • apache2ctl: Front-end for the Apache HTTP server.
  • -S: Dumps parsed VirtualHost settings (IP addresses, ports, server names, and config file paths).
  • configtest (or -t): Parses configuration files and reports syntax errors.

Output & Analysis

admin@i-0786b0f0e548dc52b:~$ sudo apache2ctl -S
VirtualHost configuration:
*:80                    localhost (/etc/apache2/sites-enabled/portal.conf:1)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
User: name="www-data" id=33
Group: name="www-data" id=33

admin@i-0786b0f0e548dc52b:~$ sudo apache2ctl configtest
Syntax OK

Findings:

  1. Port 80 requests for localhost map directly to /etc/apache2/sites-enabled/portal.conf.
  2. Apache configuration syntax is valid (Syntax OK).

Decision:

Inspect /etc/apache2/sites-enabled/portal.conf, /var/www/portal/.htaccess, and /var/www/portal/status/.


Step 4: Site and Directory Directives Inspection

Objective

Examine the VirtualHost configuration, the .htaccess rules, and the target /status directory contents.

Commands Executed

cat /etc/apache2/sites-enabled/portal.conf
cat /var/www/portal/.htaccess
ls -la /var/www/portal/status/

Output & Analysis

admin@i-0786b0f0e548dc52b:~$ cat /etc/apache2/sites-enabled/portal.conf 
<VirtualHost *:80>
    ServerName localhost
    DocumentRoot /var/www/portal

    <Directory /var/www/portal>
        Options -Indexes +FollowSymLinks
        AllowOverride None
        Require all granted
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/portal-error.log
    CustomLog ${APACHE_LOG_DIR}/portal-access.log combined
</VirtualHost>

admin@i-0786b0f0e548dc52b:~$ cat /var/www/portal/.htaccess 
# Legacy bookmark: /reports should reach the status page (applied only when AllowOverride permits)
RedirectMatch ^/reports/?$ /status/

admin@i-0786b0f0e548dc52b:~$ ls -la /var/www/portal/status/
total 12
drwxr-xr-x 2 www-data www-data 4096 Jun 16 16:07 .
drwxr-xr-x 3 www-data www-data 4096 Jun 16 16:07 ..
-rw-r--r-- 1 www-data www-data   23 Jun 16 16:07 index.html

Findings:

  1. Root 403 Forbidden Cause: The directory block contains Options -Indexes. When Apache receives a request for / and no index document (like index.html) exists in /var/www/portal, it attempts directory listing. Because -Indexes disables directory listings, Apache returns 403 Forbidden.
  2. Legacy Redirect 404 Not Found Cause: .htaccess contains RedirectMatch ^/reports/?$ /status/. However, portal.conf sets AllowOverride None for <Directory /var/www/portal>. This forces Apache to bypass .htaccess completely, making /reports unmapped and returning 404 Not Found.

Step 5: Applying Initial Fixes

Objective

Create a default homepage file and update portal.conf to enable .htaccess processing.

Commands Executed

# 1. Create root index.html file
echo "<h1>Apache Portal Homepage</h1>" | sudo tee /var/www/portal/index.html
sudo chown www-data:www-data /var/www/portal/index.html

# 2. Update AllowOverride from None to All in VirtualHost config
sudo sed -i 's/AllowOverride None/AllowOverride All/' /etc/apache2/sites-enabled/portal.conf

# 3. Verify configuration syntax and reload Apache
sudo apache2ctl configtest
sudo systemctl reload apache2

Command Explanations

  • tee: Reads from standard input and writes to standard output and files. Using sudo tee grants root privileges to write output piped from non-root shells.
  • chown www-data:www-data: Sets user and group ownership of the file to Apache’s runtime account.
  • sed -i 's/search/replace/': In-place editing using stream editor string substitution.
  • systemctl reload apache2: Sends a SIGHUP signal to reload configuration files without interrupting active connection sockets.

Output & Analysis

admin@i-0786b0f0e548dc52b:~$ echo "<h1>Apache Portal Homepage</h1>" | sudo tee /var/www/portal/index.html
<h1>Apache Portal Homepage</h1>
admin@i-0786b0f0e548dc52b:~$ sudo chown www-data:www-data /var/www/portal/index.html 
admin@i-0786b0f0e548dc52b:~$ curl -I http://localhost/
HTTP/1.1 200 OK
Date: Mon, 09 Nov 2026 05:15:03 GMT
Server: Apache/2.4.67 (Debian)
Last-Modified: Mon, 09 Nov 2026 05:14:35 GMT
ETag: "20-65fd109abe1d1"
Accept-Ranges: bytes
Content-Length: 32
Content-Type: text/html

admin@i-0786b0f0e548dc52b:~$ sudo sed -i 's/AllowOverride None/AllowOverride All/' /etc/apache2/sites-enabled/portal.conf
admin@i-0786b0f0e548dc52b:~$ sudo apache2ctl configtest
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using fe80::8ff:c1ff:fec0:4869%ens5. Set the 'ServerName' directive globally to suppress this message
Syntax OK
admin@i-0786b0f0e548dc52b:~$ sudo systemctl reload apache2
admin@i-0786b0f0e548dc52b:~$ curl -Lv http://localhost/reports
* Connected to localhost (::1) port 80
> GET /reports HTTP/1.1
> Host: localhost

< HTTP/1.1 302 Found
< Location: http://localhost/status/
* Issue another request to this URL: 'http://localhost/status/'
> GET /status/ HTTP/1.1
> Host: localhost

< HTTP/1.1 200 OK
SadServers - Status OK

Findings:

  • Root / now returns 200 OK.
  • /reports successfully issues an HTTP 302 Found redirect to /status/, which responds with 200 OK and "SadServers - Status OK".

Step 6: Test Harness Validation and Script Inspection

Objective

Run the automated solution check script /home/admin/agent/check.sh.

Commands Executed

/home/admin/agent/check.sh
cat /home/admin/agent/check.sh

Output & Analysis

admin@i-0786b0f0e548dc52b:~$ /home/admin/agent/check.sh 
NO

admin@i-0786b0f0e548dc52b:~$ cat /home/admin/agent/check.sh
#!/usr/bin/env bash
# DO NOT MODIFY THIS FILE ("Check My Solution" will fail)

fail() {
  echo -n "NO"
  exit 0
}

root=$(curl -s -m 2 http://127.0.0.1/ | head -n 1)
if [ "$root" != "SadServers - Portal Ready" ]; then
  fail
fi

status=$(curl -s -m 2 -L http://127.0.0.1/reports | head -n 1)
if [ "$status" != "SadServers - Status OK" ]; then
  fail
fi

echo -n "OK"

Findings:

  1. The script failed (NO).
  2. Analysis of check.sh shows that the first line of output from [http://127.0.0.1/](http://127.0.0.1/) must match "SadServers - Portal Ready".
  3. Our created /var/www/portal/index.html contained <h1>Apache Portal Homepage</h1>.

Step 7: Final Content Adjustment and Solution Verification

Objective

Update /var/www/portal/index.html to meet test requirements and re-evaluate.

Commands Executed

echo "SadServers - Portal Ready" | sudo tee /var/www/portal/index.html
/home/admin/agent/check.sh

Output & Analysis

admin@i-0786b0f0e548dc52b:~$ echo "SadServers - Portal Ready" | sudo tee /var/www/portal/index.html
SadServers - Portal Ready
admin@i-0786b0f0e548dc52b:~$ /home/admin/agent/check.sh
OK

Findings:

The validation script returned OK, confirming that both functional web server issues and validation criteria are satisfied.


5. Root Cause Analysis

                              [ HTTP Request ]
                                     |
              +----------------------+----------------------+
              |                                             |
       GET / (Web Root)                             GET /reports
              |                                             |
   DocumentRoot: /var/www/portal                 Target path: /reports
              |                                             |
  Look for default Index File                    Check VirtualHost Config
    (index.html missing)                        (AllowOverride None set)
              |                                             |
   Check Directory Options                       Ignore /var/www/portal/.htaccess
    (Options -Indexes set)                       (RedirectMatch rule bypassed)
              |                                             |
   [ HTTP 403 Forbidden ]                        [ HTTP 404 Not Found ]
              |                                             |
   ==================== RESOLUTION APPLIED ====================
              |                                             |
   Create index.html with:                       Change to AllowOverride All
   "SadServers - Portal Ready"                   Apache parses .htaccess rule
              |                                             |
   [ HTTP 200 OK ]                               [ HTTP 302 Redirect -> /status/ ]

Detailed Breakdown

  1. Root Path Failure (HTTP 403 Forbidden):
  • Apache uses the DirectoryIndex directive (defaulting to index.html) to serve index documents when directories are requested.
  • When /var/www/portal/index.html was missing, Apache attempted to fall back to generating an automatic directory listing.
  • Because /etc/apache2/sites-enabled/portal.conf specified Options -Indexes, Apache explicitly prohibited directory browsing.
  • In compliance with security defaults, Apache rejected the client request with HTTP 403 Forbidden.
  1. Legacy Bookmark Failure (HTTP 404 Not Found):
  • The redirection rule RedirectMatch ^/reports/?$ /status/ was located in /var/www/portal/.htaccess.
  • The active site configuration /etc/apache2/sites-enabled/portal.conf defined AllowOverride None for <Directory /var/www/portal>.
  • The AllowOverride None directive disables .htaccess parsing to optimize performance. Consequently, Apache never processed the redirection rule, resulting in an HTTP 404 Not Found response.

6. Solution and Remediation

Actions Executed

  1. Create Index Document: Created /var/www/portal/index.html containing "SadServers - Portal Ready" and set ownership to www-data:www-data.
  2. Enable Directory Overrides: Updated /etc/apache2/sites-enabled/portal.conf to change AllowOverride None to AllowOverride All.
  3. Reload Web Server: Tested configuration syntax (sudo apache2ctl configtest) and gracefully reloaded Apache (sudo systemctl reload apache2).

Configuration Diff

--- /etc/apache2/sites-enabled/portal.conf.orig
+++ /etc/apache2/sites-enabled/portal.conf
@@ -4,7 +4,7 @@
 
     <Directory /var/www/portal>
         Options -Indexes +FollowSymLinks
-        AllowOverride None
+        AllowOverride All
         Require all granted
     </Directory>

7. Verification and Validation

Verification steps confirmed that fixes resolved the underlying problems and persisted across operational reloads:

  1. Root URL Check:
curl -s http://localhost/
  • Output: SadServers - Portal Ready (200 OK).
  1. Redirect URL Check:
curl -Lv http://localhost/reports
  • Output: Returns HTTP 302 Found with Location: http://localhost/status/, followed by HTTP 200 OK and "SadServers - Status OK".
  1. Validation Harness Execution:
/home/admin/agent/check.sh
  • Output: OK.

8. Alternative Troubleshooting Approaches

While editing .htaccess resolved the issue, production environments often use different diagnostic and remediation techniques.

Production Alternative: Native VirtualHost Directives

In high-traffic production environments, relying on .htaccess introduces performance penalties because Apache must check for .htaccess files on every file path request. An SRE best practice is to move redirects into the main VirtualHost file while keeping AllowOverride None enabled.

<VirtualHost *:80>
    ServerName localhost
    DocumentRoot /var/www/portal

    # Native VirtualHost redirect (bypasses .htaccess)
    RedirectMatch 301 ^/reports/?$ /status/

    <Directory /var/www/portal>
        Options -Indexes +FollowSymLinks
        AllowOverride None
        Require all granted
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/portal-error.log
    CustomLog ${APACHE_LOG_DIR}/portal-access.log combined
</VirtualHost>

Comparison Matrix of Diagnostic Methods

Method Utility / Tool Target Information Advantages Limitations
HTTP Inspection curl -Iv <URL> Status codes, headers, redirect paths. Quick, non-intrusive, isolated client view. Does not reveal server-side config causes directly.
Log Analysis tail -f /var/log/apache2/portal-error.log Server internal errors, permission denials. Direct insight into Apache engine decisions. Log level settings must be sufficiently verbose.
Process / Socket Audit ss -tulpn or lsof -i :80 PID, listening ports, process bindings. Identifies port conflicts or wrong daemon binaries. Doesn’t inspect application-layer HTTP logic.
VirtualHost Dump apache2ctl -S Active VHosts, file locations, document roots. Rapid structural layout mapping. Does not evaluate .htaccess rules.

9. Linux Command Deep Dive

Command 1: curl

Purpose: Command-line network tool for transferring data using protocols including HTTP, HTTPS, FTP, and SSH.

General Syntax:

curl [options] [URL]

Options Reference:

Option Long Option Description
-I --head Fetch headers only (uses HTTP HEAD method).
-v --verbose Output detailed request/response headers and connection metadata.
-L --location Automatically follow HTTP 3xx redirects.
-s --silent Silent mode; suppresses progress meters and messages.
-m <secs> --max-time Maximum time in seconds allowed for the transfer.
-k --insecure Allow insecure SSL/TLS connections (bypasses certificate validation).

Examples:

# Debug redirect headers and final page content
curl -Lv http://localhost/reports

# Check response time and HTTP status code silently
curl -s -o /dev/null -w "%{http_code}\n" http://localhost/

Command 2: apache2ctl / httpd

Purpose: Administrative control utility for the Apache HTTP server daemon.

General Syntax:

apache2ctl [command] [options]

Options & Commands Reference:

Command/Option Description
-S (or DUMP_VHOSTS) Displays parsed VirtualHost configuration details and line numbers.
configtest (or -t) Validates configuration files for syntax errors without restarting.
-M (or DUMP_MODULES) Lists all loaded static and shared Apache modules.
graceful Reloads configuration files without terminating open connections.

Examples:

# Verify loaded modules to confirm mod_alias or mod_rewrite availability
sudo apache2ctl -M | grep -E 'alias|rewrite'

# Test syntax before reloading daemon
sudo apache2ctl configtest && sudo systemctl reload apache2

Command 3: sed (Stream Editor)

Purpose: Non-interactive text stream editor used for filtering and transforming text files.

General Syntax:

sed [options] 'script' [input-file]

Options Reference:

Option Description
-i[SUFFIX] Edits files in place (creates backup if suffix is provided, e.g., -i.bak).
-e script Adds script commands to be executed.
-n Suppresses automatic printing of pattern space.

Examples:

# Substitute string in file in-place with backup file creation
sudo sed -i.bak 's/AllowOverride None/AllowOverride All/' /etc/apache2/sites-enabled/portal.conf

# Print lines 10 through 20 of a configuration file
sed -n '10,20p' /etc/apache2/apache2.conf

10. Underlying Linux Concepts and Architecture

1. Apache Directory Indexing Engine

When Apache processes a request mapping to a directory path:

  1. It queries the DirectoryIndex directive (typically defaults to index.html index.php).
  2. If an index file is found in the target directory, it serves the file with an HTTP 200 OK status.
  3. If no index file exists, Apache evaluates directory options:
  • Options +Indexes: Generates an automated HTML file directory listing (HTTP 200 OK).
  • Options -Indexes: Prohibits directory listing generation and issues an HTTP 403 Forbidden error.

2. The .htaccess Mechanism and AllowOverride

.htaccess files allow decentralized configuration per directory tree:

  • When a request arrives, Apache checks every parent directory down to the target path for a .htaccess file.
  • Performance Impact: Enabling .htaccess lookups causes disk I/O performance penalties because Apache performs multiple filesystem checks per HTTP request.
  • AllowOverride Scope:
  • None: Apache completely ignores .htaccess files (best performance).
  • FileInfo: Enables rewrite/redirect directives (Redirect, RewriteRule).
  • All: Permits any valid .htaccess directive.

3. Graceful Reloads (SIGHUP) vs. Restarts (SIGTERM/SIGKILL)

  • systemctl reload apache2: Sends a SIGHUP signal. The main process re-reads configuration files while worker child processes finish handling active requests before restarting. This prevents dropped connections.
  • systemctl restart apache2: Kills active worker threads immediately and launches a fresh daemon, causing brief connection drops.

11. Real-World Sysadmin and SRE Applications

Production Incident Prevention & Observability

  1. Automated Synthetic Monitoring: Configure synthetic health probes (e.g., Datadog, Prometheus Blackbox Exporter) to continuously check status endpoints (/status/) and root paths for expected string matches like "SadServers - Portal Ready".
  2. Centralized Logging: Configure custom log formatting in portal.conf to track redirection metrics:
CustomLog ${APACHE_LOG_DIR}/portal-access.log "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
  1. Configuration Management (Ansible): Ensure standard web server configurations prevent drift:
- name: Ensure portal configuration enforces AllowOverride
  ansible.builtin.lineinfile:
    path: /etc/apache2/sites-available/portal.conf
    regexp: '^\s*AllowOverride'
    line: '        AllowOverride All'
  notify: Reload Apache

12. Further Learning and Experiments

Hands-on Experiments

Experiment 1: Measuring .htaccess Performance Overhead

  • Goal: Observe the performance cost of setting AllowOverride All versus AllowOverride None.
  • Steps:
  1. Install apachebench (sudo apt install apache2-utils).
  2. Measure requests per second with AllowOverride All:
ab -n 10000 -c 10 http://localhost/
  1. Move redirect rules into portal.conf, switch AllowOverride None, reload Apache, and repeat the benchmark.
  • Learning: Quantify latency and disk I/O differences when bypassing .htaccess.

Experiment 2: Custom DirectoryIndex Precedence

  • Goal: Test Apache’s fallback sequence across index file types.
  • Steps:
  1. Add DirectoryIndex index.php index.html to portal.conf.
  2. Create both /var/www/portal/index.php and /var/www/portal/index.html.
  3. Query http://localhost/ and verify which file takes precedence.

13. Key Takeaways

  1. 403 Forbidden on Directory Paths: Frequently caused by missing index files (index.html) coupled with Options -Indexes.
  2. Ignored .htaccess Rules: If .htaccess rules fail to execute, check for AllowOverride None in parent <Directory> blocks.
  3. Always Run Syntax Checks: Validate configuration files using apache2ctl configtest prior to reloading services.
  4. Use Graceful Reloads: Favor systemctl reload over restart in production to avoid dropping active client sockets.
  5. Inspect Test Scripts: When troubleshooting lab scenarios, read validation scripts (check.sh) to understand explicit content matching requirements.

14. Quick Troubleshooting Reference / Runbook

                         WEBSERVER TROUBLESHOOTING RUNBOOK
                         =================================

1. INSPECT CLIENT HTTP RESPONSE
   $ curl -Iv http://<TARGET_HOST>/<PATH>
   * Check status codes: 403 (Permissions/Indexes), 404 (Missing File/Redirect), 500 (Config Error)

2. AUDIT VIRTUALHOST MAPPINGS & SYNTAX
   $ sudo apache2ctl -S
   $ sudo apache2ctl configtest

3. VERIFY FILESYSTEM PATHS & PERMISSIONS
   $ ls -la /var/www/<TARGET_DIR>
   * Ensure www-data user has read/execute (+rx) rights.
   * Verify index.html existence if directory root is queried.

4. CHECK DIRECTORY OVERRIDES & REDIRECTS
   $ cat /etc/apache2/sites-enabled/<SITE_CONFIG>.conf
   * If using .htaccess: Ensure 'AllowOverride All' or 'AllowOverride FileInfo' is set.

5. APPLY FIX & RELOAD SAFELY
   $ sudo apache2ctl configtest
   $ sudo systemctl reload apache2

6. VERIFY END-TO-END HEALTH
   $ curl -Lv http://<TARGET_HOST>/<PATH>