s3-proxy

A small self-hosted S3/S3-compatible caching reverse proxy designed to run as one Docker image configured entirely through environment variables.

Features

• S3-compatible origin over HTTP(S), including AWS SigV4 and anonymous/public buckets
• ETag revalidation (If-None-Match) on stale metadata
• Cache-Control aware TTL (s-maxage, max-age, no-store, private, no-cache)
• custom per-path cache rules
• canonical query-string sorting for cache keys
• directory index documents, so / and /docs/ serve index.html
• SPA fallback mode that does not turn missing assets into index.html
• custom 404 and custom 4xx/5xx pages from S3 while preserving the real status code
• 8 MiB block / 2 MiB segment disk cache by default
• Range support: only missing aligned segments are fetched from S3; contiguous misses are coalesced into bounded origin fetch runs
• full GET and single Range GET reuse cached segments and use a bounded download window; no whole-object buffering in RAM
• disk max size with scan-resistant probation/protected block-level eviction
• short metadata-only negative caching for origin 404 and 403 lookups
• stale-if-error for completely cached objects
• local health endpoint, access log, and graceful shutdown that drains in-flight downloads
• GET/HEAD/OPTIONS only

Block and segment model

Default configuration:

CACHE_BLOCK_SIZE=8MiB
CACHE_SEGMENT_SIZE=2MiB

A block is the management/eviction unit. A segment is the fetch/cache unit.

8 MiB block

0 MiB      2          4          6          8 MiB
|----------|----------|----------|----------|
 segment 0  segment 1  segment 2  segment 3
   2 MiB      2 MiB      2 MiB      2 MiB

A block does not need to be complete. If only segments 1 and 3 have been requested, the cache may contain:

block 0
[ MISS ][ HIT ][ MISS ][ HIT ]

The missing segments simply remain absent until a later request needs them.

Example: Range: 2MiB-10MiB

With 8 MiB blocks and 2 MiB segments, the requested bytes align to exactly four segments:

block 0
0----2====4====6====8 MiB
     [ S1 ][ S2 ][ S3 ]

block 1
8====10---12---14---16 MiB
[ S0 ]

The four missing segments are contiguous, so the proxy issues one origin request:

Range: bytes=2097152-10485759

The 8 MiB response is streamed into four independent 2 MiB segment files. A partial-hit layout such as [HIT][MISS][MISS][HIT][MISS][MISS][MISS] produces two origin requests, one for each contiguous missing run. Cached gaps are never downloaded merely to join the requests, and a whole eviction block is not used as the origin I/O unit.

For a non-aligned request such as 2.3MiB-10.1MiB, the proxy fetches the aligned 2 MiB segments that overlap it. The maximum unused prefix before the first requested byte is therefore less than 2 MiB.

Large-file memory behavior

Full GET and single Range GET share the same segment pipeline. Cached segments are read locally; adjacent missing segments within the active window are fetched with one aligned origin Range GET and committed separately. Disjoint missing runs in a window may execute in parallel. Responses reach the client in byte order even when origin requests finish out of order. The current segment streams as bytes arrive, without waiting for its commit or for later segments.

# Window includes the current segment: current + next 3 by default.
CACHE_DOWNLOAD_CONCURRENCY=4
# Upper bound for one contiguous origin Range GET. The active window may impose
# a smaller bound (4 x 2 MiB = 8 MiB with the defaults).
ORIGIN_FETCH_MAX_SIZE=16MiB
# Across all clients, cap active origin fetch runs; excess runs wait.
ORIGIN_MAX_CONCURRENT_REQUESTS=32

The concurrency settings accept 1–256. ORIGIN_FETCH_MAX_SIZE must be at least one segment. The global limit counts origin fetch runs rather than individual segments; metadata HEAD requests and uncached BYPASS requests are outside this limit. Waiting is cancellable, but does not promise strict FIFO ordering. Only the current window is scheduled, not all segments in a large file.

Each missing segment temporarily retains its bytes in memory so concurrent readers can consume the same in-flight download at different speeds. At the default settings, each request retains at most 4 × 2 MiB = 8 MiB of body data, plus allocation and network buffers. Memory scales with active clients and the configured window/segment size, not the size of the complete object. Shared fills retain one copy of the data. Completed read-ahead buffers held by slow clients are not covered by the active-origin limit of 32.

A full GET reuses a cached prefix even if the rest of the object is missing. X-Cache: MISS can therefore include cached bytes. A single Range GET fetches only missing aligned segments overlapping its requested range. Full cache hits continue to use the disk-only path.

When a client disconnects, its subscriptions and queued work are cancelled. Completed segments stay cached; incomplete segments are discarded. A shared fill continues as long as another client still needs it. The proxy does not finish downloading the whole object in the background after a disconnect. Only the first chunk of each streamed segment is explicitly flushed.

Disk eviction

CACHE_BLOCK_SIZE=8MiB
CACHE_SEGMENT_SIZE=2MiB
CACHE_MAX_DISK_SIZE=100GB

Disk accounting is based on the segment files actually present. Eviction, however, is performed at the 8 MiB block level.

For example, if a block currently contains only two cached 2 MiB segments, it consumes roughly 4 MiB. If that block becomes the LRU victim, both cached segments are removed together.

New blocks enter a probation tier. A later cache access promotes a block into a protected tier capped at 80% of the disk budget. Eviction chooses the oldest probation block first, then the oldest protected block. A one-time sequential download therefore churns primarily within the probation area instead of displacing repeatedly used assets. The tier bit and access times are in-memory metadata; after restart all scanned blocks begin on probation and can be promoted again on access.

An object larger than the entire cache can still be streamed. As the disk limit is reached, older blocks are evicted while newer/hotter blocks remain.

ETag / object version behavior

ETag, object size, and Last-Modified belong to the parent object, not to individual segments.

When revalidation reports the same ETag, existing segments remain valid. If the object version changes, every segment belonging to the previous version becomes logically inaccessible immediately. New Range requests populate segments under the new version.

Old-version segment files remain eligible for disk LRU cleanup.

Origin segment requests send If-Match when metadata has an ETag. A failed precondition or mismatched response ETag expires the metadata and fails the response, preventing old cached bytes from being combined with a new version. This applies equally to full GET and Range GET. If bytes have already reached the client, the response is truncated; the next request revalidates metadata. The origin Content-Range, Content-Length (when supplied), and actual segment length are also checked before the segment is committed.

Negative metadata cache

CACHE_NEGATIVE_TTL_404=30s
CACHE_NEGATIVE_TTL_403=10s

An origin HEAD returning 404 or 403 stores only the lookup status and its short expiry in the normal metadata namespace. No error body is cached. 5xx responses are never negative-cached. Set either duration to 0s to disable that status. An object created or made accessible immediately afterward can remain hidden until the corresponding TTL expires. Negative metadata is capped at 4096 entries; the oldest entries are removed first, including across restarts, so a crawler cannot grow missing-key metadata without bound.

SPA routing and custom error pages run after the object lookup, so a cached negative result still performs the normal SPA fallback decision and still serves the configured custom error object. The fallback/error object has its own cache key and metadata.

Directory index

S3 has no directories, so a request for a trailing-slash path has nothing to return. INDEX_DOCUMENT resolves it the way static website hosting does:

INDEX_DOCUMENT=index.html

• / -> /index.html
• /docs/ -> /docs/index.html

The resolved path is what gets cached, so / and /index.html share one cache entry instead of storing the object twice.

Only trailing-slash paths are resolved. /docs without the slash is still looked up as an object, because deciding otherwise would need a bucket listing on every miss.

Set INDEX_DOCUMENT= (empty) to disable and pass the bare path through to S3. Without it the origin answers a bucket or prefix listing for /, which is not a page and -- being a 200 -- never reaches the SPA fallback either.

SPA and 404 behavior

With SPA_MODE=true, a missing path falls back to SPA_INDEX only when the request looks like browser navigation (Accept: text/html or Sec-Fetch-Dest: document). By default, paths with a file extension do not SPA-fallback.

Examples:

• /dashboard/users/123 + HTML navigation -> /index.html, status 200
• /missing.js -> custom/default 404, status 404
• /missing.png -> custom/default 404, status 404
• /does-not-exist requested as JSON/API -> 404, not SPA fallback

Set SPA_ALLOW_DOTTED_ROUTES=true if your SPA intentionally uses dotted client-side routes.

For a custom 404 page:

ERROR_PAGE_404=/404.html

For other statuses:

ERROR_PAGES_JSON={"403":"/403.html","404":"/404.html","500":"/50x.html","502":"/50x.html","503":"/50x.html","504":"/50x.html"}

These are S3 object paths. The page body is custom, but the HTTP status remains the original status. If the custom error object cannot be loaded, the proxy emits a small built-in HTML error page.

Query sorting

Default:

CACHE_QUERY_MODE=sort

These map to the same cache key:

/image.jpg?w=100&format=webp
/image.jpg?format=webp&w=100

Modes:

• sort: parse and canonicalize parameters
• include: preserve raw query order
• ignore: do not include query string in the cache key

The query string is a cache-key concern; it is not appended to the S3 object key.

Start

cp .env.example .env
# edit .env
docker compose up --build

Then:

curl -i http://localhost:8080/path/to/object.jpg

Cache headers

X-Cache values currently include:

• HIT
• MISS
• RANGE-HIT
• RANGE-MISS
• BYPASS
• STALE
• META

Custom cache rules

First matching rule wins:

CACHE_RULES_JSON=[
  {"prefix":"/assets/","ttl":"24h","browser_ttl":"1h"},
  {"prefix":"/private/","bypass":true},
  {"suffix":".mp4","ttl":"6h"},
  {"prefix":"/avatars/","ttl":"10m","ignore_query":true},
  {"ttl":"5m"}
]

A rule with neither prefix nor suffix is a catch-all. Because the first match wins, putting one last -- as above -- makes it the default for everything the earlier rules did not claim.

bypass wins over a ttl set on the same rule: {"prefix":"/private/","bypass":true,"ttl":"10m"} stays uncached. Use separate rules if you meant otherwise.

Durations inside rules are validated at startup, so a typo such as "24hours" fails the process rather than silently leaving the rule inert.

Operations

Health endpoint

HEALTH_PATH=/healthz

Answered locally, never forwarded to S3, and kept out of the access log:

{"status":"ok","cache_bytes":51539607552,"cache_max_bytes":107374182400,"cache_blocks":6144}

The image ships a HEALTHCHECK pointing at it. Set HEALTH_PATH= (empty) to disable if you need to serve an object at that path.

Access log

ACCESS_LOG=true

One line per request -- method, path, status, bytes written, X-Cache, duration:

GET /assets/app.js 200 48213 HIT 1ms
GET /video.mp4 206 2097152 RANGE-MISS 84ms

The query string is deliberately not logged: it commonly carries signed URL tokens. The path is logged in its percent-encoded form so a request path cannot inject newlines into the log.

Graceful shutdown

SHUTDOWN_TIMEOUT=30s

On SIGTERM or SIGINT the listener stops accepting connections while in-flight downloads are given up to this long to finish, so a redeploy does not cut every large transfer mid-body.

Notes

• CACHE_BLOCK_SIZE must be an exact multiple of CACHE_SEGMENT_SIZE.
• Default: 8 MiB block / 2 MiB segment = 4 segments per block.
• Multi-range (Range: bytes=0-1,5-6) requests are streamed directly from origin instead of being segment-cached.
• Full-object misses use bounded contiguous fetch runs and split each run into segment files while forwarding it.
• Sparse Range misses fetch only aligned segments overlapping the client range and coalesce only adjacent misses.
• Existing cached segments are reused; only absent overlapping segments are fetched.
• Old object versions become unreachable when ETag/size/Last-Modified changes and are eventually removed by disk eviction.
• Whether an object is fully cached is answered from in-memory block accounting rather than by stat-ing every segment, so the check does not get slower as objects get larger. If a segment file disappears underneath the proxy, the affected block is dropped and refetched on the next request.