siftp Specification — Super Intelligent FTP

• Project: siftp (Super Intelligent FTP), https://siftp.org
• Version: 0.2.1
• License: MIT OR Apache-2.0
• Platforms: Linux x86_64 and aarch64 (primary: assembly kernels, sendfile/splice, optional io_uring). macOS runs the same code with the pooled data path and is supported for small deployments. Other Unix targets build with the pure-Rust fallbacks; Windows compiles but is not a supported deployment target (§12).
• Language/runtime: Rust (edition 2021, MSRV 1.89), tokio multi-thread runtime; the io-uring cargo feature enables the io_uring data path.

This document is normative. Code, configs/siftp.example.toml, the admin OpenAPI document, the MCP tool schemas, llms.txt, and the website must not contradict it; when they disagree, fix the code or amend this spec in the same change.

1. Goals and non-goals

1.1 Goals

1. One daemon, siftpd, serving FTP, explicit and implicit FTPS, SSH/SFTP v3, and a read-only WebDAV/HTTP gateway against the same users, virtual roots, and permissions.
2. Throughput close to kernel limits: zero-copy file transfer (sendfile/splice), pooled buffers, backpressure everywhere, an optional io_uring data plane, and hand-written assembly kernels for checksum/scanning hot paths with bit-identical Rust fallbacks.
3. Secure by default: argon2id password hashing, SSH public-key auth, per-user virtual-root jails with per-directory permissions, rate limits, login bans, TLS 1.2+, post-quantum SSH key exchange.
4. LLM-operable ("Super Intelligent"): every error is stable structured JSON {code,message,hint,docs_url}; a capability document at /.well-known/siftp.json; an OpenAPI 3.1 admin REST API; MCP tools mirroring the admin API; llms.txt; a JSON-lines audit log.
5. Boring operations: one TOML file with SIFTP_* environment overrides, a durable store for API-managed users, graceful reload and drain, health and readiness probes, Prometheus metrics.

1.2 Non-goals

• FTP extensions beyond §4 (no MFCT, no COMB, no mode B/C, no FXP server-to-server transfers — active-mode PORT to a third party is rejected as an anti-bounce measure).
• SFTP versions other than 3; legacy SCP protocol (OpenSSH scp in its default SFTP mode works); shell access; port or agent forwarding; SSH subsystems other than sftp.
• Writes over WebDAV; clustering/HA; OIDC (§12).
• A web admin GUI. The website is static documentation only.

2. Architecture

            ┌──────────────────────────────────────────────────────┐
 clients ──▶│ siftpd (crates/siftpd) — tokio runtime               │
            │  ┌──────────┐ ┌───────────┐ ┌────────────┐ ┌───────┐ │
            │  │siftp-ftp │ │siftp-sftp │ │siftp-webdav│ │siftp- │ │
            │  │FTP/FTPS  │ │SSH/SFTP v3│ │read-only   │ │admin  │ │
            │  └────┬─────┘ └─────┬─────┘ └─────┬──────┘ └───┬───┘ │
            │       └─────────────┴──────┬──────┘     siftp-mcp│     │
            │                       siftp-core ◀───────────────┘     │
            │   (auth, jail, Storage trait, errors, config, audit)   │
            │   siftp-s3 (S3 Storage)   siftp-ldap (identity source)  │
            │   siftp-asm (kernels + Rust fallback)                   │
            └──────────────────────────────────────────────────────┘

• All protocol front-ends share one siftp-core session/authorisation model: authenticate → bind a Storage (virtual root) + Access (permissions per path) → every file operation is resolved inside the root, permission-checked, and audited.
• The data plane never copies file bytes through user space on Linux when TLS is off and the root is local (sendfile/splice); with TLS, SSH, rate limits, or remote storage it uses pooled buffers. Backpressure: a transfer reads only when the socket (or ring) accepts more data; bounded queues everywhere.
• Admin and MCP are control plane only: they never move file bytes except directory listings and audit tails.

2.1 Crate layout

Dependency rule: protocol crates (ftp, sftp, webdav), backends (s3, ldap), and siftp-tier depend on siftp-core (+ siftp-asm) only; siftp-mcp depends on siftp-admin; only siftpd depends on everything and wires backends into the context (register_storage, set_identity_provider). No crate may depend "upwards" or sideways (dev-dependencies excepted).

2.2 Concurrency model

• tokio multi-thread runtime, one task per control connection, one task per data transfer. Admission is shared by FTP, FTPS, SFTP, and WebDAV: server.max_connections total and server.max_connections_per_ip per source address, checked at accept together with the ban list; per-user max_sessions is enforced at login.
• Blocking filesystem and object-storage calls run on spawn_blocking or io_uring (feature io-uring) so runtime threads never stall.
• Shared state (config, user table, session registry, ban list, counters, drain flag) lives in ServerContext; reload swaps Arc snapshots atomically without dropping sessions.

3. Virtual filesystem and jail semantics

• Every user has a virtual_root. Client paths are virtual: / is the virtual root. Resolution splits on / (and \ for FTP clients), drops empty/. segments, and rejects any .. segment that would leave the root and any NUL byte with invalid_path (FTP 553 / SSH_FX_PERMISSION_DENIED) before touching storage. Reference: siftp_core::Jail.
• Local roots: after resolution, the canonical host path must still be inside the canonical root (symlink escape check) at use time; symlinks pointing outside are followed only if they land back inside.
• Creating a symlink (SFTP SYMLINK, local roots): a relative target without .. is stored as given. Any other target is resolved as a virtual path, a relative one from the link's directory; a target that leaves the root is invalid_path, and one inside is stored as its host path under the canonical root. READLINK returns such targets as virtual paths. No link created through siftp points outside its root, also after the link or a directory above it is renamed, so a host process that follows links under the root cannot reach another user's files through one. Links placed on the host by other means are still checked at use.
• Filenames are UTF-8 (FTP OPTS UTF8 ON is a no-op success; FEAT advertises UTF8). Non-UTF-8 host filenames are listed lossily.
• Quotas (quota_bytes, 0 = unlimited) are checked on STOR/APPE, SFTP write, and SFTP SETSTAT size growth: exceeding the quota fails with quota_exceeded and removes the partial file unless the client was extending an existing file (REST, APPE).

3.1 Storage backends

Protocols program against siftp_core::Storage; a backend implements metadata, listing, positional open/read/write with explicit commit/abort, rename (with and without overwrite), mkdir/rmdir, remove, symlinks, usage, attribute changes, free space, and a canonical virtual path for permission checks.

S3 semantics: virtual /a/b is object prefix/a/b; directories are key prefixes, with an empty dir/ marker object for directories created by mkdir; reads are ranged GETs with an 8 MiB window; writes are staged in [s3].spool_dir and published on commit (single PUT up to part_size_bytes, multipart above) — an aborted upload publishes nothing and the previous object survives; rename is copy + delete (directories up to 10 000 objects); symlinks are invalid_command; modification times and mode bits cannot be set (requests succeed and are ignored); free space is reported as 1 PiB (quotas still apply). [s3].credentials selects static keys, the AWS_* environment, or the EC2 instance role (IMDSv2, refreshed five minutes before expiry).

3.2 Tiered storage

With [tier].remote set (s3://bucket/prefix, Unix only), the local roots under [server].root are a cache over an object store (siftp-tier).

• Copies. A regular file unchanged for upload_after_secs is uploaded to objects/<32 hex>, a random key, so renames never touch the store. It carries [tier].storage_class (instant-retrieval classes only) and its root-relative path as x-amz-meta-siftp-path. Hard-linked files, files open for writing, and directories on another filesystem are skipped.
• State. Each file carries one extended attribute, user.siftp.tier (58 bytes: local or remote, object id, size, mtime in ns, inode, a SHA-256 prefix, and the remote's id), written only through an open descriptor; user.siftp.tier.read records the last read. A file whose size, mtime, or inode no longer match its attribute has changed and is uploaded again under a new key.
• Eviction. A copied file nobody read or wrote for evict_after_days is replaced by a placeholder, and so are the least recently used ones while disk usage is above high_watermark_percent, until it is below low_watermark_percent. A placeholder is a sparse file with the same name, size, mtime, and mode and no data. The file is hashed again and the object's size checked first. Files below min_evict_bytes, files written within upload_after_secs, and files leased by an open writer or a pending sendfile or splice are kept. Listings, SIZE, MDTM, MLSD facts (except unique, the inode), quotas, rename, delete, and WebDAV ETags behave as for local files.
• Fetch. Opening a placeholder to read, append, or write in place, local_path (sendfile), and a SETSTAT that changes its size first download the object to a temporary file on the same filesystem, check length and hash, fsync, and rename it over the placeholder under a per-path lock. A truncating open replaces a placeholder without a fetch; STOR onto a placeholder takes the pooled path. An mtime change on a placeholder updates the placeholder and its attribute together.
• No zeros uploaded. A placeholder is never uploaded. One whose size, mtime, or inode changed outside siftp is quarantined: never uploaded or fetched over, refused with internal when opened, and counted. A non-empty file without data extents (SEEK_DATA) is never uploaded.
• Guards. <root>/.siftp-tier/ holds an exclusive lock (one daemon or restore at a time), temporary files (cleared at start), and root-id, which must equal the store's root-id. A mismatch (an empty or replaced disk, or the wrong store) makes tiering read-only: fetches work; nothing is uploaded, evicted, or written to manifests. The directory is hidden from users rooted at [server].root. Startup fails if the filesystem lacks user extended attributes or, on Linux, hole reporting.
• Manifests and restore. Each scan that found a change writes manifests/<time>-<hash>.jsonl (files with keys, sizes, mtimes, modes and hashes; directories; symlinks); the newest 48 are kept. siftpd tier-restore, with the daemon stopped, recreates the newest manifest as placeholders without replacing anything that exists, puts objects no manifest names under siftp-recovered/<recorded path>, and adopts the store's root-id.
• Nothing deleted. Objects no file refers to are counted (tier_orphans), not deleted; the bucket's lifecycle rules expire noncurrent versions and incomplete uploads.
• Worker. One scan every scan_interval_secs, sooner when a 10-second check finds disk usage above the high watermark; it stops between files on drain. Data movement is reported in tracing and the tier_* counters, not the audit log: it changes nothing a user sees.

4. Protocol matrix

FTP details:

• Greeting 220 uses [ftp].greeting; multi-line replies use the 220-…220 convention. Command lines are capped at 4 KiB (invalid_command beyond that); verbs are case-insensitive.
• LIST emits Unix-style lines; MLSD/MLST are the machine-readable formats. perm facts reflect the rights for each entry's own path.
• REST applies to the next RETR/STOR only and resets after any transfer command. TYPE A performs CRLF translation; TYPE I is byte-exact and the only mode eligible for kernel fast paths.
• HOST <name> must precede USER (503 otherwise). With [ftp].virtual_hosts set, unknown names get 504 (not_found); users with hosts = [...] may log in only under one of those names.
• MFMT YYYYMMDDHHMMSS <path> (UTC) sets the modification time; it needs the w bit and replies 213 Modify=<time>; <path>.
• RNTO replaces an existing target (rename(2) semantics).
• Passive data connections must come from the control connection's address unless [ftp].passive_promiscuous = true; others are refused and audited (DATA_REFUSED). Active mode targets only the control peer, on a port ≥ 1024.
• Control idle: [server].idle_timeout_secs; the same limit applies to each data-channel read and write (stalled transfers end with rate_limited), and TLS handshakes are bounded at 15 s.
• Anonymous FTP is off unless allow_anonymous = true; the anonymous user maps to anonymous_root with read+list permissions only.
• TLS data channels resume sessions issued on the control connection but never issue tickets themselves (an upload-only client would leave them unread and its close would reset the connection mid-file).

SFTP details:

• RENAME fails with already_exists if the target exists; posix-rename@openssh.com replaces it.
• statvfs/fstatvfs report the user's quota as the filesystem size when a quota is set (so df in sftp shows what is left).
• At most 256 open handles per session (rate_limited beyond); limits@openssh.com advertises 256 KiB packets and 255 KiB reads/writes.
• SETSTAT/FSETSTAT apply size, mtime, and permission bits; setuid, setgid, and sticky bits are stripped; uid/gid are ignored.

4.1 Read-only WebDAV/HTTP gateway

Enabled by [webdav].bind (HTTPS with [webdav].tls = true, using the FTPS certificate). Users, credentials (HTTP Basic, verified through the shared authentication path), storage, per-path permissions, rate limits, bans, and audit are the same as for FTP/SFTP. GET of a file needs r; PROPFIND and directory GET need l (HTML index, or JSON with Accept: application/json). Every write method gets permission_denied (403). File responses carry X-Content-Type-Options: nosniff and Content-Security-Policy: sandbox. Verified credentials are cached for 60 s under a keyed digest from the login verification cache's process-only key (§5.1), with the stored hash in the input, so a changed password or a deleted user stops working at once.

5. Authentication and authorisation

5.1 Passwords

• Stored only as argon2id PHC strings ($argon2id$v=19$…), generated with siftpd hash-password or server-side by the admin API (defaults m=64 MiB, t=3, p=4; never below m=19 MiB, t=2, p=1 — config load rejects weaker hashes and --check-config fails).
• Verification is constant-time (argon2's verify; bearer tokens via siftp_asm::ct_eq over SHA-256 digests). Unknown and key-only users run a dummy verification against a hash with the default parameters, so response timing does not enumerate accounts whose hashes use the defaults. Custom-cost hashes are faster or slower to reject than unknown names, in proportion to their cost.
• Verification cache: after argon2id accepts a password, the server keeps a BLAKE2b keyed digest of (user name, stored PHC string, password) for server.login_cache_ttl_secs (default 300; 0 turns the cache off), and a repeat of exactly that credential within the window is accepted without recomputing argon2id. The 256-bit key is drawn from the OS RNG at start and lives only in process memory, so neither the user store nor the cache contents can be used to test guesses offline. Only correct credentials are ever stored; a wrong password, an unknown name, or a key-only user always pays one full argon2id computation. The cache holds at most 8192 entries and is cleared whenever the user table is replaced (reload or admin change); the PHC string in the digest means a changed password misses anyway. The password-age policy is applied to cache hits too. Cached logins are counted in logins_cached as well as logins_ok.
• Concurrency: at most server.max_concurrent_password_checks argon2id computations run at once (0, the default, means one per CPU); further logins and admin hash requests wait for a slot. Each computation holds the hash's memory while it runs, so this is the memory cap for password checking.
• Age: with server.password_max_age_days > 0, a password older than that (measured from the user's password_set_unix) fails like a wrong password on the wire and is logged as expired. Users without password_set_unix never expire (--check-config warns).
• Rotation: POST /api/v1/users/{name}/password (MCP siftp_set_password) sets a given password or generates a 24-character one and returns it once; password_set_unix is updated.
• Plaintext passwords are never logged or echoed, including in audit events and API responses.

5.2 SSH public keys

• ssh_public_keys holds OpenSSH lines (ssh-ed25519, ssh-rsa, ecdsa-sha2-nistp256/384/521), optionally prefixed by options: expiry-time="YYYYMMDD[HHMM[SS]]" (UTC; the key stops working at that time) and the no-op restrictions restrict, no-pty, no-port-forwarding, no-agent-forwarding, no-x11-forwarding, no-user-rc, no-touch-required. Any other option (command=, from=, …) is a config error rather than silently ignored.
• server.revoked_keys_file lists revoked keys (OpenSSH lines or SHA256: fingerprints, # comments); it is re-read on reload. Revoked and expired keys are refused at both the offer and the signed request.
• Key offers are answered without a signature check per RFC 4252 §7; the signature is verified on the signed request. Either a password or a key authenticates (no MFA).

5.3 Permissions, groups, and directory rules

• Permission bits: r read (RETR/SFTP read/WebDAV GET), w write (STOR/APPE/SFTP write/SETSTAT/MFMT/SYMLINK), d delete (DELE/RMD/ REMOVE), n rename (both ends), m mkdir, l list (LIST/MLSD/ READDIR/STAT/PROPFIND). Missing the bit ⇒ permission_denied before any I/O.
• Base bits: the user's permissions, else the union of their [[groups]] permissions, else rl.
• Directory rules ([[users.directories]], [[groups.directories]], {path, permissions}) cover a virtual path and everything below it. The longest matching rule wins; a user rule beats group rules at the same path; group rules at the same path are unioned; rules replace the base for their subtree (they can grant or restrict).
• The check runs on the lexical path first (no I/O); when a user has rules, the operation is re-checked against the symlink-resolved path, so a link cannot reach a restricted directory with another directory's rights. Reference: siftp_core::Access.
• A session is bound to one root and one Access snapshot for its lifetime; CWD changes a virtual prefix only.

5.4 Rate limits and throttling

• rate_limit_bps (per user, 0 = unlimited) token-buckets data bytes in both directions on every protocol; rate-limited users never take the kernel fast paths (which cannot pace bytes).
• Failed logins (any protocol, including wrong admin/MCP bearer tokens): after max_failed_logins failures from one IP within ban_window_secs, the IP is refused at accept time (and by the admin API with 429) for ban_duration_secs, logged as auth_ban. Success clears the counter.
• max_connections and max_connections_per_ip (default 512 / 32) refuse excess connections with FTP 421 rate_limited (SSH: closed before the banner).

5.5 Users from the durable store

See §7.1. Store users are validated exactly like config users.

5.6 External identity: LDAP

With [ldap].url set, a login name that is not a local user (config or store) is checked against LDAP: search-then-bind (bind_dn + base_dn + user_filter, filter-escaped) or direct bind (user_dn_template, DN-escaped), over ldaps:// or StartTLS. Empty passwords are refused before any network I/O (anonymous-bind bypass). Group DNs from group_attribute map onto [[groups]] through group_map; without a mapped group the user gets default_permissions, or is refused with require_group = true. The virtual root is virtual_root_template ({user} substituted) or <server.root>/<user>. LDAP users are session-only (never stored) and go through the same jail, permissions, quotas, and audit. An unreachable directory fails logins as auth_failed on the wire (audited as internal).

6. Performance design

6.1 Zero-copy data plane (Linux)

• Plaintext RETR/TYPE I on a local root: sendfile64 loop with explicit offsets on a blocking thread. STOR: splice socket→pipe→ file, with the pipe grown to 1 MiB (F_SETPIPE_SZ) where the kernel allows it and the default 64 KiB otherwise. TLS/SSH channels and remote storage use the buffer pool. On a tiered root (§3.2), RETR of a placeholder fetches it before sendfile, and STOR/APPE onto a placeholder use the buffer pool.
• [performance].sendfile/splice disable each path; every path falls back automatically on EINVAL/ENOSYS/EOPNOTSUPP (and EPERM from seccomp) before any byte moves. Blocking sockets carry SO_SNDTIMEO/SO_RCVTIMEO equal to the idle timeout. The chosen path is counted in /api/v1/stats and /metrics.

6.2 io_uring (feature io-uring, Linux ≥ 5.6)

With the feature compiled and [performance].io_uring = true: downloads read the file with IORING_OP_READ (after sendfile in preference) and uploads write it with IORING_OP_WRITE (after splice), one ring per transfer, one entry in flight. If ring setup fails (old kernel, seccomp — Docker's default profile blocks io_uring), that transfer falls back. The feature never changes wire behaviour.

6.3 Buffer pools and backpressure

• A byte-budget pool (buffer_pool_bytes, default 64 MiB) hands out transfer_chunk_bytes (default 256 KiB) buffers; acquiring waits when exhausted — that wait is the backpressure signal.
• The pooled FTP paths overlap socket and file work. An upload fills a whole buffer from the socket (a TLS read returns one 16 KiB record) before one blocking write, and that write runs while the next buffer fills. A download reads the next chunk on the blocking pool while the current one is written (and encrypted) to the socket, but only when a second buffer is free at once; otherwise the current chunk is sent first. A pooled transfer holds up to two buffers and never waits for one while holding another, so any pool of at least one buffer makes progress.
• Control channels are line-buffered with a 4 KiB cap; SFTP packets are capped at 256 KiB (siftp_sftp::MAX_PACKET) before allocation.

6.4 Assembly kernels (siftp-asm)

Sources are .s files under crates/siftp-asm/asm/<arch>/, assembled via build.rs when the asm feature is on, target_os ∈ {linux, macos}, and target_arch ∈ {x86_64, aarch64}. Every kernel has a pure- Rust twin that is always compiled and tested and is the reference for differential tests. Runtime feature detection (SSE4.2 cpuid, aarch64 HWCAP_CRC32) gates each kernel; absent the feature, Rust serves.

Rules: (1) assembly is bit-identical to the Rust reference on all inputs; (2) no kernel allocates, panics, or reads past len; (3) without the feature backend_name() returns "rust"; (4) unsafe is confined to the FFI shims in siftp-asm and the one io_uring submission helper in siftp-ftp::fastpath, with its safety argument written out. Everything else is #![forbid(unsafe_code)] (the FTP crate uses deny).

7. Configuration

• File format: TOML. Discovery: --config, else ./siftp.toml, else configs/siftp.toml. configs/siftp.example.toml covers every key. Unknown keys are an error.
• Environment overrides: SIFTP_<SECTION>_<KEY>, one for every scalar key (list: siftp_core::config::ENV_OVERRIDE_NAMES; a test checks it against the config structs, and that website/docs/configuration.md documents every key). Precedence: env > file > default. ${ENV:NAME} interpolation works inside string values (tokens, LDAP and S3 secrets, hashes).
• Sections: [server], [ftp], [ftps], [sftp], [admin], [mcp], [logging], [performance], [s3], [ldap], [webdav], [[groups]], [[users]].
• [[users]] keys: name, password_hash, password_set_unix, ssh_public_keys, virtual_root, permissions, groups, directories, hosts, max_sessions, quota_bytes, rate_limit_bps. User names are 1–255 characters, not ./.., with no whitespace, /, \, :, or control characters.

7.1 Durable user store

• [server].user_store (default /var/lib/siftp/users.toml) holds users created or changed through the admin API or MCP, in the same [[users]] format (file mode 0600). An empty value keeps API users in memory only (warned).
• Precedence: config-file users are operator-owned and read-only through the API (permission_denied); a store entry with a config user's name is shadowed and reported by --check-config.
• Writes are atomic: temp file in the same directory, fsync, rename, directory fsync. A crash leaves the old or the new file, never a torn one; stray temp files are swept on the next write.
• Mutations take an exclusive OS lock on <store>.lock, re-read the file, apply the change, and write, so concurrent writers (the daemon and siftpd mcp-stdio) cannot lose updates.

7.2 Lifecycle

• Reload (SIGHUP or POST /api/v1/reload): re-reads config, the user store, and the revoked-key list; swaps snapshots; reopens the audit file (log rotation); reloads the FTPS certificate (a bad pair keeps the old one and the reload reports it). Live sessions keep their snapshot; listener address changes need a restart.
• Readiness: GET /readyz is 200 once every listener is bound and 503 while starting or draining; GET /healthz is 200 while the process runs.
• Drain (SIGTERM/SIGINT): stop accepting on every listener, mark not ready, tell idle sessions 421 and close them, let transfers and open SFTP uploads finish, then exit 0 — bounded by server.shutdown_grace_secs (default 30). A second signal skips the wait.

8. LLM contracts

8.1 Structured errors — everywhere

Every error on every surface derives from siftp_core::ApiError and serializes to exactly:

{"code":"permission_denied","message":"user bob lacks write permission","hint":"grant the 'w' permission bit or choose a writable path","docs_url":"https://siftp.org/docs/errors#permission_denied"}

• code is from a closed, append-only set (auth_failed, permission_denied, not_found, already_exists, invalid_path, invalid_command, rate_limited, quota_exceeded, tls_required, internal). Clients treat unknown codes as internal.
• message may change; hint says what to do next; docs_url anchors into the (unversioned) error reference.
• Mapping: admin API, MCP, and WebDAV return the JSON object (HTTP 401 auth_failed, 403 permission_denied, 404 not_found, 409 already_exists, 400 invalid_*, 429 rate_limited, 507 quota_exceeded, 426 tls_required, 500 internal; 413 for oversized admin bodies). FTP sends ErrorCode::ftp_reply with text "<code>: <message>"; SFTP sends the mapped SSH_FX_* status with "<code>: …".

8.2 Capability document

GET /.well-known/siftp.json on the admin listener (unauthenticated):

{"name":"siftp","version":"0.2.1","protocols":["ftp","ftps-explicit","ftps-implicit","sftp","webdav-readonly"],"docs":"https://siftp.org/v/0.2.1/docs/","llms_txt":"https://siftp.org/v/0.2.1/llms.txt","docs_latest":"https://siftp.org/docs","openapi":"/api/v1/openapi.json","mcp":"/mcp","metrics":"/metrics","health":"/healthz","ready":"/readyz"}

protocols lists what this server serves with its current config (FTPS only with a certificate, WebDAV only when bound). docs and llms_txt point at the documentation for this exact version (§9.1). mcp is null when [mcp].transport = "stdio". An agent's boot sequence: this document → llms_txt → openapi → operate.

8.3 Admin REST API + OpenAPI

Base: [admin].bind (default 127.0.0.1:8080), JSON bodies up to 1 MiB, Authorization: Bearer <token> on everything except /.well-known/siftp.json, /healthz, /readyz, and /api/v1/openapi.json. At most 64 concurrent requests (429 beyond) and 30 s per request. Responses carry X-Content-Type-Options: nosniff, Cache-Control: no-store, X-Frame-Options: DENY. The route table is siftp_admin::ROUTES: each route carries its parameters, request body, success status, and response schema, and the OpenAPI document is generated from it (every $ref resolves; a test enforces it).

8.4 MCP tools

Transports: stdio (siftpd mcp-stdio, always available) and streamable HTTP at POST /mcp (same bearer token and ban accounting), which [mcp].transport = "stdio" turns off: requests then get not_found. The setting is read per request, so a reload applies it. Results are the admin JSON payloads; errors are the §8.1 object in an error result. Registry: siftp_mcp::TOOLS (names are stable API).

8.5 Audit log

[logging].audit_file is append-only JSON-lines (mode 0640): {"ts":<unix>,"user":…,"action":"RETR|STOR|LOGIN|…","path":…, "result":"ok|<error code>","remote":"ip:port","seq":<n>,"chain":"<hex>"}. Passwords, keys, tokens, and file contents never appear. One line per:

Failed transfers and changes are logged with their error code as result. path is the resolved virtual path (absolute, starting with /) on every protocol; when a path does not resolve (an escape attempt), it is the text the client sent. Lookups (FTP CWD, SIZE, MDTM, LIST, MLSD, STAT; SFTP STAT, REALPATH, OPENDIR, READDIR; admin GET routes) are not logged, including when they fail.

• seq increases by one per line for the life of the chain (across restarts and rotations); chain is hex(sha256(previous chain || "\n" || line up to ",\"chain\"")), starting from 64 zeros. Editing, inserting, or deleting a line breaks every later value; siftpd audit-verify <file> [--anchor <chain>] (exit 2 on tampering) and GET /api/v1/audit/verify check it. Truncating the end cannot be detected from the file alone.
• Rotation: logrotate create + reload; the chain continues into the new file.

8.6 llms.txt rules

website/llms.txt (llmstxt.org format): an H1, a blockquote summary, then H2 sections of links to the spec, config, error, OpenAPI, and MCP references, each with one line. Every link resolves; no marketing copy; updated with any contract change; under ~200 lines. A copy is published per version (§9.1).

8.7 Observability

• GET /metrics (bearer) exposes Prometheus text: every counter in /api/v1/stats as siftp_*_total (tiered storage under tier in the JSON, as siftp_tier_* in Prometheus), transfers by data path (siftp_transfers_total{path=…}), gauges for sessions, connections, transfers in flight, banned addresses, buffer pool use, readiness, start time, and siftp_build_info. Example alerts: packaging/prometheus/alerts.yml.
• Logs go to stderr, as text or JSON ([logging].format); every event carries its span: per FTP/SFTP connection peer, session, user (and host after FTP HOST), plus one transfer complete|failed event per transfer with verb, path, bytes, data path, and duration.

9. Website information architecture (siftp.org)

Static files only. Markdown files are the canonical sources and the served .md mirrors; website/build.py (stdlib-only) renders HTML, sitemap.xml, and llms-full.txt:

website/
  index.md            what siftp is, quick start, measured numbers
  llms.txt, llms-full.txt
  docs/               quickstart, configuration, protocols, storage,
                      identity, operations, llm, performance, security,
                      errors
  spec.md             verbatim copy of SPEC.md
  download.md         releases, verification, source builds
  v/<version>/        frozen copy of the docs for each release (§9.1)

Rules: canonical URL per topic; every error code anchorable at /docs/errors#<code>; no JavaScript required; code samples runnable.

9.1 Versioned documentation

Each release freezes the docs under /v/<version>/ (HTML, Markdown mirrors, and its own llms.txt/llms-full.txt); versions.json lists them. Pages under /v/<version>/ that are not the latest carry a banner linking to the current page. The capability document (§8.2) points agents at the version they are talking to.

10. Security considerations

• Jail first: path resolution and permission checks precede all I/O; symlink escapes and symlink laundering across directory rules are re-checked at use (§3, §5.3). Run unprivileged; bind low ports with CAP_NET_BIND_SERVICE (the systemd unit does this).
• Credentials: argon2id only; SSH keys from a strict allow-list with expiry and revocation; admin token compared in constant time, required on every non-public route, and failing closed when unset; wrong tokens count toward bans.
• Transport: rustls TLS ≥ 1.2, no plaintext fallback with require_tls; optional enforced TLS session reuse on data channels; SSH with AEAD ciphers, post-quantum hybrid kex, and strict KEX (Terrapin countermeasure). RSA host keys are refused: the available RSA implementation is not constant-time for private-key operations (RUSTSEC-2023-0071); client RSA keys are verify-only.
• Abuse resistance: bans, shared connection caps, command/packet/ handle caps, idle and handshake timeouts, data-peer pinning, anti-bounce PORT/EPRT, admin request limits, transient accept errors retried rather than fatal.
• No secret leakage: passwords, keys, tokens, and file bytes never enter logs, errors, or API responses; WebDAV caches only credential digests.
• Supply chain: Cargo.lock committed; cargo deny (advisories, licenses, bans — OpenSSL banned, sources) and cargo audit in CI; cargo-fuzz targets for FTP lines, SFTP packets, config, and paths run in CI and their seed corpora replay in cargo test; releases are signed (Sigstore) with SBOMs and provenance attestations.

11. Acceptance tests

Run in CI on Linux x86_64 and aarch64 (and macOS for the dev lane):

1. cargo build/test/clippy -D warnings on the workspace, also with --features asm (siftp-asm) and --features io-uring (siftp-ftp).
2. FTP with curl, lftp, and Python ftplib (tests/interop/run.sh): login, PASV/EPSV transfers byte-identical, MLSD/SIZE/MDTM/MFMT, REST resume, mirror both ways, .. refused 553, read-only STOR 550.
3. FTPS explicit (curl, lftp, ftplib with PROT P) and implicit; with require_tls, plaintext PASS gets 534; session reuse enforced when configured.
4. SFTP with OpenSSH sftp/scp and a russh client: pubkey + password, put/get byte-identical, readdir, rename semantics, statvfs, symlinks, jail escape refused, oversized packet rejected, expired/revoked keys refused.
5. Auth/abuse: bans trip and expire; quota exceeded removes the partial file; caps refuse; drain completes an in-flight upload.
6. LLM surface: capability document; create a user via the admin API and log in over FTP; every MCP tool once over stdio; error bodies match §8.1; audit lines parse and the chain verifies.
7. Performance smoke (Linux): sendfile/splice/io_uring counters advance on real transfers; asm kernels match references; bench/run.sh reproduces the published table.
8. Config: --check-config accepts the substituted example and rejects unknown keys and weak hashes; env overrides win; reload adds a user without dropping a session; API users survive restart, reload, and SIGKILL mid-write.
9. Storage: the FTP and SFTP suites pass on S3 roots (in-process mock in cargo test; VersityGW in CI).

12. Roadmap

• OIDC for the admin API; writes over WebDAV; per-user bandwidth pools shared across sessions.
• Clustering (shared user store and ban list).
• Windows service integration (the workspace compiles for Windows; it is not a supported deployment target).

13. Stage map

Stage 1 (spec + scaffold) → config/auth core → FTP/FTPS → SSH/SFTP → data-plane fast paths → admin API → MCP → website → production (storage backends, identity, durability, lifecycle, observability, hardening, packaging, CI, benchmarks). No stage merges with a red build or a placeholder where this spec defines behaviour.