# siftp Specification — Super Intelligent FTP

- Project: **siftp** (Super Intelligent FTP), https://siftp.org
- Version: 0.2.1
- License: MIT OR Apache-2.0
- Platforms: Linux x86_64 and aarch64 (primary: assembly kernels,
  `sendfile`/`splice`, optional io_uring). macOS runs the same code with
  the pooled data path and is supported for small deployments. Other
  Unix targets build with the pure-Rust fallbacks; Windows compiles but
  is not a supported deployment target (§12).
- Language/runtime: Rust (edition 2021, MSRV 1.89), tokio multi-thread
  runtime; the `io-uring` cargo feature enables the io_uring data path.

This document is normative. Code, `configs/siftp.example.toml`, the admin
OpenAPI document, the MCP tool schemas, `llms.txt`, and the website must
not contradict it; when they disagree, fix the code or amend this spec in
the same change.

## 1. Goals and non-goals

### 1.1 Goals

1. One daemon, `siftpd`, serving FTP, explicit and implicit FTPS,
   SSH/SFTP v3, and a read-only WebDAV/HTTP gateway against the same
   users, virtual roots, and permissions.
2. Throughput close to kernel limits: zero-copy file transfer
   (`sendfile`/`splice`), pooled buffers, backpressure everywhere, an
   optional io_uring data plane, and hand-written assembly kernels for
   checksum/scanning hot paths with bit-identical Rust fallbacks.
3. Secure by default: argon2id password hashing, SSH public-key auth,
   per-user virtual-root jails with per-directory permissions, rate
   limits, login bans, TLS 1.2+, post-quantum SSH key exchange.
4. LLM-operable ("Super Intelligent"): every error is stable structured
   JSON `{code,message,hint,docs_url}`; a capability document at
   `/.well-known/siftp.json`; an OpenAPI 3.1 admin REST API; MCP tools
   mirroring the admin API; `llms.txt`; a JSON-lines audit log.
5. Boring operations: one TOML file with `SIFTP_*` environment
   overrides, a durable store for API-managed users, graceful reload and
   drain, health and readiness probes, Prometheus metrics.

### 1.2 Non-goals

- FTP extensions beyond §4 (no `MFCT`, no `COMB`, no mode B/C, no FXP
  server-to-server transfers — active-mode `PORT` to a third party is
  rejected as an anti-bounce measure).
- SFTP versions other than 3; legacy SCP protocol (OpenSSH `scp` in its
  default SFTP mode works); shell access; port or agent forwarding; SSH
  subsystems other than `sftp`.
- Writes over WebDAV; clustering/HA; OIDC (§12).
- A web admin GUI. The website is static documentation only.

## 2. Architecture

```
            ┌──────────────────────────────────────────────────────┐
 clients ──▶│ siftpd (crates/siftpd) — tokio runtime               │
            │  ┌──────────┐ ┌───────────┐ ┌────────────┐ ┌───────┐ │
            │  │siftp-ftp │ │siftp-sftp │ │siftp-webdav│ │siftp- │ │
            │  │FTP/FTPS  │ │SSH/SFTP v3│ │read-only   │ │admin  │ │
            │  └────┬─────┘ └─────┬─────┘ └─────┬──────┘ └───┬───┘ │
            │       └─────────────┴──────┬──────┘     siftp-mcp│     │
            │                       siftp-core ◀───────────────┘     │
            │   (auth, jail, Storage trait, errors, config, audit)   │
            │   siftp-s3 (S3 Storage)   siftp-ldap (identity source)  │
            │   siftp-asm (kernels + Rust fallback)                   │
            └──────────────────────────────────────────────────────┘
```

- All protocol front-ends share one `siftp-core` session/authorisation
  model: authenticate → bind a `Storage` (virtual root) + `Access`
  (permissions per path) → every file operation is resolved inside the
  root, permission-checked, and audited.
- The data plane never copies file bytes through user space on Linux
  when TLS is off and the root is local (`sendfile`/`splice`); with TLS,
  SSH, rate limits, or remote storage it uses pooled buffers.
  Backpressure: a transfer reads only when the socket (or ring) accepts
  more data; bounded queues everywhere.
- Admin and MCP are *control plane only*: they never move file bytes
  except directory listings and audit tails.

### 2.1 Crate layout

| Crate | Responsibility | Key public surface |
| --- | --- | --- |
| `siftp-core` | Error contract, jail and `Storage` trait, permissions and `Access`, users, durable store, config, context, audit, stats | `ApiError`, `ErrorCode`, `Storage`, `JailedFs`, `Access`, `ServerContext`, `AuditLog`, `capability_document` |
| `siftp-asm` | Assembly kernels + pure-Rust fallbacks, backend selection | `crc32c`, `find_newline`, `ct_eq`, `backend_name`; `asm/<arch>/*.s` |
| `siftp-ftp` | FTP/FTPS control + data channels, TLS context | `Command`, `SUPPORTED_COMMANDS`, `serve_ftp`, `serve_implicit_ftps`, `FtpsTls` |
| `siftp-sftp` | SSH transport + SFTP v3 subsystem | `PacketType`, `StatusCode`, `serve_sftp`, `load_or_generate_host_key`, `EXTENSIONS` |
| `siftp-webdav` | Read-only WebDAV/HTTP gateway | `serve_webdav`, `router` |
| `siftp-s3` | S3-compatible `Storage` backend and tier object store (+ test mock) | `S3Storage`, `S3Factory`, `S3ObjectStore`, `CredentialProvider` |
| `siftp-tier` | Tiered storage: local roots as a cache over an object store (§3.2) | `TierManager`, `TieredStorage`, `restore` |
| `siftp-ldap` | LDAP identity source | `LdapProvider` |
| `siftp-admin` | Admin REST API, OpenAPI, health/readiness, metrics | `ROUTES`, `openapi_document`, `router`, `service::*` |
| `siftp-mcp` | MCP tool registry and dispatch over the admin service | `TOOLS`, `call_tool`, `serve_stdio`, `http_router` |
| `siftpd` | Binary: config, listeners, signals, subcommands | `serve`, `--check-config`, `hash-password`, `mcp-stdio`, `audit-verify`, `health`, `tier-restore` |

Dependency rule: protocol crates (`ftp`, `sftp`, `webdav`), backends
(`s3`, `ldap`), and `siftp-tier` depend on `siftp-core` (+ `siftp-asm`)
only; `siftp-mcp`
depends on `siftp-admin`; only `siftpd` depends on everything and wires
backends into the context (`register_storage`, `set_identity_provider`).
No crate may depend "upwards" or sideways (dev-dependencies excepted).

### 2.2 Concurrency model

- tokio multi-thread runtime, one task per control connection, one task
  per data transfer. Admission is shared by FTP, FTPS, SFTP, and WebDAV:
  `server.max_connections` total and `server.max_connections_per_ip`
  per source address, checked at accept together with the ban list;
  per-user `max_sessions` is enforced at login.
- Blocking filesystem and object-storage calls run on `spawn_blocking`
  or io_uring (feature `io-uring`) so runtime threads never stall.
- Shared state (config, user table, session registry, ban list,
  counters, drain flag) lives in `ServerContext`; reload swaps `Arc`
  snapshots atomically without dropping sessions.

## 3. Virtual filesystem and jail semantics

- Every user has a `virtual_root`. Client paths are *virtual*: `/` is the
  virtual root. Resolution splits on `/` (and `\` for FTP clients), drops
  empty/`.` segments, and rejects any `..` segment that would leave the
  root and any NUL byte with `invalid_path` (FTP 553 /
  SSH_FX_PERMISSION_DENIED) *before* touching storage. Reference:
  `siftp_core::Jail`.
- Local roots: after resolution, the canonical host path must still be
  inside the canonical root (symlink escape check) at use time; symlinks
  pointing outside are followed only if they land back inside.
- Creating a symlink (SFTP `SYMLINK`, local roots): a relative target
  without `..` is stored as given. Any other target is resolved as a
  virtual path, a relative one from the link's directory; a target that
  leaves the root is `invalid_path`, and one inside is stored as its host
  path under the canonical root. `READLINK` returns such targets as
  virtual paths. No link created through siftp points outside its root,
  also after the link or a directory above it is renamed, so a host
  process that follows links under the root cannot reach another user's
  files through one. Links placed on the host by other means are still
  checked at use.
- Filenames are UTF-8 (FTP `OPTS UTF8 ON` is a no-op success; FEAT
  advertises `UTF8`). Non-UTF-8 host filenames are listed lossily.
- Quotas (`quota_bytes`, 0 = unlimited) are checked on `STOR`/`APPE`,
  SFTP write, and SFTP `SETSTAT` size growth: exceeding the quota fails
  with `quota_exceeded` and removes the partial file unless the client
  was extending an existing file (`REST`, `APPE`).

### 3.1 Storage backends

Protocols program against `siftp_core::Storage`; a backend implements
metadata, listing, positional open/read/write with explicit
`commit`/`abort`, rename (with and without overwrite), mkdir/rmdir,
remove, symlinks, usage, attribute changes, free space, and a canonical
virtual path for permission checks.

| `virtual_root` | Backend | Notes |
| --- | --- | --- |
| a host path | `JailedFs` (local) | symlink-proof jail; kernel fast paths available |
| `s3://bucket/prefix` | `S3Storage` (`siftp-s3`) | `[s3]` endpoint and credentials; SigV4 presigned requests over rustls |

S3 semantics: virtual `/a/b` is object `prefix/a/b`; directories are key
prefixes, with an empty `dir/` marker object for directories created by
mkdir; reads are ranged GETs with an 8 MiB window; writes are staged in
`[s3].spool_dir` and published on commit (single PUT up to
`part_size_bytes`, multipart above) — an aborted upload publishes
nothing and the previous object survives; rename is copy + delete
(directories up to 10 000 objects); symlinks are `invalid_command`;
modification times and mode bits cannot be set (requests succeed and
are ignored); free space is reported as 1 PiB (quotas still apply).
`[s3].credentials` selects static keys, the `AWS_*` environment, or the
EC2 instance role (IMDSv2, refreshed five minutes before expiry).

### 3.2 Tiered storage

With `[tier].remote` set (`s3://bucket/prefix`, Unix only), the local
roots under `[server].root` are a cache over an object store
(`siftp-tier`).

- **Copies.** A regular file unchanged for `upload_after_secs` is
  uploaded to `objects/<32 hex>`, a random key, so renames never touch
  the store. It carries `[tier].storage_class` (instant-retrieval
  classes only) and its root-relative path as `x-amz-meta-siftp-path`.
  Hard-linked files, files open for writing, and directories on another
  filesystem are skipped.
- **State.** Each file carries one extended attribute, `user.siftp.tier`
  (58 bytes: local or remote, object id, size, mtime in ns, inode, a
  SHA-256 prefix, and the remote's id), written only through an open
  descriptor; `user.siftp.tier.read` records the last read. A file whose
  size, mtime, or inode no longer match its attribute has changed and is
  uploaded again under a new key.
- **Eviction.** A copied file nobody read or wrote for
  `evict_after_days` is replaced by a placeholder, and so are the least
  recently used ones while disk usage is above `high_watermark_percent`,
  until it is below `low_watermark_percent`. A placeholder is a sparse
  file with the same name, size, mtime, and mode and no data. The file is
  hashed again and the object's size checked first. Files below
  `min_evict_bytes`, files written within `upload_after_secs`, and files
  leased by an open writer or a pending sendfile or splice are kept.
  Listings, `SIZE`, `MDTM`, MLSD facts (except `unique`, the inode),
  quotas, rename, delete, and WebDAV ETags behave as for local files.
- **Fetch.** Opening a placeholder to read, append, or write in place,
  `local_path` (sendfile), and a SETSTAT that changes its size first
  download the object to a temporary file on the same filesystem, check
  length and hash, fsync, and rename it over the placeholder under a
  per-path lock. A truncating open replaces a placeholder without a
  fetch; STOR onto a placeholder takes the pooled path. An mtime change
  on a placeholder updates the placeholder and its attribute together.
- **No zeros uploaded.** A placeholder is never uploaded. One whose size,
  mtime, or inode changed outside siftp is quarantined: never uploaded or
  fetched over, refused with `internal` when opened, and counted. A
  non-empty file without data extents (`SEEK_DATA`) is never uploaded.
- **Guards.** `<root>/.siftp-tier/` holds an exclusive lock (one daemon
  or restore at a time), temporary files (cleared at start), and
  `root-id`, which must equal the store's `root-id`. A mismatch (an
  empty or replaced disk, or the wrong store) makes tiering read-only:
  fetches work; nothing is uploaded, evicted, or written to manifests.
  The directory is hidden from users rooted at `[server].root`. Startup
  fails if the filesystem lacks user extended attributes or, on Linux,
  hole reporting.
- **Manifests and restore.** Each scan that found a change writes
  `manifests/<time>-<hash>.jsonl` (files with keys, sizes, mtimes, modes
  and hashes; directories; symlinks); the newest 48 are kept.
  `siftpd tier-restore`, with the daemon stopped, recreates the newest
  manifest as placeholders without replacing anything that exists, puts
  objects no manifest names under `siftp-recovered/<recorded path>`, and
  adopts the store's `root-id`.
- **Nothing deleted.** Objects no file refers to are counted
  (`tier_orphans`), not deleted; the bucket's lifecycle rules expire
  noncurrent versions and incomplete uploads.
- **Worker.** One scan every `scan_interval_secs`, sooner when a
  10-second check finds disk usage above the high watermark; it stops
  between files on drain. Data movement is reported in tracing and the
  `tier_*` counters, not the audit log: it changes nothing a user sees.

## 4. Protocol matrix

| Area | Standard | Scope |
| --- | --- | --- |
| FTP core | RFC 959 | USER PASS ACCT QUIT SYST NOOP HELP SITE; PWD/XPWD CWD/XCWD CDUP/XCUP; TYPE A/I, MODE S, STRU F; PASV, PORT; RETR STOR STOU APPE REST ABOR; DELE RMD/XRMD MKD/XMKD RNFR RNTO; LIST NLST |
| FTP extensions | RFC 3659, RFC 7151, draft-somers-ftp-mfxx | MLSD MLST (facts type, size, modify, perm, unique; `OPTS MLST` selects facts), SIZE, MDTM, MFMT, TVFS; FEAT, OPTS (UTF8, MLST); HOST (virtual hosts) |
| IPv6 / NAT | RFC 2428 | EPSV (incl. `EPSV ALL`), EPRT; passive port range + external IP config |
| FTPS | RFC 4217 (+ RFC 2228 verbs) | Explicit: AUTH TLS, PBSZ 0, PROT P/C (CCC refused by policy, §10); implicit FTPS listener; TLS 1.2/1.3 via rustls; `require_tls` rejects plaintext PASS; data connections resume the control session's TLS session, enforced with `require_session_reuse` |
| SSH | RFC 4251–4254 | SSH-2.0; host keys Ed25519 (generated) or ECDSA, RSA host keys refused (§10); kex mlkem768x25519-sha256, curve25519-sha256, ecdh-sha2-nistp256, with ext-info and OpenSSH strict KEX; ciphers aes256-gcm and aes128-gcm, with chacha20-poly1305 offered first only when the CPU lacks AES and carry-less multiply instructions (clients pick it whenever it is offered, and AES-GCM is several times faster with them), or the `[sftp].ciphers` list; auth publickey (Ed25519, ECDSA, RSA-SHA2 verify-only) and password; only `session` channels with subsystem `sftp` |
| SFTP | draft-ietf-secsh-filexfer-02 (v3) | INIT/VERSION, OPEN CLOSE READ WRITE, LSTAT FSTAT STAT SETSTAT FSETSTAT, OPENDIR READDIR, REMOVE MKDIR RMDIR RENAME, READLINK SYMLINK, REALPATH; extensions posix-rename, statvfs, fstatvfs, expand-path, limits, fsync (`@openssh.com`); OpenSSH wire numbering (REALPATH 16, STAT 17, SYMLINK 20, NAME 104, ATTRS 105), SYMLINK target first; exit-status 0 sent when the subsystem ends |
| WebDAV/HTTP | RFC 4918 (class 1, read-only) | OPTIONS, PROPFIND (Depth 0/1), GET/HEAD with single Range; HTTP Basic auth; write methods refused (§4.1) |
| Admin API | OpenAPI 3.1 (§8.3) | REST JSON over HTTP/1.1 on the admin listener (loopback by default), bearer token |
| MCP | Model Context Protocol (2025-06-18) | `tools/list`, `tools/call` over stdio or streamable HTTP at `/mcp`; tools in §8.4 |

FTP details:

- Greeting `220` uses `[ftp].greeting`; multi-line replies use the
  `220-`…`220 ` convention. Command lines are capped at 4 KiB
  (`invalid_command` beyond that); verbs are case-insensitive.
- `LIST` emits Unix-style lines; `MLSD`/`MLST` are the machine-readable
  formats. `perm` facts reflect the rights for each entry's own path.
- `REST` applies to the next RETR/STOR only and resets after any transfer
  command. `TYPE A` performs CRLF translation; `TYPE I` is byte-exact and
  the only mode eligible for kernel fast paths.
- `HOST <name>` must precede `USER` (503 otherwise). With
  `[ftp].virtual_hosts` set, unknown names get 504 (`not_found`); users
  with `hosts = [...]` may log in only under one of those names.
- `MFMT YYYYMMDDHHMMSS <path>` (UTC) sets the modification time; it needs
  the `w` bit and replies `213 Modify=<time>; <path>`.
- `RNTO` replaces an existing target (rename(2) semantics).
- Passive data connections must come from the control connection's
  address unless `[ftp].passive_promiscuous = true`; others are refused
  and audited (`DATA_REFUSED`). Active mode targets only the control
  peer, on a port ≥ 1024.
- Control idle: `[server].idle_timeout_secs`; the same limit applies to
  each data-channel read and write (stalled transfers end with
  `rate_limited`), and TLS handshakes are bounded at 15 s.
- Anonymous FTP is off unless `allow_anonymous = true`; the anonymous
  user maps to `anonymous_root` with read+list permissions only.
- TLS data channels resume sessions issued on the control connection
  but never issue tickets themselves (an upload-only client would leave
  them unread and its close would reset the connection mid-file).

SFTP details:

- RENAME fails with `already_exists` if the target exists;
  `posix-rename@openssh.com` replaces it.
- `statvfs`/`fstatvfs` report the user's quota as the filesystem size
  when a quota is set (so `df` in `sftp` shows what is left).
- At most 256 open handles per session (`rate_limited` beyond);
  `limits@openssh.com` advertises 256 KiB packets and 255 KiB reads/writes.
- SETSTAT/FSETSTAT apply size, mtime, and permission bits; setuid,
  setgid, and sticky bits are stripped; uid/gid are ignored.

### 4.1 Read-only WebDAV/HTTP gateway

Enabled by `[webdav].bind` (HTTPS with `[webdav].tls = true`, using the
FTPS certificate). Users, credentials (HTTP Basic, verified through the
shared authentication path), storage, per-path permissions, rate
limits, bans, and audit are the same as for FTP/SFTP. `GET` of a file
needs `r`; `PROPFIND` and directory `GET` need `l` (HTML index, or JSON
with `Accept: application/json`). Every write method gets
`permission_denied` (403). File responses carry
`X-Content-Type-Options: nosniff` and `Content-Security-Policy: sandbox`.
Verified credentials are cached for 60 s under a keyed digest from the
login verification cache's process-only key (§5.1), with the stored
hash in the input, so a changed password or a deleted user stops
working at once.

## 5. Authentication and authorisation

### 5.1 Passwords

- Stored only as argon2id PHC strings (`$argon2id$v=19$…`), generated
  with `siftpd hash-password` or server-side by the admin API (defaults
  m=64 MiB, t=3, p=4; never below m=19 MiB, t=2, p=1 — config load
  rejects weaker hashes and `--check-config` fails).
- Verification is constant-time (argon2's verify; bearer tokens via
  `siftp_asm::ct_eq` over SHA-256 digests). Unknown and key-only users
  run a dummy verification against a hash with the default parameters,
  so response timing does not enumerate accounts whose hashes use the
  defaults. Custom-cost hashes are faster or slower to reject than
  unknown names, in proportion to their cost.
- Verification cache: after argon2id accepts a password, the server
  keeps a BLAKE2b keyed digest of `(user name, stored PHC string,
  password)` for `server.login_cache_ttl_secs` (default 300; 0 turns
  the cache off), and a repeat of exactly that credential within the
  window is accepted without recomputing argon2id. The 256-bit key is
  drawn from the OS RNG at start and lives only in process memory, so
  neither the user store nor the cache contents can be used to test
  guesses offline. Only correct credentials are ever stored; a wrong
  password, an unknown name, or a key-only user always pays one full
  argon2id computation. The cache holds at most 8192 entries and is
  cleared whenever the user table is replaced (reload or admin change);
  the PHC string in the digest means a changed password misses anyway.
  The password-age policy is applied to cache hits too. Cached logins
  are counted in `logins_cached` as well as `logins_ok`.
- Concurrency: at most `server.max_concurrent_password_checks` argon2id
  computations run at once (0, the default, means one per CPU); further
  logins and admin hash requests wait for a slot. Each computation
  holds the hash's memory while it runs, so this is the memory cap for
  password checking.
- Age: with `server.password_max_age_days` > 0, a password older than
  that (measured from the user's `password_set_unix`) fails like a wrong
  password on the wire and is logged as expired. Users without
  `password_set_unix` never expire (`--check-config` warns).
- Rotation: `POST /api/v1/users/{name}/password` (MCP
  `siftp_set_password`) sets a given password or generates a 24-character
  one and returns it once; `password_set_unix` is updated.
- Plaintext passwords are never logged or echoed, including in audit
  events and API responses.

### 5.2 SSH public keys

- `ssh_public_keys` holds OpenSSH lines (`ssh-ed25519`, `ssh-rsa`,
  `ecdsa-sha2-nistp256/384/521`), optionally prefixed by options:
  `expiry-time="YYYYMMDD[HHMM[SS]]"` (UTC; the key stops working at that
  time) and the no-op restrictions `restrict`, `no-pty`,
  `no-port-forwarding`, `no-agent-forwarding`, `no-x11-forwarding`,
  `no-user-rc`, `no-touch-required`. Any other option (`command=`,
  `from=`, …) is a config error rather than silently ignored.
- `server.revoked_keys_file` lists revoked keys (OpenSSH lines or
  `SHA256:` fingerprints, `#` comments); it is re-read on reload.
  Revoked and expired keys are refused at both the offer and the signed
  request.
- Key offers are answered without a signature check per RFC 4252 §7; the
  signature is verified on the signed request. Either a password or a
  key authenticates (no MFA).

### 5.3 Permissions, groups, and directory rules

- Permission bits: `r` read (RETR/SFTP read/WebDAV GET), `w` write
  (STOR/APPE/SFTP write/SETSTAT/MFMT/SYMLINK), `d` delete (DELE/RMD/
  REMOVE), `n` rename (both ends), `m` mkdir, `l` list (LIST/MLSD/
  READDIR/STAT/PROPFIND). Missing the bit ⇒ `permission_denied` before
  any I/O.
- Base bits: the user's `permissions`, else the union of their
  `[[groups]]` permissions, else `rl`.
- Directory rules (`[[users.directories]]`, `[[groups.directories]]`,
  `{path, permissions}`) cover a virtual path and everything below it.
  The longest matching rule wins; a user rule beats group rules at the
  same path; group rules at the same path are unioned; rules replace the
  base for their subtree (they can grant or restrict).
- The check runs on the lexical path first (no I/O); when a user has
  rules, the operation is re-checked against the symlink-resolved path,
  so a link cannot reach a restricted directory with another directory's
  rights. Reference: `siftp_core::Access`.
- A session is bound to one root and one `Access` snapshot for its
  lifetime; `CWD` changes a virtual prefix only.

### 5.4 Rate limits and throttling

- `rate_limit_bps` (per user, 0 = unlimited) token-buckets data bytes in
  both directions on every protocol; rate-limited users never take the
  kernel fast paths (which cannot pace bytes).
- Failed logins (any protocol, including wrong admin/MCP bearer tokens):
  after `max_failed_logins` failures from one IP within
  `ban_window_secs`, the IP is refused at accept time (and by the admin
  API with 429) for `ban_duration_secs`, logged as `auth_ban`. Success
  clears the counter.
- `max_connections` and `max_connections_per_ip` (default 512 / 32)
  refuse excess connections with FTP 421 `rate_limited` (SSH: closed
  before the banner).

### 5.5 Users from the durable store

See §7.1. Store users are validated exactly like config users.

### 5.6 External identity: LDAP

With `[ldap].url` set, a login name that is not a local user (config or
store) is checked against LDAP: search-then-bind (`bind_dn` + `base_dn`
+ `user_filter`, filter-escaped) or direct bind (`user_dn_template`,
DN-escaped), over `ldaps://` or StartTLS. Empty passwords are refused
before any network I/O (anonymous-bind bypass). Group DNs from
`group_attribute` map onto `[[groups]]` through `group_map`; without a
mapped group the user gets `default_permissions`, or is refused with
`require_group = true`. The virtual root is `virtual_root_template`
(`{user}` substituted) or `<server.root>/<user>`. LDAP users are
session-only (never stored) and go through the same jail, permissions,
quotas, and audit. An unreachable directory fails logins as
`auth_failed` on the wire (audited as `internal`).

## 6. Performance design

### 6.1 Zero-copy data plane (Linux)

- Plaintext `RETR`/`TYPE I` on a local root: `sendfile64` loop with
  explicit offsets on a blocking thread. `STOR`: `splice` socket→pipe→
  file, with the pipe grown to 1 MiB (`F_SETPIPE_SZ`) where the kernel
  allows it and the default 64 KiB otherwise. TLS/SSH channels and
  remote storage use the buffer pool. On a tiered root (§3.2), `RETR`
  of a placeholder fetches it before sendfile, and `STOR`/`APPE` onto a
  placeholder use the buffer pool.
- `[performance].sendfile`/`splice` disable each path; every path falls
  back automatically on `EINVAL`/`ENOSYS`/`EOPNOTSUPP` (and `EPERM` from
  seccomp) before any byte moves. Blocking sockets carry
  `SO_SNDTIMEO`/`SO_RCVTIMEO` equal to the idle timeout. The chosen path
  is counted in `/api/v1/stats` and `/metrics`.

### 6.2 io_uring (feature `io-uring`, Linux ≥ 5.6)

With the feature compiled and `[performance].io_uring = true`:
downloads read the file with `IORING_OP_READ` (after sendfile in
preference) and uploads write it with `IORING_OP_WRITE` (after splice),
one ring per transfer, one entry in flight. If ring setup fails (old
kernel, seccomp — Docker's default profile blocks io_uring), that
transfer falls back. The feature never changes wire behaviour.

### 6.3 Buffer pools and backpressure

- A byte-budget pool (`buffer_pool_bytes`, default 64 MiB) hands out
  `transfer_chunk_bytes` (default 256 KiB) buffers; acquiring waits when
  exhausted — that wait *is* the backpressure signal.
- The pooled FTP paths overlap socket and file work. An upload fills a
  whole buffer from the socket (a TLS read returns one 16 KiB record)
  before one blocking write, and that write runs while the next buffer
  fills. A download reads the next chunk on the blocking pool while the
  current one is written (and encrypted) to the socket, but only when a
  second buffer is free at once; otherwise the current chunk is sent
  first. A pooled transfer holds up to two buffers and never waits for
  one while holding another, so any pool of at least one buffer makes
  progress.
- Control channels are line-buffered with a 4 KiB cap; SFTP packets are
  capped at 256 KiB (`siftp_sftp::MAX_PACKET`) before allocation.

### 6.4 Assembly kernels (`siftp-asm`)

Sources are `.s` files under `crates/siftp-asm/asm/<arch>/`, assembled
via `build.rs` when the `asm` feature is on, `target_os ∈ {linux,
macos}`, and `target_arch ∈ {x86_64, aarch64}`. Every kernel has a pure-
Rust twin that is always compiled and tested and is the reference for
differential tests. Runtime feature detection (SSE4.2 `cpuid`, aarch64
`HWCAP_CRC32`) gates each kernel; absent the feature, Rust serves.

| Kernel | Signature `extern "C"` | Use | Rust reference |
| --- | --- | --- | --- |
| `siftp_crc32c_<arch>` | `fn(*const u8, usize, u32) -> u32` | integrity helpers | `crc32c_rust` |
| `siftp_find_newline_<arch>` | `fn(*const u8, usize) -> usize` | FTP command-line scanning | `find_newline_rust` |
| `siftp_ct_eq_<arch>` | `fn(*const u8, *const u8, usize) -> u8` | token comparison | `ct_eq_rust` |
| `siftp_find_byte_<arch>` | `fn(*const u8, usize, u8) -> usize` | byte scanning | `find_byte_rust` |
| `siftp_adler32_<arch>` | `fn(*const u8, usize, u32) -> u32` | weak checksums | `adler32_rust` |

Rules: (1) assembly is bit-identical to the Rust reference on all
inputs; (2) no kernel allocates, panics, or reads past `len`; (3)
without the feature `backend_name()` returns `"rust"`; (4) `unsafe` is
confined to the FFI shims in `siftp-asm` and the one io_uring submission
helper in `siftp-ftp::fastpath`, with its safety argument written out.
Everything else is `#![forbid(unsafe_code)]` (the FTP crate uses `deny`).

## 7. Configuration

- File format: TOML. Discovery: `--config`, else `./siftp.toml`, else
  `configs/siftp.toml`. `configs/siftp.example.toml` covers every key.
  Unknown keys are an error.
- Environment overrides: `SIFTP_<SECTION>_<KEY>`, one for every scalar
  key (list: `siftp_core::config::ENV_OVERRIDE_NAMES`; a test checks it
  against the config structs, and that `website/docs/configuration.md`
  documents every key). Precedence: env > file >
  default. `${ENV:NAME}` interpolation works inside string values
  (tokens, LDAP and S3 secrets, hashes).
- Sections: `[server]`, `[ftp]`, `[ftps]`, `[sftp]`, `[admin]`, `[mcp]`,
  `[logging]`, `[performance]`, `[s3]`, `[ldap]`, `[webdav]`,
  `[[groups]]`, `[[users]]`.
- `[[users]]` keys: `name`, `password_hash`, `password_set_unix`,
  `ssh_public_keys`, `virtual_root`, `permissions`, `groups`,
  `directories`, `hosts`, `max_sessions`, `quota_bytes`,
  `rate_limit_bps`. User names are 1–255 characters, not `.`/`..`, with
  no whitespace, `/`, `\`, `:`, or control characters.

### 7.1 Durable user store

- `[server].user_store` (default `/var/lib/siftp/users.toml`) holds users
  created or changed through the admin API or MCP, in the same
  `[[users]]` format (file mode 0600). An empty value keeps API users in
  memory only (warned).
- Precedence: config-file users are operator-owned and read-only through
  the API (`permission_denied`); a store entry with a config user's name
  is shadowed and reported by `--check-config`.
- Writes are atomic: temp file in the same directory, `fsync`, rename,
  directory `fsync`. A crash leaves the old or the new file, never a torn
  one; stray temp files are swept on the next write.
- Mutations take an exclusive OS lock on `<store>.lock`, re-read the
  file, apply the change, and write, so concurrent writers (the daemon
  and `siftpd mcp-stdio`) cannot lose updates.

### 7.2 Lifecycle

- Reload (`SIGHUP` or `POST /api/v1/reload`): re-reads config, the user
  store, and the revoked-key list; swaps snapshots; reopens the audit
  file (log rotation); reloads the FTPS certificate (a bad pair keeps the
  old one and the reload reports it). Live sessions keep their snapshot;
  listener address changes need a restart.
- Readiness: `GET /readyz` is 200 once every listener is bound and 503
  while starting or draining; `GET /healthz` is 200 while the process
  runs.
- Drain (`SIGTERM`/`SIGINT`): stop accepting on every listener, mark not
  ready, tell idle sessions `421` and close them, let transfers and open
  SFTP uploads finish, then exit 0 — bounded by
  `server.shutdown_grace_secs` (default 30). A second signal skips the
  wait.

## 8. LLM contracts

### 8.1 Structured errors — everywhere

Every error on every surface derives from `siftp_core::ApiError` and
serializes to exactly:

```json
{"code":"permission_denied","message":"user bob lacks write permission","hint":"grant the 'w' permission bit or choose a writable path","docs_url":"https://siftp.org/docs/errors#permission_denied"}
```

- `code` is from a closed, append-only set (`auth_failed`,
  `permission_denied`, `not_found`, `already_exists`, `invalid_path`,
  `invalid_command`, `rate_limited`, `quota_exceeded`, `tls_required`,
  `internal`). Clients treat unknown codes as `internal`.
- `message` may change; `hint` says what to do next; `docs_url` anchors
  into the (unversioned) error reference.
- Mapping: admin API, MCP, and WebDAV return the JSON object (HTTP 401
  auth_failed, 403 permission_denied, 404 not_found, 409 already_exists,
  400 invalid_*, 429 rate_limited, 507 quota_exceeded, 426 tls_required,
  500 internal; 413 for oversized admin bodies). FTP sends
  `ErrorCode::ftp_reply` with text `"<code>: <message>"`; SFTP sends the
  mapped `SSH_FX_*` status with `"<code>: …"`.

### 8.2 Capability document

`GET /.well-known/siftp.json` on the admin listener (unauthenticated):

```json
{"name":"siftp","version":"0.2.1","protocols":["ftp","ftps-explicit","ftps-implicit","sftp","webdav-readonly"],"docs":"https://siftp.org/v/0.2.1/docs/","llms_txt":"https://siftp.org/v/0.2.1/llms.txt","docs_latest":"https://siftp.org/docs","openapi":"/api/v1/openapi.json","mcp":"/mcp","metrics":"/metrics","health":"/healthz","ready":"/readyz"}
```

`protocols` lists what this server serves with its current config
(FTPS only with a certificate, WebDAV only when bound). `docs` and
`llms_txt` point at the documentation for this exact version (§9.1).
`mcp` is null when `[mcp].transport = "stdio"`.
An agent's boot sequence: this document → `llms_txt` → `openapi` →
operate.

### 8.3 Admin REST API + OpenAPI

Base: `[admin].bind` (default `127.0.0.1:8080`), JSON bodies up to 1 MiB,
`Authorization: Bearer <token>` on everything except
`/.well-known/siftp.json`, `/healthz`, `/readyz`, and
`/api/v1/openapi.json`. At most 64 concurrent requests (429 beyond) and
30 s per request. Responses carry `X-Content-Type-Options: nosniff`,
`Cache-Control: no-store`, `X-Frame-Options: DENY`. The route table is
`siftp_admin::ROUTES`: each route carries its parameters, request body,
success status, and response schema, and the OpenAPI document is
generated from it (every `$ref` resolves; a test enforces it).

| Method & path | Purpose |
| --- | --- |
| `GET /api/v1/users` · `POST /api/v1/users` | List / create users (create hashes a plaintext password server-side; persisted in the store) |
| `GET/PUT/DELETE /api/v1/users/{name}` | Get / replace / delete one user (`password_hash` is always `null`; config users are read-only) |
| `POST /api/v1/users/{name}/password` | Set or rotate a password (§5.1) |
| `GET /api/v1/sessions` · `DELETE /api/v1/sessions/{id}` | List / terminate sessions |
| `GET /api/v1/files?user=&path=` | Directory listing inside a user's root |
| `GET /api/v1/audit?lines=` · `GET /api/v1/audit/verify` | Tail the audit log / verify its hash chain |
| `GET /api/v1/stats` · `GET /metrics` | Counters as JSON / Prometheus text (§8.7) |
| `POST /api/v1/reload` | Reload (§7.2) |
| `GET /healthz` · `GET /readyz` | Liveness / readiness |

### 8.4 MCP tools

Transports: stdio (`siftpd mcp-stdio`, always available) and streamable
HTTP at `POST /mcp` (same bearer token and ban accounting), which
`[mcp].transport = "stdio"` turns off: requests then get `not_found`.
The setting is read per request, so a reload applies it. Results are the admin JSON
payloads; errors are the §8.1 object in an error result. Registry:
`siftp_mcp::TOOLS` (names are stable API).

| Tool | Mirrors |
| --- | --- |
| `siftp_capabilities` | §8.2 document |
| `siftp_list_users` · `siftp_get_user` | `GET /api/v1/users[/{name}]` |
| `siftp_create_user` · `siftp_update_user` · `siftp_delete_user` | `POST`/`PUT`/`DELETE` users |
| `siftp_set_password` | `POST /api/v1/users/{name}/password` |
| `siftp_list_files` | `GET /api/v1/files` |
| `siftp_list_sessions` · `siftp_kill_session` | sessions |
| `siftp_tail_audit` · `siftp_verify_audit` | audit tail / verify |
| `siftp_stats` | `GET /api/v1/stats` |
| `siftp_reload_config` | `POST /api/v1/reload` |

### 8.5 Audit log

`[logging].audit_file` is append-only JSON-lines (mode 0640):
`{"ts":<unix>,"user":…,"action":"RETR|STOR|LOGIN|…","path":…,
"result":"ok|<error code>","remote":"ip:port","seq":<n>,"chain":"<hex>"}`.
Passwords, keys, tokens, and file contents never appear. One line per:

| Event | `action` values |
| --- | --- |
| Login, success or failure | `LOGIN` (FTP, SFTP, WebDAV), `ADMIN_AUTH` (wrong admin token) |
| Ban | `auth_ban` |
| Transfer | FTP `RETR`, `STOR`, `STOU`, `APPE`; SFTP `READ`, `WRITE` (at CLOSE); WebDAV `GET` |
| File change | FTP `MKD`, `RMD`, `DELE`, `RNTO`, `MFMT`; SFTP `MKDIR`, `RMDIR`, `REMOVE`, `RENAME`, `SYMLINK`, `SETSTAT` |
| Admin change (REST or MCP; `remote` is `admin-api`) | `USER_CREATE`, `USER_REPLACE`, `USER_DELETE`, `PASSWORD_SET`, `SESSION_KILL`, `RELOAD` |
| Other | `RELOAD` on SIGHUP (`remote` is `signal`), `DATA_REFUSED` (FTP data connection from a foreign address), `PROT` with `path` `C` (FTPS data channel set to clear) |

Failed transfers and changes are logged with their error code as
`result`. `path` is the resolved virtual path (absolute, starting with
`/`) on every protocol; when a path does not resolve (an escape
attempt), it is the text the client sent. Lookups (FTP `CWD`, `SIZE`, `MDTM`, `LIST`, `MLSD`, `STAT`;
SFTP `STAT`, `REALPATH`, `OPENDIR`, `READDIR`; admin `GET` routes) are
not logged, including when they fail.

- `seq` increases by one per line for the life of the chain (across
  restarts and rotations); `chain` is
  `hex(sha256(previous chain || "\n" || line up to ",\"chain\""))`,
  starting from 64 zeros. Editing, inserting, or deleting a line breaks
  every later value; `siftpd audit-verify <file> [--anchor <chain>]`
  (exit 2 on tampering) and `GET /api/v1/audit/verify` check it.
  Truncating the end cannot be detected from the file alone.
- Rotation: logrotate `create` + reload; the chain continues into the new
  file.

### 8.6 `llms.txt` rules

`website/llms.txt` (llmstxt.org format): an H1, a blockquote summary,
then H2 sections of links to the spec, config, error, OpenAPI, and MCP
references, each with one line. Every link resolves; no marketing copy;
updated with any contract change; under ~200 lines. A copy is published
per version (§9.1).

### 8.7 Observability

- `GET /metrics` (bearer) exposes Prometheus text: every counter in
  `/api/v1/stats` as `siftp_*_total` (tiered storage under `tier` in
  the JSON, as `siftp_tier_*` in Prometheus), transfers by data path
  (`siftp_transfers_total{path=…}`), gauges for sessions, connections,
  transfers in flight, banned addresses, buffer pool use, readiness,
  start time, and `siftp_build_info`. Example alerts:
  `packaging/prometheus/alerts.yml`.
- Logs go to stderr, as text or JSON (`[logging].format`); every event
  carries its span: per FTP/SFTP connection `peer`, `session`, `user`
  (and `host` after FTP `HOST`), plus one `transfer complete|failed`
  event per transfer with verb, path, bytes, data path, and duration.

## 9. Website information architecture (siftp.org)

Static files only. Markdown files are the canonical sources and the
served `.md` mirrors; `website/build.py` (stdlib-only) renders HTML,
`sitemap.xml`, and `llms-full.txt`:

```
website/
  index.md            what siftp is, quick start, measured numbers
  llms.txt, llms-full.txt
  docs/               quickstart, configuration, protocols, storage,
                      identity, operations, llm, performance, security,
                      errors
  spec.md             verbatim copy of SPEC.md
  download.md         releases, verification, source builds
  v/<version>/        frozen copy of the docs for each release (§9.1)
```

Rules: canonical URL per topic; every error code anchorable at
`/docs/errors#<code>`; no JavaScript required; code samples runnable.

### 9.1 Versioned documentation

Each release freezes the docs under `/v/<version>/` (HTML, Markdown
mirrors, and its own `llms.txt`/`llms-full.txt`); `versions.json` lists
them. Pages under `/v/<version>/` that are not the latest carry a banner
linking to the current page. The capability document (§8.2) points
agents at the version they are talking to.

## 10. Security considerations

- **Jail first**: path resolution and permission checks precede all I/O;
  symlink escapes and symlink laundering across directory rules are
  re-checked at use (§3, §5.3). Run unprivileged; bind low ports with
  `CAP_NET_BIND_SERVICE` (the systemd unit does this).
- **Credentials**: argon2id only; SSH keys from a strict allow-list with
  expiry and revocation; admin token compared in constant time,
  required on every non-public route, and failing closed when unset;
  wrong tokens count toward bans.
- **Transport**: rustls TLS ≥ 1.2, no plaintext fallback with
  `require_tls`; optional enforced TLS session reuse on data channels;
  SSH with AEAD ciphers, post-quantum hybrid kex, and strict KEX
  (Terrapin countermeasure). RSA host keys are refused: the available
  RSA implementation is not constant-time for private-key operations
  (RUSTSEC-2023-0071); client RSA keys are verify-only.
- **Abuse resistance**: bans, shared connection caps, command/packet/
  handle caps, idle and handshake timeouts, data-peer pinning, anti-bounce
  `PORT`/`EPRT`, admin request limits, transient accept errors retried
  rather than fatal.
- **No secret leakage**: passwords, keys, tokens, and file bytes never
  enter logs, errors, or API responses; WebDAV caches only credential
  digests.
- **Supply chain**: `Cargo.lock` committed; `cargo deny` (advisories,
  licenses, bans — OpenSSL banned, sources) and `cargo audit` in CI;
  cargo-fuzz targets for FTP lines, SFTP packets, config, and paths run
  in CI and their seed corpora replay in `cargo test`; releases are
  signed (Sigstore) with SBOMs and provenance attestations.

## 11. Acceptance tests

Run in CI on Linux x86_64 and aarch64 (and macOS for the dev lane):

1. `cargo build/test/clippy -D warnings` on the workspace, also with
   `--features asm` (siftp-asm) and `--features io-uring` (siftp-ftp).
2. FTP with curl, lftp, and Python ftplib (`tests/interop/run.sh`):
   login, PASV/EPSV transfers byte-identical, MLSD/SIZE/MDTM/MFMT, REST
   resume, mirror both ways, `..` refused 553, read-only STOR 550.
3. FTPS explicit (curl, lftp, ftplib with PROT P) and implicit; with
   `require_tls`, plaintext PASS gets 534; session reuse enforced when
   configured.
4. SFTP with OpenSSH `sftp`/`scp` and a russh client: pubkey + password,
   put/get byte-identical, readdir, rename semantics, statvfs, symlinks,
   jail escape refused, oversized packet rejected, expired/revoked keys
   refused.
5. Auth/abuse: bans trip and expire; quota exceeded removes the partial
   file; caps refuse; drain completes an in-flight upload.
6. LLM surface: capability document; create a user via the admin API and
   log in over FTP; every MCP tool once over stdio; error bodies match
   §8.1; audit lines parse and the chain verifies.
7. Performance smoke (Linux): sendfile/splice/io_uring counters advance
   on real transfers; asm kernels match references; `bench/run.sh`
   reproduces the published table.
8. Config: `--check-config` accepts the substituted example and rejects
   unknown keys and weak hashes; env overrides win; reload adds a user
   without dropping a session; API users survive restart, reload, and
   SIGKILL mid-write.
9. Storage: the FTP and SFTP suites pass on S3 roots (in-process mock in
   `cargo test`; VersityGW in CI).

## 12. Roadmap

- OIDC for the admin API; writes over WebDAV; per-user bandwidth pools
  shared across sessions.
- Clustering (shared user store and ban list).
- Windows service integration (the workspace compiles for Windows; it is
  not a supported deployment target).

## 13. Stage map

Stage 1 (spec + scaffold) → config/auth core → FTP/FTPS → SSH/SFTP →
data-plane fast paths → admin API → MCP → website → production (storage
backends, identity, durability, lifecycle, observability, hardening,
packaging, CI, benchmarks). No stage merges with a red build or a
placeholder where this spec defines behaviour.