lib: Update FRCP implementation

The Flow and Retransmission Control Protocol (FRCP) runs end-to-end
between two peers over a flow. It provides reliability, in-order
delivery, flow control, and liveness. Note that congestion avoidance
is orthogonal to FRCP and handled in the IPCP.

A fixed 16-octet header, network byte order, is prefixed to every FRCP
packet:

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |             flags             |              hcs              |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                            window                             |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                            seqno                              |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                            ackno                              |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                     payload (variable) ...
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

hcs is a CRC-16-CCITT-FALSE checksum over the PCI (and the stream
extension when present), verified before any flag-driven dispatch.  A
single packet can simultaneously carry DATA + ACK + FC + RXM by OR-ing
flag bits. An optional CRC trailer covers the body on DATA when qs.ber
== 0, and on every SACK packet; an optional AEAD wrap (per-flow keys)
sits outermost.

Flag bits (MSB-first; bits 13..15 reserved, MUST be zero):

    +------+--------+--------+----------------------------------------+
    | Bit  | Mask   | Name   | Meaning                                |
    +------+--------+--------+----------------------------------------+
    |   0  | 0x8000 | DATA   | Carries caller payload                 |
    |   1  | 0x4000 | DRF    | Start of a fresh data run              |
    |   2  | 0x2000 | ACK    | ackno field valid                      |
    |   3  | 0x1000 | NACK   | Pre-DRF nudge (seqno informational)    |
    |   4  | 0x0800 | FC     | window field valid (rwe advertisement) |
    |   5  | 0x0400 | RDVS   | Rendezvous probe (window-closed)       |
    |   6  | 0x0200 | FFGM   | First Fragment of a multi-fragment SDU |
    |   7  | 0x0100 | LFGM   | Last Fragment of a multi-fragment SDU  |
    |   8  | 0x0080 | RXM    | Retransmission                         |
    |   9  | 0x0040 | SACK   | Block list follows in payload          |
    |  10  | 0x0020 | RTTP   | RTT probe / echo (payload follows)     |
    |  11  | 0x0010 | KA     | Keepalive                              |
    |  12  | 0x0008 | FIN    | End of stream marker                   |
    | 13-15|   --   |   --   | Reserved (MUST be zero)                |
    +------+--------+--------+----------------------------------------+

(FFGM, LFGM) encodes the fragment role of a DATA packet (SCTP-style
B/E): 11=SOLE, 10=FIRST, 00=MID, 01=LAST. Each fragment carries its
own seqno; Retransmission recovers fragments individually, reassembly
runs at consume time. In stream mode FFGM/LFGM are unused; per-byte
position is carried by the stream extension below and end-of-stream is
signalled by FIN on a 0-byte DATA packet.

SACK payload (FRCT_ACK | FRCT_FC | FRCT_SACK):

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |           n_blocks            |        padding (2 octets)     |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                           start[0]                            |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                            end[0]                             |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                        ... n_blocks pairs total ...

Each block describes a *present* (received) range strictly above the
cumulative ACK in the PCI ackno. D-SACK (RFC 2883) is signalled
in-band as block[0] - no flag bit, no extra framing - and consumed by
the RACK reo_wnd_mult scaler (RFC 8985 sec. 7.2).

RTTP payload (FRCT_RTTP only; 24 octets):

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                          probe_id                             |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                          echo_id                              |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                                                               |
    +                  nonce (16 octets, echoed verbatim)           +
    |                                                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Stream PCI extension (in_order == STREAM only; 8 octets after the base
PCI on every DATA packet):

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                            start                              |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                             end                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

start, end are monotonic 32-bit byte offsets; end - start equals the
on-wire payload length. Stream mode is negotiated at flow allocation;
the extension is present iff stream mode is in use, never on a
per-packet basis.

Service modes are an orthogonal (in_order, loss, ber) vector selected
at flow_alloc; the cubes above map to the axes:

    +----------------+---------+------+-----+-----------------------+
    | Cube           | in_order| loss | ber | Engaged               |
    +----------------+---------+------+-----+-----------------------+
    | qos_raw        |    0    |   1  |   1 | Raw passthrough       |
    | qos_raw_safe   |    0    |   1  |   0 | Raw + CRC trailer     |
    | qos_rt         |    1    |   1  |   1 | FRCP, no FRTX, no CRC |
    | qos_rt_safe    |    1    |   1  |   0 | FRCP, no FRTX, CRC    |
    | qos_msg        |    1    |   0  |   0 | FRCP + FRTX           |
    | qos_stream     |    2    |   0  |   0 | FRCP + FRTX, stream   |
    +----------------+---------+------+-----+-----------------------+

in_order=0 sends raw datagrams with no PCI (UDP-equivalent);
in_order=1 engages FRCP with SDU framing; in_order=2 (stream) requires
loss=0 and is rejected otherwise. loss=0 engages the FRTX retransmit
machinery. ber=0 appends the CRC-32 trailer; QOS_DISABLE_CRC at build
time forces ber=1 for development. Encryption is a separate per-flow
attribute layered as an AEAD wrap outside the FRCP packet.

Heritage: delta-t (Watson 1981) supplies timer-based connection
management - no SYN/FIN handshake, the DRF marker, the t_mpl / t_a /
t_r timers. RINA (Day 2008) supplies the unified flow_alloc(name, qos,
...) primitive and the orthogonal QoS-cube axes.  Loss detection
follows TCP/QUIC practice (RFCs 2018, 2883, 6582, 6298, 8985); RTT
probing is nonce-authenticated like QUIC PATH_CHALLENGE.

Adds oftp, a minimal file-transfer tool over an FRCP stream flow. The
client reads from stdin or --in FILE and writes through a
flow_alloc(qos_stream); the server (--listen) calls flow_accept and
writes to stdout or --out FILE. Both sides compute a CRC-64/NVMe over
the bytes they handle and print the result. The server rejects flows
whose negotiated qs.in_order != STREAM.

Two FRCP knobs are exposed via env vars on either side:
  OFTP_FRCT_RTO_MIN         fccntl FRCTSRTOMIN  (ns)
  OFTP_FRCT_STREAM_RING_SZ  fccntl FRCTSRRINGSZ (octets)

The ocbr_client gains an OCBR_QOS env var to pick the cube the client
uses for flow_alloc; recognised values are raw, safe, rt, rt_safe,
msg, stream. Unknown values fall back to raw with a warning on
stderr. Without the env set behaviour is unchanged.

Removes the deprecated lib/timerwheel.c

Signed-off-by: Dimitri Staessens <dimitri@ouroboros.rocks>
Signed-off-by: Sander Vrijders <sander@ouroboros.rocks>

3 files changed, 144 insertions, 11 deletions

diff --git a/doc/man/flow_alloc.3 b/doc/man/flow_alloc.3
index dbe5323c..8a9b5f5b 100644
--- a/doc/man/flow_alloc.3
+++ b/doc/man/flow_alloc.3
@@ -62,10 +62,60 @@ The \fBflow_dealloc\fR() function will release any resources
 associated with the flow. This call may block and keep reliable flows
 active until all packets are acknowledged.
 
-A \fBqosspec_t\fR specifies the following QoS characteristics of a
-flow:
-
-TODO: specify a qosspec_t
+A \fBqosspec_t\fR specifies the QoS characteristics of a flow.
+The fields are:
+
+.TP
+\fBdelay\fR (ms)
+Maximum one-way delay.
+.TP
+\fBbandwidth\fR (bits/s)
+Minimum bandwidth.
+.TP
+\fBavailability\fR
+Class of 9s (e.g. 5 = 99.999%).
+.TP
+\fBloss\fR
+Tolerated packet loss; 0 selects reliable delivery.
+.TP
+\fBber\fR
+Tolerated bit error rate (errors per billion bits); 0 enables an
+end-to-end integrity check (corrupted packets are dropped).
+.TP
+\fBservice\fR
+Framing / reliability class: \fBSVC_RAW\fR (0) disables FRCT;
+\fBSVC_MESSAGE\fR (1) preserves SDU boundaries; \fBSVC_STREAM\fR (2) is
+a byte stream with no SDU boundaries. \fBSVC_STREAM\fR requires
+\fIloss\fR = 0; otherwise
+\fBflow_alloc\fR()/\fBflow_accept\fR() returns \fB-EINVAL\fR.
+.TP
+\fBmax_gap\fR (ms)
+Maximum tolerated inter-packet gap. Packets exceeding the gap
+budget are dropped under the real-time cubes.
+.TP
+\fBtimeout\fR (ms)
+Peer-liveness timeout; 0 disables. Only applies when FRCT is
+enabled (service > 0).
+
+.PP
+The library provides predefined cubes:
+
+.TP
+\fBqos_raw\fR
+No guarantees, no integrity check.
+.TP
+\fBqos_raw_safe\fR
+Best-effort with end-to-end integrity (ber = 0).
+.TP
+\fBqos_rt\fR / \fBqos_rt_safe\fR
+Real-time messages, optimised for latency over reliability;
+\fBqos_rt_safe\fR adds an end-to-end integrity check.
+.TP
+\fBqos_msg\fR
+Reliable, SDU-preserving delivery.
+.TP
+\fBqos_stream\fR
+Reliable byte stream; no SDU boundaries are preserved.
 
 .SH RETURN VALUE
 
@@ -117,13 +167,39 @@ _
 \fBflow_dealloc\fR() & Thread safety & MT-Safe
 .TE
 
+.SH NOTES
+The returned file descriptor is subject to a single-reader and
+single-writer discipline \(em at most one thread may call
+.BR flow_read ()
+(or monitor the fd via
+.BR fevent ())
+and at most one thread may call
+.BR flow_write ()
+concurrently. See
+.BR flow_read (3),
+.BR flow_write (3),
+and
+.BR fevent (3)
+for details.
+.PP
+.BR flow_dealloc ()
+must not be called concurrently with any thread that is inside
+.BR flow_read (),
+.BR flow_write (),
+.BR fevent (),
+or any other Ouroboros library call on the same fd; the result is
+undefined behaviour. Applications must serialise teardown with
+in-flight use, e.g. by signalling worker threads to drop the fd
+before calling
+.BR flow_dealloc ().
+
 .SH TERMINOLOGY
 Please see \fBouroboros-glossary\fR(7).
 
 .SH SEE ALSO
 
-.BR fccntl "(3), " flow_read "(3), " fqueue "(3), " fset "(3), " \
-ouroboros (8)
+.BR fccntl "(3), " fevent "(3), " flow_read "(3), " flow_write "(3), " \
+fqueue "(3), " fset "(3), " ouroboros (8)
 
 .SH COLOPHON
 This page is part of the Ouroboros project, found at
diff --git a/doc/man/flow_read.3 b/doc/man/flow_read.3
index acc1f61e..d4a5e883 100644
--- a/doc/man/flow_read.3
+++ b/doc/man/flow_read.3
@@ -39,8 +39,7 @@ end of the datagram.
 
 On success, \fBflow_write\fR() returns the number of bytes written. On
 failure, a negative value indicating the error will be returned.
-Partial writes needs to be explicitly enabled. Passing a
-NULL pointer for \fIbuf\fR returns 0 with no other effects.
+Passing a NULL pointer for \fIbuf\fR returns 0 with no other effects.
 
 .SH ERRORS
 .B -EINVAL
@@ -62,7 +61,8 @@ The flow has been reported down.
 The flow's peer is unresponsive (flow timed out).
 
 .B -EMSGSIZE
-The buffer was too large to be written.
+The received packet does not fit in the caller's buffer and partial
+reads are disabled (see \fBfccntl\fR(3), \fBFLOWFRNOPART\fR).
 
 .SH ATTRIBUTES
 
@@ -74,11 +74,47 @@ LB|LB|LB
 L|L|L.
 Interface & Attribute & Value
 _
-\fBflow_read\fR() & Thread safety & MT-Safe
+\fBflow_read\fR() & Thread safety & MT-Safe race:fd
 _
-\fBflow_write\fR() & Thread safety & MT-Safe
+\fBflow_write\fR() & Thread safety & MT-Safe race:fd
 .TE
 
+.SH THREAD SAFETY
+Only one thread may call
+.BR flow_read ()
+on a given file descriptor at any time. Partial-read state kept
+across calls assumes a single logical reader; two threads racing
+.BR flow_read ()
+on the same fd is undefined behaviour. Likewise, only one thread
+may call
+.BR flow_write ()
+on a given fd at a time; two writer threads on the same fd is
+undefined behaviour.
+.PP
+Combining a writer thread with a reader thread (one thread calling
+.BR flow_write (),
+another calling
+.BR flow_read ()
+or
+.BR fevent ())
+is permitted and safe. The writer does not need a dedicated reader
+thread \(em when the FRCT send window fills,
+.BR flow_write ()
+drives its own inbound rx draining internally to process incoming
+ACKs and reopen the window, clamped by the caller's
+.BR fccntl (3)
+send-timeout if any.
+.PP
+Monitoring the same fd via
+.BR fevent ()
+from a different thread is well-defined but races: events reported
+by
+.BR fevent ()
+may already have been consumed by the racing
+.BR flow_read (),
+so the second reader may then block. See
+.BR fevent (3).
+
 .SH TERMINOLOGY
 Please see \fBouroboros-glossary\fR(7).
 
diff --git a/doc/man/fqueue.3 b/doc/man/fqueue.3
index 72a0bc25..f2fb8c9f 100644
--- a/doc/man/fqueue.3
+++ b/doc/man/fqueue.3
@@ -116,6 +116,27 @@ _
 \fBfevent\fR() & Thread safety & MT-Safe
 .TE
 
+.SH THREAD SAFETY
+.BR fevent ()
+and
+.BR flow_read ()
+on the same fd from distinct threads is well-defined but races:
+events reported by
+.BR fevent ()
+may already have been consumed by the racing
+.BR flow_read (),
+so the reader may then block. Same shape as
+.BR select (2)
++
+.BR read (2)
+from distinct threads. The intended pattern is that the thread
+invoking
+.BR fevent ()
+is the same thread that calls
+.BR flow_read ()
+on the fds returned by
+.BR fqueue_next ().
+
 .SH TERMINOLOGY
 Please see \fBouroboros-glossary\fR(7).