Projects / FF00 Studio / Technical appendix

Experimental Pre-Alpha / Rack, worker, Surge authority, and Windows package under proof

Why FF00 Studio is being built this way

FF00 Studio is an attempt to make a local music collaborator that can hear context, explain its decisions, and actually perform bounded work without allowing a language model onto the realtime audio thread or pretending that a proposal is an action.

This is the architectural record for the project, not a description of a finished product. The contracts are written before the most dangerous parts exist so that later implementation can be tested against something stronger than intention. A statement here is either verified, planned, unsupported, or unknown. Planned behavior is not presented as working.

Current target: Windows, Waveform 14.0.49 Free + Welcome, Surge XT 1.3.4, and a replaceable local Ollama model. Linux portability shapes interfaces from the first commit, but Linux audio support is not yet claimed. An unsigned Windows Experimental Pre-Alpha is publicly available with a warning gate, checksum, manual, build manifest, and its exact matching AGPL source commit.

1. Product boundary

The product has two visible working surfaces. FF00 Master holds the project conversation, references, analysis, generated assets, and reviewed actions. FF00 Rack is inserted on a Waveform track and defines the place where FF00 may process audio, host child plugins, and expose controlled parameters.

It is deliberately not a general desktop-driving chatbot. UI automation and invented Waveform commands are too weak a foundation for audio work. A model may reason about a desired sound; deterministic executors must discover the available parameters, validate a proposed change, perform it, and report what actually happened.

The initial non-goals are equally important: no direct .tracktionedit rewriting, no cloud requirement, no feedback graph, no promise that arbitrary VST3 plugins are safe, and no claim that Waveform scripting can do something until it has been executed on the installed version.

2. Process topology and ownership

Waveform 14 process
  └─ FF00 Rack VST3
       ├─ fixed realtime bridge ── shared memory ── FF00 worker
       └─ control/state channel ── named pipe ──────┘

FF00 Master ── local broker/control protocol ── registered Rack instances
     └─ deterministic tools ── validated proposals ── Ollama on loopback

The Rack is the smallest host-resident component. One worker belongs to one Rack, contains the child plugin graph, and can be terminated without placing child DSP inside Waveform. Master stays outside the realtime path. Ollama is a replaceable reasoning provider, not a state owner.

Process isolation limits blast radius; it does not make the Rack infallible. A bug in the Rack, its host-facing VST3 implementation, shared-memory bounds, or host-side state decoder can still destabilise Waveform. Native child-plugin editors must therefore be worker-owned windows, not code loaded into the host.

3. Realtime audio contract

The audio callback must never wait for a worker, pipe, model, file, allocator, or UI. Cross-process DSP therefore uses a pipeline, not a synchronous remote procedure call. Input block n is submitted into a fixed slot while an output from an earlier sequence is consumed. The configured safety horizon is one or more processing periods and its total latency is reported to Waveform in samples for plugin delay compensation.

“One block” is not a contractual promise on Windows. Scheduler jitter, process wake-up time, driver conditions, and competing work can make that false even when average processing time is low. The contract is instead: a deterministic, configured pipeline latency; no callback waiting; sequence and generation validation; and defined output when a deadline is missed.

The alpha must impulse-test reported latency at buffer sizes from 64 through 1024 samples. Offline rendering is not assumed equivalent to realtime. Until a synchronous watchdog mode is experimentally safe, unsupported hosts or modes must require realtime/1× rendering rather than silently produce incomplete audio.

4. Waveform ownership and project context

Waveform owns tracks, clips, automation, native plugins, saving, and rendering. FF00 Studio permanently treats .tracktionedit as read-only context under the current architecture. The Rust core has stable snapshot primitives, but the Waveform XML snapshot parser is not integrated yet. Its contract is to copy a stable snapshot, parse only a version-aware whitelist, treat unknown XML and plugin state as opaque, and never reconstruct or serialise the project file.

This removes the central corruption risk of a third-party serializer dropping fields it does not understand. It does not make reads automatically safe: a live file may be replaced during save, so parsing occurs only from a copied, stable snapshot with pre/post metadata checks and bounded retries.

Waveform scripts are capability-gated. Track creation, MIDI clips, notes, plugins, presets, parameters, render/export, Step Clips, Clip Launcher, and Micro Drum Sampler operations remain unknown until exercised through Waveform 14.0.49’s own installed discovery and disposable-project tests. A prepared script is not a successful action: preparation, user execution, and verified result are separate states.

5. VST3 hosting and the Surge proof target

JUCE supplies useful hosting abstractions, but it cannot guarantee honest metadata, stable behavior, deterministic state, or safe editors for every plugin. The current Surge proof uses a sacrificial inspection process; a general scanner service with timeout and persistent quarantine remains planned. The worker already owns tested Surge instantiation, mono/stereo audio, MIDI, child UI, parameters, opaque state, and teardown. General latency, tail, bypass, multi-bus, and plugin-specific behavior remain compatibility work.

The alpha narrows compatibility before claiming arbitrary hosting. Plugins may have unstable parameter names, mode-dependent controls, multiple buses, sidechains, offline-only behavior, file-dialog parameters, hidden state, hangs, or crashes. Each tested binary receives a fingerprint and compatibility record.

Surge XT 1.3.4 is the first deep adapter. Save/reload does not require arbitrary reserialisation to be byte-identical. The correct distinctions are:

• preserve an untouched opaque blob exactly when no transformation is needed;
• verify parameter and preset equivalence after restoration;
• render a controlled MIDI/audio fixture and compare within declared tolerance;
• record plugin version and fingerprint with every result.

Literal byte equality is useful only where the plugin itself guarantees it. Semantic and controlled-render equivalence are the portable acceptance tests.

6. Constrained Rack graph

The long-term graph supports splits, merges, reordering, MIDI, auxiliary buses, sidechains, bypass, and per-node latency. The Rust core currently validates the constrained graph contract; worker-side runtime graph execution is not built. Alpha intentionally permits mono or stereo main I/O, one optional mono/stereo sidechain per node, fixed layouts, and no feedback.

The required runtime design builds and prepares graphs away from the audio thread as immutable plans. The future executor must compute path latency, insert compensation on shorter branches, define deterministic MIDI ordering, then publish a revision. Block-boundary swaps, safe topology crossfades, and deferred retirement remain acceptance requirements rather than implemented claims.

An atomic pointer swap alone is insufficient if plugin preparation, state migration, bus negotiation, or destruction occurs on the callback. Those tasks remain outside it. Dynamic layouts and feedback are deferred because they add host negotiation and cycle-delay semantics before the basic bridge is proven.

7. Local protocol and Rack identity

Realtime audio and bounded MIDI events belong in fixed shared-memory slots. Graph state, parameter changes, capability discovery, jobs, errors, and snapshots belong on control IPC: Windows named pipes first, Unix sockets later.

Every implemented connection is identified by Rack UUID, process ID, process-start nonce, connection generation, and protocol version. The current Windows endpoint still inherits the process/default DACL; an explicit current-user-only pipe ACL remains an open release gate. Major/minor capability negotiation, asynchronous jobs, and cancellation are also planned rather than claimed. Expected graph revisions already reject stale mutation requests.

Master currently discovers multiple Racks through one ephemeral, versioned registry record per prepared Rack. Discovery is not authority: Rack UUID, generation, authority mode, and committed base revision are checked again at the endpoint. A future per-user broker may replace the registry rendezvous. “Accepted” is never reported as completed; current synchronous commands report the deterministic executor result.

8. Discuss, Preview, and Apply

For hosted parameters, “permanent” means present in committed Rack state. For a graph, it means the committed graph revision. For generated audio/MIDI, it means a provenance-bearing immutable asset promoted from preview. For Waveform, it means the user executed a reviewed action and FF00 subsequently verified the observable result. Writing a script file alone is never success.

Preview changes cannot leak into Waveform automation, committed Rack state, or durable generated assets. Apply uses revision preconditions so a model cannot approve an old candidate after the user or plugin state has changed.

This boundary now has production-path evidence. Master can turn up to three of nine curated Surge directions into an exact, read-only ParamID/value plan. A separate action creates Preview; Discard restores committed state; Apply advances the committed revision once. A headless Rack/worker/Surge transaction verified six parameter readbacks, exact replanning after Discard, revision 1 to 2 on Apply, and rejection of the original stale plan. Human Waveform audition and semantic-batch save/reopen remain open.

9. Session sidecar and provenance

<edit-name>.ff00/
  session.toml
  SESSION.md
  history/events.jsonl
  generated/audio/
  generated/midi/
  snapshots/
  cache/

The Rust sidecar crate implements the core manifest, atomic-replacement, JSONL, immutable-asset identity, and stable source-snapshot contracts. Integration into Master and a live Waveform project remains planned. The intended runtime has one writer; session.toml uses atomic replacement; events.jsonl recovers to the final complete record; and locks include PID plus process-start nonce.

References remain linked by path, SHA-256, size, timestamps, and optionally a media fingerprint. Moved or changed media cannot be silently substituted. Portable copies require explicit approval. Cache entries are disposable and hash-keyed; original input audio is never rewritten.

10. Ollama and deterministic authority

The first implemented provider is Ollama on canonical loopback. Master lists locally installed models and currently prefers the machine-tested qwen3:8b; qwen3:14b remains selectable and model choice remains replaceable. The client does not follow redirects or proxies, bounds prompt/context/response sizes, and rejects non-loopback endpoints and cloud-labelled model names.

The model may interpret language, compare options, draft recipes, and explain trade-offs. Rust/C++ code owns capability discovery, schema validation, parameter identity, range checks, state revisions, execution, cancellation, and truthful results. Schema-constrained output gets at most one bounded repair attempt; repeated invalid output becomes a visible failure, not an improvised tool call.

The first provider slice is Discuss-only. It receives no executor tools and its schema contains only a summary, observations, non-binding options, and unknowns. Rust validates the returned shape and bounds before display. The first direct structured-output probe completed with qwen3:8b in 23.22 seconds; the 14B model exceeded the 180-second usability timeout on the same machine. Neither result is a permanent quality ranking.

Long work will be represented as asynchronous jobs. Request cancellation is not implemented yet. Context summaries are provenance-bearing inputs, not proof of current plugin state, and the current model response has no route into Preview or Apply.

11. PluginKnowledge and semantic sound design

PluginKnowledge is keyed by exact plugin fingerprint and stable host ParamID. Evidence order is: user-confirmed project knowledge, user-global corrections, curated adapter facts, direct observation, controlled inference, then model hypothesis. Higher evidence always wins; confidence without evidence cannot.

Unknown controls are not casually swept. Parameters can interact, be discrete, change meaning by mode, load files, open dialogs, destabilise output, or expose misleading names. Controlled auditioning occurs only in Preview, one bounded change at a time, around an exact state snapshot and captured test signal.

The first versioned Surge XT 1.3.4 adapter is implemented for metallic, hollow, shorter, wider, harsher, cleaner, more movement, less transient, and heavier, each at three bounded intensities. It resolves only against the complete 2,855-entry unique ParamID catalogue for the tested build, combines at most three unique directions, reports saturation, and rejects ID/name mismatches. Plans list exact before/requested values and reasons. Persistent PluginKnowledge, user correction layers, and controlled unknown-plugin audition remain planned.

12. Transcription, analysis, and UVR

Basic Pitch will use a pinned official ONNX model and a Rust implementation of the pinned reference preprocessing, windowing, stitching, note decoding, and pitch-bend behavior. A golden corpus must compare timings, pitches, confidence, and bends against the Python reference before parity is claimed.

Loudness, true peak, spectrum, and other defined measurements can be deterministic for declared algorithms and versions. Key, chord, section, timbre, masking, and artistic similarity remain heuristic and carry confidence and evidence. Multichannel sources retain channel-specific analysis; downmix proxies may assist algorithms but cannot replace information they destroy.

UVR remains a separate optional application. FF00 consumes stems as ordinary references and records their provenance; it does not bundle UVR, manage its models, or make separation a runtime dependency. Later convenience integration may discover user-provided outputs without changing that boundary.

13. Failure and recovery semantics

healthy → degraded → faulted → restarting → healthy
                              └──────────→ quarantined

A missed deadline yields deterministic fallback and a visible degraded state. Repeated misses or a dead worker latch a fault. Restart is bounded and backed off; the last committed graph is restored only after state validation. The initial policy under test is three consecutive misses before latching and at most two automatic restart attempts. A state that repeatedly crashes is quarantined instead of creating a restart loop.

Effects fall back to latency-aligned dry audio where possible. Instruments emit silence, all-notes-off, and reset state. Stale shared-memory generations and late IPC replies are discarded. Recovery never promotes Preview state or tells the user an unverified restoration succeeded.

14. What “local-only” must mean

FF00 telemetry, crash upload, remote web content, and automatic network updates are disabled for alpha. Ordinary FF00 processing must pass a network-observation test showing no non-loopback traffic from FF00 processes. Model downloads, operating-system traffic, WebView behavior, Ollama behavior, and independently networked child plugins are documented separately rather than hidden inside the claim.

References are never uploaded by default. Logs and public validation records must remove private paths, keys, tokens, personal media names, opaque plugin state, and machine identifiers. Local-only describes FF00’s behavior, not a promise about every third-party binary a user loads.

15. Windows first, Linux from the boundary

IPC, shared memory, supervision, realtime scheduling, clocks, locks, paths, plugin discovery, and native editor handling receive explicit platform interfaces from the first commit. Windows remains the only supported audio runtime until the same contracts are tested on Linux. Linux CI may compile and exercise non-audio core logic without implying Waveform or VST3 UI support.

The public Forgejo repository is licensed AGPL-3.0-or-later, aligned with the other FF00 applications and contributor agreement. The public unsigned Windows package installs Master plus the development Rack/worker for the current user. It is paired with the exact source commit, checksum, build manifest, manual, licence, and third-party notices. Dependency/legal review, code signing, and the actual install/uninstall acceptance test remain open. Waveform, Surge XT, UVR, Ollama models, and reference material are not bundled.

16. Evidence, unknowns, and milestone gates

As reconciled on 2026-09-12 from evidence recorded through 2026-09-10, the development Rack runs one external worker per retained Rack through fixed shared-memory audio/MIDI slots and named-pipe control. Component tests cover fixed blocks from 64 through 1024 samples, delayed-dry fallback, bounded replacement, restart limiting, quarantine, and state restoration after worker termination.

Waveform 14.0.49 has now hosted Surge XT 1.3.4 through that worker. Master discovered the Rack, enumerated 2,855 host parameters, and completed exact Preview, Discard, and Apply transitions. Applied Surge state survived save, close, and reopen. Two Rack instances remained independently addressable. Killing one exact worker during playback left Waveform and its sibling worker alive, produced no audible change in the owner’s listening check, and caused a bounded replacement. Closing the edit removed all workers and registrations. The listening result is not a captured proof of click-free recovery.

The first loopback-only Ollama Discuss path returned schema-valid advice without receiving mutation tools. Deterministic normalization now prevents repeated model direction IDs from stacking. The nine-direction Surge adapter, exact-plan review UI, model-free selector, and headless Preview/Discard/Apply transaction advance Milestone 6 without completing persistent PluginKnowledge or live semantic audition.

The complete Rust workspace passes 16 tests and the optimized native path passes all five CTest entries. The first public 0.0.1 experimental-pre-alpha installer is 4,246,035 bytes with SHA-256 CFDC80DCF1D7A2697EEDA94C2DC3E2A2217A1EC55403FC8CA7038ABBEFC9BD2D. Its static manifest/checksum validation passed; the installer itself has not yet completed the close-host install/uninstall acceptance procedure.

Still unverified or incomplete: the remaining 256- and 1024-sample full-duration host PDC/null runs, host-level callback allocation instrumentation, variable-block and offline rendering, explicit pipe ACL proof, the constrained split/merge graph and sidechain, Waveform script operations, request cancellation, persistent PluginKnowledge, live semantic audition/save-reopen, transcription, observed non-loopback network behavior, installer/uninstaller acceptance, and warning-free linting. The 64-, 128-, and 512-sample host renders are exploratory passes under the recorded shortened-render limitation.

The architecture rejects three tempting shortcuts: blocking until the worker is ready, treating model text as execution, and rewriting Waveform XML. Each makes an early demo easier while moving uncertainty into realtime safety, truth, or project integrity. The project advances only when those boundaries survive the recorded tests.