Projects / FF00 Studio / Technical appendix

Milestone 2 / worker bridge under proof

Why FF00 Studio is being built this way

FF00 Studio is an attempt to make a local music collaborator that can hear context, explain its decisions, and actually perform bounded work without allowing a language model onto the realtime audio thread or pretending that a proposal is an action.

This is the architectural record for the project, not a description of a finished product. The contracts are written before the most dangerous parts exist so that later implementation can be tested against something stronger than intention. A statement here is either verified, planned, unsupported, or unknown. Planned behavior is not presented as working.

Current target: Windows, Waveform 14.0.49 Free + Welcome, Surge XT 1.3.4, and a replaceable local Ollama model. Linux portability shapes interfaces from the first commit, but Linux audio support is not yet claimed.

1. Product boundary

The product has two visible working surfaces. FF00 Master holds the project conversation, references, analysis, generated assets, and reviewed actions. FF00 Rack is inserted on a Waveform track and defines the place where FF00 may process audio, host child plugins, and expose controlled parameters.

It is deliberately not a general desktop-driving chatbot. UI automation and invented Waveform commands are too weak a foundation for audio work. A model may reason about a desired sound; deterministic executors must discover the available parameters, validate a proposed change, perform it, and report what actually happened.

The initial non-goals are equally important: no direct .tracktionedit rewriting, no cloud requirement, no feedback graph, no promise that arbitrary VST3 plugins are safe, and no claim that Waveform scripting can do something until it has been executed on the installed version.

2. Process topology and ownership

Waveform 14 process
  └─ FF00 Rack VST3
       ├─ fixed realtime bridge ── shared memory ── FF00 worker
       └─ control/state channel ── named pipe ──────┘

FF00 Master ── local broker/control protocol ── registered Rack instances
     └─ deterministic tools ── validated proposals ── Ollama on loopback

The Rack is the smallest host-resident component. One worker belongs to one Rack, contains the child plugin graph, and can be terminated without placing child DSP inside Waveform. Master stays outside the realtime path. Ollama is a replaceable reasoning provider, not a state owner.

Process isolation limits blast radius; it does not make the Rack infallible. A bug in the Rack, its host-facing VST3 implementation, shared-memory bounds, or host-side state decoder can still destabilise Waveform. Native child-plugin editors must therefore be worker-owned windows, not code loaded into the host.

3. Realtime audio contract

The audio callback must never wait for a worker, pipe, model, file, allocator, or UI. Cross-process DSP therefore uses a pipeline, not a synchronous remote procedure call. Input block n is submitted into a fixed slot while an output from an earlier sequence is consumed. The configured safety horizon is one or more processing periods and its total latency is reported to Waveform in samples for plugin delay compensation.

“One block” is not a contractual promise on Windows. Scheduler jitter, process wake-up time, driver conditions, and competing work can make that false even when average processing time is low. The contract is instead: a deterministic, configured pipeline latency; no callback waiting; sequence and generation validation; and defined output when a deadline is missed.

The alpha must impulse-test reported latency at buffer sizes from 64 through 1024 samples. Offline rendering is not assumed equivalent to realtime. Until a synchronous watchdog mode is experimentally safe, unsupported hosts or modes must require realtime/1× rendering rather than silently produce incomplete audio.

4. Waveform ownership and project context

Waveform owns tracks, clips, automation, native plugins, saving, and rendering. FF00 Studio permanently treats .tracktionedit as read-only context under the current architecture. It copies a stable snapshot, parses only a version-aware whitelist, and treats unknown XML and plugin state as opaque. It never reconstructs or serialises the project file.

This removes the central corruption risk of a third-party serializer dropping fields it does not understand. It does not make reads automatically safe: a live file may be replaced during save, so parsing occurs only from a copied, stable snapshot with pre/post metadata checks and bounded retries.

Waveform scripts are capability-gated. Track creation, MIDI clips, notes, plugins, presets, parameters, render/export, Step Clips, Clip Launcher, and Micro Drum Sampler operations remain unknown until exercised through Waveform 14.0.49’s own installed discovery and disposable-project tests. A prepared script is not a successful action: preparation, user execution, and verified result are separate states.

5. VST3 hosting and the Surge proof target

JUCE supplies useful hosting abstractions, but it cannot guarantee honest metadata, stable behavior, deterministic state, or safe editors for every plugin. Discovery runs in a sacrificial scanner with timeout and quarantine. The worker owns instantiation, audio/MIDI buses, child UI, latency, tail, bypass, state, and teardown.

The alpha narrows compatibility before claiming arbitrary hosting. Plugins may have unstable parameter names, mode-dependent controls, multiple buses, sidechains, offline-only behavior, file-dialog parameters, hidden state, hangs, or crashes. Each tested binary receives a fingerprint and compatibility record.

Surge XT 1.3.4 is the first deep adapter. Save/reload does not require arbitrary reserialisation to be byte-identical. The correct distinctions are:

• preserve an untouched opaque blob exactly when no transformation is needed;
• verify parameter and preset equivalence after restoration;
• render a controlled MIDI/audio fixture and compare within declared tolerance;
• record plugin version and fingerprint with every result.

Literal byte equality is useful only where the plugin itself guarantees it. Semantic and controlled-render equivalence are the portable acceptance tests.

6. Constrained Rack graph

The long-term graph supports splits, merges, reordering, MIDI, auxiliary buses, sidechains, bypass, and per-node latency. Alpha intentionally permits mono or stereo main I/O, one optional mono/stereo sidechain per node, fixed layouts, and no feedback.

Graphs are built and prepared away from the audio thread as immutable plans. The executor computes path latency, inserts compensation on shorter branches, defines deterministic MIDI ordering, then publishes a revision. A block-boundary swap changes the active plan; topology changes crossfade where safe, while old plans are retired only after no realtime reader can reference them.

An atomic pointer swap alone is insufficient if plugin preparation, state migration, bus negotiation, or destruction occurs on the callback. Those tasks remain outside it. Dynamic layouts and feedback are deferred because they add host negotiation and cycle-delay semantics before the basic bridge is proven.

7. Local protocol and Rack identity

Realtime audio and bounded MIDI events belong in fixed shared-memory slots. Graph state, parameter changes, capability discovery, jobs, errors, and snapshots belong on control IPC: Windows named pipes first, Unix sockets later.

Every connection is identified by Rack UUID, process ID, process-start nonce, connection generation, and protocol version. Named pipes are restricted to the current user. Major versions reject incompatibility; minor versions negotiate capabilities. Operation IDs, expected graph revisions, deadlines, and cancellation tokens prevent a reply from an old connection applying to a new session.

Master discovers multiple Racks through a per-user broker. Discovery is not authority: a Rack still authenticates the connection and rejects stale or mis-scoped commands. “Accepted” means queued and validated, not completed. Completion is reported only from executor state.

8. Discuss, Preview, and Apply

For hosted parameters, “permanent” means present in committed Rack state. For a graph, it means the committed graph revision. For generated audio/MIDI, it means a provenance-bearing immutable asset promoted from preview. For Waveform, it means the user executed a reviewed action and FF00 subsequently verified the observable result. Writing a script file alone is never success.

Preview changes cannot leak into Waveform automation, committed Rack state, or durable generated assets. Apply uses revision preconditions so a model cannot approve an old candidate after the user or plugin state has changed.

9. Session sidecar and provenance

<edit-name>.ff00/
  session.toml
  SESSION.md
  history/events.jsonl
  generated/audio/
  generated/midi/
  snapshots/
  cache/

The sidecar has one writer. session.toml uses atomic replacement; events.jsonl is recoverable to the final complete record; snapshots and generated assets have immutable IDs rather than filename identity. Locks include PID and process-start nonce so recycled PIDs do not inherit ownership.

References remain linked by path, SHA-256, size, timestamps, and optionally a media fingerprint. Moved or changed media cannot be silently substituted. Portable copies require explicit approval. Cache entries are disposable and hash-keyed; original input audio is never rewritten.

10. Ollama and deterministic authority

The initial development model is qwen3:14b, accessed through a configurable loopback Ollama provider. Model choice remains replaceable. In strict-local mode, non-loopback endpoints are rejected.

The model may interpret language, compare options, draft recipes, and explain trade-offs. Rust/C++ code owns capability discovery, schema validation, parameter identity, range checks, state revisions, execution, cancellation, and truthful results. Schema-constrained output gets at most one bounded repair attempt; repeated invalid output becomes a visible failure, not an improvised tool call.

Long work is represented as asynchronous jobs. Cancellation is explicit, and a changing model response cannot overwrite an in-flight decision. Context summaries are provenance-bearing inputs, not proof of current plugin state.

11. PluginKnowledge and semantic sound design

PluginKnowledge is keyed by exact plugin fingerprint and stable host ParamID. Evidence order is: user-confirmed project knowledge, user-global corrections, curated adapter facts, direct observation, controlled inference, then model hypothesis. Higher evidence always wins; confidence without evidence cannot.

Unknown controls are not casually swept. Parameters can interact, be discrete, change meaning by mode, load files, open dialogs, destabilise output, or expose misleading names. Controlled auditioning occurs only in Preview, one bounded change at a time, around an exact state snapshot and captured test signal.

Terms such as metallic, hollow, shorter, or heavier map to versioned Surge recipes: coordinated oscillator, filter, envelope, modulation, effect, and level changes with guardrails. A recipe reports the exact before/after parameters and never claims there is one objectively correct meaning for a subjective word.

12. Transcription, analysis, and UVR

Basic Pitch will use a pinned official ONNX model and a Rust implementation of the pinned reference preprocessing, windowing, stitching, note decoding, and pitch-bend behavior. A golden corpus must compare timings, pitches, confidence, and bends against the Python reference before parity is claimed.

Loudness, true peak, spectrum, and other defined measurements can be deterministic for declared algorithms and versions. Key, chord, section, timbre, masking, and artistic similarity remain heuristic and carry confidence and evidence. Multichannel sources retain channel-specific analysis; downmix proxies may assist algorithms but cannot replace information they destroy.

UVR remains a separate optional application. FF00 consumes stems as ordinary references and records their provenance; it does not bundle UVR, manage its models, or make separation a runtime dependency. Later convenience integration may discover user-provided outputs without changing that boundary.

13. Failure and recovery semantics

healthy → degraded → faulted → restarting → healthy
                              └──────────→ quarantined

A missed deadline yields deterministic fallback and a visible degraded state. Repeated misses or a dead worker latch a fault. Restart is bounded and backed off; the last committed graph is restored only after state validation. The initial policy under test is three consecutive misses before latching and at most two automatic restart attempts. A state that repeatedly crashes is quarantined instead of creating a restart loop.

Effects fall back to latency-aligned dry audio where possible. Instruments emit silence, all-notes-off, and reset state. Stale shared-memory generations and late IPC replies are discarded. Recovery never promotes Preview state or tells the user an unverified restoration succeeded.

14. What “local-only” must mean

FF00 telemetry, crash upload, remote web content, and automatic network updates are disabled for alpha. Ordinary FF00 processing must pass a network-observation test showing no non-loopback traffic from FF00 processes. Model downloads, operating-system traffic, WebView behavior, Ollama behavior, and independently networked child plugins are documented separately rather than hidden inside the claim.

References are never uploaded by default. Logs and public validation records must remove private paths, keys, tokens, personal media names, opaque plugin state, and machine identifiers. Local-only describes FF00’s behavior, not a promise about every third-party binary a user loads.

15. Windows first, Linux from the boundary

IPC, shared memory, supervision, realtime scheduling, clocks, locks, paths, plugin discovery, and native editor handling receive explicit platform interfaces from the first commit. Windows remains the only supported audio runtime until the same contracts are tested on Linux. Linux CI may compile and exercise non-audio core logic without implying Waveform or VST3 UI support.

The intended repository licence is AGPL-3.0-or-later, aligned with the other FF00 applications and contributor agreement. Release work still requires a dependency and asset inventory separating source licence, model licence, runtime dependency, dynamically loaded plugin, and material that cannot be redistributed. Waveform, Surge XT, UVR, Ollama models, and reference material are not bundled.

16. Evidence, unknowns, and milestone gates

As verified on 2026-08-20, the development Rack runs one external worker per retained Rack through fixed shared-memory audio/MIDI slots and named-pipe lifecycle control. Component tests cover a two-block impulse delay from 64 through 1024 samples and delayed-dry fallback after worker termination.

Waveform 14.0.49 loaded the Rack, created independent workers after duplication, cleaned all workers up when the edit closed, and restored one fresh worker after reopen. Deliberately killing the live worker did not kill Waveform; playback had no obvious fault in a non-critical listening check, and edit close/reopen restored the worker. This is a Milestone 2 host-lifecycle checkpoint, not completion of the milestone.

Still unverified or unimplemented: measured host PDC/null accuracy, callback allocation instrumentation, automatic restart and quarantine orchestration, variable-block and offline rendering, pipe ACL proof, Surge hosting and native UI, Waveform script operations, corrupt child-state recovery, and observed non-loopback network behavior. Warning-free linting also remains work.

The architecture rejects three tempting shortcuts: blocking until the worker is ready, treating model text as execution, and rewriting Waveform XML. Each makes an early demo easier while moving uncertainty into realtime safety, truth, or project integrity. The project advances only when those boundaries survive the recorded tests.