veffects: Turning Any Track Into Math-Only Visuals — Inside the Score Format, the Plugin ABI and the Renderer

Executive Summary

Most music visualizers are a shader plus an FFT, welded together in one process and one render loop. veffects takes the pieces apart. It splits the problem into an analysis stage that turns any track — mp3, wav, flac or midi — into a compact numeric “score”, and a rendering stage that is a pure consumer of that score. The score is 160 bytes per data frame at 60 frames per second: eight normalized scalars (loudness, bass, mid, treble, spectral centroid, spectral flux, onset strength, peak loudness) plus 32 log-spaced spectrum bands. A 21-minute track becomes a 12 MB file of nothing but floats — no audio, no imagery, no assets.

The player is a plugin host. Every one of the 28 bundled scenes is a shared library (.dylib / .so / .dll) that exports four C functions and draws into a shared additive HDR frame buffer; the host owns envelope smoothing, scene crossfades, bloom, chromatic aberration, beat-driven camera shake, film grain and tone mapping. Everything is drawn in software on the CPU at 640×480, with no GPU shader stack, no texture assets and no runtime dependencies beyond SDL2. That design has two consequences worth the article: rendering is deterministic and seekable (any frame can be produced from time alone, so offline mp4 export and scrubbing are trivial), and adding a new visual is a self-contained 200-line C++ file that you drop into plugins/. This piece walks the format, the analyzer, the plugin ABI, the renderer and the build glue — then makes the case for where you would actually use this, and invites you to write a scene.

The Core Idea: Separate the Listening From the Drawing

The architectural decision that shapes everything else is stated in the first lines of docs/FORMAT.md:

A .veffects file is a compact, precomputed “mathematical score” of a track: per-frame audio features that a player turns into visuals. It contains no audio and no imagery — only normalized numbers. Roughly 9.6 KB per second of audio.

docs/FORMAT.md

Two processes fall out of that. veffects_gen is the analyzer: it decodes a file to PCM, runs a windowed FFT at a fixed 60 data-frames per second, extracts features and writes the score. veffects_play is the host: it loads a score (or analyzes a track in-process), samples it by time rather than by frame index, and hands the sampled values to whichever scene plugin is active. The two never share state beyond the file format.

The payoff is that the render frame rate is decoupled from the analysis frame rate. The score is 60 Hz; the player might run at 60 fps on screen, 30 fps into an mp4 export, or produce one single frame at t = 30.05 s for a screenshot. All three go through the same interpolating accessors, so all three agree.

The .veffects Score: 160 Bytes per Frame

The container is deliberately boring: little-endian binary, a packed header, then frameCount fixed-size frames. Each frame is 8 scalar floats followed by bandCount spectrum floats. At the default 32 bands that is (8 + 32) × 4 = 160 bytes per frame, and at 60 frames per second exactly 9,600 bytes per second of audio.

The header and the scalar enumeration are the whole schema:

#pragma pack(push, 1)
struct VfxHeader {
    char     magic[4];      // "VFX1"
    uint32_t version;       // 1
    uint32_t fps;           // data frame rate (usually 60)
    uint32_t frameCount;    // number of data frames
    uint32_t bandCount;     // spectrum bands per frame (usually 32)
    uint32_t sampleRate;    // source audio sample rate
    float    duration;      // track length, seconds
    float    bpm;           // tempo estimate (0 if unknown)
    uint32_t reserved[6];   // reserved for future use
};
#pragma pack(pop)

// Scalar parameters of one frame (indices into the scalars array).
enum VfxScalar {
    VFX_RMS      = 0,  // overall loudness (smoothed)
    VFX_BASS     = 1,  // low-band energy   (~40-160 Hz)
    VFX_MID      = 2,  // mid-band energy   (~160-2000 Hz)
    VFX_TREBLE   = 3,  // high-band energy  (~2-16 kHz)
    VFX_CENTROID = 4,  // spectral centroid ("brightness" of timbre)
    VFX_FLUX     = 5,  // spectral flux     (rate of spectral change)
    VFX_BEAT     = 6,  // onset/beat strength for this frame (peaks = beats)
    VFX_LOUD     = 7,  // peak loudness (fast, unsmoothed)
    VFX_NSCALARS = 8
};

Here are the two tables from docs/FORMAT.md, reproduced as-is.

The repository ships a sample score, track.veffects, and reading its header gives concrete numbers for the claim above: magic VFX1, version 1, 60 fps, 76,350 frames, 32 bands, 44,100 Hz source, duration 1272.50 s (21:12), estimated tempo 61.0 BPM. File size on disk: 12,216,056 bytes, which is 9,600.02 bytes per second of audio — the header rounding error and nothing else.

The reason a player can sample a 60 Hz score at any frame rate is these two accessors. Note that they clamp rather than fail, so a scene that asks for a time past the end of the track gets the last frame instead of a crash:

    // Linear interpolation of a scalar at an arbitrary time t (seconds).
    float scalarLerp(double t, VfxScalar s) const {
        double f = t * header.fps;
        if (f < 0) f = 0;
        uint32_t i0 = (uint32_t)f;
        uint32_t i1 = i0 + 1;
        float a = (float)(f - i0);
        return scalarAt(i0, s) * (1.f - a) + scalarAt(i1, s) * a;
    }

vfxSave() and vfxLoad() in the same header are a page of fwrite / fread with a magic-and-version check. The entire format — structs, I/O, interpolation — is one self-contained header with no dependencies beyond the C++ standard library, which means anything that wants to consume a score can vendor a single file.

Inside the Analyzer: FFT, Bands, Flux, Onsets, Tempo

include/veffects_analyze.h is the other self-contained header: decode plus analysis, shared by the CLI analyzer and the player. Input dispatch is by file extension, with mp3 as the fallback:

// Decode any supported audio file into PCM (dispatch by extension).
inline bool vfxDecodeFile(const std::string& path, VfxAudioPCM& pcm, std::string* err) {
    if (vfxHasExt(path, ".wav"))  return vfxDecodeWav(path, pcm, err);
    if (vfxHasExt(path, ".flac")) return vfxDecodeFlac(path, pcm, err);
    if (vfxHasExt(path, ".mid") || vfxHasExt(path, ".midi")) return vfxDecodeMidi(path, pcm, err);
    return vfxDecodeMp3(path, pcm, err);   // default: mp3
}

mp3 goes through minimp3, wav through dr_wav, flac through dr_flac. MIDI is the interesting one: tml.h parses the events, and the project renders them with a ~50-line built-in synth — a sine plus a saw with an attack/sustain/release envelope, channel 9 switched to an LCG noise burst for percussion. No soundfont, no external synth. It sounds like a chiptune, which is exactly enough to drive visuals.

A hand-rolled iterative FFT

There is no FFTW, no KissFFT, no dependency at all. The transform is a standard bit-reversal permutation followed by an iterative Cooley–Tukey butterfly, 24 lines, operating in place on separate real and imaginary vectors:

inline void vfxa_fft(std::vector<float>& re, std::vector<float>& im) {
    const int n = (int)re.size();
    for (int i = 1, j = 0; i < n; i++) {
        int bit = n >> 1;
        for (; j & bit; bit >>= 1) j ^= bit;
        j ^= bit;
        if (i < j) { std::swap(re[i], re[j]); std::swap(im[i], im[j]); }
    }
    for (int len = 2; len <= n; len <<= 1) {
        float ang = -2.f * (float)M_PI / len;
        float wr = cosf(ang), wi = sinf(ang);
        for (int i = 0; i < n; i += len) {
            float cr = 1.f, ci = 0.f;
            for (int k = 0; k < len / 2; k++) {
                float ur = re[i+k],          ui = im[i+k];
                float vr = re[i+k+len/2]*cr - im[i+k+len/2]*ci;
                float vi = re[i+k+len/2]*ci + im[i+k+len/2]*cr;
                re[i+k] = ur + vr;  im[i+k] = ui + vi;
                re[i+k+len/2] = ur - vr;  im[i+k+len/2] = ui - vi;
                float ncr = cr*wr - ci*wi;  ci = cr*wi + ci*wr;  cr = ncr;
            }
        }
    }
}

Band edges are laid out logarithmically from 40 Hz to 16 kHz, then de-duplicated so that no two bands can collapse onto the same FFT bin at low frequencies — a real problem at 2048 samples and 44.1 kHz, where the first few log bands would otherwise all land in bin 1. The three fixed bass/mid/treble ranges are pinned to the same bin mapping:

    std::vector<int> bandEdge(BANDS + 1);
    for (int b = 0; b <= BANDS; b++) {
        float f = VFXA_FMIN * powf(VFXA_FMAX / VFXA_FMIN, (float)b / BANDS);
        int bin = (int)(f * FFT_SIZE / sr);
        bandEdge[b] = std::min(std::max(bin, 1), FFT_SIZE / 2 - 1);
    }
    for (int b = 1; b <= BANDS; b++)
        if (bandEdge[b] <= bandEdge[b-1]) bandEdge[b] = bandEdge[b-1] + 1;

    auto hz2bin = [&](float f){ return std::min(std::max((int)(f * FFT_SIZE / sr), 1), FFT_SIZE/2 - 1); };
    const int bBass0 = hz2bin(40),   bBass1 = hz2bin(160);
    const int bMid0  = hz2bin(160),  bMid1  = hz2bin(2000);
    const int bTre0  = hz2bin(2000), bTre1  = hz2bin(16000);

The per-frame loop

One iteration per data frame. The hop is sampleRate / 60 samples; the FFT window is centred on the hop start (wstart = start - FFT_SIZE/2) and multiplied by a Hann window. RMS and peak come from the raw hop, not the window. Then band energies, the spectral centroid as a magnitude-weighted mean bin converted to Hz, and spectral flux as the sum of positive magnitude deltas — with a frequency weighting that deliberately favours low frequencies, because kick drums are what you want to see on screen:

    for (uint32_t fi = 0; fi < frames; fi++) {
        size_t start = (size_t)(fi * hop);
        float rms = 0, peak = 0;
        size_t hopN = (size_t)hop;
        for (size_t i = 0; i < hopN && start + i < nSamples; i++) {
            float v = mono[start + i];
            rms += v * v;
            peak = std::max(peak, fabsf(v));
        }
        sRms[fi] = sqrtf(rms / std::max<size_t>(hopN, 1));
        sLoud[fi] = peak;

        long wstart = (long)start - FFT_SIZE / 2;
        for (int i = 0; i < FFT_SIZE; i++) {
            long idx = wstart + i;
            float v = (idx >= 0 && idx < (long)nSamples) ? mono[idx] : 0.f;
            re[i] = v * hann[i];
            im[i] = 0.f;
        }
        vfxa_fft(re, im);
        for (int i = 0; i < FFT_SIZE / 2; i++)
            mag[i] = sqrtf(re[i]*re[i] + im[i]*im[i]);

        auto sumRange = [&](int a, int b){
            float s = 0; for (int i = a; i < b; i++) s += mag[i]*mag[i]; return s;
        };
        sBass[fi] = sqrtf(sumRange(bBass0, bBass1));
        sMid[fi]  = sqrtf(sumRange(bMid0,  bMid1));
        sTre[fi]  = sqrtf(sumRange(bTre0,  bTre1));

        float num = 0, den = 0;
        for (int i = 1; i < FFT_SIZE / 2; i++) { num += i * mag[i]; den += mag[i]; }
        sCen[fi] = den > 1e-9f ? (num / den) * ((float)sr / FFT_SIZE) : 0.f;

        float flux = 0;
        for (int i = 1; i < FFT_SIZE / 2; i++) {
            float dm = mag[i] - prevMag[i];
            if (dm > 0) {
                float freqW = 1.f / (1.f + (float)i / bBass1);
                flux += dm * (0.4f + 1.6f * freqW);
            }
        }
        sFlux[fi] = flux;
        std::swap(mag, prevMag);

        for (int b = 0; b < BANDS; b++) {
            float s = 0;
            for (int i = bandEdge[b]; i < bandEdge[b+1]; i++) s += prevMag[i]*prevMag[i];
            bandsRaw[(size_t)fi * BANDS + b] = sqrtf(s / (bandEdge[b+1] - bandEdge[b]));
        }
    }

Note std::swap(mag, prevMag) at the end of the flux computation: the previous frame’s spectrum is recycled as this frame’s buffer, so the loop allocates nothing. The band pass immediately afterwards reads prevMag, which is now the current frame’s magnitudes.

Onsets and tempo

Onset detection is adaptive-threshold peak picking: subtract 1.3× the local mean flux over a ±15-frame (quarter-second) window, half-wave rectify, then keep only local maxima. Tempo is estimated by autocorrelating the onset envelope over lags corresponding to 60–200 BPM and taking the strongest lag — brute force, O(frames × lags), and entirely adequate for a one-shot offline pass:

    const int Wl = VFXA_FPS / 4;
    for (uint32_t i = 0; i < frames; i++) {
        float mean = 0; int cnt = 0;
        for (int j = -Wl; j <= Wl; j++) {
            long k = (long)i + j;
            if (k >= 0 && k < (long)frames) { mean += sFlux[k]; cnt++; }
        }
        mean /= std::max(cnt, 1);
        float v = sFlux[i] - 1.3f * mean;
        sOnset[i] = v > 0 ? v : 0;
    }
    std::vector<float> onsetPeaks(frames, 0.f);
    for (uint32_t i = 1; i + 1 < frames; i++)
        if (sOnset[i] > sOnset[i-1] && sOnset[i] >= sOnset[i+1])
            onsetPeaks[i] = sOnset[i];

    float bpm = 0;
    {
        int lagMin = (int)(60.0 / 200.0 * VFXA_FPS);
        int lagMax = (int)(60.0 / 60.0  * VFXA_FPS);
        float best = 0; int bestLag = 0;
        for (int lag = lagMin; lag <= lagMax && lag < (int)frames; lag++) {
            double s = 0;
            for (uint32_t i = 0; i + lag < frames; i++) s += (double)sOnset[i] * sOnset[i + lag];
            if (s > best) { best = (float)s; bestLag = lag; }
        }
        if (bestLag > 0) bpm = 60.f * VFXA_FPS / bestLag;
    }

Finally, normalization. Every feature is divided by its own 98th percentile (99.5th for onset peaks) and clamped to 1.0, so a quiet track and a loud master both fill the [0..1] range and scenes never have to guess at absolute levels. The band values get an extra treble tilt and a pow(x, 0.6) curve, which is what stops a spectrum display from looking like a wall on the left and a flat line on the right:

    auto normalize = [&](std::vector<float>& v) {
        float p = vfxa_percentile(v, 0.98f);
        for (float& x : v) x = std::min(x / p, 1.f);
    };
    normalize(sRms); normalize(sBass); normalize(sMid); normalize(sTre);
    normalize(sFlux); normalize(sLoud);
    { float p = vfxa_percentile(onsetPeaks, 0.995f);
      for (float& x : onsetPeaks) x = std::min(x / p, 1.f); }
    for (float& x : sCen) x = std::min(x / 8000.f, 1.f);
    { float p = vfxa_percentile(bandsRaw, 0.98f);
      for (uint32_t fi = 0; fi < frames; fi++)
        for (int b = 0; b < BANDS; b++) {
            float tilt = 1.f + 1.5f * (float)b / BANDS;
            float& x = bandsRaw[(size_t)fi * BANDS + b];
            x = std::min(x * tilt / p, 1.f);
            x = powf(x, 0.6f);
        } }

Percentiles use std::nth_element on a copy — O(n) selection instead of a full sort. That matters: a 21-minute track is 76,350 frames × 32 bands = 2.4 million band values to normalize.

The Plugin ABI: Four Functions and an HDR Buffer

This is the contract that makes the project extensible, and it is worth reading in full because it is remarkably small. A scene is a shared library that exports four C-linkage functions:

const VfxPluginInfo* vfx_plugin_info(void);
void* vfx_plugin_create(int width, int height);            // returns scene state (or NULL)
void  vfx_plugin_destroy(void* state);
void  vfx_plugin_render(void* state, const VfxCanvas* canvas, const VfxParams* p);

Everything a scene knows about the music arrives in one struct, per frame. Nothing is pulled; nothing is global:

typedef struct VfxParams {
    double time;        // track time, seconds
    double dt;          // seconds since previous frame
    float  alpha;       // crossfade weight [0..1] -- multiply your output by it

    float  bass, mid, treble;   // low / mid / high band energy (smoothed)
    float  rms;                 // overall loudness (smoothed)
    float  loud;                // peak loudness (fast)
    float  centroid;            // spectral centroid ("brightness" of timbre)
    float  flux;                // spectral flux
    float  beat;                // beat strength (smoothed envelope)
    float  onset;               // instantaneous onset for this frame (peaks = beats)

    float  bpm;                 // tempo estimate (0 if unknown)
    float  duration;            // track length, seconds
    int    frameNo;             // render frame number (for pseudo-randomness)

    int          bandCount;     // number of spectrum bands
    const float* bands;         // bandCount smoothed band energies [0..1]

    int    width, height;       // frame dimensions

    // Optional input image for image-driven scenes (loaded via the GUI "Open
    // image" button or by dropping a jpg/png/bmp onto the window). NULL when no
    // image is loaded. Pixels are row-major, 8-bit, imageChannels bytes each.
    const unsigned char* image;
    int    imageW, imageH, imageChannels;
} VfxParams;

And everything a scene can draw with arrives in the other struct. The canvas is a raw width × height × 3 float buffer, additive and unclamped, plus four function pointers. A plugin may use the primitives, write into fb directly, or mix both:

typedef struct VfxCanvas {
    int    width, height;
    float* fb;                  // width*height*3 floats, additive HDR, RGB row-major
    void*  impl;                // engine-internal

    // Primitives (pixel coordinates; k = brightness/weight; r,g,b linear [0..~]).
    void (*add_px)  (const struct VfxCanvas*, int x, int y,
                     float r, float g, float b, float k);
    void (*add_glow)(const struct VfxCanvas*, float x, float y,
                     float r, float g, float b, float k, float rad);
    void (*add_line)(const struct VfxCanvas*, float x0, float y0, float x1, float y1,
                     float r, float g, float b, float k, float w);
    // Utility: HSV(h,s,v) -> linear RGB. h wraps.
    void (*hsv)(float h, float s, float v, float* r, float* g, float* b);
} VfxCanvas;

The host installs trampolines for those pointers that forward into its own renderer, so the plugin never links against the host binary — it only needs the header:

// ---- canvas trampolines ----
static void tramp_add_px(const VfxCanvas* c, int x, int y, float r, float g, float b, float k) {
    ((Renderer*)c->impl)->addPx(x, y, RGB{r, g, b}, k);
}
static void tramp_add_glow(const VfxCanvas* c, float x, float y, float r, float g, float b, float k, float rad) {
    ((Renderer*)c->impl)->addGlow(x, y, RGB{r, g, b}, k, rad);
}
static void tramp_add_line(const VfxCanvas* c, float x0, float y0, float x1, float y1, float r, float g, float b, float k, float w) {
    ((Renderer*)c->impl)->addLine(x0, y0, x1, y1, RGB{r, g, b}, k, w);
}
static void tramp_hsv(float h, float s, float v, float* r, float* g, float* b) {
    RGB c = hsvf(h, s, v); *r = c.r; *g = c.g; *b = c.b;
}

A complete, do-nothing scene is therefore about ten lines:

#include "veffects_plugin.h"

static VfxPluginInfo INFO = {
    VFX_PLUGIN_ABI, "My Scene", "you", "One-line description."
};
VFX_EXPORT const VfxPluginInfo* vfx_plugin_info(void) { return &INFO; }
VFX_EXPORT void* vfx_plugin_create(int W, int H) { /* ... */ return state; }
VFX_EXPORT void  vfx_plugin_destroy(void* s) { /* ... */ }
VFX_EXPORT void  vfx_plugin_render(void* s, const VfxCanvas* cv, const VfxParams* p) { /* draw */ }

The two rules that actually matter

docs/PLUGINS.md is blunt about the only two things a scene author can get wrong at the architectural level:

1. Multiply every contribution by p->alpha. The player crossfades scenes by rendering two of them into the same buffer with complementary weights. A scene that ignores alpha will punch through every transition.
2. Derive animation from p->time, not from accumulated per-frame state. This is what makes seeking and offline rendering correct. If you keep state, make it a pure function of p->time — the way the rain columns in the Ghost in the Shell scene keep a per-column phase and speed, but compute the head position from fmodf(phase * travel + t * speed, travel).

There is a third, subtler rule: destructive full-buffer operations — row shifts, multiplicative scanlines, anything that reads back what is already in fb — are only allowed when p->alpha > 0.98, because during a crossfade the buffer also contains the other scene, and mangling it corrupts both.

The Host: Envelopes, Crossfades and Reactive Cuts

The score is raw feature data; it is deliberately not smoothed for display. The host applies asymmetric attack/release envelopes on top — fast rise, slow fall — which is what makes a bass hit snap and then decay rather than flicker:

    void frame(double t, double dt) {
        frameNo++;
        std::fill(fb.begin(), fb.end(), 0.f);
        if (hasData()) {
            float bass = data.scalarLerp(t, VFX_BASS), rms = data.scalarLerp(t, VFX_RMS);
            float mid  = data.scalarLerp(t, VFX_MID),  tre = data.scalarLerp(t, VFX_TREBLE);
            float cen  = data.scalarLerp(t, VFX_CENTROID), beat = data.scalarLerp(t, VFX_BEAT);
            auto env = [&](float& e, float v, float up, float dn) {
                e = v > e ? lerpf(e, v, up) : lerpf(e, v, dn);
            };
            env(envBass, bass, 0.5f, 0.06f);  env(envRms, rms, 0.4f, 0.05f);
            env(envMid,  mid,  0.5f, 0.07f);  env(envTre, tre, 0.5f, 0.10f);
            env(envCen,  cen,  0.05f, 0.05f); env(envBeat, beat, 0.8f, 0.10f);
            for (uint32_t b = 0; b < data.header.bandCount; b++)
                env(bandEnv[b], data.bandLerp(t, b), 0.6f, 0.12f);
        }
        int total = totalScenes();
        if (total > 0 && hasData()) {
            if (forceScene >= 0) {
                renderEntry(forceScene < total ? forceScene : 0, t, dt, 1.f);
            } else {
                autoRender(t, dt);
            }
        }
        postFX(t);
    }

Read the envelope constants: bass and mid rise at 0.5 and fall at 0.06/0.07 per frame; the beat envelope rises at 0.8 and falls at 0.10; the spectral centroid is symmetric at 0.05 both ways, because timbre “colour” should drift, not twitch. Those six numbers are most of the perceived feel of the whole program.

Reactive scene changes derived from the score

Auto-rotation has three modes: timed (a fixed 38-second segment), shuffle (hash-ordered), and reactive, which is the interesting one. Cut points are precomputed once per track by heavily smoothing RMS (a one-pole filter at 0.02, roughly a one-second time constant) and placing a cut wherever the smoothed loudness has drifted more than 0.18 from the level at the last cut, subject to a 12-second minimum and a 38-second maximum gap. In practice that lands cuts on build-ups and drops:

    // Reactive mode: derive scene-change times from the track's energy structure.
    // A cut is placed where the smoothed loudness has drifted noticeably since the
    // last cut (a build-up or drop), spaced within [minGap, maxGap] seconds.
    void computeCuts() {
        cuts.clear();
        if (!hasData()) return;
        int N = (int)data.header.frameCount, fps = (int)data.header.fps;
        if (N < 2 || fps <= 0) return;
        std::vector<float> e(N);
        float acc = 0;
        for (int i = 0; i < N; i++) {
            float v = data.scalars[(size_t)i * VFX_NSCALARS + VFX_RMS];
            acc = lerpf(acc, v, 0.02f); e[i] = acc;    // ~1s smoothing
        }
        const double minGap = 12.0, maxGap = 38.0;
        cuts.push_back(0.f);
        double lastCut = 0; float refE = e[0];
        for (int i = 1; i < N; i++) {
            double ti = (double)i / fps;
            if ((fabsf(e[i] - refE) > 0.18f && ti - lastCut > minGap) ||
                (ti - lastCut > maxGap)) {
                cuts.push_back((float)ti); lastCut = ti; refE = e[i];
            }
        }
        cuts.push_back((float)data.header.duration + 1.f);
        fprintf(stderr, "reactive: %zu cut points\n", cuts.size());
    }

Rendering a segment boundary is then just rendering both scenes with complementary smoothstep weights into the same additive buffer — there is no separate compositing pass, because the buffer is linear and additive by construction:

    // Auto rotation: timed (fixed length) or reactive (score-driven cut points),
    // crossfading into the new scene at each segment boundary.
    // pool of scene indices to cycle: favorites only, or all
    void buildPool(std::vector<int>& pool) {
        int total = totalScenes();
        for (int i = 0; i < total; i++)
            if (!favoritesOnly || favSet.count(plugins[i].name)) pool.push_back(i);
        if (pool.empty()) for (int i = 0; i < total; i++) pool.push_back(i);
    }
    int poolPick(long k, int ps) {
        if (ps <= 0) return 0;
        if (autoMode == 2) {                      // shuffle: pseudo-random order
            int idx = (int)(hashu32((uint32_t)(k * 2654435761u)) % (uint32_t)ps);
            if (ps > 1) {
                int prev = (int)(hashu32((uint32_t)((k - 1) * 2654435761u)) % (uint32_t)ps);
                if (idx == prev) idx = (idx + 1) % ps;
            }
            return idx;
        }
        return (int)(((k % ps) + ps) % ps);
    }
    void autoRender(double t, double dt) {
        int total = totalScenes(); if (total <= 0) return;
        std::vector<int> pool; buildPool(pool);
        int ps = (int)pool.size();
        long k; double s0;
        if (autoMode == 1 && cuts.size() >= 2) {   // reactive: score-derived cut points
            k = 0;
            while (k + 1 < (long)cuts.size() && cuts[k + 1] <= t) k++;
            if (k < 0) k = 0;
            s0 = cuts[k];
        } else {                                   // timed / shuffle: fixed intervals
            k = (long)(t / SCENE_LEN); s0 = k * (double)SCENE_LEN;
        }
        int cur = pool[poolPick(k, ps)];
        float local = (float)(t - s0);
        if (local < FADE && k > 0 && ps > 1) {
            float a = smoothstepf(0.f, FADE, local);
            int prev = pool[poolPick(k - 1, ps)];
            renderEntry(prev, t, dt, 1.f - a);
            renderEntry(cur, t, dt, a);
        } else {
            renderEntry(cur, t, dt, 1.f);
        }
    }

Post-Processing: Bloom, Aberration, Shake, Grain, Tone Map

Scenes draw unbounded linear light. Everything cinematic happens afterwards, in the host, identically for every scene — which is why a 200-line plugin can look like it has a render pipeline behind it. Bloom is a quarter-resolution downsample with a smoothstep brightness threshold, followed by two separable 5-tap box passes (four passes total, horizontal and vertical, twice) which approximates a wide Gaussian for the cost of 20 taps:

    void postFX(double t) {
        (void)t;
        for (int y = 0; y < BH; y++)
            for (int x = 0; x < BW; x++) {
                RGB s = {0,0,0};
                for (int dy = 0; dy < 4; dy++)
                    for (int dx = 0; dx < 4; dx++) {
                        size_t i = (((size_t)(y*4+dy)) * W + (x*4+dx)) * 3;
                        s.r += fb[i]; s.g += fb[i+1]; s.b += fb[i+2];
                    }
                size_t o = ((size_t)y * BW + x) * 3;
                float lum = (s.r + s.g + s.b) / 48.f;
                float k = smoothstepf(0.55f, 1.4f, lum);
                bloomA[o] = s.r / 16.f * k; bloomA[o+1] = s.g / 16.f * k; bloomA[o+2] = s.b / 16.f * k;
            }
        for (int pass = 0; pass < 2; pass++) {
            for (int y = 0; y < BH; y++)
                for (int x = 0; x < BW; x++) {
                    RGB s = {0,0,0};
                    for (int k = -2; k <= 2; k++) {
                        int xx = std::clamp(x + k, 0, BW - 1);
                        size_t i = ((size_t)y * BW + xx) * 3;
                        s.r += bloomA[i]; s.g += bloomA[i+1]; s.b += bloomA[i+2];
                    }
                    size_t o = ((size_t)y * BW + x) * 3;
                    bloomB[o] = s.r / 5; bloomB[o+1] = s.g / 5; bloomB[o+2] = s.b / 5;
                }
            for (int y = 0; y < BH; y++)
                for (int x = 0; x < BW; x++) {
                    RGB s = {0,0,0};
                    for (int k = -2; k <= 2; k++) {
                        int yy = std::clamp(y + k, 0, BH - 1);
                        size_t i = ((size_t)yy * BW + x) * 3;
                        s.r += bloomB[i]; s.g += bloomB[i+1]; s.b += bloomB[i+2];
                    }
                    size_t o = ((size_t)y * BW + x) * 3;
                    bloomA[o] = s.r / 5; bloomA[o+1] = s.g / 5; bloomA[o+2] = s.b / 5;
                }
        }

Then camera shake and chromatic aberration are applied in a single resampling pass. Shake is quadratic in the beat envelope (5.5 * envBeat²), so quiet passages are perfectly still and a hit throws the frame; aberration scales each colour channel about the frame centre by a slightly different factor, which is the cheap and correct way to fake lateral chromatic aberration. Bloom is added back bilinearly from the quarter-res buffer in the same loop, and the whole thing is row-parallel across hardware threads:

        float shake = 5.5f * envBeat * envBeat;
        float shx = (hash21(frameNo, 41) - 0.5f) * 2.f * shake;
        float shy = (hash21(frameNo, 42) - 0.5f) * 2.f * shake;
        float ca = 0.0025f + 0.008f * envBeat;
        auto sampleFB = [&](float fx, float fy, int c) -> float {
            int x0 = (int)floorf(fx), y0 = (int)floorf(fy);
            float ax = fx - x0, ay = fy - y0;
            x0 = std::clamp(x0, 0, W - 2); y0 = std::clamp(y0, 0, H - 2);
            size_t i00 = ((size_t)y0 * W + x0) * 3 + c;
            return lerpf(lerpf(fb[i00], fb[i00 + 3], ax),
                         lerpf(fb[i00 + (size_t)W * 3], fb[i00 + (size_t)W * 3 + 3], ax), ay);
        };
        parallelRows([&](int y) {
            for (int x = 0; x < W; x++) {
                float dx = x - W * 0.5f, dy = y - H * 0.5f;
                size_t o = ((size_t)y * W + x) * 3;
                float off[3] = { 1.f + ca, 1.f, 1.f - ca };
                for (int c = 0; c < 3; c++) {
                    float sx2 = W * 0.5f + dx * off[c] + shx;
                    float sy2 = H * 0.5f + dy * off[c] + shy;
                    resolve[o + c] = sampleFB(sx2, sy2, c);
                }
                float bx = clampf((float)x / 4.f - 0.5f, 0.f, BW - 1.001f);
                float by = clampf((float)y / 4.f - 0.5f, 0.f, BH - 1.001f);
                int bx0 = (int)bx, by0 = (int)by;
                float axc = bx - bx0, ayc = by - by0;
                size_t b00 = ((size_t)by0 * BW + bx0) * 3;
                for (int c = 0; c < 3; c++) {
                    float v = lerpf(lerpf(bloomA[b00+c], bloomA[b00+3+c], axc),
                                    lerpf(bloomA[b00+(size_t)BW*3+c], bloomA[b00+(size_t)BW*3+3+c], axc), ayc);
                    resolve[o + c] += v * 0.9f;
                }
            }
        });
    }

The final resolve is a vignette, a Reinhard-ish exponential tone map (1 - exp(-v * 1.55)), a gamma 2.2 encode and hashed film grain, straight to RGB24:

    // tonemap + vignette + grain -> RGB24
    void toRGB24(std::vector<uint8_t>& out) {
        out.resize((size_t)W * H * 3);
        parallelRows([&](int y) {
            float vy = (float)y / H - 0.48f;
            for (int x = 0; x < W; x++) {
                float vx = (float)x / W - 0.5f;
                float vig = clampf(1.f - 1.10f * (vx * vx + vy * vy), 0.f, 1.f);
                float grain = (hash21(x + frameNo * 613, y) - 0.5f) * 0.020f;
                size_t i = ((size_t)y * W + x) * 3;
                for (int c = 0; c < 3; c++) {
                    float v = resolve[i + c] * vig;
                    v = 1.f - expf(-v * 1.55f);
                    v = powf(v, 1.f / 2.2f) + grain;
                    out[i + c] = (uint8_t)(clampf(v, 0.f, 1.f) * 255.f + 0.5f);
                }
            }
        });
    }

That is the entire “engine”: about 90 lines of post-processing running on the CPU at 640×480. No GPU, no shader compiler, no driver surprises — and the same code path produces the on-screen frame and the mp4 frame, so what you export is exactly what you saw.

Anatomy of a Scene: Ghost in the Shell

plugins/ghost_in_the_shell.cpp is the documented reference implementation, and it exercises nearly every part of the ABI in 261 lines. Its file header states the contract it keeps:

All animation is derived from the absolute time p->time, so any frame renders correctly without simulation history (offline render and seeking both work).

plugins/ghost_in_the_shell.cpp

Setup: per-column rain state and a procedural skull

vfx_plugin_create allocates the scene state once. The rain columns get a hashed speed, length, seed and phase — deterministic, derived from the column index, so there is no RNG to reseed. The “skull” is an ellipsoid mesh tapered toward the chin by a per-latitude profile function, with edges generated as a lat/lon wireframe:

VFX_EXPORT void* vfx_plugin_create(int W, int H){
    State* s = new State();
    s->W = W; s->H = H; s->cols = W / s->cw;
    s->rain.resize(s->cols);
    for(int c=0;c<s->cols;c++){
        RainCol& r = s->rain[c];
        r.speed = 45.f + hashf(c,71)*130.f;
        r.len   = 8 + (int)(hashf(c,72)*18);
        r.seed  = hashu(c*2654435761U + 12345U);
        r.phase = hashf(c,73);
        r.flick = hashf(c,74)*100.f;
    }
    // "skull": ellipsoid tapered toward the chin
    const int LAT = 11, LON = 20;
    auto vidx = [&](int i,int j){ return i*LON + (j%LON); };
    for(int i=0;i<LAT;i++){
        float t = (float)i/(LAT-1);
        float lat = (t - 0.5f) * (float)M_PI;
        float low = clampf((0.5f - t)*2.f, 0.f, 1.f);
        float top = clampf((t - 0.72f)/0.28f, 0.f, 1.f);
        float prof = 1.f - 0.45f*low*low - 0.25f*top;
        float rx = 0.72f*prof, ry = 1.0f, rz = 0.80f*prof;
        for(int j=0;j<LON;j++){
            float lon = (float)j/LON * 2.f*(float)M_PI;
            s->verts.push_back({ rx*cosf(lat)*cosf(lon), ry*sinf(lat), rz*cosf(lat)*sinf(lon) });
        }
    }
    for(int i=0;i<LAT;i++)
        for(int j=0;j<LON;j++){
            s->edges.push_back({ vidx(i,j), vidx(i,j+1) });
            if(i+1<LAT) s->edges.push_back({ vidx(i,j), vidx(i+1,j) });
        }
    s->rowtmp.resize((size_t)W*3);
    return s;
}

The digital rain

Each column is a head glyph plus a fading tail. The head position is a pure function of time; the glyph seed mixes the column seed, the cell row and a quantized time term, so glyphs flicker and change without any stored state. The head cell is brighter, gets a treble-scaled boost and an explicit add_glow call through the canvas primitives:

    // 1) digital rain
    for(int c=0;c<s->cols;c++){
        RainCol& r = s->rain[c];
        float tailPx = r.len * s->ch;
        float travel = H + tailPx;
        float headY  = fmodf(r.phase*travel + (float)t*r.speed, travel) - tailPx;
        int px = c * s->cw;
        int headCell = (int)floorf(headY / s->ch);
        for(int q=0;q<r.len;q++){
            int cellY = headCell - q;
            int py = cellY * s->ch;
            if(py < -s->ch || py > H) continue;
            float fade = 1.f - (float)q / r.len; fade *= fade;
            uint32_t gs = hashu(r.seed ^ (uint32_t)(cellY*2246822519U) ^ (uint32_t)((t*7.0 + r.flick)));
            float rr,gg,bb,k;
            if(q==0){ rr=0.85f; gg=1.0f; bb=0.85f; k=(1.7f + 0.9f*tre);
                      cv->add_glow(cv, px+s->cw*0.5f, py+s->ch*0.5f, 0.4f,1.0f,0.5f, 0.5f*A, 5.f); }
            else    { rr=0.10f; gg=1.0f; bb=0.30f; k=(0.40f + 0.85f*fade)*(0.85f+0.5f*tre); }
            drawGlyph(fb,W,H, px,py, s->cw,s->ch, gs, rr,gg,bb, k*A);
        }
    }

The glyphs themselves are procedural — two to four horizontal strokes, one to three vertical strokes and an occasional diagonal, all positioned by hashing the seed. That is the entire “katakana” font: no glyph atlas, no font file, no texture.

A 3D wireframe with a five-line projector

The skull is rotated by yaw (time-driven) and pitch (mid-band-driven), scaled by the beat envelope, and projected with a hand-written perspective divide. Edge brightness is modulated by depth so back-facing geometry recedes without a depth buffer:

    // 2) wireframe cyber-skull
    float cx = W*0.5f, cy = H*0.46f;
    float yaw = (float)t*0.55f;
    float pitch = 0.12f*sinf((float)t*0.35f) + 0.30f*mid;
    float scale = 1.f + 0.09f*beat;
    float cyS=cosf(yaw), syS=sinf(yaw), cpS=cosf(pitch), spS=sinf(pitch);
    float f=360.f, camZ=3.2f;
    auto project = [&](Vec3 v, float& sx, float& sy, float& depth){
        v.x*=scale; v.y*=scale; v.z*=scale;
        float x1=v.x*cyS+v.z*syS, z1=-v.x*syS+v.z*cyS;
        float y1=v.y*cpS-z1*spS,  z2= v.y*spS+z1*cpS;
        float den=camZ-z2; if(den<0.2f)den=0.2f;
        sx=cx+f*x1/den; sy=cy-f*y1/den; depth=z2;
    };
    for(auto& e : s->edges){
        float ax,ay,ad,bx,by,bd;
        project(s->verts[e.first], ax,ay,ad);
        project(s->verts[e.second], bx,by,bd);
        float md=(ad+bd)*0.5f;
        float front=clampf(0.5f+md*0.9f,0.05f,1.f);
        float k=(0.05f+0.16f*front)*(0.8f+0.5f*rms);
        lineAdd(fb,W,H, ax,ay,bx,by, 0.15f*front,0.9f,0.75f, k*A);
    }
    for(int e=0;e<2;e++){
        Vec3 ev={ (e?0.28f:-0.28f),0.06f,0.72f };
        float sx,sy,dep; project(ev,sx,sy,dep);
        if(dep>0.05f){
            float k=(0.5f+2.5f*beat)*A;
            cv->add_glow(cv, sx,sy, 0.6f,1.0f,0.7f, k, 3.5f);
            for(int a=0;a<4;a++){
                float an=a*1.5708f+(float)t*1.5f; float r0=5,r1=8;
                lineAdd(fb,W,H, sx+cosf(an)*r0,sy+sinf(an)*r0, sx+cosf(an)*r1,sy+sinf(an)*r1, 0.2f,1.f,0.6f, 0.5f*A);
            }
        }
    }

Destructive effects, gated on alpha

Scanlines multiply every third row by 0.55, and the beat glitch shifts random horizontal slices sideways by copying through a scratch row. Both read back the frame buffer, so both are gated behind A > 0.98f exactly as the plugin documentation requires:

    // destructive post-effects only when the scene is fully visible
    if(A > 0.98f){
        for(int y=0;y<H;y+=3){
            float* row=&fb[(size_t)y*W*3];
            for(int x=0;x<W*3;x++) row[x]*=0.55f;               // CRT scanlines
        }
        if(glitch > 0.15f){                                     // horizontal slice glitch
            int slices=1+(int)(glitch*4);
            for(int q=0;q<slices;q++){
                uint32_t hs=hashu((uint32_t)(p->frameNo*131 + q*977));
                int y0=(int)(hashf(hs,1)*H);
                int hgt=4+(int)(hashf(hs,2)*22);
                int dx=(int)((hashf(hs,3)-0.5f)*60.f*glitch);
                for(int y=y0; y<y0+hgt && y<H; y++){
                    float* row=&fb[(size_t)y*W*3];
                    memcpy(s->rowtmp.data(), row, (size_t)W*3*sizeof(float));
                    for(int x=0;x<W;x++){
                        int sx=x-dx; if(sx<0)sx+=W; if(sx>=W)sx-=W;
                        row[x*3+0]=s->rowtmp[sx*3+0];
                        row[x*3+1]=s->rowtmp[sx*3+1];
                        row[x*3+2]=s->rowtmp[sx*3+2];
                    }
                }
            }
        }
    }

Image-Driven Scenes: Feed It a Photo

The ABI carries an optional input image — p->image, imageW, imageH, imageChannels, populated when the user opens or drops a jpg/png/bmp (loaded through stb_image, forced to three channels). Two bundled scenes build a world out of it, and they are the shortest plugins in the tree.

Photo Particles (2D)

A 150×110 sampling grid over the image. Each cell becomes a glowing dot at its “home” position, coloured by the pixel. A scatter term built from the beat envelope and RMS displaces every particle along a per-particle hashed direction, with a slow sinusoidal wander and a slight pull toward the centre so the explosion “breathes” instead of just expanding. Quiet passages let the picture reform:

    const unsigned char* img=p->image; int iw=p->imageW, ih=p->imageH, ic=p->imageChannels;
    // aspect-fit the image into the frame
    float scale=fminf((float)W/iw,(float)H/ih);
    float dW=iw*scale, dH=ih*scale, oX=(W-dW)*0.5f, oY=(H-dH)*0.5f;

    // scatter amount: quiet -> reformed photo, loud/beat -> exploded
    float scatter = 0.6f*beat + 0.25f*rms + 0.15f;
    float cx=W*0.5f, cy=H*0.5f;

    for(int gy=0; gy<s->GY; gy++){
        for(int gx=0; gx<s->GX; gx++){
            float u=(gx+0.5f)/s->GX, vv=(gy+0.5f)/s->GY;
            int ix=(int)(u*iw); if(ix>=iw)ix=iw-1;
            int iy=(int)(vv*ih); if(iy>=ih)iy=ih-1;
            const unsigned char* px=&img[((size_t)iy*iw+ix)*ic];
            float r=px[0]/255.f, g=px[1]/255.f, b=px[2]/255.f;

            float bx=oX+u*dW, by=oY+vv*dH;                 // home position (the photo)
            // per-particle scatter direction + gentle time wander
            uint32_t h=hashu(gx*73856093u ^ gy*19349663u);
            float ang=hashf(h,1)*6.2831853f;
            float dist=hashf(h,2);
            float outx=cosf(ang), outy=sinf(ang);
            float wob=sinf((float)t*(1.5f+2.f*hashf(h,3))+hashf(h,4)*6.28f);
            float disp=scatter*(30.f+120.f*dist)*(0.7f+0.6f*wob);
            // pull toward center a touch as it scatters, so it "breathes"
            float px2=bx + outx*disp + (cx-bx)*0.15f*scatter;
            float py2=by + outy*disp + (cy-by)*0.15f*scatter;

            float bright=(0.5f+0.9f*rms)*(0.8f+0.5f*tre);
            float sparkle = (hashf(h,(uint32_t)(t*6)) < tre*0.15f) ? 1.8f : 1.f;
            float rad=1.3f + 1.2f*(1.f-scatter*0.5f);
            addGlowFB(fb,W,H, px2,py2, r,g,b, bright*sparkle*0.6f*A, rad);
        }
    }

Image World 3D (isometric voxels)

The same image read as a heightmap: luminance (0.299R + 0.587G + 0.114B) becomes column height, pixel colour becomes the tile tint, and a sine wave driven by the beat envelope ripples across the map. A 64×48 grid is rotated in world space by a slowly orbiting yaw, projected isometrically and painter-sorted far-to-near — no depth buffer, just std::sort on rx + rz:

    // build tiles
    for(int j=0;j<GY;j++){
        for(int i=0;i<GX;i++){
            int idx=j*GX+i;
            Tile& T=s->tiles[idx];
            float u=(i+0.5f)/GX, v=(j+0.5f)/GY;
            float r,g,b,h;
            if(haveImg){
                int ix=(int)(u*iw); if(ix>=iw)ix=iw-1;
                int iy=(int)(v*ih); if(iy>=ih)iy=ih-1;
                const unsigned char* px=&img[((size_t)iy*iw+ix)*ic];
                r=px[0]/255.f; g=px[1]/255.f; b=px[2]/255.f;
                h=0.299f*r+0.587f*g+0.114f*b;
            } else {
                // placeholder: rolling sine terrain, teal palette
                h=0.5f+0.5f*sinf(u*10+ (float)t)*cosf(v*10-(float)t*0.6f);
                r=0.1f+0.3f*h; g=0.5f+0.5f*h; b=0.6f+0.3f*h;
            }
            float cxw=(i-GX*0.5f), czw=(j-GY*0.5f);
            float rx=cxw*cyaw - czw*syaw;
            float rz=cxw*syaw + czw*cyaw;
            // ripple wave across the map on beats
            float wave = sinf((cxw+czw)*0.35f - (float)t*2.2f)*0.18f*beat;
            T.rx=rx; T.rz=rz; T.depth=rx+rz;
            T.r=r; T.g=g; T.b=b; T.h=clampf(h+wave,0.f,1.4f);
            s->order[idx]=idx;
        }
    }
    // far -> near
    std::sort(s->order.begin(), s->order.end(),
              [&](int a,int b){ return s->tiles[a].depth < s->tiles[b].depth; });

Each tile then draws as an extruded column (a darker side quad) plus a lit top diamond, with a rim highlight on tall tiles driven by the mid band. Vertical exaggeration pumps with bass: heightPx = 40 * (1 + 0.6 * bass).

    for(int oi=0; oi<(int)s->order.size(); oi++){
        Tile& T=s->tiles[s->order[oi]];
        float sideH = T.h*heightPx;
        float sx = W*0.5f + (T.rx - T.rz)*tw;
        float sy = baseY + (T.rx + T.rz)*th - sideH;
        if(sx< -10||sx>W+10||sy<-10||sy>H+80) continue;
        float lit = 0.22f + 0.5f*T.h;                  // brighter = higher
        float glow = (0.45f + 0.4f*rms)*A;
        // side (extruded column), darker
        float sr=T.r*0.45f, sg=T.g*0.45f, sb=T.b*0.55f;
        for(int dx=-(int)drawR; dx<=(int)drawR; dx++){
            float edge = drawTh*(1.f - fabsf((float)dx)/drawR);
            int y0=(int)(sy+edge);
            for(int yy=0; yy<(int)sideH; yy++)
                putAdd(fb,W,H, (int)sx+dx, y0+yy, sr,sg,sb, 0.22f*glow);
        }
        // top diamond, lit
        for(int dy=-(int)drawTh; dy<=(int)drawTh; dy++){
            float span=drawR*(1.f - fabsf((float)dy)/drawTh);
            for(int dx=-(int)span; dx<=(int)span; dx++)
                putAdd(fb,W,H, (int)sx+dx, (int)sy+dy, T.r,T.g,T.b, lit*glow);
        }
        // rim highlight on tall tiles hit by mid
        if(T.h>0.75f)
            putAdd(fb,W,H,(int)sx,(int)(sy-drawTh), 1.f,1.f,1.f, 0.3f*mid*A);
    }

Build Glue: Every .cpp in plugins/ Is a Scene

There is no plugin registry, no manifest and no list to edit. CMake globs the directory and builds each file as a MODULE library straight into build/bin/plugins/, with visibility hidden so only the four exported symbols are visible, and with the suffix forced to .dylib on macOS to match what the loader looks for:

# ---------------- scene plugins ----------------
file(GLOB VEFFECTS_PLUGIN_SRC "${CMAKE_SOURCE_DIR}/plugins/*.cpp")
foreach(psrc ${VEFFECTS_PLUGIN_SRC})
  get_filename_component(pname "${psrc}" NAME_WE)
  add_library(${pname} MODULE "${psrc}")
  target_include_directories(${pname} PRIVATE ${VEFFECTS_INCLUDES})
  set_target_properties(${pname} PROPERTIES
    PREFIX ""
    LIBRARY_OUTPUT_DIRECTORY "${VEFFECTS_PLUGINS}"
    RUNTIME_OUTPUT_DIRECTORY "${VEFFECTS_PLUGINS}")
  foreach(cfg DEBUG RELEASE RELWITHDEBINFO MINSIZEREL)
    set_target_properties(${pname} PROPERTIES
      LIBRARY_OUTPUT_DIRECTORY_${cfg} "${VEFFECTS_PLUGINS}"
      RUNTIME_OUTPUT_DIRECTORY_${cfg} "${VEFFECTS_PLUGINS}")
  endforeach()
  if(APPLE)
    set_target_properties(${pname} PROPERTIES SUFFIX ".dylib")  # match the loader
  endif()
  if(MSVC)
    target_compile_options(${pname} PRIVATE /O2)
  else()
    target_compile_options(${pname} PRIVATE -O3 -fvisibility=hidden)
  endif()
endforeach()

On the loading side, the host scans its plugin directories, sorts for a deterministic order, dlopens each candidate, resolves the four symbols and — critically — checks the ABI version before trusting anything else. A stale plugin from an older ABI is skipped with a warning rather than being called into:

// ---- plugin discovery/loading ----
static bool loadPluginHandle(const std::string& path, PluginHandle& out) {
    void* dl = dynOpen(path.c_str());
    if (!dl) return false;
    auto info = (vfx_info_fn)   dynSym(dl, "vfx_plugin_info");
    auto cr   = (vfx_create_fn) dynSym(dl, "vfx_plugin_create");
    auto de   = (vfx_destroy_fn)dynSym(dl, "vfx_plugin_destroy");
    auto rn   = (vfx_render_fn) dynSym(dl, "vfx_plugin_render");
    if (!info || !cr || !de || !rn) { dynClose(dl); return false; }
    const VfxPluginInfo* pi = info();
    if (!pi || pi->abi != VFX_PLUGIN_ABI) { dynClose(dl); return false; }
    out.dl = dl; out.info = pi; out.create = cr; out.destroy = de; out.render = rn;
    out.name = pi->name ? pi->name : "plugin";
    return true;
}

The platform layer above it is thirty lines of #if defined(_WIN32): LoadLibraryA/GetProcAddress/FreeLibrary versus dlopen/dlsym/dlclose, plus an exeDir() implemented three ways (GetModuleFileNameA, _NSGetExecutablePath, /proc/self/exe) so the binary can find its own plugins/ folder regardless of the working directory.

Building a plugin by hand, without CMake, is a single command:

clang++ -O3 -std=c++17 -dynamiclib -fvisibility=hidden -Iinclude \
    -o build/bin/plugins/my_scene.dylib plugins/my_scene.cpp

Playback, Sync and Offline Export

Audio/video sync is solved by making the audio device the clock. The SDL audio callback advances a sample position — and keeps advancing it while muted, so “watch the visuals in silence” does not stop time:

// ---- audio context (callback-driven, mute-aware, drives the master clock) ----
struct AudioCtx {
    std::vector<short> pcm;   // interleaved 16-bit
    int hz = 0, ch = 2;
    std::atomic<size_t> pos{0};
    std::atomic<bool>   muted{false};
    std::atomic<bool>   hasAudio{false};
};
static void audioCB(void* ud, Uint8* stream, int len) {
    AudioCtx* a = (AudioCtx*)ud;
    Sint16* out = (Sint16*)stream;
    int n = len / (int)sizeof(Sint16);
    size_t p = a->pos.load(std::memory_order_relaxed);
    size_t sz = a->pcm.size();
    bool m = a->muted.load(std::memory_order_relaxed);
    for (int i = 0; i < n; i++) {
        if (p < sz) { out[i] = m ? 0 : a->pcm[p]; p++; }   // advance even when muted
        else out[i] = 0;
    }
    a->pos.store(p, std::memory_order_relaxed);
}

The main loop then derives track time from that atomic position, falling back to a performance counter when there is no audio at all:

        // ---- master clock ----
        double t;
        if (actx.hasAudio.load()) {
            size_t p = actx.pos.load();
            t = (double)(p / std::max(1, actx.ch)) / std::max(1, actx.hz);
            if (R.hasData() && t >= R.data.header.duration - 0.02) {   // loop
                actx.pos.store(0); t = 0;
            }
        } else {
            Uint64 now = SDL_GetPerformanceCounter();
            if (!paused) tManual += (double)(now - t0) / pfreq;
            t0 = now;
            t = tManual;
            if (R.hasData() && t > R.data.header.duration) { tManual = 0; t = 0; }
        }

Export is a pipe. The renderer produces RGB24 frames at 30 fps and writes them into a popen’d ffmpeg that muxes the original audio alongside — and because every scene is a pure function of time, the exported frames are bit-identical to what an on-screen playback would have produced at those timestamps:

// Render the whole track (one scene) to an mp4 via ffmpeg. Returns 1 ok, 2 no ffmpeg, 3 aborted.
static int exportMp4(Renderer& R, const std::string& audio, const std::string& outPath,
                     int sceneIdx, std::atomic<bool>* abort = nullptr,
                     std::atomic<int>* progress = nullptr) {
    if (!R.hasData()) return 2;
    std::string cmd = "ffmpeg -y -loglevel error -f rawvideo -pix_fmt rgb24 -s 640x480 -r 30 -i - ";
    bool ha = vfxHasExt(audio, ".mp3") || vfxHasExt(audio, ".wav") || vfxHasExt(audio, ".flac");
    if (ha) cmd += "-i \"" + audio + "\" ";
    cmd += "-c:v libx264 -pix_fmt yuv420p ";
    if (ha) cmd += "-c:a aac -shortest ";
    cmd += "\"" + outPath + "\"";
    FILE* pipe = VFX_POPEN(cmd.c_str(), "w");
    if (!pipe) return 2;
    int fps = 30, nF = (int)(R.data.header.duration * fps); double ddt = 1.0 / fps;
    int saved = R.forceScene; R.forceScene = sceneIdx;
    std::vector<uint8_t> buf;
    for (int i = 0; i < nF; i++) {
        if (abort && abort->load()) break;
        R.frame(i * ddt, ddt); R.toRGB24(buf);
        fwrite(buf.data(), 1, buf.size(), pipe);
        if (progress && (i & 15) == 0) progress->store((int)(100.0 * i / std::max(nF, 1)));
    }
    R.forceScene = saved;
    VFX_PCLOSE(pipe);
    return (abort && abort->load()) ? 3 : 1;
}

The same machinery is available headless from the command line, which is what makes the project usable as a batch renderer:

# analyze a track to a .veffects score
./build/bin/veffects_gen track.mp3 track.veffects

# play a score with synced audio, forcing a scene
./build/bin/veffects_play track.veffects --audio track.mp3 --scene-name "Blade Runner"

# or just hand the player the mp3 and let it analyze in-process
./build/bin/veffects_play track.mp3

# build a world from an image
./build/bin/veffects_play track.mp3 --image photo.jpg --scene-name "Image World 3D"

And the raw-frame path, for piping into whatever encoder you prefer:

./build/bin/veffects_play track.veffects --render --fps 30 --start 0 --end 60 \
    --scene-name "Ghost in the Shell" \
  | ffmpeg -f rawvideo -pix_fmt rgb24 -s 640x480 -r 30 -i - \
           -ss 0 -t 60 -i track.mp3 \
           -c:v libx264 -pix_fmt yuv420p -c:a aac -shortest out.mp4

The Scene Library

Twenty-eight scenes ship in the box — twenty-six themed worlds and two image-driven ones. Every entry below is a single self-contained .cpp in plugins/, between 100 and 456 lines. The descriptions are the ones the plugins report through VfxPluginInfo:

Where and How You Would Actually Use This

The interesting part of veffects is not the scene catalogue — it is that the analysis output is a documented file format and the renderer is a plugin host. That combination makes it usable in places a monolithic visualizer is not.

Things it does today, out of the box

• Live visuals and VJ-style playback. Open a track, pick Auto with Reactive mode, star the scenes you want with the favorites toggle, and the rotation stays inside your set and cuts on the track’s own build-ups and drops. Keys 1–9 jump to scenes, 0 returns to auto.
• Music-video and lyric-video backdrops. One command renders a whole track with audio to an mp4 at a fixed scene. Deterministic rendering means you can re-render a single 10-second range and cut it in without seams.
• Batch/headless rendering on a server. --render writes raw RGB24 to stdout; there is no window, no GPU and no display server involved. That is a CI job or a queue worker, not a desktop app.
• Turning a photo into a world. Drop a jpg on the window and the same track now drives either a particle field of that image or an isometric voxel city built from its brightness. Album art in, visualizer out.
• Installations and kiosks. 640×480 software rendering with no GPU requirement runs on a Raspberry-Pi-class box, a museum kiosk, an old laptop driving a projector, or a machine with no working graphics drivers at all.
• MIDI visualization without a synth stack. Hand it a .mid and the built-in chiptune renderer gives you both audible playback and a feature stream.

Things the architecture invites

• Teaching material for graphics and DSP. A student can read the entire signal chain — FFT, band split, flux, onset picking, autocorrelation tempo — in one 394-line header with no library indirection, then see each feature move something on screen. The same is true of the renderer: bloom, tone mapping and chromatic aberration in ninety readable lines of scalar C++.
• A feature-extraction front-end for something else entirely. The score is the point. veffects_format.h is one dependency-free header; anything — a game, a web player, a lighting rig, a Unity project, a plotting script — can vendor it and read a 9.6 KB/s stream of loudness, bands, onsets and tempo without ever touching an audio decoder.
• Stage and DMX lighting cues. Precomputed onsets, BPM and band energies with stable, percentile-normalized ranges are exactly what a lighting cue engine wants, and precomputation means zero-latency lookahead: you know the drop is coming before it lands.
• Game and demo-scene prototyping. The plugin ABI is a clean sandbox for trying a procedural effect — write it against a flat float buffer, iterate in a hot reload cycle of “rebuild one .cpp, restart the player”, and port to a shader once the idea holds up.
• Accessibility and audio inspection. A precise, time-aligned visual encoding of a track is useful anywhere the audio itself cannot be heard — silent displays, ambient screens, or simply an engineer eyeballing whether the onset detector agrees with the beat.

What it is not

It is a 640×480 CPU renderer with a precomputed feature file. It does not do live microphone input, it does not render at 4K, and it does not use your GPU. Those are design choices that buy determinism, portability and a very low barrier to writing a scene — and, as it happens, they are also three of the most obvious things a contributor could change.

Join the Project

veffects is MIT-licensed and open to contributions at github.com/oxfemale/veffects. The contribution surface is unusually friendly, because the most valuable thing you can add — a new scene — touches exactly one new file and zero existing ones.

The five-minute path in: write a scene

1. Copy plugins/ghost_in_the_shell.cpp (the commented reference) or the 100-line plugins/photo_particles.cpp if you want the shortest possible starting point.
2. Implement the four exports. Draw with add_px / add_glow / add_line, or write into cv->fb directly.
3. Multiply everything by p->alpha; drive motion from p->time.
4. Drop the file in plugins/, reconfigure CMake, rebuild. It appears in the scene dropdown — no registration, no player changes, no rebuild of anything else.
5. Sanity-check a single frame straight to PNG with the one-liner in docs/PLUGINS.md, then open a pull request.

If you have ever wanted to write a demo-scene effect but bounced off shader toolchains and windowing boilerplate, this is that, minus all of it: a float buffer, a time value, eight audio numbers and a spectrum.

Bigger pieces that are wide open

• A GPU backend. The canvas abstraction is already the seam — the four primitives and an additive HDR target map cleanly onto a compute or fragment pass. Someone could keep the plugin ABI and add an optional GPU path for higher resolutions.
• Higher resolution and aspect handling. W and H are compile-time constants (640×480) in the player. Making them runtime and letting scenes scale is a well-scoped refactor.
• Live audio input. The analyzer is an offline batch pass today. A streaming variant that fills a rolling window would unlock microphone and line-in visualization.
• A WebAssembly / browser port. Software rendering, no GPU dependency and a tiny score format make this a very natural wasm target; the score could even be served separately from the audio.
• Better MIDI. The built-in synth is deliberately minimal. SoundFont support via TinySoundFont (the same author as the bundled tml.h) is an obvious upgrade.
• More decoders. ogg/opus/m4a are missing; the decoder dispatch is a six-line function, so adding one is mechanical.
• Format v2. There are six reserved uint32s in the header. Chroma features, key detection, beat-grid positions and section labels would all be at home there.
• Tests and Windows QA. CI builds all three platforms and smoke-tests --list-scenes; there is room for golden-frame regression tests, and for people who actually run the MSVC build day to day.
• Documentation and examples. A gallery of community scenes, a tutorial that builds one scene from an empty file, or a worked example of consuming .veffects from another language.

Issues, pull requests, scene submissions and format proposals are all welcome. If you build something on top of the score format — in any language — that is arguably the most interesting contribution of all, and worth an issue just to say so.

Key Takeaways

• Precomputing the analysis into a documented file format is the whole design. 160 bytes per frame, 9.6 KB/s, no audio inside — and every consumer downstream becomes trivially seekable and deterministic.
• Sampling by time, not by frame index (scalarLerp / bandLerp) is what decouples a 60 Hz score from a 30 fps export and a 60 fps display.
• The plugin ABI is four C functions and two structs. Version-checked at load (VFX_PLUGIN_ABI), trampoline-dispatched, and small enough that a scene never links against the host.
• Additive linear HDR plus shared post-processing means a 200-line scene inherits bloom, chromatic aberration, beat shake, grain and tone mapping for free — and crossfades are literally just two renders with complementary alphas into the same buffer.
• Percentile normalization (98th / 99.5th) is what makes scenes portable across quiet masters and loud ones; scenes can assume [0..1] and never calibrate.
• Pure-time animation is a hard rule, not a style preference. It is what makes seeking, offline rendering and single-frame screenshots all produce the same pixels.
• Zero heavy dependencies. A hand-rolled FFT, vendored single-header decoders, SDL2 and Dear ImGui — and CMake globs plugins/*.cpp, so contributing a scene means adding one file.

Hardening Checklist

Nothing here is a vulnerability report against a hobby visualizer — but the project loads native code and parses untrusted media, so if you deploy it on a kiosk, a shared render box or anything facing users, these are the edges worth knowing about.

• Plugin loading is arbitrary code execution by design. The host dlopens every .dylib/.so/.dll found in its plugin directories, which include exeDir()/../plugins. Treat that directory as part of your trusted computing base: make it non-writable by the runtime user, and prefer an explicit --plugins path over the search defaults on multi-user machines.
• Quote or reject shell metacharacters in export paths. exportMp4() builds an ffmpeg command line by string concatenation and runs it through popen(), i.e. through /bin/sh. Double-quoting the operands stops spaces but not a filename that itself contains a double quote. If paths can ever come from somewhere other than the local file dialog, switch to fork/exec with an argv array, or validate hard.
• Media parsing is your attack surface, not the visuals. minimp3, dr_wav, dr_flac, tml and stb_image all parse fully untrusted input. Keep the vendored copies in third_party/ current, and if you accept files from users, run the analyzer as a separate short-lived low-privilege process rather than inside the GUI.
• Validate score headers if you accept .veffects files from others. vfxLoad() checks magic and version, then resizes vectors from frameCount and bandCount read straight out of the file — a crafted header is a large-allocation denial of service. A sanity bound on both fields is a two-line fix.
• Fuzz the two parsers you own. vfxLoad() and the MIDI note-rendering path are small, self-contained and ideal libFuzzer targets; that would be a genuinely useful first contribution.
• Build with sanitizers during scene development. Plugins write into a raw float buffer through their own inlined helpers; an off-by-one in a scene’s bounds check corrupts the host’s heap, and -fsanitize=address,undefined finds it in seconds.
• Pin the ABI when you distribute binaries. VFX_PLUGIN_ABI is checked at load, so a mismatched plugin is skipped rather than mis-called — keep that check in place if you fork the loader.

Credits and Licenses

veffects itself is MIT-licensed, Copyright © 2026 veffects contributors. The vendored third-party code in third_party/ keeps its own licenses, and all of it is worth knowing about independently:

Scene names referencing films, anime, novels and games are affectionate homages rendered entirely in code — there are no assets, frames or trademarked artwork from any of those works anywhere in the repository.

Conclusion

veffects is a small project with an unusually clean idea at the centre: analyze once into a documented numeric score, render many times from it, and make every visual a hot-discovered plugin that only depends on a single header. That separation is what buys deterministic seeking, honest offline export, cross-platform builds with no GPU stack, and a contribution path where a new visual is one new file. Whether you want music-reactive visuals for a stream, a batch renderer for videos, a teaching example of a full DSP and rendering chain in readable C++, or just a place to write the procedural effect you have been carrying around in your head — clone it, build it, and drop a .cpp into plugins/.

Original text: the veffects repository by veffects contributors on GitHub, MIT-licensed.