Signature Projects

The MIDI/Piano Generation Pipeline

The ask was precise and, at the time, appeared simple: take raw quantized MIDI — the kind with no rubato, no dynamics, no pedal, dead flat and metronomic — and make it sound as though a real pianist had played it. That became midi-transformer. The first two generations were disposable experiments, built to be thrown away and duly thrown away; gen-3, nicknamed PerformanceNet, was the one that worked — a ~10M-parameter model that learned expressive timing and dynamics well enough to serve as the project's baseline for months.

Then came the ambitious part. Gens 4 through 7 all tried to push past gen-3 — bigger models, up to ~90M params, bf16 autocast, gradient checkpointing, and a custom "velocity trajectory loss" aimed at the one thing gen-3 never nailed, velocity correlation. Almost none of it worked. The trajectory loss proved structurally incapable of improving velocity at all, which is a clean and genuinely useful negative result; it is also still a negative result. Rubato modeling failed less gracefully: what looked for a while like emergent expressive timing turned out, on a close inspection of the note counts, to be fully degenerate output — not subtle humanization but the model collapsing outright. A separate gen-7 attempt pretrained a GPT-style tokenized model on the ARIA-MIDI corpus, 80× larger than anything the project had used before, and got further than the rest; it banked roughly 5 hours of pretraining before hitting its time budget at about 11% of the planned run. Between the failed experiments and the half-finished pretrain, the file tree had become, in the phrase used at the time, "a complete mess."

The fix was to stop trying to salvage any of it in place. Gen-3 was declared the sole production model and the gen-4 through gen-7 checkpoints were deleted outright — deprecated rather than hoarded, on the theory that a checkpoint nobody will ever load again is simply a claim on disk. The one promising idea, the GPT-style humanizer, got a clean fork of its own in piano-humanizer, carrying over the model and runtime code and none of the wreckage. Partway into that rebuild a second realization arrived, and it was rather more inconvenient than the first: much of the crowdsourced, rough MIDI the humanizer was meant to run on was itself full of quantization noise, so the noise had to go before anything could be humanized at all. That spun off a third project, piano-normalizer — an ASAP-trained denoiser built around a deliberately constrained "anchored edit-transducer," which makes timing-only add/subtract edits, always preserves the input pitches, and is capped at one edit per bar. The constraint was the point; having already watched a free-form model degenerate into noise, I chose a narrow and interpretable edit space on purpose.

A fourth project, piano-generator, took a different branch entirely — a from-scratch generator that reuses neither the normalizer nor the humanizer, pretrained on ARIA performance data with its own tokenizer. Its first overnight baseline ran in June and promptly found the same ~28M-effective-capacity ceiling the earlier models had found: loss bouncing inside a tight band no matter how the architecture was tweaked, RoPE and form mixing included. Some limits are about capacity, not cleverness.

By the time four related projects existed, the housekeeping bill arrived: over 50GB combined, duplicate token caches, and a midi-transformer .git folder bloated to 2.22GB by an early accidental git add -A of the entire dataset. The bloat turned out to be 236,000 unreachable loose objects rather than real history, and the distinction mattered a great deal; it meant a single git gc --prune=now could bring the repository down to 13MB without disturbing the actual 7-commit history. The datasets were then consolidated into one canonical ~/midi-corpus, which the other three projects reach through directory junctions instead of copies: one source of truth, and no more silent duplication.

nyc-weather: An Hourly Forecaster That Learned to Look Upstream

The weather work began as two warm-ups. The first was pure data plumbing: an ERA5 pipeline out of Copernicus's CDS, which came with war stories of its own — a 403 required licence error that turns out to be fixable only by clicking "accept" on each dataset's own web page, accumulated-versus-snapshot variables that would have silently corrupted precipitation had they gone unnoticed, and a pool-of-5 concurrent downloader without which multi-decade pulls are not practical. The second was a daily, single-location toy forecaster, weather-model, whose only job was to prove the pipeline end to end. nyc-weather then raised the bar considerably: hourly forecasts for New York City, twelve surface variables over a 500-km box, and probability of rain as the product that actually matters to anybody.

Gen1 looked respectable on temperature and hopeless on rain, and an adversarial design review established that the fault lay with the experiment rather than with the model. Five failures were verified. The precipitation transform was a no-op, log1p having been applied to values in metres, which left 37σ spikes standing in the inputs; a class weight inflated every printed rain probability by ~3.5×, so that a true 14% risk displayed as roughly 50% and lost to climatology by construction; rain was being scored through a mean-feedback rollout, which blurs into a climatological smear within a few steps; the training loss had diverged, so the deployed checkpoint was effectively 5%-trained, and it had been selected on a single random validation batch; and NYC itself accounted for 3.8% of the loss. The lesson shaped everything afterward: fix the measurement before touching the model.

Gen2 was a ladder of small, individually measurable fixes — precipitation in millimetres, the class weighting dropped on the grounds that calibration is the product, β-NLL with a sigma floor and EMA to stop the divergence, a fixed deterministic validation set, and checkpoint selection on a baseline-normalized skill composite rather than raw loss. One architectural pivot stuck, and it was the removal of a component rather than the addition of one: rain stopped being scored through the rollout at all. Direct heads now predict rain probability at eight leads plus four windowed events — "any rain tomorrow afternoon?" — in a single forward pass. Twenty-four-hour rain skill went from below climatology to genuinely positive, and temperature's long-lead divergence disappeared.

Gen3 scaled up to 114M parameters, a 336-hour context, and fifteen years of data, but its most consequential change was once again a matter of evaluation discipline: splits were rebuilt as blocked-interspersed 56-day blocks with a 168-hour embargo, so that validation and test sample every season of eleven different years, and the test set was placed under a one-shot rule — read once per generation, after every decision has been frozen on validation. It beat a logistic-regression floor at every lead through 24h. Rain at 48h and beyond, however, sat at zero skill, and the uncertainty bands covered 31–37% of outcomes while claiming 80%.

Gen4 was won on geometry, and the decision was made with cheap experiments before anything at all was downloaded. Occlusion tests showed the model barely used context beyond ~24 real hours, which closed off longer windows as a dead end; the same tests drew attention to the fact that NYC sat only ~255 km from the western edge of the data box, which is five to ten hours of upstream visibility in weather that mostly arrives from the west. The unlock was noticing that CDS's request quota counts fields, not area — the same price therefore buys pressure-level fields (geopotential, humidity, wind at 500/850 hPa) over a Midwest→Gulf→Atlantic box that actually contains tomorrow's weather. When CDS rejected the full 10-field request, temperature and u-wind were the two cut, as the most physically recoverable of the set: layer thickness and geostrophy carry them implicitly. The same run added dense distributional heads, a Gaussian per variable per lead across all 72 hours, retiring the mean-feedback rollout entirely, with per-lead sigma calibration fit on validation before the test read.

The one-shot test read landed on July 6th. Calibration verified out-of-sample: 80% bands now cover 76–84% at every lead and every variable, which closes gen3's worst defect. Temperature error dropped 28% at both 24h and 48h, beating climatology at 48 hours for the first time in the project's life; twenty-four-hour rain skill doubled, and the 25–48h windowed rain product improved by ~2.6×. The gains concentrated in exactly the 12–48h advective band the wide box was meant to feed, which is meteorologically textbook and, I admit, considerably more satisfying for having been predicted in advance. The honest ledger is shorter but real: 1-hour rain sits marginally under the logistic floor, the cost of a shared trunk, and 72-hour hourly rain remains parked at the intrinsic predictability floor. A Flask demo on port 5054 serves sub-second forecasts with the calibrated bands, sampling real held-out test mornings.

yelp-gpt: A GPT-2 Trained from Scratch on Yelp Reviews

The raw material was already sitting on disk: the Yelp Open Dataset, converted earlier into five Parquet tables of 7M reviews and 2M users, in that format specifically because the working set then fits comfortably in RAM and can be explored interactively through D-Tale's spreadsheet-like web UI. The new question was more ambitious. Could a GPT-2-scale model be trained from scratch, nanoGPT-style, on nothing but that review text?

The hardware reality check came first — an RTX 3060 Ti with 8GB of VRAM, running torch 2.12.1 with CUDA confirmed working. Eight gigabytes is not a great deal of room for a 124M-parameter model, so gradient checkpointing was switched on defensively from the start, before any real tuning, on the safe assumption that memory would be the binding constraint.

The batch-size tuning that followed overturned that assumption. Naive intuition says a larger batch trains faster up to whatever the GPU can physically hold; in practice, batch sizes of 8 and above spilled out of dedicated VRAM into shared system memory, and the spillover made throughput worse rather than better, despite the configuration technically "fitting." The fastest configuration was a small batch of 4 combined with gradient checkpointing and gradient accumulation — 32 micro-batches averaged into an effective batch of 128 — and it could only have been found by empirical sweeping, since the bigger-is-faster assumption pointed the wrong way. A second look afterward revealed that the checkpointing was barely needed at that batch size: the randomly initialized model was using 2.92GB out of 8, leaving some 5GB of headroom sitting idle while checkpointing quietly spent part of the speed that had just been won. That tension — safety margin bought at the price of throughput — was weighed directly rather than assumed away, since an overly aggressive configuration risks an out-of-memory crash hours into an unattended overnight run.

One operational loose end remained before committing to the real run: a stray training process from an earlier attempt was still technically alive. It had to be confirmed killed and the GPU verified genuinely free — down to the memory used by the display itself — so that the notification announcing the old task's failure could be read correctly as the expected side effect of the kill, rather than as the symptom of an unrelated crash contaminating the new run.

With the environment sorted, the three options were laid out side by side: a tight small-model quick check, a full from-scratch GPT-2 Small (124M params, 12 layers, 768 dimensions, a 512-token context) at roughly 12–20 hours with gradient checkpointing and accumulation, and a warm-started fine-tune of a pretrained GPT-2 as the faster alternative. The from-scratch run was chosen deliberately as the best from-scratch path rather than as the shortcut, over roughly 1.33B tokens — about one full epoch across the 7M reviews — reaching a validation loss of 1.65. Measured against the two projects running in parallel at the time, the MIDI model's 54.9M-token performance corpus and the weather model's ERA5-Land grid, yelp-gpt had by a wide margin the largest token budget of the three.

midi-transformer: From a Deterministic Regressor to Gen-3

The project's first model, gen-1, was about as simple as a "humanizer" can be: PerformanceRegressor, deterministic, trained with plain L1 loss. Give it a score and it predicts one specific timing, velocity, and pedal value per note. That works, and it is also the whole problem; a phrase that could reasonably be played several different ways still gets exactly one output, and there is no room anywhere in the architecture for interpretation.

Gen-2 tried to fix that by rebuilding the model as PerformanceCVAE, a conditional VAE with a heteroscedastic output head, so that it could both sample different plausible performances from a latent z and report its own uncertainty per note. Half of that worked; the heteroscedastic head produced a genuinely useful per-note confidence signal. The latent side collapsed. val_kl sat pinned at exactly the free-bits floor every epoch, and sampling different values of z barely moved the output at all. The cause was structural rather than a tuning miss: the score features alone were already predictable enough that the decoder had no incentive to listen to z, and the conditioning path was a single additive bias, washed out across six transformer layers.

That failure only became visible because of an eval harness built specifically to catch it. Validation NLL alone was actively misleading — it fell to 0.011 while the model was quietly collapsing — so gen-2's harness added a second measurement: the smoothed Pearson correlation between the model's deadpan prediction and the real human performance, averaged over a rolling window of notes. Sweeping that window size located the problem precisely. Velocity correlation held steady at ~0.62–0.65 regardless of window size, while rubato correlation fell apart as the window widened: 0.72 raw, 0.46 at 8 notes, 0.33 at 32. The model was reproducing local, score-driven timing and had no notion whatever of phrase-level rubato — the slow accelerando and ritardando arc a real pianist shapes across a passage.

Gen-3, PerformanceNet, was built to attack that specific finding rather than simply to be a bigger model. Its most important fix sat almost embarrassingly upstream of the network itself: gen-2's timing target, the onset offset from the nearest metric grid point, was capped at ±half a grid step by construction, and that cap physically deleted phrase-level rubato from the training data before the model ever saw it. No amount of retraining could have recovered it. Gen-3 re-represented timing as a local-tempo deviation, log(perf_IOI / score_IOI), an unbounded target that round-trips exactly; it replaced the collapsed VAE latent with five observed, supervised style descriptors — rubato magnitude, dynamic range, articulation, pedal amount, tempo — a conditioning signal that cannot collapse because it is never sampled; and it changed how progress itself was measured, selecting the best checkpoint by a composite "expression score," the average of smoothed velocity and rubato correlation, instead of by NLL, precisely because NLL had already been shown blind to the thing that mattered.

To make that comparison visible rather than merely tabulated, a dedicated eval viewer (eval_ui.py) was built: pick any of 177 held-out test pieces and it plots ground truth against gen-2 and gen-3 side by side. Below is that tool running live on a real held-out piece. The middle panel is gen-3's unbounded rubato representation, tracing an actual phrase arc; the bottom panel is gen-2's bounded representation, visibly flat by construction, unable to show a phrase arc even in principle.