Making an AY chip imitate a voice

This is an experiment that follows on from VTX Speech. That project builds speech from text: words become phonemes, and phonemes become AY register settings using the chip's envelope as the buzz of a voice. Here the starting point is a recording instead. For every 50th of a second, the tool searches for the AY register settings that sound closest to the recording, then plays the result through an AY emulator so you can compare.

Everything runs in your browser. Load a WAV file, record your own voice or try the examples; nothing is uploaded. For noisy recordings, clean them first with NVIDIA RE-USE (section 4). There is no cartridge or Spectrum player yet: the output is register data, checked only in emulation.

Open the tool Source on GitHub

Listen

The openings of five public-domain LibriVox readings (clean recordings, voices from about 86 to 200 Hz) and two unaccompanied songs, one in a high voice, each beside the AY version made from it: 50 updates a second on a Vectrex's AY clock, free search with the current defaults (exact squares and noise × tone, section 3b), no gate; denoise only on the nursery rhyme (2). The tool's example menu loads them.

Reading	AY STOI	4-band vocoder	8-band vocoder
“Kearney at Seven Pines” Edmund Clarence Stedman; read for LibriVox; voice about 200 Hz	0.764	0.693	0.757
The Gettysburg Address Abraham Lincoln; read by John Greenman for LibriVox; about 120 Hz	0.705	0.752	0.808
John Adams to John Quincy Adams, 27 July 1777 read for LibriVox; about 170 Hz	0.686	0.672	0.776
“The Seven Ages of Man” William Shakespeare; read for LibriVox; about 145 Hz	0.693	0.685	0.767
The Swiss Family Robinson, opening Johann David Wyss; read by Mark F. Smith for LibriVox; a deep voice, about 86 Hz	0.678	0.718	0.806
“Twinkle, Twinkle, Little Star”, sung unaccompanied, by Wikimedia Commons user Dcoetzee (CC0); about 180 Hz, pitch followed every update	0.665	0.634	0.692
“A chantar m’er”, sung 12th-century song by the Comtessa de Dia, unaccompanied; recorded by Wikimedia Commons user Makemi (public domain); a high voice, about 350 Hz	0.475	0.427	0.490

The vocoder columns are the same readings through 4- and 8-band noise vocoders, for scale (section 2): the AY versions sit around the 4-band level, the highest speaking voice beats 8 bands, and both songs sit between 4 and 8 bands. The deep voice does worst against the references although its pitch is the most accurate (7 cents): at 86 Hz there are about 70 harmonics under 6 kHz, too dense a spectrum for three channels to follow, and the bare envelope buzz does much of the work. Pitch is followed every update: in the nursery rhyme the envelope's steps land 14 cents from the sung notes on average (40 at worst); in the high voice, where each envelope step is about 6%, 29 cents (62 at worst). STOI rates the long sustained notes of the second song lower overall, vocoders included. Noisy sources, such as the Apollo radio recordings, score far lower: clean them first, ideally with NVIDIA RE-USE (section 4). Sources and licences are under Credits.

1. How it works

The AY-3-8910 has three square-wave channels, one noise source and one envelope generator. VTX Speech showed that the envelope, run as a fast sawtooth, makes a good voice buzz: it is rich in harmonics, like the vocal folds, and its pitch is set by the envelope period. Tones locked to exact harmonics of that buzz then shape it into vowels. Voice matching uses the same parts, but decides them from the recording, one update at a time:

Pitch. The recording's pitch is found for each update (the YIN method), and the envelope period is set to match: EP = clock ÷ (256 × pitch). Pitch follows the recording every update.
Harmonic slots. The recording's energy is measured around each harmonic of that pitch, up to 6 kHz.
Candidates. Each channel can be off; the envelope buzz with its tone off, or gated by a tone at harmonic n (tone period = 16 × EP ÷ n, so it stays locked to the buzz); a square at harmonic n at one of 15 fixed volumes; or noise at a fixed volume, with one noise period shared between channels.
Search. The three channels are chosen together, to bring the AY's harmonic energies as close as possible to the recording's in decibels. The search fits each frame's peaks, as LPC speech coders do, and ignores slots more than 30 dB below the loudest. It also tries several overall levels, because the envelope only ever plays at full volume.
Render and measure. The chosen registers are played through Ayumi, an accurate AY emulator, and the result is measured against the recording the same way.

A second mode, Speech, adds speech-coder rules: LPC formants place channel A near the first formant and B near the second, quiet high-frequency frames become noise only, and voiced frames get no noise. It places formants more accurately, but it measures as less intelligible than the free search (section 3).

The tool also shows each update's harmonic energies (recording, model and rendered side by side) and its 14 register values, and exports them for a player.

2. Measuring intelligibility

A small spectral error is not the same as speech you can understand, so every result is scored with STOI (short-time objective intelligibility, Taal et al., 2011). It is a standard measure of whether speech survives processing: it compares how the loudness of 15 frequency bands rises and falls over about 0.4 seconds in the original and the copy. It runs from 0 to 1, where 1 means identical.

To give the numbers a scale, the same clips are also put through noise vocoders. A noise vocoder keeps only the loudness of a few frequency bands and rebuilds the sound from noise. It is a standard way to simulate cochlear implants, and a few bands are known to be enough for much speech to be understood. They are references, not rivals: a vocoder controls many bands independently and updates over three times as often as 50 Hz.

Two cautions. STOI assumes its reference is clean, so measuring against a noisy recording penalises any cleaning (section 4). And it is not a listening test: the tool has an A/B button (the whole recording, then the whole AY version) for the final judgement.

3. Results so far

Four test recordings, none of which are included here: a spoken film line over a music bed, an a cappella singing clip, and two short game voice clips (8-bit, 22 and 11 kHz). All at 50 Hz with no smoothing between updates; gates set per clip.

STOI	film line	singing	game voice 1	game voice 2
noise vocoder, 16 bands (reference)	0.777	0.765	0.615	0.722
noise vocoder, 8 bands (reference)	0.688	0.683	0.521	0.636
noise vocoder, 4 bands (reference)	0.604	0.595	0.425	0.563
AY, free search	0.550	0.624	0.382	0.686
AY, built-in denoise first (best of strength 2 or 4)	0.622	0.643	0.480	0.679
AY, speech mode	0.503	0.587	0.340	0.537

Noise carries speech. In an earlier run (smoothing 10), taking noise away dropped the film line from 0.519 to 0.306 but the singing only from 0.632 to 0.611. Consonants are noise.
Free search beats the speech rules on every clip, even though the speech rules place formants better. Forcing quiet frames to be all noise and voiced frames to have none costs more than it gains.
All three channels earn their place, and so do the fixed-volume squares: without the squares the film line drops from 0.519 to 0.494 and the singing from 0.632 to 0.534 (smoothing 10); in speech mode, dropping channel C costs 0.03–0.07.
Fast changes help. No smoothing between updates scored best on speech (0.550 against 0.519 at smoothing 10), at the cost of more register writes.
With cleaning, the AY passes the 4-band vocoder on the noisy clips. The clean game voice already did.

Full tables: results/sweep.json, made by tools/sweep.js. These tables, and those in sections 4 and 5, were measured before the two defaults chosen in section 3b; those changed the figures by about ±0.02.

3b. Choosing by ear

STOI is a good guide, but it is not a listener. The later experiments were decided by blind A/B tests: each pair of AY versions played in random order, the listener picking the better one or “can't tell”, with the answers revealed only at the end.

Experiment	Blind result	STOI change	Now
Exact squares: square tones tuned to the recording's own pitch instead of the envelope's rounded pitch	preferred on the high voice; a tie on speech and the lower song	−0.02 to +0.02	default
Noise × tone: noise ANDed with a 1–6 kHz tone, moving the hiss up the spectrum for consonants	preferred 4 of 4 on noisy and cleaned speech; a tie on clean speech	−0.02 to +0.05	default
Triangle envelope (shape 10) as the buzz	the usual sawtooth preferred 3 to 1	+0.002 to +0.010	rejected
Pitch dithering: alternating envelope periods so the average pitch lands on the note	never preferred	−0.03 to +0.01	option
Formant pairs: two channels on neighbouring harmonics placing the first formant between them	can't tell	up to +0.018 on the high voice	option
Phase lock: only harmonics that keep a fixed phase against the envelope	can't tell	−0.007 to −0.020	rejected

The triangle envelope is the lesson: STOI scored it better on every clip, and it sounded worse. Anything that changes the timbre needs listening, not only a score. Each default costs some data: together they add about 10–60 register writes a second.

4. Clean the audio first

Anything in the recording that is not the voice (music, room noise, hiss) is something the AY will try to imitate with its three channels. Removing it first helps more than any other setting tried.

Built-in denoise. The tool can subtract the background's spectrum before matching. It estimates the background from the quietest tenth of the recording. On the film line it lifts STOI from 0.550 to 0.622, and on the noisier game voice from 0.382 to 0.480. On a clean recording it costs a little, so leave it at 0 there. A gate silences frames below a set level; it helps on top of denoise.

Recommended: clean noisy recordings with NVIDIA RE-USE first. RE-USE is a universal speech-enhancement model: it repairs noise, reverberation, clipping, narrow bandwidth and codec damage while keeping the speaker's voice, and accepts recordings at 8 to 48 kHz. It runs on Hugging Face or on your own machine. Neural enhancement like this separates a voice from its background far better than the tool's spectral subtraction. One noisy game clip, matched before and after cleaning with RE-USE:

AY matched to	STOI vs original	STOI vs cleaned
the original	0.382	0.563
the original, built-in denoise	0.480	0.570
the RE-USE cleaned copy	0.354	0.725
(the cleaned copy itself)	0.522	1

Read the columns with care. The cleaned copy itself only scores 0.522 against the noisy original, because STOI counts the removed noise as an error. When a clean copy exists it is the better yardstick, and against it the AY version made from the cleaned copy is the best result so far. Each version naturally scores best against the file it was matched to, so listen to decide.

How: run the recording through RE-USE, load the cleaned file in the tool with denoise at 0 and the gate off (60), and use score against to compare with the original if you like. Other tools for the same job include DeepFilterNet (open-source noise suppression), Demucs (separating vocals from music) and Adobe Podcast's Enhance Speech. Clean recordings, such as the examples above, need no cleaning.

5. Is 50 Hz enough?

One update per 50 Hz frame is what a game can afford: the register writes happen once per frame, between the drawing. Faster updates were measured to see what they would buy (free search, film line with its gate; a clean game clip):

updates a second	film line STOI	register writes a second	clean clip STOI
50	0.550	178	0.686
100	0.558	307	0.652
200	0.588	514	0.678
400	0.611	615	0.697
800	0.618	819	0.694

The film line gains 0.06 by 400 Hz and then levels off; the clean clip shows no trend. Faster rates mean writing registers in the middle of a frame from a timer, and 3–5 times the data, so 50 Hz stays the target. Cleaning the audio gains more than any update rate.

6. Playing it back on a Vectrex or a Spectrum

Vectrex: a player exists. In the tool, match at 50 Hz with machine Vectrex, then Save Vectrex .asm (the whole match) or drag across the spectrogram and save just that part. vectrex/vtx_voice_player.asm (193 bytes of 6809 code, 4 bytes of RAM) plays it: each update is a 2-byte mask of the registers that changed, then their values, about 250–350 bytes a second of speech. vectrex/voice_cart.asm is a complete example cartridge. The player matches the tool update by update in MAME, and MAME's sound matches the tool's Ayumi render; nobody has heard it on a real Vectrex yet.

For other machines, Save registers .bin writes 14 bytes per update, R0 to R13 in order. R13 (the envelope shape) is $FF when it must not be written, because writing it restarts the envelope; it is written once, when the buzz starts. A player only has to copy one 14-byte record to the AY every frame.

Match for the right machine. Every period register depends on the chip's clock, so choose the machine in the tool before matching. The same file played on a machine with a different clock would come out at the wrong pitch.

Machine	AY clock	What the player does each frame
Vectrex	1.5 MHz	Copy the record into the BIOS sound buffer (`Vec_Music_Work`) and call `Do_Sound`, which writes the registers, once per `Wait_Recal`. VTX Speech's cartridge player already works this way.
ZX Spectrum 128, +2, +3	1.7734 MHz	In the 50 Hz interrupt, for each register: select it on port `$FFFD`, write the value on `$BFFD`. The 48K Spectrum has no AY.
Amstrad CPC	1 MHz	Writes go through the PPI, so each register costs more instructions; still small at 50 Hz.
MSX	1.79 MHz	As above through the PSG ports. 60 Hz machines would need matching at 60 updates a second, which the tool does not offer yet.
Atari ST	2 MHz (YM2149)	The YM's envelope has 32 steps; the model assumes the AY's 16, so expect small differences.

Data size. At 50 Hz the raw stream is 700 bytes a second, so a 32 KB cartridge holds about 45 seconds. Most registers do not change every update: storing a 2-byte mask of the changed registers plus their values takes 300–350 bytes a second on the test clips. A real format would compress further. The tool shows both figures for each match.

CPU time is small: writing 14 registers is a few hundred cycles, a percent or two of a 50 Hz frame.

For the phoneme-based speech player, which stores words instead of frames and is much smaller, see VTX Speech.

7. How it differs from other projects

Nick Bild's AY-3-8910 speech synthesis (2021, GPL-3.0) is the closest idea. He recorded every tone and noise setting the chip can make on one channel, without envelopes, then split the target speech ("Greetings, Professor Falken") into 30 ms pieces. Each piece got the recorded sound whose spectrogram was closest (mean squared error), played from a Raspberry Pi; he describes the result as rough but distinguishable. Voice matching is the same "find the closest chip sound" approach, but with a much larger set of sounds to choose from:

all three channels, chosen together, instead of one tone channel;
the envelope as a voice buzz at the speaker's own pitch, tracked every frame, which his library leaves out;
tones locked to exact harmonics of the buzz, so all three channels reinforce one pitch;
a search over register settings rather than a fixed library of recordings, and an intelligibility score to compare settings.

audio2ay by Jari Komppa (2025, zlib licence) converts audio to AY data with a short-time FFT, giving up to three tone channels the strongest peaks between 300 and 2,000 Hz, with an optional arpeggio on the last channel. It writes 2 bytes per channel per frame and comes with a ZX Spectrum player. Its author says speech does not really work with it; it is aimed at melodic sounds. Its documentation does not mention noise or the envelope, which section 3 shows matter most for speech.

Neither project has been run here, so their quality is described in their authors' own words, not measured with STOI.

8. Limits and next steps

Hardware. The Vectrex player works in MAME; it still needs a listen on a real Vectrex. No Spectrum player yet.
A compact format, to fit more than seconds of speech.
More from the noise channel: the envelope ANDed with noise for breathy and voiced sounds (z, v), and noise ANDed with a tone to shape fricatives (s against sh).
An extra update only where the sound changes fast (consonant onsets), instead of a faster rate everywhere.
Planning a whole phrase at once rather than frame by frame.
The model's own round trip (an AY sound, matched and replayed) comes back about 3 dB off, mostly because the phase between a tone and the envelope cannot be controlled on the chip.

Credits and licences

Ayumi, the AY-3-8910/YM2149 emulator used for every sound here: Peter Sovietov; JavaScript port by Alexander Kovalenko. MIT licence, included unmodified: vendor/ayumi.js, licence.
STOI: C. H. Taal, R. C. Hendriks, R. Heusdens and J. Jensen, "An algorithm for intelligibility prediction of time–frequency weighted noisy speech", IEEE Transactions on Audio, Speech, and Language Processing 19(7), 2011. Implemented from the paper.
YIN pitch tracking: A. de Cheveigné and H. Kawahara, "YIN, a fundamental frequency estimator for speech and music", JASA 111(4), 2002.
Noise vocoders as an intelligibility reference: R. V. Shannon et al., "Speech recognition with primarily temporal cues", Science 270, 1995.
Related projects: Nick Bild's AY-3-8910 speech synthesis and Jari Komppa's audio2ay, as described above. No code from either is used.
Examples (examples/): openings of LibriVox recordings, which LibriVox dedicates to the public domain, of public-domain texts. The Gettysburg Address (Abraham Lincoln) read by John Greenman (archive.org gettysburg_johng_librivox); "The Seven Ages of Man" (William Shakespeare), John Adams's letter to John Quincy Adams of 27 July 1777, and "Kearney at Seven Pines" (Edmund Clarence Stedman), read by LibriVox volunteers for the 7th Anniversary Collection (archive.org 7th_anniversary_collection_1208_librivox, sections 2, 3 and 8). Cut to 8–11 seconds and converted to WAV. Also “Twinkle, Twinkle, Little Star” sung unaccompanied by Wikimedia Commons user Dcoetzee, released under CC0 (File:Twinkle Twinkle Little Star - sung with full lyrics.ogg): the first two lines. And “A chantar m’er de so qu’eu no volria” (Comtessa de Dia, 12th century) sung unaccompanied, recorded by Wikimedia Commons user Makemi and released into the public domain (File:A Chantar2.ogg): the first two phrases. Also the opening of The Swiss Family Robinson (Johann David Wyss) read by Mark F. Smith for LibriVox (archive.org swiss_family_robinson_librivox, 1.1–11.1 s). examples/ay/ holds this tool's output from them.
NVIDIA RE-USE is recommended for cleaning noisy recordings first, and was used to clean one test clip. It is a separate model under its own licence, not included or required.
Voice matching grew out of VTX Speech, made for VTX, a language and toolchain for the Vectrex.