Coding Guide 6 — Give your Butter Bot a new voice

Reshape the robot's pitch, speed, and intonation with three lines of code — then go further and teach it an entirely new voice. The advanced guide to Butter Bot's text-to-speech.

Every word your Butter Bot speaks comes from flite, a tiny text-to-speech engine from Carnegie Mellon University that runs entirely on the robot — no cloud, no account, no internet. Guides 2–5 changed what the robot says. This guide changes how it sounds, in two escalating stages:

This is the most advanced guide in the series, but "advanced" mostly means unfamiliar territory, not lots of code: the whole voice swap is a folder copy plus three changed lines. It's designed that way on purpose.

Prefer to let an AI do the typing?Guide 3 (Claude Code) and Guide 4 (Codex) show how. Like Guide 5, every code block here names its file, so you can paste a whole section into Claude Code or Codex and say "do this".

Everything from Topic 2 onward exists only on the voice-modding branch of ButterBot-Firmware. The master branch — the one Guides 2–5 use, and the one your robot shipped with — can no longer receive changes, and it is missing the engine-room work that new voices depend on. If you try to run one of flite's nicer voices on master, the robot doesn't just sound wrong — it crashes mid-sentence with a task_wdt: IDLE1 watchdog error (the "why" is in Topic 2).

Switching is one command inside your existing ButterBot-Firmware folder from Guide 2, plus the usual submodule refresh:

Or, if you're starting from scratch, clone the branch directly:

Everything else about the workflow is unchanged: idf.py build, idf.py -p PORT flash, exactly as in Guide 2. To go back to the standard firmware later, git checkout master (plus the same git submodule update line), rebuild, reflash.

Before installing a new voice, let's bend the one you have. The stock voice is called cmu_us_kal16, and like every flite voice it carries a small set of tunable features — named values you can override from the firmware with one function call each. No voice files get touched.

The voice is created once, in the SpeechGen constructor. Right after the register_cmu_us_kal16(nullptr) call is where your overrides go — they apply to every sentence the robot speaks from then on:

The values above are exactly what the stock voice already uses, so this block changes nothing yet — it's your baseline. What each one does:

Two ready-made personalities to try. Build, flash, and short-press the power button to make it talk (or ask for a joke):

These features are looked up by type. Set a float feature with feat_set_int() (or vice versa) and flite silently ignores your value and uses the default — no error, no warning, just a robot that stubbornly sounds the same. All four knobs above are floats; write the f suffix (95.0f) and use feat_set_float. The one integer knob you'll meet later (mlsa_speed_param, Topic 4) uses feat_set_int.

What about volume? Careful readers will notice there's no loudness knob in the table. That's flite being honest: the engine has no per-voice volume feature. The stock voice's loudness is what it is; the new-voice family in Topic 3 does have a gain setting, covered in Topic 4.

Bonus: everything in this topic also works on the regular master branch — these knobs are plain flite features that the stock firmware already understands. It's the rest of the guide that needs voice-modding.

To install a voice you only need to know three things about flite's world:

1. A voice is generated C code. There are no audio files, no models to download at runtime. A flite voice is a folder of .c files — enormous machine-generated tables baked straight into the firmware. That's why installing one is a folder copy: the build system compiles whatever voices are present, and the firmware registers the one it wants by calling that voice's register_...() function.

2. There are two voice families.Diphone voices (like the stock cmu_us_kal16) glue together tiny recorded snippets — cheap to run, unmistakably robotic. ClusterGen voices (like cmu_us_slt, this guide's worked example) generate speech from a statistical model — noticeably smoother and more human, at the cost of a lot more math per syllable.

3. That math is exactly why the branch exists. ClusterGen voices upstream do their number-crunching in double-precision floating point. The robot's ESP32-S3 chip only has single-precision hardware — doubles get emulated in software, which made ClusterGen synthesis slower than real time. The speech thread would hog its entire CPU core, a watchdog would notice the core never idles, and the robot would reboot mid-word: the task_wdt: IDLE1 crash mentioned earlier. The voice-modding branch converts flite's two synthesis engines (components/flite/src/cg/cst_mlsa.c and cst_mlpg.c) to single-precision — a one-time fix that makes every ClusterGen voice work, which is what makes the rest of this guide a folder-copy exercise.

One hard rule survives all of this:

The robot's entire audio chain — microphone, effects, amplifier — is hardwired to a 16 kHz sample rate. A voice recorded at any other rate will compile and run, but come out sounding like a broken tape deck. Every voice recommended in this guide is 16 kHz; if you go hunting for others, check before you copy (the voice's ..._cg.c file contains its sample rate — look for 16000).

Where voices live in this project:

Here's the main event. The recipe has four steps, and because the branch did the hard work already, none of them involves editing a single line inside the voice you're installing.

Ready-made voices live in the flite project's source tree, in the lang/ directory: github.com/festvox/flite. Download the repository (the ZIP button is fine here — flite has no submodules) and look inside lang/. Three are 16 kHz ClusterGen voices ready for the robot:

Avoid cmu_us_kal (that's the 8 kHz cousin of the stock voice — wrong sample rate) and cmu_time_awb (a specialist voice that only tells the time). The truly adventurous can build a voice of their own with the FestVox tools — as long as the result is 16 kHz US English C sources, the robot will take it.

Copy the whole lang/cmu_us_slt/ folder from the flite download into components/flite/src/lang/cmu_us_slt/. Then move its one odd file out: voxdefs.h goes to components/flite/include/lang/cmu_us_slt/voxdefs.h. (That header is part of flite's own build system and the robot's build never reads it — parking it under include/ just keeps the source folders tidy and every installed voice looking the same.)

That's the entire copy step. No renaming, no editing, no build-file surgery — the flite component's build automatically compiles every .c file under src/.

One command after copying: the build system's list of source files is cached, exactly like in Guide 5. After adding the new files, run idf.py reconfigure once, or the build will end with undefined reference to 'register_cmu_us_slt'. (Editing existing files never needs this.)

The firmware picks its voice in one place: the top of SpeechGen.cpp. The register/unregister functions are always named after the voice's folder — register_cmu_us_slt for cmu_us_slt, and so on:

(Installing cmu_us_rms or cmu_us_awb instead? The exact same three lines, with rms or awb in the names.)

The stock voice's data tables are enormous — around 18 MB of generated C — and once nothing references them the linker drops them from the robot's flash image anyway. But they still get compiled every build, which costs time. One line in the flite component's build file skips them:

The sources stay on disk, so switching back later is just deleting the line. (If you ever exclude a voice that's still registered in SpeechGen.cpp, the build fails with undefined reference to 'register_cmu_us_kal16' — the fix is obvious in hindsight: don't evict the tenant who lives there.)

Two things you'll notice with a ClusterGen voice, both normal:

ClusterGen voices respond to every knob from Topic 1 — duration_stretch, int_f0_target_mean, int_f0_target_stddev, f0_shift all work, with the voice's own recorded character as the baseline (pitch comes out clamped to a sane 50–700 Hz, so you can't push a voice into ultrasound by accident). But this family brings two knobs of its own, and both are about the same trade: sound quality versus CPU time.

The robot is telling you the vocoder can't keep up in real time. Two remedies, in order:

Remedy 1 — mlsa_speed_param (an integer!). It tells the vocoder to ignore the N subtlest ingredients of the sound. Values 5–15 are reasonable; each step is a little less CPU and a slightly duller, more muffled voice. Set it right next to your other feature lines:

(Tuning from the app side like this leaves the voice files completely untouched — which keeps your next voice swap as clean as the first one.)

Remedy 2 — drop the vocoder's precision order. A deeper cut, worth about 25% of the vocoder's CPU time; flite's authors ship this exact trick on phones and say it "sounds basically the same". In init_vocoder, change the 5 to a 4:

Here is the volume knob Topic 1 promised. Every ClusterGen voice carries a gain value in its data tables — it's the last field of the big voice-description struct in the voice's ..._cg.c file. For our worked example:

Raise it for louder, lower it for quieter — small steps (1.5 → 1.8) go a long way, and too high will distort. This is the one tuning change that lives in a voice file rather than SpeechGen.cpp, because the struct is read-only at runtime. It's also the exception that proves the rule: it's a number swap, not a code change.

Your Butter Bot now speaks with whatever voice you gave it — and you've touched the deepest layer of the firmware the guides cover. Want an AI pair-programmer for your next voice experiment? Guide 3 — Claude Code and Guide 4 — OpenAI Codex show the workflow, and every section of this guide doubles as a spec you can paste straight into them.