Coding Guide 5 — Program your Butter Bot: four worked examples

Guides 2–4 taught you to build, flash, and tweak a joke. This guide goes further: by the end of it, you'll have taught your Butter Bot a whole new trick, end to end. Say "do you see a bottle" — the robot wakes its camera, actually looks, and answers out loud depending on what it saw. Then you'll trigger the same trick from the Butter Bot Controller.

One running project — we call it Bottle Detective — carries you through four skills, each building on the last:

Prefer to let an AI do the typing? Guide 3 (Claude Code) and Guide 4 (Codex) show how. This guide is written to double as a spec: every code block names its file, so you can paste a whole section straight into Claude Code or Codex and say "do this".

Before touching any code, you need one piece of architecture that the earlier guides could skip: the Butter Bot is really two devices and three codebases.

When you edit anything under components/ButterBot-Common, you are editing one copy of a file that exists in two places on your disk — once inside each firmware project. If Topic 4 is on your menu, make the identical edit in both checkouts (or commit in one and check out that commit in the other), and reflash both devices. If you only ever touch the robot, the single copy inside ButterBot-Firmware is all that matters.
​

Three enums — Phrase, BB::Action::Scenario, and Ctrl::Command — are sent between the devices as raw numbers. Their values come from their position in the list. Always add new entries at the end (right before COUNT where there is one). Inserting in the middle silently renumbers everything after it, and a robot and controller flashed from different versions will start speaking past each other in the strangest ways.

Guide 2 showed you how to slip a new joke into JokePhrases. Now let's look at what that file actually does — and create two brand-new phrase groups our Bottle Detective will speak later.

Every distinct thing the robot can say — a joke, a fact, "battery low", "shutting down" — has one entry in enum class Phrase in Phrases.h. There are about 150 of them, and the file marks exactly where yours go:

New entries go right where the // Add phrases here comment says — after the last real entry, before COUNT. That's the append-only rule from the repo map in action: the enum value is what the robot sends to the controller over Bluetooth, so existing entries must never shift.

One Phrase entry maps to an array of variants in Phrases.cpp, so the robot doesn't repeat itself. Each variant is a PhraseOutput:

Those four fields are the whole secret to how the robot's personality is tuned:

The stock firmware uses that last trick in earnest. Look at this real entry from JokePhrases — areaa is deliberate, because the speech engine rushed the correctly spelled word:

Template phrases: a few arrays contain printf-style placeholders — WhatsThisOnePhrases has lines like "It might be a %s", and the routine fills in the object name at runtime. If you add variants to one of those arrays, every variant must keep the same number of %s placeholders as its neighbors, or the robot will read garbage.

The arrays live inside a wrapper class called PhraseArrays at the top of Phrases.cpp (a C++ visibility trick — the comment in the file explains it). The link between an enum entry and its array is a single line in PhraseArrays::buildMappings(), further down the same file:

Forget this line and the phrase compiles fine but stays silent — Phrases::get() finds an empty array and returns -1.

Every routine that talks uses the same four-step pattern. Here it is, verbatim, from the joke routine — this is the most important snippet in the whole guide, and we'll reuse it shortly:

Time to write. Our routine will need two phrases: one for "I see a bottle", one for "I don't". First the enum — insert both entries at the marker:

Then the variant arrays. Put these anywhere inside class PhraseArrays in Phrases.cpp, next to the existing ones. Note how the last BottleFound variant shows off both tuning knobs at once — a 10% pick chance and the rare flag — and how one line uses shown to fix a spelling written for the speech engine's benefit:

And finally the two registration lines, added inside buildMappings() in the same file:

Run idf.py build (Guide 2, Step 6) to confirm everything compiles. The robot can't say these lines yet — nothing asks for them. That's Topic 2's job.

This is the section Guides 3 and 4 told you was "an advanced project". It is — but it's four small, understandable steps: write a routine class, give it a scenario number, map the number to the class, and register a spoken phrase that triggers it.

A routine is a plain C++ class deriving from Routine (main/src/Routines/Routine.h). The state machine creates it, calls tick(deltaTime) repeatedly, and destroys it when it reports it's finished. Your tick() returns one of:

Two mechanics are worth knowing before you write one:

There are also specialized base classes you'll meet when reading the stock code: EventRoutine (carries an event payload), PhraseEventRoutine (an event routine that just says a phrase), and PhraseListeningRoutine (lets a routine listen for speech mid-flight — DiceRollRoutine is the showcase). Plain Routine is all we need today.

Create two new files. The header is nearly nothing:

The first version of the implementation doesn't look for bottles yet — it just speaks the "no bottle" phrase from Topic 1, using the exact pattern from JokeRoutine. Camera comes in Topic 3.

No build-system edits needed — but one command is. The firmware's CMake finds everything under main/src/ by scanning the folder, and that scan is cached. After creating a brand-new .cpp, run idf.py reconfigure once so the build system notices it — otherwise the build ends with undefined reference errors at the link step. (Editing existing files never needs this.)

Everything the robot can be asked to do — by voice or by controller — is identified by a value of BB::Action::Scenario. Append ours at the end of the enum in BBData.h:

Then map the number to your class in ScenarioRoutineMappings.cpp. Add the include at the top and one row to the Mappings array — the existing rows show the pattern:

Skip this row and the robot will hear you, look up the scenario, find nothing, and log No routine mapped for scenario ... before shrugging back to idle. (The second element of the key, {}, is a ScenarioData payload — that's how one scenario like DiceRoll maps to different routines for a D6 versus a D20. Plain routines leave it empty.)

Now the fun part. Every phrase the robot understands lives in one array — activations in Scenarios.h, about 240 rows. Here's its shape, with real rows:

The mechanics behind it: when the robot starts listening, ListenState feeds this whole array into the speech-recognition engine (Espressif's MultiNet). Adding a row is genuinely all it takes — the list is rebuilt from activations every time listening starts.

The phoneme strings look like line noise (TfL Mm c qbK) because they use a compact one-letter-per-sound alphabet internal to MultiNet. The generator script ships inside the firmware project, at managed_components/espressif__esp-sr/tool/multinet_g2p.py — note it's tool, singular, and that managed_components/ only exists after your first build, because ESP-IDF downloads it then.

The script needs two Python packages (one-time setup):

If pip refuses with an externally-managed-environment error (common with recent Python on Linux and macOS), install into a virtual environment instead: python -m venv g2p-venv, then use g2p-venv/bin/pip and g2p-venv/bin/python (on Windows: g2p-venv\Scripts\pip and ...\python) for the commands in this section.

Now generate — run this from the ButterBot-Firmware folder. Commas separate synonyms of the same command; semicolons would separate different commands:

Copy each phrase's phonemes from the out: line (drop the trailing semicolon). Keep the spaces — the fuzzy matcher works word by word. To sanity-check your setup, run it once on "tell me a joke": the output must be exactly TfL Mm c qbK, the same string you saw in Scenarios.h.

If the script stops with LookupError: Resource 'averaged_perceptron_tagger_eng' not found, the phonetic library needs a one-time data download. Run this, then retry:

python -c "import nltk; nltk.download('averaged_perceptron_tagger_eng')"

With real phonemes in hand, add the activation rows anywhere in the activations array. The fuzzy core is the phoneme spelling of "bottle" — BnTcL — which you can read straight off the end of either output:

Two matchers stand between your voice and your routine. First, MultiNet itself tries for an exact command hit. If it times out without one, the firmware runs its own fuzzy phoneme matcher over what it heard — an edit-distance comparison in phoneme space, where the fuzzyCore word counts double and short filler words count half. A candidate wins if its score clears the phrase's fuzzyThreshold without being ambiguous against a different command (synonyms of the same scenario don't compete with each other).

Tuning, in practice:

See what the robot hears: the stock firmware silences the recognizer's logs. In main/main.cpp, inside begin(), find esp_log_level_set("AudioFrontend", ESP_LOG_NONE); and comment it out, rebuild, and run idf.py monitor. You'll see the recognized transcript, candidate scores, and why the fuzzy matcher accepted or rejected each attempt — invaluable while tuning.

The robot should answer with one of your BottleMissing lines — it always says "no bottle" for now, because version 1 of the routine never looks. Notice it will not repeat a line until it has used all three. That's Phrases::get() doing its no-repeat bookkeeping.

Time to make the Bottle Detective earn its name. The robot has a camera and an on-board neural network; in this topic the routine starts using them.

It's a whole-frame classifier, not a bounding-box detector: it looks at the entire camera image (a 120×120 crop of a 128×128 frame) and answers "what object is this most likely to be". It knows exactly ten things:

You can build anything you like using these ten classes, but you can't add an eleventh by editing the enum — the list mirrors the output layer of a trained neural-network model (models/objdet.tflite, flashed to its own partition). Teaching it new objects means retraining that model, which is a much bigger adventure than this guide.

The detection service is reached through ServiceLocator::ObjDetInstance, and every consumer in the stock firmware uses the same five-call lifecycle. Here it is in WhatsThisRoutine (the routine behind "what is this?"):

Mirror every loadModel() with unloadModel() and every init() with deinit() before your routine returns Done — on every code path. That megabyte of arena and the reconfigured camera don't free themselves, and the next routine that needs memory will find the robot mysteriously out of it.

Bonus reading: ObserveRoutine (main/src/Routines/ObserveRoutine.cpp) is the ambient cousin — it's in IdleState::RandomRoutines, so the robot occasionally looks around unprompted and comments on what it sees. ObjDet::ClassNames maps each ObjClass to a printable name if you need one.

Replace BottleCheckRoutine.cpp with the final version. The diff from version 1: two new includes, the ObjDet guard, the five-line lifecycle, and a one-line verdict that picks which phrase to speak.

Build, flash, and test properly this time:

What about the controller's screen? Our routine speaks but doesn't send anything to the controller, and that's perfectly fine — the controller simply stays on its home screen. If you want a custom screen for your routine one day: define a data struct in BBData.h, send it with com->sendData(...) like JokeRoutine does, and build a window for it in the controller firmware under main/src/Components/HomeWindows/. The next topic gets you into that codebase.

The Butter Bot Controller is a computer of its own, with its own firmware to hack. In this topic: how a button press becomes a robot action, and three graded examples — remap a button, put Bottle Detective in the controller's action list, and mint a brand-new Bluetooth command.

The controller has five physical buttons — ManualOverride, Poke, ShutUp, Summon, and the joystick's own press (main/src/Enums.hpp). The framework delivers exactly two events per button, Press and Release — anything fancier, like "hold for two seconds", is hand-rolled with timestamps, and you'll see how in a moment.

The journey of a press: the input service polls the pins and broadcasts a button event → HomeScreen::handleButtonEvent() decides what it means → a Com::send...() call writes a message over Bluetooth (the controller is the BLE client, the robot the server) → the robot's Com::tick() (main/src/Services/Com.cpp) parses it and re-broadcasts it as an event → whichever robot-side object bound that event reacts. The message itself is beautifully simple — a 4-byte command code, optionally followed by a raw payload struct:

Who reacts to what, on the robot:

All the button decisions live in HomeScreen::handleButtonEvent(): press-time actions in its if(action == ButtonInput::Action::Press) block, release-time actions after it, and hold detection in HomeScreen::loop() (press stores a millis() timestamp, loop() watches the clock — steal the Summon pattern if you ever add a hold). To make the Poke button ask for a joke instead of a poke:

Whenever you change what a button does, also update the controller's idle-screen cheat sheet so it doesn't lie to you:

Same rule as the robot firmware: CMake's source scan is cached, so editing existing files is fine, but after adding a new .cpp to either project, run idf.py reconfigure once — otherwise you'll get undefined reference errors at link time.

Pressing the joystick opens the controller's scrollable list of everything the robot can do. That list is generated from one array — ScenarioNameMap — and its declaration order is the display order. One new row puts our Topic-2 scenario on the menu:

For that line to compile, this checkout's copy of ButterBot-Common must contain your BottleCheck enum entry from Topic 2 — this is the two-checkouts warning from the repo map becoming real. Mirror the BBData.h edit (and the Phrases.h/Phrases.cpp ones while you're at it), build, and flash the controller. Now joystick-press → Bottle Check → the robot goes looking, no voice needed. The selection travels as Ctrl::Scenario with the scenario in its payload, lands in the same mapping table your voice command uses, and runs the same routine.

The Scenario command covers most "make the robot do X" ideas, but sometimes you want a first-class command — something the robot should react to even outside of scenarios. Three edits, three files:

1. Append it to the shared vocabulary (append-only — this enum crosses the wire raw):

2. Send it from the controller — a payload-less command needs no new plumbing, sendCommand() already handles any Ctrl::Command. Wire it to whatever trigger you like inside handleButtonEvent(), for example the Poke button again:

3. Handle it on the robot. Payload-less commands need zero changes in the robot's Com service — everything it receives is re-broadcast through its OnCommand event. You just need a listener. IdleState::onCommand (main/src/States/IdleState.cpp) already listens and is the natural home; hand the routine to the scenario system exactly the way ListenState does after a voice match:

(Add #include "Routines/BottleCheckRoutine.h" at the top of IdleState.cpp.) If your command carried a payload instead, you'd follow the Drive pattern in the robot's Com::tick() — a length-checked memcpy and a dedicated event — but start payload-less; it's nine-tenths of the use cases.

Any edit under components/ButterBot-Common — Examples B and C both make one — means rebuilding and reflashing both the robot and the controller before testing. A pair flashed from mismatched copies of CtrlData.h or BBData.h is the classic source of "the button does the wrong thing" mysteries.

Every recipe from this guide, in one table. Paths are relative to the repo named in the last column.

You now know every moving part: phrases, routines, voice commands, vision, and the controller link. Want an AI pair-programmer to do the typing while you do the ideas? Guide 3 — Claude Code and Guide 4 — OpenAI Codex show exactly that — and every section of this guide is written to work as a spec you can paste straight into them.

Ready for the deep end? Guide 6 — Give Your Butter Bot a New Voice is our advanced guide: tune the robot's pitch and speaking speed, or teach it an entirely new voice.