Most of the conversation about AI safety happens a long way from the moment that actually matters. In our product, that moment is small and specific. A six-year-old reads a word out loud, gets it wrong, and waits to find out what the app thinks of her.
That is where our AI lives. On a child’s voice, at the point in the day when she is most exposed. This month we published Version 1.0 of our Safe AI for Children policy, which sets out what our systems do, what they will never do, and who is accountable when they get something wrong.
What We Committed To
The policy starts by naming three stages an organization can occupy. Awareness means recognizing that children might use your system. Mitigation means adding safeguards after the fact, so filters, age gates and warnings. Child centricity means children were considered from the moment the thing was designed.
We placed ourselves in the third stage, and we published the policy so that claim can be checked rather than simply asserted. Bookbot has no adult product. There is no general-purpose mode that a child wandered into by accident.
Some of the commitments are about where a child’s data goes. Speech recognition, the technology that listens as a child reads aloud, runs on the device itself, without needing a network connection. Audio is processed and then discarded by default, and nothing is kept. Recording for model improvement is a separate, explicit choice made by a parent or school, never bundled into general terms of use and never switched on by default.
Others are red lines. These are things we will not build, whatever the roadmap looks like later.
We do not synthesize or clone a child’s voice. We do not infer emotion, attention or mental health from a child’s voice or face. We do not sell child data or use it for advertising or profiling. And we do not build features designed to create emotional attachment to the app or to a synthetic character.
Why It Matters
The commitment I expect to be hardest to hold is the one about accents. Speech recognition trained mostly on adult, standard-accent English performs measurably worse on children, on second-language speakers, and on regional and non-standard accents. A system that under-recognizes one child’s English is not a neutral tool. It will tell that child she is wrong more often than it tells another child he is wrong, and it will do it for reasons that have nothing to do with her reading.
So we measure accuracy separately by accent, language variety, age band and device class. Aggregate numbers hide exactly the gaps we are looking for. Where a group is served worse and we have not closed the gap yet, the policy commits us to saying so rather than burying it.
There is a related design rule I am fond of. When the recognizer is not confident about what it heard, it stays quiet instead of guessing. The asymmetry is deliberate. A missed error costs a child one unreinforced word. A false accusation, meaning being told you are wrong when you were actually right, costs confidence, and confidence is the scarce resource in early reading.
We also decided not to use a model for the decision that shapes a child’s experience most, which is choosing the book she reads next. That runs on explicit rules tied to a published scope and sequence, the set order in which phonics patterns are taught. A teacher can be told exactly why a book was suggested, in terms of the patterns the child has and has not mastered. There is no confidence score to interpret and no black box to trust.
A system that learned to recommend from historical progression data would inherit whatever inequities sit in that data. A rule cannot.
What Happens Next
Accountability sits with a named person rather than a committee. As CEO I am accountable for AI safety and data protection at Bookbot, which means there is somebody specific to come to when we get this wrong.
Before any new AI feature reaches children, we complete a child AI impact assessment. It asks seven questions, including which children will encounter this and at what ages, what a child experiences when it fails, and whether it creates dependency or builds independence. The last question asks who is accountable and how we would know if the feature were causing harm. Anything that cannot answer that one does not ship.
The policy is reviewed at least annually, and also before we deploy into a new country or language, before we release any new learner-facing AI capability, and after any incident. If you are a parent, a teacher, a researcher or a partner and you think we have something wrong, you can raise it at [email protected]. We have committed to acknowledging within five working days and to responding substantively, including where the answer is that we made a mistake.
Writing all of this down makes us easier to hold to account. That is the point of publishing it.
Reference