What actually makes a generated voice sound emotional?

A controlled experiment on the MOSS voice-acting model · 1440 clips · every combination of four factors, generated once each

The question, in plain language

A text-to-speech model that can act is given more than words. It is given a target voice, an instruction about how to perform the line, and a written text whose punctuation already suggests a tone. When a clip comes out sounding genuinely furious or genuinely delighted, it is usually not clear which of those things did the work — people change all of them at once and keep whatever sounded best.

This page separates them. Four things were varied independently, and everything else was held fixed:

Six emotions, five short utterances each, written in English and German as translations of one another, every utterance in both punctuation forms, both direction settings and six voice conditions. That is 6 × 5 × 2 × 2 × 2 × 6 = 1440 clips, each generated once. Because the grid is complete, the average score at one level of a factor averages over exactly the same mix of everything else, so a difference between two levels cannot be an accident of what else happened to be in that group.

What a direction looks like

This is one script in both arms. The square brackets are timings the model must hit — a length for each sentence and for each silence. The round brackets with a number are vocal bursts. The round brackets without a number are the delivery direction, and they are the only thing that differs between the two:

The wording of a direction is not invented for this study. The intensity adverb comes from the requested band, the emotion name from the label being asked for, and the phrase describing the stance is one of ten harvested from the training corpus by grouping the directions that real training rows already carried. The manner words at the end come from the project's own fixed vocabulary of delivery descriptors. One consequence is visible above: the corpus collapses forty emotions onto ten stance families, so the anger anchor is worded in terms of contempt. That is what the model was trained on, so it is what it is given here.

What the numbers are

• Emotion percentile — a classifier scores the clip on the emotion that was asked for, and the score is turned into a percentile against the training corpus, rescaled per emotion so that "nothing detected" is 0. Higher is more of that emotion. This is the number the question is about.
• Emotion rank in study — a companion to the percentile, added because the percentile hits its ceiling here. With the emotion adapter on and texts written to suit their emotion, most clips land above the 0.95th corpus percentile, and a number pinned near its maximum cannot show a difference that the underlying score does show. This is the rank of the clip's raw classifier score among all clips of the same emotion on this page, scaled to 0–1. It cannot saturate. It says nothing about how these clips compare with the corpus — only how they compare with each other.
• Genuineness — a separate model rating how much the delivery sounds felt rather than read, as a corpus percentile.
• Burst blend — how naturally non-speech sounds (a sigh, a gasp, a giggle) sit inside the speech, as a corpus percentile. Burst realisation is the fraction of the bursts the script asked for that a detector actually found.
• Word error rate — the clip is transcribed and compared with the script's own words. Lower is better; this is the intelligibility cost of pushing on the other dials.
• Duration error — the script states how long every sentence, silence and burst should take, and those numbers are made to sum exactly to the length budget. This is how far the finished clip lands from that budget, in seconds.

The main-effects table

Each block takes one factor and averages every clip at each of its levels over everything else. The difference row is stronger than subtracting the two averages: it pairs each clip with the one clip that differs only in that factor — same emotion, same utterance, same language, same everything else — and averages the difference inside the pair, with a 95% bootstrap confidence interval. An interval that does not cross zero is marked in colour; n.s. means it does.

What that says

None of the three factors we set out to separate moves the emotion score. The delivery direction — the factor everyone assumes is doing the work — is flat: its confidence interval straddles zero on both emotion measures. Punctuation is flat on emotion too. Supplying a voice profile actually lowers the emotion percentile slightly. The one factor that moves it a long way is the one nobody was asking about: the same utterance scores far higher in English than in German.

The honest reading of the direction result is not “directions do nothing”. It is that they do nothing here, on top of an emotion adapter that is already running at its measured optimum. The adapter is on in all 1440 clips, and the emotion percentile averages 0.84 before any of the three factors is applied — the clips are already at the top of the corpus distribution, and there is very little room left for an instruction to add anything. Two explanations fit that equally well: the direction is genuinely redundant once the adapter is loaded, or the measurement is against its ceiling. The control below separates them.

The factors do move the other numbers, and those effects are real: expressive punctuation improves how naturally vocal bursts sit in the speech, and supplying a voice profile improves both that and genuineness substantially, at a small cost in emotion percentile. Whatever a voice profile is doing, it is making the delivery sound more like a real person and slightly less like a strong emotion.

The control: what happens with the emotion adapter switched off

The result above has an obvious hole in it. The emotion adapter is loaded in every clip of the main grid, so a flat direction effect could mean the direction is useless, or it could mean the adapter had already pushed the score as high as it goes and left the direction nothing to do. Those are very different conclusions.

So the whole no-reference block was generated a second time — same 240 prompts, same seeds, same sampling — with the emotion adapter's weight set to zero and everything else identical. If directions matter at all, this is where it should show.

But the control says something larger than that, and it is the most surprising number on this page. Switching the emotion adapter off — the component the whole pipeline is built around — moves the emotion percentile only from 0.864 to 0.821, a drop of 0.043. With no adapter, no direction and flat punctuation, these clips still score near the top of the corpus distribution.

So the ceiling was never mainly the adapter's doing, and it is not the direction's either. The most likely remaining explanation is the one thing every condition shares and nothing in the grid varies: the words. The utterances were written to suit their emotions — someone saying “get out of my house right now” reads as angry before any acting is applied — and an emotion classifier listening to a clip is also, unavoidably, hearing what was said. On text chosen this way, the content sets the score and the delivery controls move it very little.

That is a limitation of this design as much as a finding, and it points at the experiment worth running next: the identical factorial grid on emotionally neutral sentences, where the classifier has nothing but the delivery to go on. That is the condition under which a delivery direction has something to do, and it is not the condition tested here.

Per emotion

The same six averages, and then the paired effect of each factor on the emotion percentile, computed separately inside each emotion.

Per language

German and English texts are translations of the same utterance and carry the same timing plan, the same bursts and the same marks, so the only thing that changes is the words.

Per voice condition

The unconditional row is the model given no reference and no voice adapter. The five named rows each had that voice's own recording in the reference slot and that voice's own adapter loaded.

The five reference voices

The clips

Grouped so that the same utterance can be heard across every condition. Open an emotion, then an utterance; inside, the clips are ordered so the factor levels sit next to each other. Under every clip is what an automatic transcriber heard, and the script exactly as the model received it — the square brackets are durations and silences, the round brackets with a number are vocal bursts, and the round brackets without a number are the delivery direction.