How machines translate speech, text and voice

1. Scope

This study examines three systems that mediate between language representations:

• Machine Translation Machine Translation;
• Speech Recognition and Automated Transcription Speech Recognition and Automated Transcription;
• Text-to-Speech and Voice Synthesis Text-to-Speech and Voice Synthesis.

Together they form a language-interface triangle:

language A text → language B text
speech → text
text → speech

Their superficial similarity can be misleading. All three transform information, but they solve different inference and generation problems. Machine translation must preserve communicative function across languages that divide meaning differently. Speech recognition must infer symbolic language from continuous and uncontrolled acoustic evidence. Text-to-speech begins from known symbols and constructs an acoustic performance that never existed as one human utterance.

The shared historical shift is from machines transporting language to machines mediating its representation.

2. Central Finding

Recognition converts uncertain evidence into symbols. Translation converts one symbolic language into another. Big-picture essays converts symbols into a generated performance.

This creates three different trust questions:

1. Recognition: What was probably said?
2. Translation: What target expression probably preserves the source meaning and function?
3. Big-picture essays: What acoustic performance should embody the supplied symbols and voice conditions?

A speech-to-speech translation system chains all three:

source speech → recognised transcript → translated text → synthetic target speech

The final audio can sound coherent even when each stage adds uncertainty. Natural output at the end can conceal recognition omissions, translation errors and synthetic identity cues inherited from earlier stages.

3. The Representation Triangle

3.1 Speech is an event

Speech is produced by a body in time. It carries words, timing, accent, emotion, hesitation, overlap and environmental context.

3.2 Text is a symbolic representation

Text stabilises selected linguistic information while leaving out much of the performance. Punctuation, paragraphing and speaker labels may be editorial reconstructions rather than directly spoken facts.

3.3 Translation is a derived symbolic representation

A translated text is not the source text wearing another script. It is a new representation created through lexical, grammatical, cultural and pragmatic choices.

3.4 Synthetic speech is a generated event

TTS does not recover the original performance. It creates a new acoustic event conditioned on text, pronunciation, prosody, voice and model behaviour.

The triangle therefore contains two derived representations and one generated performance. Provenance must travel through the entire chain.

4. Shared Pipeline Model

A general machine-language mediation pipeline is:

capture or input → normalisation → representation → model inference or generation → decoding → structural enrichment → confidence or quality estimation → human review → delivery → downstream use

Different topics emphasise different stages.

| Stage | Machine translation | Speech recognition | Text-to-speech | |---|---|---|---| | Input | Source-language text | Acoustic waveform | Text or linguistic symbols | | Normalisation | Segmentation, tokenisation, formatting | Sampling, endpointing, audio conditioning | Number, date and abbreviation expansion | | Core problem | Cross-language semantic and functional transfer | Sequence inference from noisy continuous evidence | Acoustic performance generation | | Decoding | Select target token sequence | Select transcript token sequence | Generate acoustic frames or waveform | | Structural enrichment | Formatting and terminology | Punctuation, timing, speaker labels | Prosody, duration and voice style | | Output | Target-language text | Transcript or command | Synthetic speech audio | | Main audit object | Source text | Source audio | Source text plus voice and consent records |

5. The Systems Are Not Inverses

It is tempting to treat ASR and TTS as mirror images. They are not.

5.1 Recognition is underdetermined

Many linguistic sequences can fit imperfect sound. Noise, accents and coarticulation make the input ambiguous. A language model helps choose plausible text, which can replace unusual truth with common expectation.

5.2 Big-picture essays begins with specified content

TTS already knows the text. Its problems are pronunciation, alignment, prosody, identity and waveform quality. It can still omit or repeat content, but it is not discovering the message from the sound.

5.3 Translation changes linguistic systems

Translation is neither acoustic analysis nor acoustic generation. It must navigate languages that encode tense, gender, politeness, number and conceptual distinctions differently.

5.4 Consequence

The same metric family cannot evaluate all three. Recognition needs alignment and error analysis. Translation needs source fidelity and target appropriateness. Big-picture essays needs intelligibility, pronunciation, prosody, naturalness and identity governance.

6. Shared Distinctions Added to the Map

6.1 Input, model and output are separate evidence layers

The source remains distinct from the machine-generated derivative.

6.2 Plausibility is not fidelity

A probable sequence can sound or read naturally while contradicting the source.

6.3 Content accuracy is not identity authenticity

A synthetic voice may pronounce correct words in the likeness of a person who never spoke them.

6.4 Average error is not consequence-weighted error

One substitution in a filler word and one substitution in a medication dosage contribute similarly to simple edit metrics but not to human risk.

6.5 Confidence is not calibrated certainty

Model scores depend on training, domain and decoding. They require calibration and context.

6.6 Human review is a workflow stage, not a magical guarantee

Reviewers need source access, domain expertise, time and responsibility. “Human in the loop” is meaningless unless the loop is specified.

6.7 Language coverage is not language equality

A provider may list hundreds of languages while offering radically different quality, dialect coverage and support.

6.8 Accessibility and surveillance can share infrastructure

Automated captions and searchable calls use the same basic ability to convert speech into text. Purpose and governance determine whether the capability expands autonomy or monitoring.

6.9 Naturalness can reduce healthy scepticism

Smooth translation and realistic voices encourage recipients to stop checking provenance.

6.10 A cascade inherits every upstream uncertainty

Later stages cannot recover information omitted by earlier ones unless they retain alternate hypotheses or source evidence.

7. New Language-Mediation State Ladder

This study introduces a reusable state ladder:

1. Source produced
2. Source captured or submitted
3. Input normalised
4. Machine representation created
5. Candidate outputs generated
6. One output decoded or selected
7. Structure enriched
8. Confidence or quality estimated
9. Human reviewed
10. Published or delivered
11. Perceived by recipient
12. Interpreted
13. Trusted or challenged
14. Acted upon

These states prevent “the AI translated it,” “the transcript says” or “I heard their voice” from collapsing a long chain into one authoritative event.

8. Provenance Requirements

A machine-mediated language artifact should record, where stakes justify it:

• source object and identifier;
• source language, speaker or author;
• capture conditions;
• preprocessing and segmentation;
• model and version;
• prompt, glossary or constraints;
• decoding settings;
• confidence or quality estimates;
• human reviewer and edits;
• output language or voice;
• voice-consent and licensing status;
• timestamp and later transformations.

For speech-to-speech translation, the source audio should remain reachable. A final target voice without source lineage is persuasive but weak evidence.

9. Evaluation Model

9.1 Machine translation

Evaluate:

• adequacy;
• fluency;
• terminology;
• names and numbers;
• register;
• document consistency;
• domain fitness;
• human consequence.

9.2 Speech recognition

Evaluate:

• word or character errors;
• named entities;
• semantic severity;
• timestamps;
• speaker attribution;
• overlap;
• punctuation;
• performance by language and speaker group.

9.3 Text-to-speech

Evaluate:

• intelligibility;
• pronunciation;
• prosody;
• naturalness;
• long-form robustness;
• speaker similarity;
• consent and provenance;
• listener understanding.

9.4 Shared rule

No single scalar metric should stand in for communicative success.

BLEU, WER and mean opinion score are useful engineering summaries. None proves that the recipient received the right meaning from an authorised source.

10. Error Propagation

Consider a speech-to-speech translation call:

1. ASR hears a name as a common noun.
2. Translation chooses a plausible target phrase.
3. TTS pronounces it naturally in a trusted voice.
4. The listener accepts the result because it sounds confident.

The final layer increases persuasive quality without increasing source fidelity. This creates an important map principle:

Presentation quality can amplify upstream error.

Systems should therefore preserve uncertainty rather than laundering it through fluent generation. Possible mechanisms include source display, alternative hypotheses, uncertain-word highlighting, replay controls, confidence thresholds and mandatory review for high-risk terms.

11. Accessibility and Inclusion

This study contains some of the project's strongest accessibility contributions:

• translation expands language access;
• ASR produces captions and searchable speech;
• TTS makes text audible and supports non-speaking users.

Yet inclusion depends on representation. A language unsupported by MT, an accent mishandled by ASR or a name mispronounced by TTS can turn an “accessible” system into a new gatekeeper.

Accessibility must therefore include:

• language and dialect coverage;
• correction mechanisms;
• user control over voice and speed;
• source access;
• offline or privacy-preserving options;
• community participation in evaluation;
• transparent failure reporting.

12. Identity and Authentication

Speech historically carried a strong assumption of bodily presence. Voice synthesis weakens that assumption. A familiar voice can no longer serve as sufficient authentication.

ASR also separates speech content from speaker identity. Diarisation clusters voices, speaker recognition identifies them and transcription writes their words. These functions should not be bundled without consent.

The map now distinguishes:

• linguistic content authenticity: Were these the intended words?
• speaker authenticity: Did this person produce the utterance?
• translation authenticity: Does the target preserve the source?
• performance authenticity: Is this a real recording or generated voice?
• authorisation: Was the transformation permitted?
• provenance: Can the chain be inspected?

13. Institutional Power

The three systems concentrate power in data and model providers.

• Corpus owners determine which languages and voices can be learned.
• Benchmark designers determine which performance becomes visible.
• Model providers determine supported languages, voices and policies.
• Platforms determine when translation, captions or synthesis appear.
• Employers and states determine whether speech technology is assistive or surveillant.
• Language communities bear the consequences of standardisation and error.

A low-resource language can be absent from all three interfaces at once: poorly translated, poorly recognised and poorly synthesised. This produces a complete digital participation barrier.

14. Historical Arc

The batch spans almost a century.

• The 1930s vocoder and Voder show speech as decomposable and synthesizable.
• The 1949 Weaver memorandum frames translation as computational mediation.
• The 1950s digit recognisers show sound becoming machine-readable.
• Rule-based systems place knowledge in explicit linguistic structures.
• Statistical systems shift knowledge toward corpora and probabilities.
• Neural systems shift more structure into learned parameters.
• Large multilingual models combine recognition, translation and synthesis in increasingly general interfaces.

The historical movement is:

explicit decomposition → probabilistic modelling → learned representation → integrated multimodal mediation

15. Content and Visual Opportunities

Visuals

• Language-interface triangle.
• Speech-to-speech cascade with uncertainty accumulation.
• Three-column comparison of recognition, translation and synthesis.
• Provenance chain from source event to final generated voice.
• Accessibility-versus-surveillance dual-use map.
• Metric comparison: BLEU, WER and MOS.
• Identity-authenticity matrix.
• Language coverage heat map across MT, ASR and TTS.

Content

• Why Speech Recognition and Text-to-Speech Are Not Opposites
• The Machine Language Triangle
• Fluent, Audible and Wrong: How AI Launders Errors
• Hearing Is No Longer Believing
• The Languages Machines Leave Behind
• From the Voder to Real-Time Translation

16. Decisions Produced by This study

This study supports the following framework decisions:

1. separate source objects from machine derivatives;
2. classify recognition as probabilistic interpretation, not transcription truth;
3. classify translation as target-language reconstruction, not word substitution;
4. classify TTS as generated performance, not playback;
5. separate content fidelity from surface fluency or naturalness;
6. separate speaker similarity from identity authenticity and consent;
7. add consequence-weighted error analysis;
8. require provenance through cascaded language systems;
9. distinguish language coverage from language equality;
10. evaluate accessibility and surveillance as separate institutional uses of shared capability;
11. preserve source audio and source text wherever practical;
12. treat human review as a specified stage with accountable evidence.

17. Final perspective

This study begins Era VII by moving the map from machine-managed communication toward machine-mediated meaning. These systems do not merely route a message. They infer what was said, decide how meaning should cross a language boundary or create the voice through which words will be heard.

That delegation is powerful because language interfaces are bottlenecks. People cannot read every language, manually transcribe every recording or record every sentence that a system may need to speak. Automation breaks those bottlenecks and opens extraordinary access.

The same delegation introduces a new epistemic problem. Machine outputs arrive in forms humans naturally trust: fluent prose, clean transcripts and convincing voices. The better the surface becomes, the easier it is to forget the invisible chain of models, data, decoding and institutional policy beneath it.

The central lesson is therefore simple:

A machine-generated language artifact must be judged by its relationship to the source, not by how confidently it presents itself.

Machine translation, speech recognition and voice synthesis are best understood as transformation layers with explicit provenance, uncertainty and consequence. Used that way, they expand human access. Treated as automatic truth, they become machines for producing certainty faster than understanding.