Audio samples from "Direct speech-to-speech translation with a sequence-to-sequence model" (Translatotron)

Paper: arXiv

Authors: Ye Jia *, Ron J. Weiss *, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, Yonghui Wu.

Abstract: We present an attention-based sequence-to-sequence neural network which can directly translate speech from one language into speech in another language, without relying on an intermediate text representation. The network is trained end-to-end, learning to map speech spectrograms into target spectrograms in another language, corresponding to the translated content (in a different canonical voice). We further demonstrate the ability to synthesize translated speech using the voice of the source speaker. We conduct experiments on two Spanish-to-English speech translation datasets, and find that the proposed model slightly underperforms a baseline cascade of a direct speech-to-text translation model and a text-to-speech synthesis model, demonstrating the feasibility of the approach on this very challenging task.

Contents

• Fisher Spanish-to-English
• Conversational Spanish-to-English

Fisher Spanish-to-English

These audio samples correspond to Section 3.2 in the paper. Source speech from Spanish telephone conversations is translated into English speech in a canonical voice.

Transcripts for ground truth samples come from the original data; while the transcripts for predictions are transcribed by an ASR model for evaluation (see the beginning of Section 3 in the paper).

	Ground truth		Predictions
	Source (Spanish)	Target (English)	Cascade (ST + TTS)	S2ST (canonical voice)

Conversational Spanish-to-English

These audio samples correspond to Section 3.1 and Section 3.4 in the paper. The canonical voice predictions correspond to Section 3.1, and the source voice and target voice predictions correspond to Section 3.4.

The transcripts for ground truth samples come from the original data; while the transcripts for predictions are transcribed by an ASR model used for evaluation (see the beginning of Section 3 in the paper).

The samples were cherrypicked in order to inlcude both good cases as well as typical bad cases.

	Ground truth		Predictions		Predictions with voice transfer
	Source (Spanish)	Target (English)	Cascade (ST + TTS)	S2ST (canonical voice)	S2ST (source voice)	S2ST (target voice)