SpeechPainter receives as input an utterance containing a gap of at most one second and the transcript. It must learn to identify the part of the transcript corresponding to the utterance (dark gray) and the gap (bold), and fill in the gap with the correct content, while maintaining speaker identity, prosody and recording environment conditions. The following samples for unseen speakers are randomly selected from LibriTTS (test-clean and test-other splits) and VCTK.
For the inpainted gap, the model can maintain:
| Input Sample
(transcript, gap start, gap length)
| SpeechPainter
|
---|---|---|
speaker id and prosody | |George Robertson desc|ribed the plan |as an outrageous and disg|raceful decision. 0.89, 0.84 | |
speaker id and prosody | |I am very sorry that Alastair |Campbell has taken |this decision.| 1.40, 0.84 | |
background noise | His recovery was destined to be almost as sudden |as his disappearance, and was due| directly |to the tra|mp Alex had brought to Sunnyside. 1.83, 0.75 | |
background noise | The co|mparative adva|ntages of complian|ce and non-compliance are as follows:| 0.39, 0.83 | |
bandpass filtering due to the microphone | But, as I suppose your Excellency is joking, I will add no more. |He replied that, f|ar from| joking,| he meant solemn earnest. 1.83, 0.75 | |
bandpass filtering due to the microphone | "You should go to bed," she replied,| with that ironical air |which went| so well| with her delicate and witty face. 1.70, 0.95 | |
reverberation | He went |to the window| and sat down,| scanning the groups,| and listening to what was being said around him. 0.65, 0.95 | |
reverberation | |We shall say no more, but I tru|st you understand| the responsibility| you have? 1.40, 0.84 |
The samples were collected by the authors.
Use-case
| Input Sample
| SpeechPainter
|
---|---|---|
Grammar correction | Tony and me went to the store. | Tony and I went to the store. |
Grammar correction | We swimmed in the river last weekend, and the water was cold. | We swam in the river last weekend, and the water was cold. |
Grammar correction | Yesterday, I gone to the store and I bought some milk. | Yesterday, I went to the store and I bought some milk. |
Pronunciation correction | Yesterday I bought a very nice watch. | |
Pronunciation correction | I would like the steak cooked medium-rare. | |
Pronunciation correction | Then, I suddenly shifted into third gear on the highway. | |
Background noise removal | Now, suddenly, we have this new landscape. | |
Background noise removal | The quick brown fox jumped over the lazy dog. | |
Packet loss correction | SpeechPainter can also be used for correcting packet losses. |
SpeechPainter is NOT robust to all:
| Input Sample
(transcript, gap start, gap length)
| Target
| SpeechPainter
|
---|---|---|---|
speakers | The captain behaved perfectly well in this critical instant, commanding a |dead sile|nce, and the closes|t attention to his orders.| 0.65, 0.95 | ||
background noises | Mr. |Bell said |they absolutely li|ved upon water-porridge| for years--how, he did not know; but long after the creditors had given up hope of any payment of old Mr. Thornton's debts (if, indeed, they ever had hoped at all about it, after his suicide,) this young man returned to Milton, and went quietly round to each creditor, paying him the first instalment of the money owing to him. 0.76, 1.00 | ||
content | |The project has already secured the sup|port of Sir Sean| Connery.| 1.72, 0.95 | ||
speech tempo | |Hopefully, he will be |back| in contention next week.| 1.00, 1.00 |
Sample ID
| Input Sample
(transcript, gap start, gap length)
| Target Sample
| SpeechPainter
|
---|---|---|---|
libritts_test_clean 7021_85628_000019_000000 | But just at this instant the Princ|ess came trip|ping across the y|ard. She was dresse|d in white silk with bows of ribbon. 0.62, 0.94 | ||
libritts_test_clean 2300_131720_000004_000012 | Bur|bank and his tribe represe|nt in the vegetable| worl|d, Edison in the mechanical. 1.58, 0.95 | ||
libritts_test_clean 1580_141083_000012_000000 | "The first page on the |floor, |the second in the win|dow, the third where you left |it," said he. 0.64, 1.00 | ||
libritts_test_clean 5639_40744_000020_000002 | The boy having embraced his mother, calling her his cousin, |and his grandmother,| calling her his| benefactress,| repeated his grandfather's question. 1.43, 0.82 | ||
libritts_test_clean 5142_36586_000010_000000 | |EFFECTS OF |THE INCREASED USE |AND DISUSE OF PARTS.| 0.47, 1.00 | ||
libritts_test_other 4198_61336_000001_000002 | The national exchequer had |been exhausted by the l|oss of tri|bute from revolt|ing provinces, trade was paralysed, and the industries were in a languishing condition. 1.40, 0.93 | ||
libritts_test_other 533_1066_000015_000000 | The doctor was puffing |somewhat |when we| finally came to a halt|. 0.64, 0.85 | ||
libritts_test_other 3080_5040_000000_000006 | Would it would leave me, and th|en I could believe I sh|all not always have| occasio|n for it. 1.72, 0.95 | ||
libritts_test_other 3005_163390_000002_000000 | WELL, all day him and the king was hard at it, |ri|gging up a sta|ge and a curtain and a row of ca|ndles for footlights; and that night the house was jam full of men in no time. 0.47, 1.00 | ||
libritts_test_other 4294_35475_000021_000001 | But remember that thou canst not keep them sharp |and shining, unless they are u|sed at least on|ce each| day in some unselfish service." 1.58, 0.95 | ||
vctk p277_341 | |A spokeswoman for Edin|burgh City Counc|il confirmed it|s support for the company. 0.99, 0.75 | ||
vctk p272_008 | These take the shape of a |long round a|rch, with its |path high above, and its two| ends apparently beyond the horizon. 0.47, 0.78 | ||
vctk p268_028 | |Wages and salaries account fo|r a major prop|ortion of our |expenses. 1.40, 0.93 | ||
vctk p245_081 | It just |shows the arrogance |of Labour| and the Liberal Dem|ocrats. 1.40, 0.84 | ||
vctk p265_180 | Any |change would b|e subject| to the Scottish Parliament's |approval. 0.64, 0.85 |
Sample ID
| Input Sample
(transcript, gap start, gap length)
| Target Sample
| TTS
| SpeechPainter
|
---|---|---|---|---|
libritts_test_clean 1580_141083_000010_000000 | "The mom|ent I looked at my table, I was aware| that someone had rumm|aged among my pa|pers. 1.58, 0.83 | |||
libritts_test_clean 7127_75946_000040_000001 | Suddenly, for the purpose of restoring peace and ord|er, Spring,| accompanied by| his whole court,| made his appearance. 1.28, 0.85 | |||
libritts_test_clean 5683_32866_000012_000001 | But don't these |very wi|se things| sometimes turn out very fool|ishly? 0.38, 0.78 | |||
libritts_test_clean 4970_29095_000006_000000 | |I wanted to talk with thee a |little about thy| plans.| 1.44, 0.93 | |||
libritts_test_clean 7176_88083_000015_000000 | The hawk sat u|pon the branch and watched his qua|rry swimming| beneath the surfa|ce. 1.55, 0.76 | |||
libritts_test_other 5484_24318_000068_000001 | He |was appearing before his c|ompanions| only to give truth| its just due. 1.05, 0.89 | |||
libritts_test_other 5764_299665_000104_000000 | The third stone is that matter and force cannot exist ap|art--no matter |without f|orce--n|o force without matter. 1.55, 0.76 | |||
libritts_test_other 4294_35475_000036_000003 | My friend, |the Fly, sent me |to guide you| to a place of safety.| 0.98, 0.90 | |||
libritts_test_other 7902_96592_000061_000002 | |Why, your clothes don't fit you, |and your cap's put on |all skew-rew.| 1.36, 0.92 | |||
libritts_test_other 6128_63240_000013_000002 | He threw it down at the approach of Mrs. Luna, laughed, shook hands with her, and said in answer to her last re|mark, "You |imply that |you do tell fi|bs. 1.36, 0.78 | |||
vctk p271_344 | A form|al announcement is expecte|d this morning at| a news conference.| 1.31, 0.98 | |||
vctk p306_234 | |Now, suddenly, w|e have this |new landsc|ape. 1.68, 0.85 | |||
vctk p286_119 | |Our offer represents an attra|ctive price for |the busine|ss. 1.76, 0.96 | |||
vctk p228_032 | |We must provide |a long-term so|lution to tackle this atti|tude. 0.84, 0.83 | |||
vctk p311_004 | We also |need a small plastic sna|ke and a big| toy frog for the ki|ds. 1.24, 0.87 |
Sample ID
| Input Sample
(transcript, gap start, gap length)
| Target Sample
| SpeechPainter w/o Adversarial Training
| SpeechPainter
|
---|---|---|---|---|
libritts_test_clean 8463_294825_000022_000002 | Yet this loose structure gi|ves the n|ovel |an air of documentary realism.| 0.28, 0.91 | |||
libritts_test_clean 5639_40744_000000_000004 | But Rodolfo had been struck by the great beauty of Leocadia, the hidalgo's daughter, |and present|ly he began to entertain the idea| of enjoying it at all hazards. 0.38, 0.90 | |||
libritts_test_clean 2300_131720_000016_000013 | And|rews skipped from under; |he obeyed orders;| I did| not. 1.36, 0.96 | |||
libritts_test_clean 4970_29095_000008_000000 | "I know," said Margaret Bolton, with a half anxious smile,| "thee chafe|s against all the |ways of Friend|s, but what will thee do? 1.08, 0.96 | |||
libritts_test_clean 7176_88083_000023_000002 | At the same mo|ment he felt the light |restraint of the al|most invisible |leader upon his wings, where the other two flies had affixed themselves. 1.11, 0.99 | |||
libritts_test_clean 7021_79740_000007_000000 | Now Delia |contrived to |obtain a great |influence and ascende|ncy over the minds of the children by means of these dolls. 0.66, 0.95 | |||
libritts_test_clean 8555_284447_000035_000001 | Then she |seated her|self in an out-of-|the-way place and quietly wai|ted. 0.43, 0.88 | |||
libritts_test_clean 8230_279154_000013_000001 | There may be a specific feeling which could be called the f|eeling of "pa|stness,"| especially where immediate mem|ory is concerned. 0.54, 0.92 | |||
libritts_test_clean 7729_102255_000010_000014 | |They were not only well armed |and supplied,| but wrought| up to the highest pitch of partisan excitement. 1.31, 0.93 | |||
libritts_test_clean 237_134500_000003_000002 | She |hated to see the Sunday new|spapers come in|to the house.| 1.29, 1.00 |