Long-Form Simultaneous Speech Translation

Abstract

Simulta ́nnı ́ pr ̌eklad r ̌ec ̌i (SimulST) si klade za cı ́l pr ̌ekla ́dat mluveny ́ ja- zyk do psane ́ho textu, zatı ́mco mluvc ̌ı ́ jes ̌te ̌ hovor ̌ı ́. Tento poz ̌adavek na zpra- cova ́nı ́ v rea ́lne ́m c ̌ase vytva ́r ̌ı ́ za ́kladnı ́ kompromis mezi kvalitou a latencı ́: dr ̌ı ́ve ̌js ̌ı ́ vyda ́nı ́ vy 'stupu sniz ̌uje latenci, ale zvys ̌uje riziko za ́vazku k nesta- bilnı ́m hypote ́za ́m, zejme ́na kdyz ̌ je vstup neu ́plny ́ a por ̌adı ́dek slov se lis ̌ı ́ mezi jazyky. V te ́to disertac ̌nı ́ pra ́ci posouva ́me SimulST dve ̌ma dopln ̌ujı ́cı ́mi se sme ̌ry. Za prve ́, vyvı ́jı ́me prakticke ́ metody pro ovladatelne ́ a efektivnı ́ deko ́dova ́nı ́ v be ̌z ̌ne ̌ pouz ̌ı ́vane ́m kra ́tke ́m forma ́tu, vc ̌etne ̌ ra ́mce pro inkre- menta ́lnı ́ deko ́dova ́nı ́ neza ́visle ́ho na modelu, ktery ́ pr ̌eva ́dı ́ offline modely na simulta ́nnı ́; vyleps ̌ene ́ho beam searchu, ktery ́ umoz ̌n ̌uje skutec ̌ne ̌ inkre- menta ́lnı ́ vy 'stup a explicitnı ́ kontrolu kvality a latence; a deko ́dovacı ́ stra- tegie zaloz ̌ene ́ na CTC, ktera ́ sniz ̌uje opakovane ́ deko ́dovanı ́. Za druhe ́, po- souva ́me SimulST sme ̌rem k realisticke ́mu dlouhe ́mu vstupu, kde hranice ve ̌t nejsou da ́ny pr ̌edem. Navrhujeme segmentaci r ̌ı ́zenou koncovy ́mi signa ́ly ve ̌t v generovane ́m pr ̌ekladu a latentnı ́mi...

Simultaneous speech translation (SimulST) aims to translate spoken lan- guage into written text while the speaker is still talking. This real-time requirement creates a fundamental quality-latency trade-off: emitting out- put earlier reduces delay but increases the risk of committing to unstable hypotheses, especially when the input is incomplete and word order dif- fers across languages. In this thesis, we advance SimulST along two com- plementary directions. First, we develop practical methods for controllable and efficient decoding in the commonly used short-form setting, including a model-agnostic incremental decoding framework that onlinizes offline speech translation models; improved blockwise beam search that enables truly incre- mental output and explicit quality-latency control; and a CTC-based stabil- ity policy that reduces repeated decoding passes. Second, we move SimulST towards realistic long-form input, where sentence boundaries are not given a priori, by proposing on-the-fly segmentation driven by end-of-sentence cues in the generated translation and latent alignments extracted from speech- translation CTC, enabling a single end-to-end model to translate and segment simultaneously without an external segmentation module. Finally, we ana- lyze how SimulST is studied and evaluated in practice...