The multi-domain international search on speech 2020 ALBAYZIN evaluation: Overview, systems, results, discussion and post-evaluation analyses
Entity
UAM. Departamento de Tecnología Electrónica y de las ComunicacionesPublisher
MDPIDate
2021-09-14Citation
10.3390/app11188519
Applied Sciences-Basel 11.18 (2021): 8519
ISSN
2076-3417 (online)DOI
10.3390/app11188519Funded by
This research was funded by the Ministry of Science, Innovation and Universities of Spain, grant number RTI2018-095324-B-I00, and project DSForSec (grant number RTI2018-098091-B-I00). The APC was funded by the project DSForSec (grant number RTI2018-098091-B-I00) from the Ministry of Science, Innovation and Universities of SpainProject
Gobierno de España. RTI2018-095324-B-I00; Gobierno de España. RTI2018-098091-B-I00Editor's Version
https://doi.org/10.3390/app11188519Subjects
International evaluation; Query-by-example spoken term detection; Search on speech; Spanish language; Spoken term detection; TelecomunicacionesRights
© The author(s)Abstract
The large amount of information stored in audio and video repositories makes search on speech (SoS) a challenging area that is continuously receiving much interest. Within SoS, spoken term detection (STD) aims to retrieve speech data given a text-based representation of a search query (which can include one or more words). On the other hand, query-by-example spoken term detection (QbE STD) aims to retrieve speech data given an acoustic representation of a search query. This is the first paper that presents an internationally open multi-domain evaluation for SoS in Spanish that includes both STD and QbE STD tasks. The evaluation was carefully designed so that several post-evaluation analyses of the main results could be carried out. The evaluation tasks aim to retrieve the speech files that contain the queries, providing their start and end times and a score that reflects how likely the detection within the given time intervals and speech file is. Three different speech databases in Spanish that comprise different domains were employed in the evaluation: the MAVIR database, which comprises a set of talks from workshops; the RTVE database, which includes broadcast news programs; and the SPARL20 database, which contains Spanish parliament sessions. We present the evaluation itself, the three databases, the evaluation metric, the systems submitted to the evaluation, the evaluation results and some detailed post-evaluation analyses based on specific query properties (in-vocabulary/out-of-vocabulary queries, single-word/multi-word queries and native/foreign queries). The most novel features of the submitted systems are a data augmentation technique for the STD task and an end-to-end system for the QbE STD task. The obtained results suggest that there is clearly room for improvement in the SoS task and that performance is highly sensitive to changes in the data domain
Files in this item
Google Scholar:Tejedor, Javier
-
Toledano, Doroteo T.
-
Ramirez, Jose M.
-
Montalvo, Ana R.
-
Alvarez-Trejos, Juan Ignacio
This item appears in the following Collection(s)
Related items
Showing items related by title, author, creator and subject.
-
Spoken term detection ALBAYZIN 2014 evaluation: overview, systems, results, and discussion
Tejedor, Javier; Toledano, Doroteo T.; Lopez-Otero, Paula; Docío-Fernández, Laura; García-Mateo, Carmen García; Cardenal, Antonio; Echeverry-Correa, Julián David; Coucheiro-Limeres, Alejandro; Olcoz, Julia; Miguel, Antonio
2015-12-08 -
Comparison of ALBAYZIN query-by-example spoken term detection 2012 and 2014 evaluations
Tejedor, Javier; Toledano, Doroteo T.; Lopez-Otero, Paula; Docío-Fernández, Laura; García-Mateo, Carmen García
2016-12