Akademik Veri Yönetim Sistemi
Engineering Science and Technology, an International Journal, cilt.35, 2022 (SCI-Expanded)
© 2022 Karabuk UniversityIn recent years, lip-reading has been one of the studies whose importance has increased considerably, especially with the spread of deep learning applications. In this topic, researchers try to detect what a person says from video frames without sound. When the previous studies are analysed, it is seen that automatic lip-reading systems have been developed for various languages such as Chinese, Korean, English and German. However, these studies reveal that the development of the system is difficult because lip-reading from video frame images without audio data depends on many parameters such as light, shooting distance, and the gender of the person. Lip-reading systems were first developed using classical machine learning methods. However, especially in recent years, with the popularity of deep learning applications, this subject has started to be studied more than before and studies reveal that in general, deep learning-based lip-reading gives more successful results. Even though there are studies in this field in different languages, there is no current study and dataset in Turkish. Therefore, this study aims to investigate the performances of the state-of-the art deep learning models on Turkish lip-reading. To this aim, two new datasets, one with 111 words and other with 113 sentences were created using image processing techniques. The model used in this study to perform lip-reading extracts features from video frames using CNN based models and performs classification using Bidirectional Long Short-Term Memory (Bi-LSTM). Results of experiments reveal that, ResNet-18 and Bi-LSTM pair gives the best results in both word and sentence datasets with accuracy values 84.5% and 88.55%, respectively. It is also observed that, better performances are obtained in sentence recognition than word recognition in almost every model implemented.