TURKISH-TO-ENGLISH SHORT STORY TRANSLATION BY DEEPL: HUMAN EVALUATION BY TRAINEES AND TRANSLATION PROFESSIONALS VS. AUTOMATIC EVALUATION
Vol.11, Issue 1, 2025, pp. 17-42 Full text: PDF . HTML
DOI: https://doi.org/10.33919/esnbu.25.1.2
Web of Science: 001528232700003
Author: Halise Gülmüş Sırkıntı https://orcid.org/0000-0002-6585-5961
Affiliation: Marmara University, Istanbul, Türkiye 02kswqa67
Abstract
[abstract=This mixed-methods study aims to evaluate the quality of Turkish-to-English literary machine translation by DeepL, incorporating both human and automatic evaluation metrics while engaging translation trainees and professional translators. Raw MT output of two short stories, Mendil Altında and Kabak Çekirdekçi, evaluated by both groups via TAUS DQF tool and evaluators wrote reports on the detected errors. Additionally, BLEU was employed for automatic evaluation. The results indicate a consensus between trainees and professionals in assessing MT accuracy and fluency. Accuracy rates were 80.59% and 80.50% for Mendil Altında, and 73.08% and 82.35% for Kabak Çekirdekçi. Fluency rates were similarly close, 71.96% and 72.32% for Mendil Altında, and 66.81% and 62.09% for Kabak Çekirdekçi. Bleu scores, particularly 1-gram results, align with the human evaluators' results. Furthermore, reports show that trainees provided more detailed analysis, frequently using meta-language, suggesting that increased exposure to metrics enhances trainees' ability to identify fine-grained MT errors.]
Keywords: literary translation, machine translation evaluation, human evaluation, automatic evaluation, BLEU
Article history:
Submitted: 20 February 2025
Reviewed: 31 March 2025
Accepted: 03 May 2025
Published: 30 June 2025
Citation (APA):
Gülmüş Sırkıntı, H. (2025). Turkish-to-English short story translation by DeepL: Human evaluation by trainees and translation professionals vs. automatic evaluation. English Studies at NBU, 11(1), 17-42. https://doi.org/10.33919/esnbu.25.1.2
Copyright © 2025 Halise Gülmüş Sırkıntı
This is an Open Access article published and distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Funding:
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
[[refs=]]
Review:
1. Reviewer's name: Name undisclosed
Review Content: Undisclosed
Review Verified on Publons
2. Reviewer's name: Name Undisclosed
Review Content: Undisclosed
Review Verified on Publons
Handling Editor: Boris Naimushin, New Bulgarian University
Verified Editor Record on Publons