Lost in Semantics: Assessing the Precision of Gemini's Arabic to Indonesian Translation
DOI:
https://doi.org/10.36668/wa0n1080Keywords:
Artificial Intelligence; Google Gemini; Gemini translation performance, Arabic interpretation, AI’s contextual understandingAbstract
This research critically evaluated Google Gemini's performance in translating and interpreting classical Arabic literature, specifically focusing on the books Alfiyah Ibn Malik and Nadham al-Imrithi, into Indonesian. A collect-measure-repeat methodology was employed as a taxonomic investigation, comparing AI-generated translations with those from verified translation books and human experts to assess both literal accuracy and contextual comprehension. The findings highlighted that while Gemini consistently provided precise word-for-word translations, it frequently overlooked contextual nuances and detailed interpretations. Despite demonstrating a solid understanding of overarching themes, the model struggled to capture the nuanced meanings and subtle contextual elements inherent in classical Arabic texts. An observed inconsistency in language output such as the occasional inclusion of English responses despite Indonesian input suggested limitations in linguistic fidelity. From a semantic standpoint, Gemini excelled in lexical semantics by accurately rendering individual words and phrases. However, it fell short in pragmatic and contextual semantics, failing to convey implied meanings and cultural references effectively. This limitation hindered its ability to faithfully reproduce the deeper thematic interconnections present in the original texts. Nevertheless, the translations generally provided a foundational understanding of the texts' basic messages, indicating considerable potential for the model. Future refinement efforts should prioritize improving contextual understanding and enhancing the depth of interpretation to make Gemini a more dependable tool for translating intricate literary texts. Future research is recommended to conduct comparative evaluations across different AI translation models, explore pragmatic and cultural semantics more deeply, and develop interdisciplinary frameworks that integrate linguistic, pedagogical, and technological perspectives
Downloads
References
Ahmed, I., Kajol, M., Hasan, U., & Datta, P. P. (2023). ChatGPT vs. Bard: A Comparative Study [Preprint]. https://doi.org/10.36227/techrxiv.23536290.v1
Ajao, E. (2023). Benefits, risks of Google AI chatbot’s Bard Extensions | TechTarget. Enterprise AI. https://www.techtarget.com/searchenterpriseai/news/366552898/Benefits-risks-of-Google-AI-chatbots-Bard-Extensions
Akter, S. N., Yu, Z., Muhamed, A., Ou, T., Bäuerle, A., Cabrera, Á. A., Dholakia, K., Xiong, C., & Neubig, G. (2023). An In-depth Look at Gemini’s Language Abilities. arXiv Preprint arXiv:2312.11444. https://arxiv.org/abs/2312.11444
Almelhes, S. A. (2023). A Review of Artificial Intelligence Adoption in Second-Language Learning. Theory and Practice in Language Studies, 13(5), 1259–1269. https://doi.org/10.17507/tpls.1305.21
Alruqi, T. N., & Alzahrani, S. M. (2023). Evaluation of an Arabic Chatbot Based on Extractive Question-Answering Transfer Learning and Language Transformers. AI, 4(3), 667–691. https://doi.org/10.3390/ai4030035
Annamalai, N., Rashid, R. A., Munir Hashmi, U., Mohamed, M., Harb Alqaryouti, M., & Eddin Sadeq, A. (2023). Using chatbots for English language learning in higher education. Computers and Education: Artificial Intelligence, 5, 100153. https://doi.org/10.1016/j.caeai.2023.100153
Atwell, E. S., Brierley, C., Dukes, K., Sawalha, M., & Sharaf, A. (2011). An artificial intelligence approach to Arabic and Islamic content on the internet [Proceedings Paper]. Leeds. NITS 2011 3rd National Information Technology Symposium. Proceedings of NITS 3rd National Information Technology Symposium. http://www.kokannews.org/wp-content/uploads/2014/05/An-Artificial-Intelligence.pdf
Azmi, A. M., Al-Qabbany, A. O., & Hussain, A. (2019). Computational and natural language processing based studies of hadith literature: A survey. Artificial Intelligence Review, 52, 1369–1414.
Bandi, A., Adapa, P. V. S. R., & Kuchi, Y. E. V. P. K. (2023). The Power of Generative AI: A Review of Requirements, Models, Input–Output Formats, Evaluation Metrics, and Challenges. Future Internet, 15(8), Article 8. https://doi.org/10.3390/fi15080260
Cantú-Ortiz, F. J., Galeano Sánchez, N., Garrido, L., Terashima-Marin, H., & Brena, R. F. (2020). An artificial intelligence educational strategy for the digital transformation. International Journal on Interactive Design and Manufacturing (IJIDeM), 14, 1195–1209. https://doi.org/https://doi.org/10.1007/s12008-020-00702-8
Dalayli, F. (2023). Use of NLP Techniques in Translation by ChatGPT: Case Study. In A. H. Haddad, A. R. Terryn, R. Mitkov, R. Rapp, P. Zweigenbaum, & S. Sharoff (Eds.), Proceedings of the Workshop on Computational Terminology in NLP and Translation Studies (ConTeNTS) Incorporating the 16th Workshop on Building and Using Comparable Corpora (BUCC) (pp. 19–25). INCOMA Ltd., Shoumen, Bulgaria. https://aclanthology.org/2023.contents-1.3
Feng, Z., Zhang, Y., Li, H., Liu, W., Lang, J., Feng, Y., Wu, J., & Liu, Z. (2024). Improving LLM-based Machine Translation with Systematic Self-Correction. arXiv Preprint arXiv:2402.16379. https://arxiv.org/pdf/2402.16379
Fuad, A., & Al-Yahya, M. (2022). Recent Developments in Arabic Conversational AI: A Literature Review. IEEE Access, 10, 23842–23859. https://doi.org/10.1109/ACCESS.2022.3155521
Garrido, N. (2017). The method of James Spradley in qualitative research. Enfermería: Cuidados Humanizados, 6(SPE), 37–42.
Gemini Team, Reid, M., Savinov, N., Teplyashin, D., Dmitry, Lepikhin, Lillicrap, T., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., Antonoglou, I., Anil, R., Borgeaud, S., Dai, A., Millican, K., Dyer, E., Glaese, M., … Vinyals, O. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context (arXiv:2403.05530). arXiv. https://doi.org/10.48550/arXiv.2403.05530
Hajir, B., Rasman, R., & McInerney, W. (2022). Digital translanguaging and Arabic-English transliteration (Arabizi): Insights from Syria and Lebanon. Cambridge Educational Reseach E-Journal, 9.
Haq, Y. N. (2022). Manhaj al-Imam Ibn Aqil fi Syarhi Alfiah al-Imam Ibn Malik [bachelorThesis, Fakultas Dirasat Islamiah]. https://repository.uinjkt.ac.id/dspace/handle/123456789/61799
Hidayatullah, A. S., & Fauji, I. (2023). Bridging Theory and Practice in Arabic Language Education | Indonesian Journal of Islamic Studies. https://ijis.umsida.ac.id/index.php/ijis/article/view/1724
Huang, X., Zou, D., Cheng, G., Chen, X., & Xie, H. (2023). Trends, Research Issues and Applications of Artificial Intelligence in Language Education. Educational Technology & Society, 26(1), 112–131. https://www.jstor.org/stable/48707971
Imran, M., & Almusharraf, N. (2024). Google Gemini as a next generation AI educational tool: A review of emerging educational technology. Smart Learning Environments, 11(1), 22. https://doi.org/10.1186/s40561-024-00310-z
Inel, O., Draws, T., & Aroyo, L. (2023). Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 11(1), Article 1. https://doi.org/10.1609/hcomp.v11i1.27547
Khoshafah, F. (2023). ChatGPT for Arabic-English Translation: Evaluating the Accuracy. Ministry of Education, Yemen. https://doi.org/10.21203/rs.3.rs-2814154/v2
Killman, J. (2023). Machine translation and legal terminology: Data-driven approaches to contextual accuracy. In Handbook of Terminology (pp. 485–510). John Benjamins. https://www.jbe-platform.com/content/books/9789027249388-hot.3.mac2
Kishore, K. S., & Shaik, R. (2024). Evaluating Telugu Proficiency in Large Language Models_ A Comparative Analysis of ChatGPT and Gemini (arXiv:2404.19369). arXiv. https://doi.org/10.48550/arXiv.2404.19369
Koco?, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szyd?o, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., Koco?, A., Koptyra, B., Mieleszczenko-Kowszewicz, W., Mi?kowski, P., Oleksy, M., Piasecki, M., Radli?ski, ?., Wojtasik, K., Wo?niak, S., & Kazienko, P. (2023). ChatGPT: Jack of all trades, master of none. Information Fusion, 99, 101861. https://doi.org/10.1016/j.inffus.2023.101861
Kolhar, M., & Alameen, A. (2021). Artificial Intelligence Based Language Translation Platform. Intelligent Automation & Soft Computing, 28(1). https://doi.org/http://dx.doi.org/10.32604/iasc.2021.014995
Labadze, L., Grigolia, M., & Machaidze, L. (2023). Role of AI chatbots in education: Systematic literature review. International Journal of Educational Technology in Higher Education, 20(1), 56. https://doi.org/10.1186/s41239-023-00426-1
Läubli, S., Castilho, S., Neubig, G., Sennrich, R., Shen, Q., & Toral, A. (2020). A set of recommendations for assessing human–machine parity in language translation. Journal of Artificial Intelligence Research, 67, 653–672. https://doi.org/https://doi.org/10.1613/jair.1.11371
Lu, Q., Qiu, B., Ding, L., Xie, L., & Tao, D. (2023). Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models: A Case Study on ChatGPT (No. 2023030255). Preprints. https://doi.org/10.20944/preprints202303.0255.v1
Ram, B., & Verma, P. (2023). Artificial intelligence AI-based Chatbot study of ChatGPT, Google AI Bard and Baidu AI. World Journal of Advanced Engineering Technology and Sciences, 8(1), 258–261. https://doi.org/10.30574/wjaets.2023.8.1.0045
Tiawaldi, A., & Abdul Wahab, M. (2017). PERKEMBANGAN BAHASA ARAB MODERN DALAM PERSPEKTIF SINTAKSIS DAN SEMANTIK PADA MAJALAH ALJAZEERA. Arabiyat : Jurnal Pendidikan Bahasa Arab Dan Kebahasaaraban, 4(1). https://doi.org/10.15408/a.v4i1.5328
Zaimah, N. R., Fatchiatuzahro, & Budi Hartanto, E. (2024). ENHANCING WRITING COMPREHENSION IN L2 ARABIC LEARNERS THROUGH AI-BASED TRANSLANGUAGING CHATBOTS. Al-Mubin: Islamic Scientific Journal, 7(1), 21–34. https://doi.org/10.51192/almubin.v7i1.753
Zaimah, N. R., Hartanto, E. B., & Zahro, F. (2024). Acceptability and Effectiveness Analysis of Large Language Model-Based Artificial Intelligence Chatbot Among Arabic Learners. Mantiqu Tayr: Journal of Arabic Language, 4(1), Article 1. https://doi.org/10.25217/mantiqutayr.v4i1.3951
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Nely Rahmawati Zaimah, Alam Budi Kusuma
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
