MULTILINGUAL GENERATIVE AI AVATAR: EDGE AND CLOUD-BASED REAL-TIME DEPLOYMENTS

Auyeskhan Alibekov; Galammadin Askar; Zhanat Makhataeva; Guldana Nauryzbaikyzy; Kulzat Sadirova; Huseyin Atakan Varol

doi:10.26577/jpcsit5320262

Authors

Auyeskhan Alibekov Private Institution “Institute of Smart Systems and Artificial Intelligence” (ISSAI), Nazarbayev University, Astana, Kazakhstan , Al-Farabi Kazakh National University, Almaty, Kazakhstan https://orcid.org/0009-0005-7551-668X (unauthenticated)
Galammadin Askar Private Institution “Institute of Smart Systems and Artificial Intelligence” (ISSAI), Nazarbayev University, Astana, Kazakhstan , Al-Farabi Kazakh National University, Almaty, Kazakhstan https://orcid.org/0009-0001-6304-3003 (unauthenticated)
Zhanat Makhataeva Private Institution “Institute of Smart Systems and Artificial Intelligence” (ISSAI), Nazarbayev University, Astana, Kazakhstan , Al-Farabi Kazakh National University, Almaty, Kazakhstan https://orcid.org/0000-0001-9366-7047 (unauthenticated)
Guldana Nauryzbaikyzy Private Institution “Institute of Smart Systems and Artificial Intelligence” (ISSAI), Nazarbayev University, Astana, Kazakhstan , K. Zhubanov Aktobe Regional University, Aktobe, Kazakhstan https://orcid.org/0000-0001-9272-8952 (unauthenticated)
Kulzat Sadirova Private Institution “Institute of Smart Systems and Artificial Intelligence” (ISSAI), Nazarbayev University, Astana, Kazakhstan , K. Zhubanov Aktobe Regional University, Aktobe, Kazakhstan https://orcid.org/0000-0001-6092-8191 (unauthenticated)
Huseyin Atakan Varol Private Institution “Institute of Smart Systems and Artificial Intelligence” (ISSAI), Nazarbayev University, Astana, Kazakhstan , Al-Farabi Kazakh National University, Almaty, Kazakhstan https://orcid.org/0000-0002-4042-425X (unauthenticated)

DOI:

https://doi.org/10.26577/jpcsit5320262

Keywords:

Human-AI Interaction, Avatars, Large Language Model (LLM), Text-to-Speech (TTS), Speech-to-Text (STT)

Abstract

This paper introduces a multilingual avatar system designed for real-time Human-AI Interaction in Kazakh, Russian, and English. AI-enhanced avatars present novel ways of communication and interaction between users and technology. This work follows current trends in Human-Computer Interaction (HCI), bringing innovations towards text-, voice-, and vision-supported embodied AI agents. We present and evaluate two deployment architectures for the proposed multilingual avatar system (i.e., cloud-based and edge-deployed real-time solutions). Both methods utilize models developed and fine-tuned at the Institute of Smart Systems and Artificial Intelligence (ISSAI) at Nazarbayev University, based in Astana, Kazakhstan. The cloud-based solution uses ISSAI’s Large Language Model (LLM) “Oylan” and the Text-to-Speech/Speech-to-Text (TTS/STT) platform “MangiSoz”, both deployed on the servers of ISSAI. On the other hand, the edge-deployed avatar solution uses the quantized Small Vision-Language Model (VLM) “Qolda” with 4.3B parameters and on-device speech processing tools (i.e., Whisper Turbo for Automatic Speech Recognition (ASR) and Matcha-TTS for speech synthesis). The avatar interface is built using the popular stylized avatar creation technology “Ready Player Me”. During the evaluation of 96 queries across three languages (specifically, 32 Kazakh-language queries, along with 32 Russian and 32 English translations), the edge-deployed avatar system showed a substantial performance advantage, reducing end-to-end response latency by 45.5% (2.28 s vs. 4.18 s) compared to the cloud-based solution. The study results highlight the potential of a small language model (SLM) integrated with speech detection and generation models on the edge to create real-time, responsive, offline communication with avatar-enhanced AI agents within the trilingual environment of Kazakhstan. The multilingual avatar system demo, implementation pipeline, model details, and other materials are publicly available at the project’s GitHub repository https://github.com/IS2AI/lightweight_avatar.

Downloads

Download data is not yet available.

Author Biographies

Auyeskhan Alibekov, Private Institution “Institute of Smart Systems and Artificial Intelligence” (ISSAI), Nazarbayev University, Astana, Kazakhstan, Al-Farabi Kazakh National University, Almaty, Kazakhstan

Auyeskhan Alibekov is a Junior Data Scientist at the Institute of Smart Systems and Artificial Intelligence (ISSAI) at Nazarbayev University (NU) (Astana, Kazakhstan, auyeskhan.alibekov@nu.edu.kz). Auyeskhan Alibekov holds a B.S. degree in Computer Science from Astana IT University, Astana, Kazakhstan. ORCID ID: 0009-0005-7551-668X.
Galammadin Askar, Private Institution “Institute of Smart Systems and Artificial Intelligence” (ISSAI), Nazarbayev University, Astana, Kazakhstan, Al-Farabi Kazakh National University, Almaty, Kazakhstan

Galammadin Askar is a Master’s student at Nazarbayev University (NU) and a Research Assistant at the Institute of Smart Systems and Artificial Intelligence (ISSAI) at NU (Astana, Kazakhstan, galammadin.askar@nu.edu.kz). Galammadin Askar holds a B.S. degree in Computer Science from Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea. ORCID ID: 0009-0001-6304-3003.
Zhanat Makhataeva, Private Institution “Institute of Smart Systems and Artificial Intelligence” (ISSAI), Nazarbayev University, Astana, Kazakhstan, Al-Farabi Kazakh National University, Almaty, Kazakhstan

Zhanat Makhataeva, PhD, is a Senior Data Scientist at the Institute of Smart Systems and Artificial Intelligence (ISSAI) of Nazarbayev University (NU) (Astana, Kazakhstan, zhanat.mahataeva@nu.edu.kz). Dr. Zhanat Makhataeva received a B.S. degree in Robotics and Mechatronics from Nazarbayev University (NU), Astana, Kazakhstan, in 2017, and the M.S. and Ph.D. degrees in Robotics Engineering from the School of Engineering and Digital Sciences at NU, in 2019 and 2023, respectively. During her doctoral studies, Zhanat became a Microsoft Research PhD Fellow 2021-2022 and had a 12-week Internship at Microsoft Research Cambridge, UK Laboratory. She was hosted by the Future of Work Research Group at Microsoft Research and presented their work at the CHI 2023 Conference in Hamburg, Germany. Her research interests include augmented reality, cognitive assistive systems, artificial intelligence, human memory augmentation, cognitive ergonomics, system testing and evaluation, and human information processing. ORCID ID: 0000-0001-9366-7047.
Guldana Nauryzbaikyzy, Private Institution “Institute of Smart Systems and Artificial Intelligence” (ISSAI), Nazarbayev University, Astana, Kazakhstan, K. Zhubanov Aktobe Regional University, Aktobe, Kazakhstan

Guldana Nauryzbaikyzy, PhD, is a Senior Lecturer at the Faculty of Foreign Languages of K. Zhubanov Aktobe Regional University (Aktobe, Kazakhstan, gnauryzbaikyzy@zhubanov.edu.kz). Dr. Guldana Nauryzbaikyzy received the B.S. and M.S. degrees in Arts with a specialization in Foreign Philology from L.N. Gumilyov Eurasian National University, Astana, Kazakhstan, in 2017 and 2020. In 2026, Dr. Guldana Nauryzbaikyzy received her PhD in Philology from K. Zhubanov Aktobe Regional University. Her research interests include linguacultural studies, cognitive linguistics, and discourse studies. ORCID ID: 0000-0001-9272-8952.
Kulzat Sadirova, Private Institution “Institute of Smart Systems and Artificial Intelligence” (ISSAI), Nazarbayev University, Astana, Kazakhstan, K. Zhubanov Aktobe Regional University, Aktobe, Kazakhstan

Kulzat Sadirova, Doctor of Philological Sciences, is a professor at the Department of Theoretical and Applied Linguistics, Faculty of Philology, at K. Zhubanov Aktobe Regional University (Aktobe, Kazakhstan, ksadirova@zhubanov.edu.kz). She received her degree in Kazakh Language and Literature from K. Zhubanov Aktobe Pedagogical Institute in 1993. She obtained her Candidate of Sciences degree in Philology in 1997 and her Doctor of Sciences degree in Philology in 2009 from the A. Baitursynuly Institute of Linguistics. She currently serves as a professor and Chair of the Dissertation Council for the PhD program “8D02304 – Philology” at K. Zhubanov Aktobe Regional University. Her research interests include linguistics, discourse studies, syntax, and turkology. She has published nearly 200 scientific and methodological works, including articles indexed in Scopus and Web of Science, as well as monographs and textbooks. She has supervised 5 PhD graduates. She was the principal investigator of the grant-funded research project “Onomastic Discourse: Mythological Aspect” (2021–2023), supported by the Science Committee of the Ministry of Education and Science of Kazakhstan, and contributed as a linguist and editor to the National Corpus of the Kazakh Language at the Til-Kazyna National Scientific and Practical Center. She has received several awards, including the State Scientific Scholarship for Leading Scientists (2024), “Best Researcher – 2024,” and the title “Best University Teacher of the Republic of Kazakhstan – 2012”. ORCID ID: 0000-0001-6092-8191.
Huseyin Atakan Varol, Private Institution “Institute of Smart Systems and Artificial Intelligence” (ISSAI), Nazarbayev University, Astana, Kazakhstan, Al-Farabi Kazakh National University, Almaty, Kazakhstan

Huseyin Atakan Varol, PhD, is a full professor of robotics at Nazarbayev University’s School of Engineering and Digital Sciences (Astana, Kazakhstan, ahvarol@nu.edu.kz). He received his BS degree in mechatronics engineering from Sabanci University, Istanbul, Turkey, in 2005. He received the MS and PhD degrees both in electrical engineering from Vanderbilt University, Nashville, TN, USA, in 2007 and 2009, respectively. From August 2009 to August 2011, he was first a postdoctoral research associate and then a research assistant professor with the Center for Intelligent Mechatronics, Department of Mechanical Engineering, Vanderbilt University, Nashville, TN, USA. In 2011, he joined the faculty of Nazarbayev University, Astana, Kazakhstan, where he is a full professor of robotics and directs the Institute of Smart Systems and Artificial Intelligence (ISSAI). His research interests include artificial intelligence, soft robotics, machine learning, sensor fusion and tensegrity. He has published more than 160 technical papers on these topics in reputable international journals and conferences. He holds multiple international patents in the area of bionics. He has attracted multiple competitively-funded research grants from the Ministry of Education and Science of Kazakhstan, World Bank and Nazarbayev University Research Fund. ORCID ID: 0000-0002-4042-425X.