1

Kakao has unveiled artificial intelligence (AI) research achievements at an international academic conference aimed at cutting large language model (LLM) training costs and reducing model footprints. The company presented technologies that predict optimal settings for training large-scale models at minimal cost, while concurrently shrinking model sizes and boosting performance.

According to Kakao on the 8th, the company presented its research findings at the main conference of the Conference on Language Modeling (COLM) 2026 and at its tokenization workshop, TokShop.

At the main conference, Kakao introduced a methodology that predicts the optimal learning rate for large-scale Mixture of Experts (MoE) models at roughly 1% of total training costs. The learning rate determines how substantially parameters are adjusted during training, directly affecting training stability and model performance.

Kakao adapted mu-parameterization (μ-Parameterization)—a technique that scales optimal training hyperparameters found on small models up to large-scale models—tailored specifically to the MoE architecture. By incrementally scaling model size and training tokens, researchers identified the appropriate learning rate and verified that configurations established over short training runs remained valid during full-scale training.

This methodology was implemented in the pre-training of Kakao's "Kanana-2.6-155b-a17b" model. Kakao explained that this approach enabled stable training up to 10 trillion tokens.

At TokShop, the company showcased the "BPE-Guided Byte Transformer" (BBT) architecture, which cuts model size while enhancing performance. Unlike conventional Byte Pair Encoding (BPE) methods that store data at the level of words or subword tokens, BBT utilizes bytes as smaller base units to reduce model size.

In comparative experiments, BBT reduced parameter counts by 20.2% to 40.4% while improving test performance by 2.7% to 6.6%. It also demonstrated enhanced robustness against typos and character perturbations, as well as superior transfer performance to unseen languages.

Kakao expects this technology can be leveraged to alleviate memory burdens for AI models in on-device environments, while delivering stable performance in multilingual inputs and conversational settings prone to frequent typographical errors.

Going forward, Kakao plans to broaden the scope of this research across diverse model architectures, multimodal systems, and multilingual environments to lower training and operational costs for large LLMs.

A Kakao official noted, "This research is significant in presenting ways to cut training and operational expenses while maintaining or improving AI model performance," adding, "We will expand our research across varied model architectures, multimodality, and multilingual environments to enhance practical applicability in real-world services."

                                                                                                           Son Kang-hoon