Is the #1 LLM on the Leaderboard Also #1 at Toss? Building Toss’s Evaluation Framework

A large language model that proudly holds the top spot on public AI leaderboards does not necessarily deliver equally impressive performance in Toss’s actual production services. Whenever a new model is released, we often check the measured scorecards first, looking at how well it solves math problems and how smoothly it writes code. It is easy to harbor rose-colored expectations that a high score means we can immediately adopt the model for our services and achieve dramatic efficiency gains. However, when these models are actually applied in real operational environments, the results often differ from expectations. Just because a model handles English fluently does not guarantee it can perfectly understand domestic financial regulations or complex policy documents. It is difficult to assert that a model scoring highly in reasoning tasks will maintain agility in production environments where real-time response is critical. Above all, the ability to perform well on public exams is fundamentally different from the ability to handle the demanding tasks of the financial and commerce domains. The Toss AI Modeling team established its own evaluation criteria to bridge this gap and select models optimized for actual production environments. This article examines in detail what criteria Toss set to use limited resources efficiently and how it refined its evaluation metrics.

=

Is the #1 LLM on the Leaderboard Also #1 at Toss? Building Toss’s Evaluation Framework

1. Real-World Production Problems Toss Solves with AI

Toss has expanded its scope beyond a simple financial application to include commerce, advertising, and internal development and infrastructure operations, utilizing large language models in various ways. In the commerce sector, for instance, Toss automatically standardizes product information registered by numerous sellers in diverse formats, grouping matching products together. A voice agent system that communicates directly with sellers on Toss Shopping to coordinate product management also runs smoothly based on AI. It is essential to understand the context of the seller’s responses in real time, flexibly adjust the flow of conversation, and accurately invoke the necessary tools. In merchant review and settlement, the system accurately extracts business information from vast amounts of documents and cross-references it with original materials. When settlement inquiries are received, it independently queries complex internal data and calculation rules to provide customers with seamless answers. In the advertising review area, it meticulously examines ad copy and images and checks whether the webpages linked upon user clicks are safe. If a sudden anomaly is detected in ad performance, it thoroughly reviews past configuration change histories and performance data to identify the cause and formulate a response. Thus, across various Toss departments, large language models have already become indispensable core workers, quietly fulfilling their roles.

2. The Gap Between Public Leaderboards and Production Environments

When introducing a new AI model, public leaderboards serve as a very useful guide for narrowing down candidates. However, if the language used in the evaluation or the reasoning setup differs from the service environment that domestic users actually face, the rankings can be completely reversed. A model that scores exceptionally high on English-based evaluations may provide irrelevant answers to Korean queries. In fact, we conducted an experiment comparing the English original and a natural Korean translation of the same problem to verify math and coding capabilities. Surprisingly, models that held the advantage in English-based evaluations were pushed down in rankings when evaluated on the Korean translations. In math problems, models that recorded higher scores on Korean problems emerged, even though they had lower English scores. In coding tests, models that led in the English environment ceded the lead to other models when processing Korean instructions. These results suggest that the relative superiority among models itself changes depending on the language, regardless of absolute score differences. Therefore, companies providing services to domestic customers must verify whether models maintain consistent performance even with translated queries.

3. The Importance of Korean Knowledge and Cultural Context

Beyond math and programming, the ability to accurately understand the everyday knowledge and unique cultural context of domestic users is crucial. Large language models developed overseas have learned from vast amounts of English data, so they may be well-versed in Western common sense but lack insight into domestic sentiments. If they fail to grasp Korea’s unique financial systems, tax laws, and the subtle nuances of everyday language, they may provide misguided advice. For example, if asked about questions related to jeonse deposits or housing lottery systems, failing to properly reflect domestic laws can cause significant confusion. When analyzing financial product terms, it is not enough to simply read the text; one must accurately pinpoint consumer protection regulations hidden between the lines. Therefore, when setting evaluation criteria, items measuring the depth of understanding of domestic knowledge levels and cultural backgrounds must be included. No matter how fast a model’s calculation speed or how strong its logic, it is not qualified for production use if it cannot gauge the sentiments of domestic customers. To overcome these realistic limitations, Toss has extensively developed Korean-specific evaluation questions and incorporated them into its internal evaluation framework.

4. Designing Toss-Domain-Specific Evaluation Criteria

Once general capabilities are verified, the next step is to precisely measure how well the model fits Toss’s actual work environment. Toss’s evaluation system is divided into finance, commerce, customer service, and internal development areas, utilizing actual work data. When simulating a mortgage consultation process, we check whether the model skillfully invokes the knowledge retrieval system, market price lookup functions, and debt service ratio calculation tools. In the customer service automation stage, it securely queries complex customer information to answer inquiries accurately and transfers sensitive cases to human agents. In the development and operations area, it meticulously reviews code changes and generates data quality inspection rules through structured query analysis. When an incident occurs, it comprehensively investigates server logs, key metrics, and source code to derive potential causes. Based on these complex real-world work scenarios, we score the model’s performance to determine if it is ready for immediate production deployment. Only models that excel not just at solving test problems but also at organically integrating with actual tools make it to the final serving list.

5. Selecting the Optimal Model Within Limited Resources

The biggest reason why we cannot indiscriminately introduce all AI models into production is the limitation of massive GPU resources and costs. Adopting a model that is too heavy solely because of its high performance can slow down response times and inflate operational costs to unmanageable levels. Therefore, Toss seeks the optimal balance by considering not only performance scores but also the efficiency of resources consumed during serving. Based on evaluation results, we select models that are lightweight yet highly efficient for specific tasks to maximize performance per cost. For services where real-time performance is critical, such as voice agents, we prioritize deploying models with low latency. Conversely, for areas requiring extreme accuracy over speed, such as in-depth analysis or terms review, we assign heavier, high-performance models. Thanks to this meticulous resource allocation strategy, Toss can operate stable and sustainable AI services without compromising quality. The process of wisely distributing limited resources is itself the key to the success or failure of LLM adoption.

6. The Future and Development Direction of Toss AI

Going forward, Toss plans to further enhance its internal evaluation system to quickly adapt to the rapidly changing AI ecosystem. Instead of manually verifying every new model that emerges daily, we will use automated pipelines to rapidly assess their excellence. Additionally, we will define the unique capabilities required for Toss’s foundation model and continuously reinforce training data and prompts to strengthen them. We aim to complete specialized models that incorporate Toss’s unique financial domain knowledge, rather than simply using external models. Through close communication with production departments, we will identify areas where AI is most urgently needed and continuously improve performance. These efforts will ultimately lay the groundwork for customers to enjoy a safer, more convenient, and intelligent financial experience when using the Toss app. A perfect model does not exist from the start; it is born only through the process of constantly refining it by incorporating the voices of production teams.

Frequently Asked Questions