Meta recently announced the new large language model called Maverick. Maverick became one of the topics that attracted attention in the field of artificial intelligence. The second place of the model on the comparative evaluation platform named LM Arena increased the expectations. But on a closer look, it was understood that Maverick used in the test was a different version. This difference was enough to confuse the developers.
Meta clearly stated in his own statement that the version used in the LM Arena was a “experimental chat version .. The table on the official website of the Llama platform included a similar statement. It was informed that the version used in the test was “optimized in terms of chat competencies”. So, the version offered to the developers is not the same.
In addition to all these, the measurement method of the LM Arena has been controversial for a long time. On the platform, models are compared by human evaluators and it stands out which model is better. This method is criticized in academic circles because it contains subjective results. Nevertheless, companies generally highlight the results measured on this platform in marketing materials.
Maverick’s version in the test was also observed to behave differently from the public version. Researchers said the version of the LM Arena uses plenty of emoji and gives extremely long answers. In contrast, it was seen that the downloadable version produced more simple outputs. This creates some kind of confusion in performance evaluations.
Such differences make it difficult for developers to predict the real performance of the model. Because the behavior seen in the test and the behavior encountered in the field may not overlap. Especially in terms of product developers, this situation can affect decision processes. In any case, if it is not clear what is evaluated in the test, the result is controversial.
Evaluation of the model with a specially set version for the test environment brings the problem of transparency. Although Meta has made it clear that it optimized the model, it is necessary to make a detailed examination to access this information. It is not always easy to understand such differences for large masses. Companies need to define the model in the evaluation environment directly and clearly.
In addition, benchmark tests have limits in terms of generalization. Because such tests cannot measure the performance of all usage scenarios of the model. It can only show how it behaves in certain contexts. In this context, Maverick’s success in the test may not show direct daily use quality.
It opens the way for developers with Meta Maverick
The Maverick model, which the company offers in open source, can be promising for many developers. However, the context of the test results should be clear so that developers can correctly set their expectations for the model. If companies do not explicitly specify which version is tested, this may be misleading. Transparency is an inevitable necessity in model comparisons that become increasingly complex.
The commodity example shows once again that measurement and comparison practices in the field of artificial intelligence should be rethinking. Developers and users should not decide without understanding the details of the test results. It will be clearer when other companies go to similar applications. In this process, it is expected that more open and honest communication forms in the sector will become widespread.