How much insight a benchmark can provide for the development of MLLMs?

With the emergence of Multimodal Large Language Models (MLLMs), hundreds of benchmarks have been developed to ensure the reliability of MLLMs in downstream tasks. However, the evaluation mechanism itself may not be reliable. For MLLM developers, a benchmark serves the MLLM, and a good benchmark should provide as many insights as possible, revealing where the model needs improvement. Therefore, we define the quality of this service as Information Density: the volume of meaningful information reflected by a certain number of samples. There are four attributes that lead to defects in information density:

• Fallacy: A sample is poorly formulated, where the information reflected is not reliable.
• Difficulty: A sample is too simple thus almost all MLLMs are correct, giving no meaningful information.
• Redundancy: A sample that can be answered correctly based on only part of the information, with the remaining part being redundant.
• Diversity: Multiple samples that are too similar, resulting in overlapping information reflected in the responses.

• [2024/6/13] 🔥 Github repo for Imfomation Density is online.
• [To Do] [ ] Update the Human Eval annotation principle.
• [To Do] [ ] Update the Model Eval code.
• [To Do] [ ] Update the Data Eval code.

Benchmarks that apply the information density principle have the following three conditions:

• Multi-modal: Each sample must contain only one image and one paragraph of text.
• Multi-choices: The correct answer is within the range of choices, rather than open-ended questions.
• Multi-task: The dataset is oriented towards general tasks, rather than specialized tasks such as mathematics/physics. (see our paper)

Additionally, this work does not involve praising or criticizing any multimodal benchmarks, nor is it a commercial leaderboard for ranking. Among the four dimensions, only the Fallacy score has an objective measure of quality (where lower values are preferable). The other three dimensions are used to represent the amount of information, instead of inherently superior or inferior values, whose appropriateness depends on the specific evaluation context.

Please contact the first author of this paper for queries.

• Chunyi Li, lcysyzxdxc@sjtu.edu.cn, @lcysyzxdxc