ChartMimic: Evaluating LMM’s Cross-Modal Reasoning Capability via Chart-to-Code Generation

Chufan Shi1*, Cheng Yang2*, Yaxin Liu1*, Bo Shui1*, Junjie Wang3*, Mohan Jing1, Linran Xu1,
Xinyu Zhu1, Siheng Li1, Yuxiang Zhang3, Gongye Liu1, Xiaomei Nie1, Deng Cai4, Yujiu Yang1†,
1Tsinghua University, 2The Chinese University of Hong Kong, 3Waseda University, 4Tencent AI Lab
*Equal contribution. †Corresponding author.
Paper arXiv Code Data
Teaser image.

The real-world example. LMMs assist scientists and researchers in understanding, interpreting and creating charts during the reading and writing of academic papers. These models serve as assistants that enhance the comprehension and presentation of data in scholarly communications.

Abstract

We introduce a new benchmark, ChartMimic, aimed at assessing the visually-grounded code generation capabilities of large multimodal models (LMMs). ChartMimic utilizes information-intensive visual charts and textual instructions as inputs, requiring LMMs to generate the corresponding code for chart rendering.

ChartMimic includes 1,000 human-curated (figure, instruction, code) triplets, which represent the authentic chart use cases found in scientific papers across various domains (e.g., Physics, Computer Science, Economics, etc). These charts span 18 regular types and 4 advanced types, diversifying into 191 subcategories.

Furthermore, we propose multi-level evaluation metrics to provide an automatic and thorough assessment of the output code and the rendered charts. Unlike existing code generation benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to harmonize a blend of cognitive capabilities, encompassing visual understanding, code generation, and cross-modal reasoning.

The evaluation of 3 proprietary models and 11 open-weight models highlights the substantial challenges posed by ChartMimic. Even the advanced GPT-4V, Claude-3-opus only achieve an average score of 73.2 and 53.7, respectively, indicating significant room for improvement. We anticipate that ChartMimic will inspire the development of LMMs, advancing the pursuit of artificial general intelligence.

Framework.

The pipeline of ChartMimic. We provide 1,000 human-curated (figure, instruction, code) triplets. We use ChartMimic to evaluate LMMs’ proficiency in the multimodal chart-to-code generation, resulting in both high-level and low-level evaluation results.

Leaderboard

We conduct examination of 14 LMMs on ChartMimic, including 3 proprietary models and 11 open-weight models.

The ChartMimic leaderboard with Direct Mimic task. We also include the code execution success rate (Exec. Rate) and model size (Params).

The ChartMimic leaderboard with Customized Mimic task. We also include the code execution success rate (Exec. Rate) and model size (Params).


Please refer to our GitHub repo to add your model to the leaderboard.

Data

You can directly download our data from Huggingface datasets. For guidance on how to access and utilize the data, please consult our instructions on Github.

BibTeX

@article{
      shi2024chartmimic,
      title={ChartMimic: Evaluating LMM’s Cross-Modal Reasoning Capability via Chart-to-Code Generation},
      author={Chufan Shi and Cheng Yang and Yaxin Liu and Bo Shui and Junjie Wang and Mohan Jing and Linran Xu and Xinyu Zhu and Siheng Li and Yuxiang Zhang and Gongye Liu and Xiaomei Nie and Deng Cai and Yujiu Yang},
      year={2024},
      journal={arXiv preprint arXiv:2406.09961},
}

Contact Us

If you have any inquiries about ChartMimic, feel free to reach out to us at chartmimic@gmail.com, {scf22, yangc21}@mails.tsinghua.edu.cn or raise an issue on Github.