mmpose-main.zip

<div align="center"> <img width="100%" src="https://github.com/open-mmlab/mmpose/assets/13503330/5b637d76-41dd-4376-9a7f-854cd120799d"/> </div> # RTMPose: Real-Time Multi-Person Pose Estimation toolkit based on MMPose > [RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose](https://arxiv.org/abs/2303.07399) <div align="center"> English | [简体中文](README_CN.md) </div> ______________________________________________________________________ ## Abstract Recent studies on 2D pose estimation have achieved excellent performance on public benchmarks, yet its application in the industrial community still suffers from heavy model parameters and high latency. In order to bridge this gap, we empirically study five aspects that affect the performance of multi-person pose estimation algorithms: paradigm, backbone network, localization algorithm, training strategy, and deployment inference, and present a high-performance real-time multi-person pose estimation framework, **RTMPose**, based on MMPose. Our RTMPose-m achieves **75.8% AP** on COCO with **90+ FPS** on an Intel i7-11700 CPU and **430+ FPS** on an NVIDIA GTX 1660 Ti GPU. To further evaluate RTMPose's capability in critical real-time applications, we also report the performance after deploying on the mobile device. Our RTMPose-s achieves **72.2% AP** on COCO with **70+ FPS** on a Snapdragon 865 chip, outperforming existing open-source libraries. With the help of MMDeploy, our project supports various platforms like CPU, GPU, NVIDIA Jetson, and mobile devices and multiple inference backends such as ONNXRuntime, TensorRT, ncnn, etc.  ______________________________________________________________________ ## 📄 Table of Contents - [🥳 🚀 What's New](#--whats-new-) - [📖 Introduction](#-introduction-) - [🙌 Community](#-community-) - [⚡ Pipeline Performance](#-pipeline-performance-) - [📊 Model Zoo](#-model-zoo-) - [👀 Visualization](#-visualization-) - [😎 Get Started](#-get-started-) - [👨‍🏫 How to Train](#-how-to-train-) - [🏗️ How to Deploy](#️-how-to-deploy-) - [📚 Common Usage](#️-common-usage-) - [🚀 Inference Speed Test](#-inference-speed-test-) - [📊 Model Test](#-model-test-) - [📜 Citation](#-citation-) ## 🥳 🚀 What's New [🔝](#-table-of-contents) - Dec. 2023: - Update RTMW models. The RTMW-l model achieves 70.1 mAP on COCO-Wholebody val set. - Sep. 2023: - Add RTMW models trained on combined datasets. The alpha version of RTMW-x model achieves 70.2 mAP on COCO-Wholebody val set. You can try it [Here](https://openxlab.org.cn/apps/detail/mmpose/RTMPose). The technical report will be released soon. - Add YOLOX and RTMDet models trained on HumanArt dataset. - Aug. 2023: - Support distilled 133-keypoint WholeBody models powered by [DWPose](https://github.com/IDEA-Research/DWPose/tree/main). - You can try DWPose/RTMPose with [sd-webui-controlnet](https://github.com/Mikubill/sd-webui-controlnet) now! Just update your sd-webui-controlnet >= v1.1237, then choose `dw_openpose_full` as preprocessor. - You can try our DWPose with this [Demo](https://openxlab.org.cn/apps/detail/mmpose/RTMPose) by choosing `wholebody`! - Jul. 2023: - Add [Online RTMPose Demo](https://openxlab.org.cn/apps/detail/mmpose/RTMPose). - Support 17-keypoint Body models trained on Human-Art. - Jun. 2023: - Release 26-keypoint Body models trained on combined datasets. - May. 2023: - Exported SDK models (ONNX, TRT, ncnn, etc.) can be downloaded from [OpenMMLab Deploee](https://platform.openmmlab.com/deploee). - [Online Conversion](https://platform.openmmlab.com/deploee/task-convert-list) of `.pth` models into SDK models (ONNX, TensorRT, ncnn, etc.). - Add [code examples](./examples/) of RTMPose, such as: - Pure Python inference without MMDeploy, MMCV etc. - C++ examples with ONNXRuntime and TensorRT backends. - Android examples with ncnn backend. - Release Hand, Face, Body models trained on combined datasets. - Mar. 2023: RTMPose is released. RTMPose-m runs at 430+ FPS and achieves 75.8 mAP on COCO val set. ## 📖 Introduction [🔝](#-table-of-contents) <div align=center> <img src="https://user-images.githubusercontent.com/13503330/221138554-110240d8-e887-4b9a-90b1-2fbdc982e9de.gif" width=400 height=300/><img src="https://user-images.githubusercontent.com/13503330/221125176-85015a13-9648-4f0d-a17c-1cbb469efacf.gif" width=250 height=300/><img src="https://user-images.githubusercontent.com/13503330/221125310-7eeb2212-907e-427f-97af-af799d70a4c5.gif" width=250 height=300/> </div> <div align=center> <img src="https://github.com/open-mmlab/mmpose/assets/13503330/38aa345e-4ceb-4e73-bc37-5e082735e336" width=450 height=300/><img src="https://user-images.githubusercontent.com/13503330/221125888-15c20faf-0ad5-4afb-828b-a71ccb064582.gif" width=450 height=300/> </div> <div align=center> <img src="https://github.com/open-mmlab/mmpose/assets/13503330/2ecbf9f4-6963-4a14-9801-da10c0a65dac" width=300 height=350/><img src="https://user-images.githubusercontent.com/13503330/221138017-10431ab4-e515-4c32-8fa7-8748e2d17a58.gif" width=600 height=350/> </div> ### ✨ Major Features - 🚀 **High efficiency and high accuracy** | Model | AP(COCO) | CPU-FPS | GPU-FPS | | :---: | :------: | :-----: | :-----: | | t | 68.5 | 300+ | 940+ | | s | 72.2 | 200+ | 710+ | | m | 75.8 | 90+ | 430+ | | l | 76.5 | 50+ | 280+ | | l-384 | 78.3 | - | 160+ | - 🛠️ **Easy to deploy** - Step-by-step deployment tutorials. - Support various backends including - ONNX - TensorRT - ncnn - OpenVINO - etc. - Support various platforms including - Linux - Windows - NVIDIA Jetson - ARM - etc. - 🏗️ **Design for practical applications** - Pipeline inference API and SDK for - Python - C++ - C# - JAVA - etc. ## 🙌 Community [🔝](#-table-of-contents) RTMPose is a long-term project dedicated to the training, optimization and deployment of high-performance real-time pose estimation algorithms in practical scenarios, so we are looking forward to the power from the community. Welcome to share the training configurations and tricks based on RTMPose in different business applications to help more community users! ✨ ✨ ✨ - **If you are a new user of RTMPose, we eagerly hope you can fill out this [Google Questionnaire](https://docs.google.com/forms/d/e/1FAIpQLSfzwWr3eNlDzhU98qzk2Eph44Zio6hi5r0iSwfO9wSARkHdWg/viewform?usp=sf_link)/[Chinese version](https://uua478.fanqier.cn/f/xxmynrki), it's very important for our work!** ✨ ✨ ✨ Feel free to join our community group for more help: - WeChat Group: <div align=left> <img src="https://user-images.githubusercontent.com/13503330/222647056-875bed70-85ec-455c-9016-c024772915c4.jpg" width=200 /> </div> - Discord Group: - 🙌 https://discord.gg/raweFPmdzG 🙌 ## ⚡ Pipeline Performance [🔝](#-table-of-contents) **Notes** - Pipeline latency is tested under skip-frame settings, the detection interval is 5 frames by defaults. - Flip test is NOT used. - Env Setup: - torch >= 1.7.1 - onnxruntime 1.12.1 - TensorRT 8.4.3.1 - ncnn 20221128 - cuDNN 8.3.2 - CUDA 11.3 - **Updates**: We recommend you to try `Body8` models trained on combined datasets, see [here](#body-2d). | Detection Config | Pose Config | Input Size<sup><br>(Det/Pose) | Model AP<sup><br>(COCO) | Pipeline AP<sup><br>(COCO) | Params (M)<sup><br>(Det/Pose) | Flops (G)<sup><br>(Det/Pose) | ORT-Latency(ms)<sup><br>(i7-11700) | TRT-FP16-Latency(ms)<sup><br>(GTX 1660Ti) |