深度学习模型优化_基于TensorRT的Caffe_TensorFlow_DarkNet_PyTorch模型加速_支持MODNe...

# tensorrt_demos This repo is a collection of examples demonstrating how to optimize Caffe/TensorFlow/DarkNet/PyTorch models with TensorRT. Highlights: * Run an optimized "MODNet" video matting model at ~21 FPS on Jetson Xavier NX. * Run an optimized "yolov4-416" object detector at ~4.6 FPS on Jetson Nano. * Run an optimized "yolov3-416" object detector at ~4.9 FPS on Jetson Nano. * Run an optimized "ssd_mobilenet_v1_coco" object detector ("trt_ssd_async.py") at 27~28 FPS on Jetson Nano. * Run an optimized "MTCNN" face detector at 6~11 FPS on Jetson Nano. * Run an optimized "GoogLeNet" image classifier at "~16 ms per image (inference only)" on Jetson Nano. Supported hardware: * NVIDIA Jetson - All NVIDIA Jetson Developer Kits, e.g. [Jetson AGX Orin DevKit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/#advanced-features), [Jetson AGX Xavier DevKit](https://developer.nvidia.com/embedded/jetson-agx-xavier-developer-kit), [Jetson Xavier NX DevKit](https://developer.nvidia.com/embedded/jetson-xavier-nx-devkit), Jetson TX2 DevKit, [Jetson Nano DevKit](https://developer.nvidia.com/embedded/jetson-nano-developer-kit). * x86_64 PC with modern NVIDIA GPU(s). Refer to [README_x86.md](https://github.com/jkjung-avt/tensorrt_demos/blob/master/README_x86.md) for more information. Table of contents ----------------- * [Prerequisite](#prerequisite) * [Demo #1: GoogLeNet](#googlenet) * [Demo #2: MTCNN](#mtcnn) * [Demo #3: SSD](#ssd) * [Demo #4: YOLOv3](#yolov3) * [Demo #5: YOLOv4](#yolov4) * [Demo #6: Using INT8 and DLA core](#int8_and_dla) * [Demo #7: MODNet](#modnet) <a name="prerequisite"></a> Prerequisite ------------ The code in this repository was tested on Jetson Nano, TX2, and Xavier NX DevKits. In order to run the demos below, first make sure you have the proper version of image (JetPack) installed on the target Jetson system. For example, [Setting up Jetson Nano: The Basics](https://jkjung-avt.github.io/setting-up-nano/) and [Setting up Jetson Xavier NX](https://jkjung-avt.github.io/setting-up-xavier-nx/). More specifically, the target Jetson system must have TensorRT libraries installed. * Demo #1 and Demo #2: works for TensorRT 3.x+, * Demo #3: requires TensoRT 5.x+, * Demo #4 and Demo #5: requires TensorRT 6.x+. * Demo #6 part 1: INT8 requires TensorRT 6.x+ and only works on GPUs with CUDA compute 6.1+. * Demo #6 part 2: DLA core requires TensorRT 7.x+ (is only tested on Jetson Xavier NX). * Demo #7: requires TensorRT 7.x+. You could check which version of TensorRT has been installed on your Jetson system by looking at file names of the libraries. For example, TensorRT v5.1.6 (JetPack-4.2.2) was present on one of my Jetson Nano DevKits. ```shell $ ls /usr/lib/aarch64-linux-gnu/libnvinfer.so* /usr/lib/aarch64-linux-gnu/libnvinfer.so /usr/lib/aarch64-linux-gnu/libnvinfer.so.5 /usr/lib/aarch64-linux-gnu/libnvinfer.so.5.1.6 ``` Furthermore, all demo programs in this repository require "cv2" (OpenCV) module for python3. You could use the "cv2" module which came in the JetPack. Or, if you'd prefer building your own, refer to [Installing OpenCV 3.4.6 on Jetson Nano](https://jkjung-avt.github.io/opencv-on-nano/) for how to build from source and install opencv-3.4.6 on your Jetson system. If you plan to run Demo #3 (SSD), you'd also need to have "tensorflow-1.x" installed. You could probably use the [official tensorflow wheels provided by NVIDIA](https://docs.nvidia.com/deeplearning/frameworks/pdf/Install-TensorFlow-Jetson-Platform.pdf), or refer to [Building TensorFlow 1.12.2 on Jetson Nano](https://jkjung-avt.github.io/build-tensorflow-1.12.2/) for how to install tensorflow-1.12.2 on the Jetson system. Or if you plan to run Demo #4 and Demo #5, you'd need to have "protobuf" installed. I recommend installing "protobuf-3.8.0" using my [install_protobuf-3.8.0.sh](https://github.com/jkjung-avt/jetson_nano/blob/master/install_protobuf-3.8.0.sh) script. This script would take a couple of hours to finish on a Jetson system. Alternatively, doing `pip3 install` with a recent version of "protobuf" should also work (but might run a little bit slowlier). In case you are setting up a Jetson Nano, TX2 or Xavier NX from scratch to run these demos, you could refer to the following blog posts. * [JetPack-4.6](https://jkjung-avt.github.io/jetpack-4.6/) * [JetPack-4.5](https://jkjung-avt.github.io/jetpack-4.5/) * [Setting up Jetson Xavier NX](https://jkjung-avt.github.io/setting-up-xavier-nx/) * [JetPack-4.4 for Jetson Nano](https://jkjung-avt.github.io/jetpack-4.4/) * [JetPack-4.3 for Jetson Nano](https://jkjung-avt.github.io/jetpack-4.3/) <a name="googlenet"></a> Demo #1: GoogLeNet ------------------ This demo illustrates how to convert a prototxt file and a caffemodel file into a TensorRT engine file, and to classify images with the optimized TensorRT engine. Step-by-step: 1. Clone this repository. ```shell $ cd ${HOME}/project $ git clone https://github.com/jkjung-avt/tensorrt_demos.git $ cd tensorrt_demos ``` 2. Build the TensorRT engine from the pre-trained googlenet (ILSVRC2012) model. Note that I downloaded the pre-trained model files from [BVLC caffe](https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet) and have put a copy of all necessary files in this repository. ```shell $ cd ${HOME}/project/tensorrt_demos/googlenet $ make $ ./create_engine ``` 3. Build the Cython code. Install Cython if not previously installed. ```shell $ sudo pip3 install Cython $ cd ${HOME}/project/tensorrt_demos $ make ``` 4. Run the "trt_googlenet.py" demo program. For example, run the demo using a USB webcam (/dev/video0) as the input. ```shell $ cd ${HOME}/project/tensorrt_demos $ python3 trt_googlenet.py --usb 0 --width 1280 --height 720 ``` Here's a screenshot of the demo (JetPack-4.2.2, i.e. TensorRT 5).  5. The demo program supports 5 different image/video inputs. You could do `python3 trt_googlenet.py --help` to read the help messages. Or more specifically, the following inputs could be specified: * `--image test_image.jpg`: an image file, e.g. jpg or png. * `--video test_video.mp4`: a video file, e.g. mp4 or ts. An optional `--video_looping` flag could be enabled if needed. * `--usb 0`: USB webcam (/dev/video0). * `--rtsp rtsp://admin:[email protected]/live.sdp`: RTSP source, e.g. an IP cam. An optional `--rtsp_latency` argument could be used to adjust the latency setting in this case. * `--onboard 0`: Jetson onboard camera. In additional, you could use `--width` and `--height` to specify the desired input image size, and use `--do_resize` to force resizing of image/video file source. The `--usb`, `--rtsp` and `--onboard` video sources usually produce image frames at 30 FPS. If the TensorRT engine inference code runs faster than that (which happens easily on a x86_64 PC with a good GPU), one particular image could be inferenced multiple times before the next image frame becomes available. This causes problem in the object detector demos, since the original image could have been altered (bounding boxes drawn) and the altered image is taken for inference again. To cope with this problem, use the optional `--copy_frame` flag to force copying/cloning image frames internally. 6. Check out my blog post for implementation details: * [Running TensorRT Optimized GoogLeNet on Jetson Nano](https://jkjung-avt.github.io/tensorrt-googlenet/) <a name="mtcnn"></a> Demo #2: MTCNN -------------- This demo builds upon the previous one. It converts 3 sets of prototxt and caffemodel files into 3 TensorRT engines, namely the PNet, RNet and ONet. Then it combines the 3 engine files to implement MTCNN, a very good face detector. Assuming this repository has been cloned at "${HOME}/project/tens