We at Deep Breathe sought to train a deep learning model for the task of automating the distinction between normal and abnormal lung tissue on lung ultrasound.

This repository contains work relating to development and validation of an A-line vs. B-line ultrasound image classifier that was used for the creation of the paper A deep learning solution for the classification of normal versus abnormal lung parenchyma on lung ultrasound: a multicentre study (a link will be added upon publication).

1. Getting Started
2. Building a Dataset
3. Use Cases
   i)Train Single Experiment
   ii) K-Fold Cross Validation
   iii) Hyper Parameter Optimization
   iv) Predictions
   v) Grad-CAM for Individual Frame Predictions
4. Project Configuration
5. Project Structure
6. Contacts

1. Clone this repository (for help see this tutorial).
2. Install the necessary dependencies (listed in requirements.txt) and ensure that your python version is 3.8. To do this, open a terminal in the root directory of the project and run the following:

   $ pip install -r requirements.txt
   

3. Obtain lung ultrasound data and preprocess it accordingly. See building a dataset for more details.
4. Update the TRAIN >> MODEL_DEF field of config.yml with the appropriate string representing the model type you wish to train. To train a model, ensure the TRAIN >> EXPERIMENT_TYPE field is set to 'train_single'.
5. Execute train.py to train your chosen model on your preprocessed data. The trained model will be serialized within results/models/, and its filename will resemble the following structure: model{yyyymmdd-hhmmss}.h5, where {yyyymmdd-hhmmss} is the current time.
   Note: When train.py is executed, the project root should be the working directory.
6. Navigate to results/logs/ to see the tensorboard log files. The folder name will be {yyyymmdd-hhmmss}. These logs can be used to create a tensorboard visualization of the training results.

Create a database_config.yml file in the root directory as follows:

USERNAME: (database user name)
PASSWORD: (database password)
HOST: (database host name)
DATABASE: data

To build a dataset run the following command:

python src/data/ab_line_dataset_creator.py

This runs a series of automated steps using the clips query specified. Note that the raw clips were scrubbed of all on-screen information (e.g. vendor logos, battery indicators, index mark, depth markers) extraneous to the ultrasound beam itself. This was done using a dedicated deep learning masking software for ultrasound (AutoMask, WaveBase Inc., Waterloo, Canada). Following this, all ultrasound clips were deconstructed into their constituent frames, and a frame table was generated linking each frame to their ground truth, associated clip, and patient id.

The following intermediate csv headers are also extracted to create the final frames table csv file to train a model:

| patient_id | a_or_b_lines | class |

Where filename is the name of the labeled clip file, patient_id is a unique patient identifier, a_or_b_lines is a string label for the clip, and class is the label as a class integer.

With a pre-processed clip dataset, you can train a frame classification model of a chosen model definition.

1. Assemble in a pre-processed clip dataset (see Building a Dataset) and set the appropriate data paths in config.yml.
2. Mask the images of extraneous information outside the ultrasound beam. In our group, this was done with proprietary software mentioned above in 'Building a Dataset'
3. Generate a frame dataset from the masked clips using build-dataset.py.
4. Set the desired data and train configuration fields in config.yml including setting a model type using the MODEL_DEF parameter and setting the EXPERIMENT_TYPE to single_train.
5. Set the associated hyperparameter values based on the chosen model definition.
6. Run train.py.
7. View all logs and trained weights in the results directory.

Note: We found that the cutoffvgg16 model definition had the best performance on our internal data.

With a pre-processed clip dataset, you can evaluate model performance using k-fold cross-validation.

1. Assemble in a pre-processed clip dataset (see Building a Dataset) and set the appropriate data paths in config.yml.
2. Generate a frame from the masked clips using build-dataset.py.
3. Set the desired data and train configuration fields in config.yml including setting a model type using the MODEL_DEF parameter and setting the EXPERIMENT_TYPE to cross_validation.
4. Set the number of folds in the train section of config.yml.
5. Set the associated hyperparameter values based on the chosen model definition.
6. Run train.py.
7. View all logs and trained weights in the results directory. The partitions from each fold can be found in the partitions folder.

With a pre-processed clip dataset, you can perform a hyperparameter search to assist with hyperparameter optimization.

1. Assemble in a pre-processed clip dataset (see Building a Dataset) and set the appropriate data paths in config.yml.
2. Generate a frame dataset from the masked clips using build-dataset.py.
3. Set the desired data and train configuration field in config.yml including setting a model type using the MODEL_DEF parameter and setting the EXPERIMENT_TYPE to hparam_search.
4. Set the hyperparameter search fields in the train section of config.yml.
5. Set the associated hyperparameter search configuration values based on the chosen model definition.
6. Run train.py.
7. View all logs in the logs folder and view bayesian hyperparameter search results in the experiments folder.

With a trained model, you can compute frame predictions and clip predictions using the following steps:

1. Set the MODEL_TO_LOAD field in config.yml to point to a trained model (in .h5 format).
2. Set the FRAME_TABLE and CLIPS_TABLE fields to the dataset of interest. Set the FRAMES field to point to the dataset's directory of LUS frames.
3. Set the CLIP_PREDICTION > ALGORITHM field to determine which algorithm is used to compute clip-wise predictions, given the clip's set of frame predictions produced by the model. Below is a brief description of each algorithm available.
   • "contiguous": If the number of contiguous frames for which the frame's predicted B-line probability meets or exceeds the classification threshold is at least the contiguity threshold, classify the clip as "B-lines".
   • "average": Compute the average prediction probabilities across the entire clip. If the B-line average probability meets or exceeds the classification threshold, classify the clip as "B-lines".
   • "sliding_window": Take the clip's B-line probability as the greatest average B-line probability present in any contiguous set of frames as large as the sliding window.
4. Execute predict.py.
5. Access the frame and corresponding clip predictions as CSV files, located in results/predictions.

With a trained model and a collection of frame data, you can apply a Grad-CAM visualization to individual frames.

1. Set the MODEL_TO_LOAD field in config.yml to point to a trained model (in .h5 format).
2. Set the FRAME_TABLE field and set the FRAMES path field to point to a directory of LUS frames .
3. Run gradcam.py.
4. Select the frame that you want to apply Grad-CAM to.
5. View Grad-CAM results in the img/heatmaps folder.

This project contains several configurable variables that are defined in the project config file: config.yml. When loaded into Python scripts, the contents of this file become a dictionary through which the developer can easily access its members.

For user convenience, the config file is organized into major components of the model development pipeline. Many fields need not be modified by the typical user, but others may be modified to suit the user's specific goals. A summary of the major configurable elements in this file is below.

The project looks similar to the directory structure below. Disregard any .gitkeep files, as their only purpose is to force Git to track empty directories. Disregard any ._init_.py files, as they are empty files that enable Python to recognize certain directories as packages.

├── img
|   ├── experiments                  <- Visualizations for experiments
|   ├── heatmaps                     <- Grad-CAM heatmap images
|   └── readme                       <- Image assets for README.md
├── results
|   ├── data                         
|   |   └── partition                <- K-fold cross-validation fold partitions
|   ├── experiments                  <- Experiment results
|   ├── figures                      <- Generated figures
|   ├── models                       <- Trained model output
|   ├── predictions                  <- Prediction output
│   └── logs                         <- TensorBoard logs
├── src
│   ├── data
|   |   ├── build-dataset.py         <- Builds a table of frame examples using a table of clip metadata
|   |   ├── database_pull.py         <- Script for pulling clip mp4 files from the cloud - step 2 (specific to our setup)
|   |   └── query_to_df.py           <- Script for pulling clip metadata from the cloud - step 1 (specific to our setup)
│   ├── explainability
|   |   └── gradcam.py               <- Script containing gradcam application and heatmap generation
│   ├── models                       
|   |   └── models.py                <- Script containing all model definitions
|   ├── visualization                
|   |   └── visualize.py             <- Script for visualization production
|   ├── predict.py                   <- Script for prediction on raw data using trained models
|   ├── deploy.py                    <- Script for confirming preprocessing for inference on the WaveBase device  
|   └── train.py                     <- Script for training experiments
|
├── .gitignore                       <- Files to be be ignored by git.
├── config.yml                       <- Values of several constants used throughout project
├── README.md                        <- Project description
└── requirements.txt                 <- Lists all dependencies and their respective versions

Robert Arntfield
Project Lead
Deep Breathe
[email protected]

Blake VanBerlo
Deep Learning Project Lead
Deep Breathe
[email protected]