Skip to content

Latest commit

 

History

History
46 lines (35 loc) · 2.06 KB

Instructions.md

File metadata and controls

46 lines (35 loc) · 2.06 KB
Raw

Instructions for running the TIL Pipeline

The TIL Pipeline [Tumor-Infiltrating Lymphocytes (TIL)] has four phases:

  1. Prediction phase

    • Run svs_2_heatmap.sh, optionally comment out upload_heatmaps.sh in heatmap_gen/start.sh
      • Tiles the svs image into PNGs and saves them to data/patches
      • Runs prediction on the patches, and saves the data as text files to data/patches
        • patch-level-color.txt
        • patch-level-lym.txt
        • patch-level-necrosis.txt
      • Generates heatmap data as JSON files in heatmap_jsons, and corresponding prediction data as text files heatmap_txt

  2. Upload/review/refine

    • Upload data (upload_heatmaps.sh) & ask pathologists to review and refine the visualized data.

  3. Retraining dataset generation phase

    • Run code in download_heatmap; specifically, download_training_patches.sh and/or download_training_tumor_patches.sh
    • It fetches the human-generated markups, generates heatmap weights, and saves the data as text in raw_marking_xy
    • Generates modified heatmaps from the weight information and stores the data as a csv file in modified_heatmaps, along with a visualization stored as PNG
    • Writes training patches as PNGs to patches_from_heatmap

    A. Manual Step

    • Create a new folder in training_data_cnn
    • Copy patches_from_heatmap/* to training_data_cnn/[new_folder]
    • Append [new_folder] to the end of file lym_data_list.txt

  4. Generate new cnn model

    • Run train_models.sh; specifically, training/lymphocyte/start_cnn_lymphocyte_training.sh
    • Generates a cnn_lym_model.pkl file in models_cnn

Setup

During the retraining phase, download_markings_weights.sh reads caseids from raw_marking_to_download_case_list/case_list.txt, so be sure to update the file when you're ready.

Modify conf/variables.sh:

  • MONGODB_HOST
  • MONGODB_PORT
  • BASE_DIR
  • USERNAME
  • HEATMAP_VERSION

Note: Whenever we update our CNN model, we need to assign a new execution id for that. This is a manual step, atm.

Put image file(s) in data/svs.

Install theano in home directory.