workflow/ghost-tree_workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ghost-tree workflow to create fungal 18S and ITS hybrid-tree\n",
    "================================\n",
    "This workflow creates a hybrid phylogenetic tree. This can be run in place with no modifications. All of the files needed to generate the tree are publicly available from their respective databases. This workflow is specific to the Silva and UNITE databases, however the commands may be adapted to create hybrid trees from other marker genes.\n",
    "\n",
    "Dependency Versions: \n",
    "------------------\n",
    "ghost-tree git@5f5d5b868fa951cecc7731ecc82f8d2798359c82  \n",
    "SUMACLUST 1.0.01  \n",
    "MUSCLE 3.8.31  \n",
    "FastTree 2.1.8  \n",
    "scikit-bio 0.2.3  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the necesary files UNITE and Silva files\n",
    "-------------------------\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--13:30:03--  http://www.arb-silva.de/fileadmin/silva_databases/release_119/Exports/SILVA_119_SSURef_Nr99_tax_silva_full_align_trunc.fasta.gz\n",
      "           => `SILVA_119_SSURef_Nr99_tax_silva_full_align_trunc.fasta.gz'\n",
      "Resolving www.arb-silva.de... 134.102.40.6\n",
      "Connecting to www.arb-silva.de[134.102.40.6]:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1,113,513,256 [text/plain]\n",
      "\n",
      "100%[==================================>] 1,113,513,256    5.72M/s    ETA 00:00\n",
      "\n",
      "13:34:03 (4.43 MB/s) - `SILVA_119_SSURef_Nr99_tax_silva_full_align_trunc.fasta.gz' saved [1113513256/1113513256]\n",
      "\n",
      "--13:34:03--  http://www.arb-silva.de/fileadmin/silva_databases/release_119/Exports/taxonomy/tax_slv_ssu_nr_119.acc_taxid\n",
      "           => `tax_slv_ssu_nr_119.acc_taxid'\n",
      "Resolving www.arb-silva.de... 134.102.40.6\n",
      "Connecting to www.arb-silva.de[134.102.40.6]:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 11,303,218 [text/plain]\n",
      "\n",
      "100%[====================================>] 11,303,218     2.83M/s    ETA 00:00\n",
      "\n",
      "13:34:09 (1.93 MB/s) - `tax_slv_ssu_nr_119.acc_taxid' saved [11303218/11303218]\n",
      "\n",
      "--13:34:09--  http://www.arb-silva.de/fileadmin/silva_databases/release_119/Exports/taxonomy/tax_slv_ssu_nr_119.txt\n",
      "           => `tax_slv_ssu_nr_119.txt'\n",
      "Resolving www.arb-silva.de... 134.102.40.6\n",
      "Connecting to www.arb-silva.de[134.102.40.6]:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1,193,845 [text/plain]\n",
      "\n",
      "100%[====================================>] 1,193,845    557.81K/s             \n",
      "\n",
      "13:34:12 (556.04 KB/s) - `tax_slv_ssu_nr_119.txt' saved [1193845/1193845]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Silva files\n",
    "!wget 'http://www.arb-silva.de/fileadmin/silva_databases/release_119/Exports/SILVA_119_SSURef_Nr99_tax_silva_full_align_trunc.fasta.gz'\n",
    "!wget 'http://www.arb-silva.de/fileadmin/silva_databases/release_119/Exports/taxonomy/tax_slv_ssu_nr_119.acc_taxid'\n",
    "!wget 'http://www.arb-silva.de/fileadmin/silva_databases/release_119/Exports/taxonomy/tax_slv_ssu_nr_119.txt'\n",
    "!gunzip SILVA_119_SSURef_Nr99_tax_silva_full_align_trunc.fasta.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--13:37:46--  https://github.com/qiime/its-reference-otus/raw/master/taxonomy/97_otu_taxonomy.txt.gz\n",
      "           => `97_otu_taxonomy.txt.gz'\n",
      "Resolving github.com... 192.30.252.128\n",
      "Connecting to github.com[192.30.252.128]:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/qiime/its-reference-otus/master/taxonomy/97_otu_taxonomy.txt.gz [following]\n",
      "--13:37:49--  https://raw.githubusercontent.com/qiime/its-reference-otus/master/taxonomy/97_otu_taxonomy.txt.gz\n",
      "           => `97_otu_taxonomy.txt.gz'\n",
      "Resolving raw.githubusercontent.com... 199.27.74.133\n",
      "Connecting to raw.githubusercontent.com[199.27.74.133]:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 655,626 [application/octet-stream]\n",
      "\n",
      "100%[====================================>] 655,626        2.11M/s             \n",
      "\n",
      "13:37:50 (2.11 MB/s) - `97_otu_taxonomy.txt.gz' saved [655626/655626]\n",
      "\n",
      "--13:37:51--  https://github.com/qiime/its-reference-otus/raw/master/rep_set/97_otus.fasta.gz\n",
      "           => `97_otus.fasta.gz'\n",
      "Resolving github.com... 192.30.252.128\n",
      "Connecting to github.com[192.30.252.128]:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/qiime/its-reference-otus/master/rep_set/97_otus.fasta.gz [following]\n",
      "--13:37:51--  https://raw.githubusercontent.com/qiime/its-reference-otus/master/rep_set/97_otus.fasta.gz\n",
      "           => `97_otus.fasta.gz'\n",
      "Resolving raw.githubusercontent.com... 199.27.74.133\n",
      "Connecting to raw.githubusercontent.com[199.27.74.133]:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 8,075,551 [application/octet-stream]\n",
      "\n",
      "100%[====================================>] 8,075,551      5.07M/s             \n",
      "\n",
      "13:37:54 (5.06 MB/s) - `97_otus.fasta.gz' saved [8075551/8075551]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#UNITE Files\n",
    "!wget 'https://github.com/qiime/its-reference-otus/raw/master/taxonomy/97_otu_taxonomy.txt.gz'\n",
    "!wget 'https://github.com/qiime/its-reference-otus/raw/master/rep_set/97_otus.fasta.gz'\n",
    "!gunzip 97_otu_taxonomy.txt.gz\n",
    "!gunzip 97_otus.fasta.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assign variables to each of the files to be used throughout the workflow\n",
    "------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "silva_aligned = 'SILVA_119_SSURef_Nr99_tax_silva_full_align_trunc.fasta'\n",
    "silva_accession = 'tax_slv_ssu_nr_119.acc_taxid'\n",
    "silva_taxonomy = 'tax_slv_ssu_nr_119.txt'\n",
    "silva_fungi_only = 'silva_fungi_only.txt'\n",
    "silva_fungi_filtered = 'silva_fungi_only_filtered.txt'\n",
    "ITS_seqs = '97_otus.fasta'\n",
    "ITS_otu_map_80 = 'ITS_otu_map_80.txt'\n",
    "ITS_tax = '97_otu_taxonomy.txt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Remove all non-fungal samples from Silva alignment\n",
    "----------------------------------------\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "real\t5m46.302s\r\n",
      "user\t5m26.687s\r\n",
      "sys\t0m14.242s\r\n"
     ]
    }
   ],
   "source": [
    "!time ghost-tree silva extract-fungi $silva_aligned $silva_accession $silva_taxonomy $silva_fungi_only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filter entropy and gap positions from Silva fungal only alignment\n",
    "------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "real\t50m25.698s\r\n",
      "user\t50m15.813s\r\n",
      "sys\t0m6.584s\r\n"
     ]
    }
   ],
   "source": [
    "!time ghost-tree filter-alignment-positions $silva_fungi_only 0.9 0.8 $silva_fungi_filtered"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Group the extension sequences (UNITE sequences clustered at 97%) at 80% identity\n",
    "--------------------------\n",
    "Decreasing the identity creates fewer total clusters with more sequences per cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "===========================================================\n",
      " SUMACLUST version 1.0.01\n",
      " Alignment using SSE2 instructions.\n",
      "===========================================================\n",
      "Reading dataset...\n",
      "55404 sequences\n",
      "Cleaning dataset... : 79472 nucleotides substituted in 6648 sequences\n",
      "Indexing dataset... : Done\n",
      "Sorting sequences by count...\n",
      "Maximum ratio between the counts of two sequences to connect them: 1.000000\n",
      "Clustering sequences when similarity >= 0.800000\n",
      "Aligning and clustering... \n",
      "Done : 100 %       18081 clusters created.                        \n",
      "Printing results in OTU table format...\n",
      "Done.\n",
      "\n",
      "real\t1012m4.275s\n",
      "user\t216m21.857s\n",
      "sys\t0m24.478s\n"
     ]
    }
   ],
   "source": [
    "!time ghost-tree extensions group-extensions $ITS_seqs 0.8 $ITS_otu_map_80"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build the hybrid-tree\n",
    "--------------------\n",
    "Steps involved in ghost-tree scaffold hybrid-tree:\n",
    "1. Build the foundation tree from 18S sequences\n",
    "2. Group ITS extension sequences by cluster  \n",
    "3. Determine consensus taxonomy for each cluster\n",
    "4. Group any clusters with the same consensus taxonomy  \n",
    "5. Align all sequences in each group \n",
    "6. Build a tree for each ITS group\n",
    "7. Graft extension trees to foundation tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `ghost-tree scaffold hybrid-tree` command uses FastTree which ignores non-nucleotide characters. FastTree generates warnings that these characters are being ignored, and while these warnings do not present a problem in the creation of the tree they can slow down the IPython notebok to the point where it crashes. This is not a problem when the command is run directly from the command line. In order to avoid this issue we recommend running the following command directly from the command line.  \n",
    "This is an open issue in a git hub; [issue-#25](https://github.com/JTFouquier/ghost-tree/issues/25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#ghost-tree scaffold hybrid-tree ITS_otu_map_80.txt 97_otu_taxonomy.txt 97_otus.fasta silva_fungi_only_filtered.txt ghost-tree-output2.nwk"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}