Integration: GATK and Galaxy
SnpEff is integrated with other tools commonly used in sequencing data analysis pipelines. Most notably Galaxy and Broad Institute's Genome Analysis Toolkit (GATK) projects support SnpEff. By using standards, such as VCF, SnpEff makes it easy to integrate with other programs.
Integration: GATK
In order to make sure SnpEff and GATK understand each other, you must activate GATK compatibility in SnpEff by using the -o gatk command line option.
The reason for using '-o gatk' is that, even though both GATK and SnpEff use VCF format, SnpEff has recently updated the EFF sub-field format and this might cause some trouble (since GATK still uses the original version).
Warning
GATK only picks one effect. Indeed, the GATK team decided to only report the effect having the highest impact. This was done intentionally for the sake of brevity, in a 'less is more' spirit. You can get the full effect by using snpEff independently, instead of using it within GATK framework.
Script example: In this example we combine SnpEff and GATK's VariantAnnotator (you can find this script in snpEff/scripts/ directory of the distribution)
#!/bin/sh
#-------------------------------------------------------------------------------
# Files
#-------------------------------------------------------------------------------
in=$1 # Input VCF file
eff=`dirname $in`/`basename $in .vcf`.ann.vcf # SnpEff annotated VCF file
out=`dirname $in`/`basename $in .vcf`.gatk.vcf # Output VCF file (annotated by GATK)
ref=$HOME/snpEff/data/genomes/hg19.fa # Reference genome file
dict=`dirname $ref`/`basename $ref .fa`.dict # Reference genome: Dictionary file
#-------------------------------------------------------------------------------
# Path to programs and libraries
#-------------------------------------------------------------------------------
gatk=$HOME/tools/gatk/GenomeAnalysisTK.jar
picard=$HOME/tools/picard/
snpeff=$HOME/snpEff/snpEff.jar
#-------------------------------------------------------------------------------
# Main
#-------------------------------------------------------------------------------
# Create genome index file
echo
echo "Indexing Genome reference FASTA file: $ref"
samtools faidx $ref
# Create dictionary
echo
echo "Creating Genome reference dictionary file: $dict"
java -jar $picard/CreateSequenceDictionary.jar R= $ref O= $dict
# Annotate
echo
echo "Annotate using SnpEff"
echo " Input file : $in"
echo " Output file : $eff"
java -Xmx8g -jar $snpeff -c $HOME/snpEff/snpEff.config -v -o gatk hg19 $in > $eff
# Use GATK
echo
echo "Annotating using GATK's VariantAnnotator:"
echo " Input file : $in"
echo " Output file : $out"
java -Xmx8g -jar $gatk \
-T VariantAnnotator \
-R $ref \
-A SnpEff \
--variant $in \
--snpEffFile $eff \
-L $in \
-o $out
Warning
Important: In order for this to work, GATK requires that the Genome Reference file should have the chromosomes in karotyping order (largest to smallest chromosomes, followed by the X, Y, and MT). Your VCF file should also respect that order.
Now we can use the script:
$ ~/snpEff/scripts/gatk.sh zzz.vcf
Indexing Genome reference FASTA file: /home/pcingola/snpEff/data/genomes/hg19.fa
Creating Genome reference dictionary file: /home/pcingola/snpEff/data/genomes/hg19.dict
[Fri Apr 12 11:23:12 EDT 2013] net.sf.picard.sam.CreateSequenceDictionary REFERENCE=/home/pcingola/snpEff/data/genomes/hg19.fa OUTPUT=/home/pcingola/snpEff/data/genomes/hg19.dict TRUNCATE_NAMES_AT_WHITESPACE=true NUM_SEQUENCES=2147483647 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
[Fri Apr 12 11:23:12 EDT 2013] Executing as pcingola@localhost.localdomain on Linux 3.6.11-4.fc16.x86_64 amd64; OpenJDK 64-Bit Server VM 1.6.0_24-b24; Picard version: 1.89(1408)
[Fri Apr 12 11:23:12 EDT 2013] net.sf.picard.sam.CreateSequenceDictionary done. Elapsed time: 0.00 minutes.
Runtime.totalMemory()=141164544
To get help, see http://picard.sourceforge.net/index.shtml#GettingHelp
Exception in thread "main" net.sf.picard.PicardException: /home/pcingola/snpEff/data/genomes/hg19.dict already exists. Delete this file and try again, or specify a different output file.
at net.sf.picard.sam.CreateSequenceDictionary.doWork(CreateSequenceDictionary.java:114)
at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:177)
at net.sf.picard.sam.CreateSequenceDictionary.main(CreateSequenceDictionary.java:93)
Annotate using SnpEff
Input file : zzz.vcf
Output file : ./zzz.ann.vcf
00:00:00.000 Reading configuration file '/home/pcingola/snpEff/snpEff.config'
00:00:00.173 done
00:00:00.173 Reading database for genome version 'hg19' from file '/home/pcingola//snpEff/data/hg19/snpEffectPredictor.bin' (this might take a while)
00:00:11.860 done
00:00:11.885 Building interval forest
00:00:17.755 done.
00:00:18.391 Genome stats :
# Genome name : 'Homo_sapiens (USCS)'
# Genome version : 'hg19'
# Has protein coding info : true
# Genes : 25933
# Protein coding genes : 20652
# Transcripts : 44253
# Avg. transcripts per gene : 1.71
# Protein coding transcripts : 36332
# Cds : 365442
# Exons : 429543
# Exons with sequence : 409789
# Exons without sequence : 19754
# Avg. exons per transcript : 9.71
# Number of chromosomes : 50
# Chromosomes names [sizes] : '1' [249250621] '2' [243199373] '3' [198022430] '4' [191154276] '5' [180915260] '6' [171115067] '7' [159138663] 'X' [155270560] '8' [146364022] '9' [141213431] '10' [135534747] '11' [135006516] '12' [133851895] '13' [115169878] '14' [107349540] '15' [102531392] '16' [90354753] '17' [81195210] '18' [78077248] '20' [63025520] 'Y' [59373566] '19' [59128983] '22' [51304566] '21' [48129895] '6_ssto_hap7' [4905564] '6_mcf_hap5' [4764535] '6_cox_hap2' [4734611] '6_mann_hap4' [4679971] '6_qbl_hap6' [4609904] '6_dbb_hap3' [4572120] '6_apd_hap1' [4383650] '17_ctg5_hap1' [1574839] '4_ctg9_hap1' [582546] 'Un_gl000220' [156152] '19_gl000209_random' [145745] 'Un_gl000213' [139339] '17_gl000205_random' [119732] 'Un_gl000223' [119730] '4_gl000194_random' [115071] 'Un_gl000228' [114676] 'Un_gl000219' [99642] 'Un_gl000218' [97454] 'Un_gl000211' [93165] 'Un_gl000222' [89310] '4_gl000193_random' [88375] '7_gl000195_random' [86719] '1_gl000192_random' [79327] 'Un_gl000212' [60768] '1_gl000191_random' [50281] 'M' [16571]
00:00:18.391 Predicting variants
00:00:20.267 Creating summary file: snpEff_summary.html
00:00:20.847 Creating genes file: snpEff_genes.txt
00:00:25.026 done.
00:00:25.036 Logging
00:00:26.037 Checking for updates...
Annotating using GATK's VariantAnnotator:
Input file : zzz.vcf
Output file : ./zzz.gatk.vcf
INFO 11:23:41,316 ArgumentTypeDescriptor - Dynamically determined type of zzz.vcf to be VCF
INFO 11:23:41,343 HelpFormatter - --------------------------------------------------------------------------------
INFO 11:23:41,344 HelpFormatter - The Genome Analysis Toolkit (GATK) v2.4-9-g532efad, Compiled 2013/03/19 07:35:36
INFO 11:23:41,344 HelpFormatter - Copyright (c) 2010 The Broad Institute
INFO 11:23:41,344 HelpFormatter - For support and documentation go to http://www.broadinstitute.org/gatk
INFO 11:23:41,347 HelpFormatter - Program Args: -T VariantAnnotator -R /home/pcingola/snpEff/data/genomes/hg19.fa -A SnpEff --variant zzz.vcf --snpEffFile ./zzz.ann.vcf -L zzz.vcf -o ./zzz.gatk.vcf
INFO 11:23:41,347 HelpFormatter - Date/Time: 2013/04/12 11:23:41
INFO 11:23:41,348 HelpFormatter - --------------------------------------------------------------------------------
INFO 11:23:41,348 HelpFormatter - --------------------------------------------------------------------------------
INFO 11:23:41,353 ArgumentTypeDescriptor - Dynamically determined type of zzz.vcf to be VCF
INFO 11:23:41,356 ArgumentTypeDescriptor - Dynamically determined type of ./zzz.ann.vcf to be VCF
INFO 11:23:41,399 GenomeAnalysisEngine - Strictness is SILENT
INFO 11:23:41,466 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 1000
INFO 11:23:41,480 RMDTrackBuilder - Loading Tribble index from disk for file zzz.vcf
INFO 11:23:41,503 RMDTrackBuilder - Loading Tribble index from disk for file ./zzz.ann.vcf
WARN 11:23:41,505 RMDTrackBuilder - Index file /data/pcingola/Documents/projects/snpEff/gatk_test/./zzz.ann.vcf.idx is out of date (index older than input file), deleting and updating the index file
INFO 11:23:41,506 RMDTrackBuilder - Creating Tribble index in memory for file ./zzz.ann.vcf
INFO 11:23:41,914 RMDTrackBuilder - Writing Tribble index to disk for file /data/pcingola/Documents/projects/snpEff/gatk_test/./zzz.ann.vcf.idx
INFO 11:23:42,076 IntervalUtils - Processing 33411 bp from intervals
INFO 11:23:42,125 GenomeAnalysisEngine - Creating shard strategy for 0 BAM files
INFO 11:23:42,134 GenomeAnalysisEngine - Done creating shard strategy
INFO 11:23:42,134 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING]
INFO 11:23:42,135 ProgressMeter - Location processed.sites runtime per.1M.sites completed total.runtime remaining
INFO 11:23:49,268 VariantAnnotator - Processed 9966 loci.
INFO 11:23:49,280 ProgressMeter - done 3.34e+04 7.0 s 3.6 m 100.0% 7.0 s 0.0 s
INFO 11:23:49,280 ProgressMeter - Total runtime 7.15 secs, 0.12 min, 0.00 hours
INFO 11:23:49,953 GATKRunReport - Uploaded run statistics report to AWS S3
Integration: Galaxy
In order to install SnpEff in your own Galaxy server, you can use the galaxy/*.xml files provided in the main distribution.
This is a screen capture from a Galaxy server (click to enlarge):
Installing SnpEff in a Galaxy server:
# Set variable to snpEff install dir (we only use it for this install script)
export snpEffDir="$HOME/snpEff"
# Go to your galaxy 'tools' dir
cd galaxy-dist/tools
# Create a directory and copy the XML config files from SnpEff's distribution
mkdir snpEff
cd snpEff/
cp $snpEffDir/galaxy/* .
# Create links to JAR files
ln -s $snpEffDir/snpEff.jar
ln -s $snpEffDir/SnpSift.jar
# Link to config file
ln -s $snpEffDir/snpEff.config
# Allow scripts execution
chmod a+x *.{pl,sh}
# Copy genomes information
cd ../..
cp $snpEffDir/galaxy/tool-data/snpEff_genomes.loc tool-data/
# Edit Galaxy's tool_conf.xml and add all the tools
vi tool_conf.xml
-------------------- Begin: Edit tool_conf.xml --------------------
<!--
Add this section to tool_conf.xml file in your galaxy distribution
Note: The following lines should be added at the end of the
file, right before "</toolbox>" line
-->
<section name="snpEff tools" id="snpEff_tools">
<tool file="snpEff/snpEff.xml" />
<tool file="snpEff/snpEff_download.xml" />
<tool file="snpEff/snpSift_annotate.xml" />
<tool file="snpEff/snpSift_caseControl.xml" />
<tool file="snpEff/snpSift_filter.xml" />
<tool file="snpEff/snpSift_int.xml" />
</section>
-------------------- End: Edit tool_conf.xml --------------------
# Run galaxy and check that the new menus appear
./run.sh