Results obtained by running the SMaSH benchmarking toolkit on various variant calling pipelines.
If you've run SMaSH, we want to see your results! Let us know at our support forum.
| Dataset | Instance Type | Cost per Hour | Aligner | Caller | Aligner Hours | Caller Hours | Total Hours | Total Cost | Precision | Recall |
| Venter | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 60 h | 7 h | 67 h | $168 | 98.7 ± 0.0 | 97.0 ± 0.0 |
| Venter | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 60 h | 26 h | 86 h | $215 | 99.3 ± 0.0 | 91.7 ± 0.0 |
| contam. Venter | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 23 h | 3 h | 26 h | $65 | 91.3 ± 0.0 | 96.7 ± 0.0 |
| contam. Venter | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 23 h | 53 h | 76 h | $190 | 98.7 ± 0.0 | 91.8 ± 0.0 |
| NA12878 | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 52 h | 5 h | 57 h | $142 | - | 98.8 ± 0.0 |
| NA12878 | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 52 h | 73 h | 125 h | $312 | - | 98.8 ± 0.0 |
| contam. NA12878 | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 45 h | 5 h | 50 h | $125 | - | 98.8 ± 0.0 |
| contam. NA12878 | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 45 h | 91 h | 136 h | $340 | - | 98.8 ± 0.0 |
| NA18507 | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 37 h | 3 h | 40 h | $100 | - | 99.0 ± 0.0 |
| NA18507 | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 37 h | 77 h | 114 h | $285 | - | 99.0 ± 0.0 |
| NA19240 | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 36 h | 12 h | 48 h | $120 | - | 98.7 ± 0.0 |
| NA19240 | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 36 h | 71 h | 107 h | $268 | - | 99.0 ± 0.0 |
| mouse | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 106 h | 9 h | 115 h | $288 | 98.4 ± 0.2 | 87.3 ± 0.2 |
| mouse | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 106 h | 112 h | 218 h | $545 | 97.8 ± 0.2 | 94.9 ± 0.2 |
| contam. mouse | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 77 h | 6 h | 83 h | $208 | 98.3 ± 0.2 | 86.7 ± 0.2 |
| contam. mouse | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 77 h | 62 h | 139 h | $348 | 97.9 ± 0.2 | 94.6 ± 0.2 |
| Dataset | Instance Type | Cost per Hour | Aligner | Caller | Aligner Hours | Caller Hours | Total Hours | Total Cost | Precision | Recall |
| Venter | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 60 h | 7 h | 67 h | $168 | 91.3 ± 0.0 | 97.0 ± 0.0 |
| Venter | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 60 h | 26 h | 86 h | $215 | 92.3 ± 0.0 | 90.9 ± 0.0 |
| Venter | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 60 h | 61 h | 121 h | $302 | 94.0 ± 0.0 | 79.8 ± 0.0 |
| contam. Venter | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 23 h | 3 h | 26 h | $65 | 91.7 ± 0.0 | 96.7 ± 0.0 |
| contam. Venter | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 23 h | 53 h | 76 h | $190 | 92.4 ± 0.0 | 90.5 ± 0.0 |
| mouse | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 106 h | 9 h | 115 h | $288 | 79.0 ± 0.4 | 87.3 ± 0.2 |
| mouse | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 106 h | 112 h | 218 h | $545 | 81.5 ± 0.4 | 95.8 ± 0.4 |
| contam. mouse | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 77 h | 6 h | 83 h | $208 | 80.4 ± 0.4 | 86.7 ± 0.2 |
| contam. mouse | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 77 h | 62 h | 139 h | $348 | 82.8 ± 0.4 | 95.6 ± 0.4 |
| contam. mouse | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 77 h | 345 h | 422 h | $1055 | 85.9 ± 0.4 | 81.3 ± 0.4 |
| Dataset | Instance Type | Cost per Hour | Aligner | Caller | Aligner Hours | Caller Hours | Total Hours | Total Cost | Precision | Recall |
| Venter | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 60 h | 7 h | 67 h | $168 | 85.8 ± 0.0 | 97.0 ± 0.0 |
| Venter | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 60 h | 26 h | 86 h | $215 | 90.6 ± 0.0 | 88.1 ± 0.0 |
| Venter | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 60 h | 61 h | 121 h | $302 | 92.5 ± 0.0 | 72.9 ± 0.0 |
| contam. Venter | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 23 h | 3 h | 26 h | $65 | 87.8 ± 0.0 | 96.7 ± 0.0 |
| contam. Venter | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 23 h | 53 h | 76 h | $190 | 90.9 ± 0.0 | 87.5 ± 0.0 |
| mouse | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 106 h | 9 h | 115 h | $288 | 87.8 ± 0.4 | 87.3 ± 0.2 |
| mouse | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 106 h | 112 h | 218 h | $545 | 91.0 ± 0.4 | 91.5 ± 0.4 |
| contam. mouse | cc2.8xlarge | $2.50 | BWA 6.1 | mpileup | 77 h | 6 h | 83 h | $208 | 88.6 ± 0.5 | 86.7 ± 0.2 |
| contam. mouse | cc2.8xlarge | $2.50 | BWA 6.1 | GATK 2.7 | 77 h | 62 h | 139 h | $348 | 91.1 ± 0.4 | 90.9 ± 0.4 |
| contam. mouse | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 77 h | 345 h | 422 h | $1055 | 92.4 ± 0.5 | 71.7 ± 0.4 |
| Dataset | Instance Type | Cost per Hour | Aligner | Caller | Aligner Hours | Caller Hours | Total Hours | Total Cost | Precision | Recall |
| Venter | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 60 h | 61 h | 121 h | $302 | 14.2 ± 0.0 | 43.3 ± 0.0 |
| Venter | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 60 h | 2 h | 62 h | $155 | - | 4.9 ± 0.0 |
| contam. Venter | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 23 h | 1 h | 24 h | $60 | - | 0.0 ± 0.0 |
| NA12878 | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 52 h | 91 h | 143 h | $358 | - | 78.3 ± 2.9 |
| NA12878 | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 52 h | 4 h | 56 h | $140 | 0.3 ± 0.0 | 95.7 ± 2.9 |
| NA18507 | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 37 h | 150 h | 187 h | $468 | - | 60.0 ± 3.0 |
| NA18507 | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 37 h | 2 h | 39 h | $98 | 0.0 ± 0.0 | 60.0 ± 3.0 |
| NA19240 | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 36 h | 221 h | 257 h | $642 | - | 70.7 ± 2.8 |
| NA19240 | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 36 h | 31 h | 67 h | $168 | 0.5 ± 0.0 | 77.6 ± 2.8 |
| mouse | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 106 h | 4 h | 110 h | $275 | - | 11.6 ± 0.8 |
| contam. mouse | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 77 h | 345 h | 422 h | $1055 | 6.6 ± 4.8 | 53.4 ± 0.8 |
| contam. mouse | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 77 h | 1 h | 78 h | $195 | - | 1.0 ± 0.8 |
| Dataset | Instance Type | Cost per Hour | Aligner | Caller | Aligner Hours | Caller Hours | Total Hours | Total Cost | Precision | Recall |
| Venter | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 60 h | 61 h | 121 h | $302 | 24.8 ± 0.0 | 10.2 ± 0.0 |
| Venter | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 60 h | 2 h | 62 h | $155 | - | 0.0 ± 0.0 |
| contam. Venter | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 23 h | 1 h | 24 h | $60 | - | 0.0 ± 0.0 |
| NA12878 | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 52 h | 91 h | 143 h | $358 | - | 0.0 ± 3.0 |
| NA12878 | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 52 h | 4 h | 56 h | $140 | 0.0 ± 0.0 | 0.0 ± 3.0 |
| NA18507 | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 37 h | 150 h | 187 h | $468 | - | 0.0 ± 5.0 |
| NA18507 | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 37 h | 2 h | 39 h | $98 | 0.0 ± 1.7 | 0.0 ± 5.0 |
| NA19240 | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 36 h | 221 h | 257 h | $642 | - | 0.0 ± 2.5 |
| NA19240 | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 36 h | 31 h | 67 h | $168 | 0.0 ± 0.0 | 0.0 ± 2.5 |
| mouse | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 106 h | 4 h | 110 h | $275 | - | 0.0 ± 0.5 |
| contam. mouse | cc2.8xlarge | $2.50 | BWA 6.1 | Pindel | 77 h | 345 h | 422 h | $1055 | 8.2 ± 82.7 | 1.3 ± 0.5 |
| contam. mouse | cc2.8xlarge | $2.50 | BWA 6.1 | Breakdancer | 77 h | 1 h | 78 h | $195 | - | 0.0 ± 0.5 |