Skip to content

rpetit3/fastq-scan

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

43 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

install with bioconda Docker Repository on Quay.io Anaconda-Server Badge

fastq-scan reads a FASTQ from STDIN and outputs summary statistics (read lengths, per-read qualities, per-base qualities) in JSON format.

fastq-scan

I wanted a quick method to output simple summary statistics of an input FASTQ in JSON format. There are (likely better) alternatives including FastQC and seqtk's fqchk, but they didn't ouput JSON which I wanted. After a brief Google search, I stumbled upon the Biostars question: "How To Efficiently Parse A Huge Fastq File?". From this question I used code from Pierre Lindenbaum's C++ solution as the base for this program.

Installation

Bioconda

fastq-scan is availble on BioConda.

conda install -c bioconda fastq-scan

From Source

git clone git@github.com:rpetit3/fastq-scan.git
cd fastq-scan
make

This will compile the program using g++, and I'll let you handle where to put it. I've testing this on gcc version 7.3.0, but would guess since its pretty basic most gcc versions should be fine. You can then run make test to make sure everything is working as expected.

make test
./fastq-scan -h
Usage: cat FASTQ | fastq-scan [options]
Version: 0.3

Optional arguments:
    -g INT   Genome size for calculating estimated sequencing coverage. (Default 1)
    -p INT   ASCII offset for input quality scores, can be 33 or 64. (Default 33)
    -v       Print version information and exit
    -h       Show this message and exit

cat example.fq | ./fastq-scan -g 150000
{
    "qc_stats": {
        "total_bp":7500,
        "coverage":0.05,
        "read_total":75,
        "read_min":100,
        "read_mean":100,
        "read_std":0,
        "read_median":100,
        "read_max":100,
        "read_25th":100,
        "read_75th":100,
        "qual_mean":34.0267,
        "qual_std":0.711306,
        "qual_median":34,
        "qual_25th":34,
        "qual_75th":34
    },
    "read_lengths": {

        "100":75
    },
    "per_base_quality": {
        "1":30.7467,        "2":31.5467,        "3":31.5467,        "4":35.44,        "5":34.24,
        "6":34.12,        "7":34.7067,        "8":34.24,        "9":36.9333,        "10":37.0667,
        "11":35.88,        "12":36.0667,        "13":36.72,        "14":38.2667,        "15":37.48,
        "16":38.2133,        "17":36.7467,        "18":37.8267,        "19":36.3333,        "20":37.2933,
        "21":37.9867,        "22":37.1067,        "23":37.4133,        "24":38.2667,        "25":36.6133,
        "26":36.2,        "27":36.3067,        "28":35.8533,        "29":36.5067,        "30":37.72,
        "31":37.3333,        "32":36.0133,        "33":37.4933,        "34":36.1067,        "35":36.76,
        "36":34.8533,        "37":36.3733,        "38":35.1867,        "39":36.0133,        "40":35.3067,
        "41":35.6,        "42":36.7867,        "43":35.52,        "44":37.3333,        "45":36.6533,
        "46":36.8,        "47":35.9867,        "48":35.4533,        "49":35.2,        "50":37.2533,
        "51":35.04,        "52":36,        "53":35.28,        "54":36.16,        "55":35.2,
        "56":33.6133,        "57":36.0533,        "58":34.4533,        "59":35.88,        "60":35.3733,
        "61":35.6933,        "62":34.8267,        "63":35.1067,        "64":35.2933,        "65":32.2667,
        "66":34.4267,        "67":33.9333,        "68":33.6667,        "69":32.6133,        "70":33.4267,
        "71":32.8267,        "72":32.96,        "73":33.5467,        "74":33.1067,        "75":31.8667,
        "76":30.72,        "77":30.6133,        "78":30.2133,        "79":31.7467,        "80":33.8933,
        "81":32.72,        "82":33.1733,        "83":31.5867,        "84":32.6933,        "85":32.0667,
        "86":32.2933,        "87":30.7467,        "88":30.6933,        "89":32.48,        "90":31.08,
        "91":31.6133,        "92":31.72,        "93":30.3867,        "94":30.7067,        "95":29.9733,
        "96":31.96,        "97":32.44,        "98":30.2267,        "99":31.2533,        "100":30.2267
    }
}

Example Usage

fastq-scan reads from STDIN, so pretty much any FASTQ output can be piped into fastq-scan. There are a few things to be aware of. I've assumed that all FASTQ entries are the four line variant, which should be a safe assumption in 2018. Also, I have a PHRED offset (33 vs 64) guesser function. By default it is set to PHRED33, it could produce errors if there are not any PHRED33 or PHRED64 specific characters in the quality scores.

Usage

./fastq-scan
Usage: cat FASTQ | fastq-scan [options]
Version: 0.3

Optional arguments:
    -g INT   Genome size for calculating estimated sequencing coverage. (Default 1)
    -p INT   ASCII offset for input quality scores, can be 33 or 64. (Default 33)
    -v       Print version information and exit
    -h       Show this message and exit

-g Genome Size

This is an optional parameter that you can use to estimate the sequencing coverage as calulated by (TOTAL_BP / GENOME_SIZE). By default, the genome size is set to 1. If a genome size is not given the total coverage will not be calculated and instead be set to 0.00 in the final JSON output.

-p ASCII Offset for Quality Scores

This optional parameter can be used to explicitely state the ASCII offset for the input quality scores. It defaults to PHRED+33 scores. Only 33 or 64 are valid inputs for this parameter.

-v Version

./fastq-scan -v
fastq-scan 0.3

example.fq

An example FASTQ file, aptly named example-q33.fq (also example-q64.fq), has been included to demonstrate usage of fastq-scan. For those interested this is a small set of simulated reads from the Lotus japonicus (NC_002694) chloroplast genome. The reads were simulated using ART (vMountRainier).

head example-q33.fq
@NC_002694.1-75
TGTATACAATAAGAATCCATTTATTGACAAATTTCATTCGAAAATTATGAAACATAAATTTTTTTTTATTGGATCAAGAATTCCAATTTTTTAAGTATAA
+
@CCDFFF*HDGHFGAIGJJJGIIHIJCHJJ*EJJHG@?GJFJGI-IBFGJ.JHGB.DDFDCEFA?)FC=JCDFF9DAA;E?>DDDDDF5?BCDDDDD><(
@NC_002694.1-74
TTTTTTGGACTTGAAGAAAAAAAAATACAACTTTGCTGACAATTATTTGTTTGGTCAGAAGAGTCCTCCAAATATTCTGATCTTATATTGATTATCATTT
+
CCCFFFFFHGF=>GJEJIJHJHBJ,)IDJGIJFIDFHBA@IGCIIBJGJIJI)JG0JJCJDIGHDIJI(9JDD?=<BHD<DCEDBBDAAD7DDDA;D?ED
@NC_002694.1-73
AAAAAGTGAAATATTCAGTTAATGAATGCCGAATCTCCGCTCTTATTCTATGAACATTTCATAATCCTATAAATTATCTTTATAAATTATCTTATAGAAT

Execution

cat example-q33.fq | ./fastq-scan -g 150000

Example Output

After the FASTQ file is read, simple summary statistics are output in JSON format. Below is the summary statistics of example.fq

{
    "qc_stats": {
        "total_bp":7500,
        "coverage":0.05,
        "read_total":75,
        "read_min":100,
        "read_mean":100,
        "read_std":0,
        "read_median":100,
        "read_max":100,
        "read_25th":100,
        "read_75th":100,
        "qual_mean":34.0267,
        "qual_std":0.711306,
        "qual_median":34,
        "qual_25th":34,
        "qual_75th":34
    },
    "read_lengths": {

        "100":75
    },
    "per_base_quality": {
        "1":30.7467,        "2":31.5467,        "3":31.5467,        "4":35.44,        "5":34.24,
        "6":34.12,        "7":34.7067,        "8":34.24,        "9":36.9333,        "10":37.0667,
        "11":35.88,        "12":36.0667,        "13":36.72,        "14":38.2667,        "15":37.48,
        "16":38.2133,        "17":36.7467,        "18":37.8267,        "19":36.3333,        "20":37.2933,
        "21":37.9867,        "22":37.1067,        "23":37.4133,        "24":38.2667,        "25":36.6133,
        "26":36.2,        "27":36.3067,        "28":35.8533,        "29":36.5067,        "30":37.72,
        "31":37.3333,        "32":36.0133,        "33":37.4933,        "34":36.1067,        "35":36.76,
        "36":34.8533,        "37":36.3733,        "38":35.1867,        "39":36.0133,        "40":35.3067,
        "41":35.6,        "42":36.7867,        "43":35.52,        "44":37.3333,        "45":36.6533,
        "46":36.8,        "47":35.9867,        "48":35.4533,        "49":35.2,        "50":37.2533,
        "51":35.04,        "52":36,        "53":35.28,        "54":36.16,        "55":35.2,
        "56":33.6133,        "57":36.0533,        "58":34.4533,        "59":35.88,        "60":35.3733,
        "61":35.6933,        "62":34.8267,        "63":35.1067,        "64":35.2933,        "65":32.2667,
        "66":34.4267,        "67":33.9333,        "68":33.6667,        "69":32.6133,        "70":33.4267,
        "71":32.8267,        "72":32.96,        "73":33.5467,        "74":33.1067,        "75":31.8667,
        "76":30.72,        "77":30.6133,        "78":30.2133,        "79":31.7467,        "80":33.8933,
        "81":32.72,        "82":33.1733,        "83":31.5867,        "84":32.6933,        "85":32.0667,
        "86":32.2933,        "87":30.7467,        "88":30.6933,        "89":32.48,        "90":31.08,
        "91":31.6133,        "92":31.72,        "93":30.3867,        "94":30.7067,        "95":29.9733,
        "96":31.96,        "97":32.44,        "98":30.2267,        "99":31.2533,        "100":30.2267
    }
}

Naming

Originally this was named fastq-stats, but after a quick Google search (which I didn't do originally!) I found another fastq-stats available from ea-utils. So I decided to rename it to fastq-scan, since its similar to the Scan ability found in some video games/movies/tv etc... Basically it 'scans' a FASTQ and provides the user with otherwise hidden information about the FASTQ reads.