a robust python module for fast random access to sequences from plain and gzipped FASTA/Q file

Introduction

The pyfastx is a lightweight Python C extension that enables users to randomly access to sequences from plain and gzipped FASTA/Q files. This module aims to provide simple APIs for users to extract seqeunce from FASTA and reads from FASTQ by identifier and index number. The pyfastx will build indexes stored in a sqlite3 database file for random access to avoid consuming excessive amount of memory. In addition, the pyfastx can parse standard (sequence is spread into multiple lines with same length) and nonstandard (sequence is spread into one or more lines with different length) FASTA format. This module used kseq.h written by @attractivechaos in klib project to parse plain FASTA/Q file and zran.c written by @pauldmccarthy in project indexed_gzip to index gzipped file for random access.

This project was heavily inspired by @mdshw5‘s project pyfaidx and @brentp‘s project pyfasta.

Features

  • Single file for the Python extension
  • Lightweight, memory efficient for parsing FASTA/Q file
  • Fast random access to sequences from gzipped FASTA/Q file
  • Read sequences from FASTA file line by line
  • Calculate N50 and L50 of sequences in FASTA file
  • Calculate GC content and nucleotides composition
  • Extract reverse, complement and antisense sequences
  • Excellent compatibility, support for parsing nonstandard FASTA file
  • Support for FASTQ quality score conversion
  • Provide command line interface for splitting FASTA/Q file

Installation

Currently, pyfastx supports Python 3.5, 3.6, 3.7, 3.8. Make sure you have installed both pip and Python before starting.

You can install pyfastx via the Python Package Index (PyPI)

pip install pyfastx

Update pyfastx module

pip install -U pyfastx

FASTA

Read FASTA file

Read plain or gzipped FASTA file and build index, support for random access to FASTA.

  1. import pyfastx
  2. fa = pyfastx.Fasta('test/data/test.fa.gz')
  3. fa

Note
Building index may take some times. The time required to build index depends on the size of FASTA file. If index built, you can randomly access to any sequences in FASTA file. The index file can be reused to save time when you read seqeunces from FASTA file next time.

FASTA records iteration

The fastest way to iterate plain or gzipped FASTA file without building index, the iteration will return a tuple contains name and sequence.

  1. import pyfastx
  2. for name, seq in pyfastx.Fasta('test/data/test.fa.gz', build_index=False):
  3.     print(name, seq)

You can also iterate sequence object from FASTA object like this:

  1. import pyfastx
  2. for seq in pyfastx.Fasta('test/data/test.fa.gz'):
  3.     print(seq.name)
  4.     print(seq.seq)
  5.     print(seq.description)

Iteration with build_index=True (default) return sequence object which allows you to access attributions of sequence. New in pyfastx 0.6.3.

Get FASTA information

  1. # get sequence counts in FASTA
  2. len(fa)
  4. # get total sequence length of FASTA
  5. fa.size
  7. # get GC content of DNA sequence of FASTA
  8. fa.gc_content
  10. # get GC skew of DNA sequences in FASTA
  11. fa.gc_skews
  13. # get composition of nucleotides in FASTA
  14. fa.composition
  16. # get fasta type (DNA, RNA, or protein)
  17. fa.type
  19. # check fasta file is gzip compressed
  20. fa.is_gzip

Get longest and shortest sequence

New in pyfastx 0.3.0

  1. # get longest sequence
  2. s = fa.longest
  3. s
  4. s.name
  5. len(s)
  7. # get shortest sequence
  8. s = fa.shortest
  9. s
  10. s.name
  11. len(s)

Calculate N50 and L50

New in pyfastx 0.3.0

Calculate assembly N50 and L50, return (N50, L50), learn more about N50,L50

  1. # get FASTA N50 and L50
  2. fa.nl(50)
  4. # get FASTA N90 and L90
  5. fa.nl(90)
  7. # get FASTA N75 and L75
  8. fa.nl(75)

Get sequence mean and median length

New in pyfastx 0.3.0

  1. # get sequence average length
  2. fa.mean
  4. # get seqeunce median length
  5. fa.median

Get sequence counts

New in pyfastx 0.3.0

Get counts of sequences whose length >= specified length

  1. # get counts of sequences with length >= 200 bp
  2. fa.count(200)
  4. # get counts of sequences with length >= 500 bp
  5. fa.count(500)

Get subsequences

Subsequences can be retrieved from FASTA file by using a list of [start, end] coordinates

  1. # get subsequence with start and end position
  2. interval = (1, 10)
  3. fa.fetch('JZ822577.1', interval)
  5. # get subsequences with a list of start and end position
  6. intervals = [(1, 10), (50, 60)]
  7. fa.fetch('JZ822577.1', intervals)
  9. # get subsequences with reverse strand
  10. fa.fetch('JZ822577.1', (1, 10), strand='-')

Key function

New in pyfastx 0.5.1

Sometimes your fasta will have a long header which contains multiple identifiers and description, for example, “>JZ822577.1 contig1 cDNA library of flower petals in tree peony by suppression subtractive hybridization Paeonia suffruticosa cDNA, mRNA sequence”. In this case, both “JZ822577.1” and “contig1” can be used as identifer. you can specify the key function to select one as identifier.

  1. #default use JZ822577.1 as identifier
  2. #specify key_func to select contig1 as identifer
  3. fa = pyfastx.Fasta('tests/data/test.fa.gz', key_func=lambda x: x.split()[1])
  4. fa

Sequence

Get a sequence from FASTA

  1. # get sequence like a dictionary by identifier
  2. s1 = fa['JZ822577.1']
  3. s1
  5. # get sequence like a list by index
  6. s2 = fa[2]
  7. s2
  9. # get last sequence
  10. s3 = fa[-1]
  11. s3
  13. # check a sequence name weather in FASTA file
  14. 'JZ822577.1' in fa

Get sequence information

  1. s = fa[-1]
  2. s
  4. # get sequence order number in FASTA file
  5. s.id
  7. # get sequence name
  8. s.name
  10. # get sequence description
  11. s.description
  13. # get sequence string
  14. s.seq
  16. # get sequence raw string, New in pyfastx 0.6.3
  17. print(s.raw)
  19. # get sequence length
  20. len(s)
  22. # get GC content if dna sequence
  23. s.gc_content
  25. # get nucleotide composition if dna sequence
  26. s.composition

Sequence slice

Sequence object can be sliced like a python string

  1. # get a sub seq from sequence
  2. s = fa[-1]
  3. ss = s[10:30]
  4. ss
  6. ss.name
  8. ss.seq
  10. ss = s[-10:]
  11. ss


> >

Note

Slicing start and end coordinates are 0-based. Currently, pyfastx does not support an optional third step or stride argument. For example ss[::-1]

Reverse and complement sequence

  1. # get sliced sequence
  2. fa[0][10:20].seq
  4. # get reverse of sliced sequence
  6. fa[0][10:20].reverse
  8. # get complement of sliced sequence
  9. fa[0][10:20].complement
  11. # get reversed complement sequence, corresponding to sequence in antisense strand
  12. fa[0][10:20].antisense

Read sequence line by line

New in pyfastx 0.3.0

The sequence object can be iterated line by line as they appear in FASTA file.

  1. for line in fa[0]:
  2.     print(line)

Note
Sliced sequence (e.g. fa[0][10:50]) cannot be read line by line

Search for subsequence

New in pyfastx 0.3.6

Search for subsequence from given sequence and get one-based start position of the first occurrence

  1. # search subsequence in sense strand
  2. fa[0].search('GCTTCAATACA')
  4. # check subsequence weather in sequence
  5. 'GCTTCAATACA' in fa[0]
  7. # search subsequence in antisense strand
  8. fa[0].search('CCTCAAGT', '-')

FASTQ

New in pyfastx 0.4.0

Read FASTQ file

Read plain or gzipped file and build index, support for random access to reads from FASTQ.

  1. import pyfastx
  2. fq = pyfastx.Fastq('tests/data/test.fq.gz')
  3. fq

FASTQ records iteration

The fastest way to parse plain or gzipped FASTQ file without building index, the iteration will return a tuple contains read name, seq and quality.

  1. import pyfastx
  2. for name,seq,qual in pyfastx.Fastq('tests/data/test.fq.gz', build_index=False):
  3.     print(name)
  4.     print(seq)
  5.     print(qual)

You can also iterate read object from FASTQ object like this:

  1. import pyfastx
  2. for read in pyfastx.Fastq('test/data/test.fq.gz'):
  3.     print(read.name)
  4.     print(read.seq)
  5.     print(read.qual)
  6.     print(read.quali)

Iteration with build_index=True (default) return read object which allows you to access attribution of read. New in pyfastx 0.6.3.

Get FASTQ information

  1. # get read counts in FASTQ
  2. len(fq)
  4. # get total bases
  5. fq.size
  7. # get GC content of FASTQ file
  8. fq.gc_content
  10. # get composition of bases in FASTQ
  11. fq.composition
  13. # get phred which affects the quality score conversion
  14. fq.phred
  16. # Guess fastq quality encoding system
  17. fq.encoding_type


> >

Read

Get read from FASTQ

  1. #get read like a dict by read name
  2. r1 = fq['A00129:183:H77K2DMXX:1:1101:4752:1047']
  3. r1
  5. # get read like a list by index
  6. r2 = fq[10]
  7. r2
  9. # get the last read
  10. r3 = fq[-1]
  11. r3
  13. # check a read weather in FASTQ file
  14. 'A00129:183:H77K2DMXX:1:1101:4752:1047' in fq

Get read information

  1. r = fq[-10]
  2. r
  4. # get read order number in FASTQ file
  5. r.id
  7. # get read name
  8. r.name
  10. # get read full header line, New in pyfastx 0.6.3
  11. r.description
  13. # get read length
  14. len(r)
  16. # get read sequence
  17. r.seq
  19. # get raw string of read, New in pyfastx 0.6.3
  20. print(r.raw)
  22. # get read quality ascii string
  23. r.qual
  25. # get read quality integer value, ascii - 33 or 64
  26. r.quali
  28. # get read length
  29. len(r)


> >

Identifiers

Get identifiers

Get all identifiers of sequence as a list-like object.

  1. ids = fa.keys()
  2. ids
  4. # get count of sequence
  5. len(ids)
  7. # get identifier by index
  8. ids[0]
  10. # check identifier where in fasta
  11. 'JZ822577.1' in ids
  13. # iter identifiers
  14. for name in ids:
  15.     print(name)
  17. # convert to a list
  18. list(ids)

Sort identifiers

Sort identifiers by sequence id, name, or length for iteration

New in pyfastx 0.5.0

  1. # sort identifiers by length with descending order
  2. for name in ids.sort(key='length', reverse=True):
  3.     print(name)
  5. # sort identifiers by name with ascending order
  6. for name in ids.sort(key='name'):
  7.     print(name)
  9. # sort identifiers by id with descending order
  10. for name in ids.sort(key='id', reverse=True)
  11.     print(name)

Filter identifiers

Filter identifiers by sequence length and name

New in pyfastx 0.5.10

  1. # get identifiers with length > 600
  2. ids.filter(ids > 600)
  4. # get identifiers with length >= 500 and <= 700
  5. ids.filter(ids>=500, ids<=700)
  7. # get identifiers with length > 500 and < 600
  8. ids.filter(500<ids<600)
  10. # get identifiers contain JZ8226
  11. ids.filter(ids % 'JZ8226')
  13. # get identifiers contain JZ8226 with length > 550
  14. ids.filter(ids % 'JZ8226', ids>550)
  16. # clear sort order and filters
  17. ids.reset()
  19. # list a filtered result
  20. ids.filter(ids % 'JZ8226', ids>730)
  21. list(ids)
  23. # list a filtered result with sort order
  24. ids.filter(ids % 'JZ8226', ids>730).sort('length', reverse=True)
  25. list(ids)
  27. ids.filter(ids % 'JZ8226', ids>730).sort('name', reverse=True)
  28. list(ids)


> >


> >

Command line interface

New in pyfastx 0.5.0

  1. pyfastx -h
  2. usage: pyfastx COMMAND [OPTIONS]
  3. A command line tool for FASTA/Q file manipulation
  4. optional arguments:
  5. -h, --help     show this help message and exit
  6. -v, --version  show program's version number and exit
  7. Commands:
  8. info         show detailed statistics information of FASTA/Q file
  9. split        split fasta file into multiple files
  10. fq2fa        Convert fastq file to fasta file
  11. subseq       Get subseqence from fasta file by id or name with region
  12. sample       randomly sample sequences from fasta or fastq file

Show statistics information

  1. pyfastx info -h
  2. usage: pyfastx info [-h] fastx
  3. positional arguments:
  4. fastx       input fasta or fastq file, gzip support
  5. optional arguments:
  6. -h, --help  show this help message and exit

Split FASTA/Q file

  1. pyfastx split -h
  2. usage: pyfastx split [-h] (-n int | -c int) [-o str] fastx
  3. positional arguments:
  4. fastx                 fasta or fastq file, gzip support
  5. optional arguments:
  6. -h, --help            show this help message and exit
  7. -n int                split a fa/q file into N new files with even size
  8. -c int                split a fa/q file into multiple files with the same
  9. sequence counts
  10. -o str, --outdir str  output directory, default is current folder

Convert FASTQ to FASTA file

  1. pyfastx fq2fa -h
  2. usage: pyfastx fq2fa [-h] [-o str] fastx
  3. positional arguments:
  4. fastx                 input fastq file, gzip support
  5. optional arguments:
  6. -h, --help            show this help message and exit
  7. -o str, --outfile str
  8. output file, default: output to stdout

Get subsequence with region

  1. pyfastx subseq -h
  2. usage: pyfastx subseq [-h] (--id int | --chr str) [-r str] fastx
  3. positional arguments:
  4. fastx                 input fasta file, gzip support
  5. optional arguments:
  6. -h, --help            show this help message and exit
  7. --id int              sequence id number in fasta file
  8. --chr str             sequence name
  9. -r str, --region str  one-based slice region, e.g. 10:20

Sample sequences

  1. pyfastx sample -h
  2. usage: pyfastx sample [-h] (-n int | -p float) [-o str] fastx
  3. positional arguments:
  4. fastx                 fasta or fastq file, gzip support
  5. optional arguments:
  6. -h, --help            show this help message and exit
  7. -n int                number of sequences to be sampled
  8. -p float              proportion of sequences to be sampled, 0~1
  9. -o str, --outfile str
  10. output file, default: output to stdout

Drawbacks

If you intensively check sequence names exists in FASTA file using in operator on FASTA object like:

fa = pyfastx.Fasta(‘tests/data/test.fa.gz’)

Suppose seqnames has 100000 names

for seqname in seqnames:
if seqname in fa:
do something

This will take a long time to finish. Becuase, pyfastx does not load the index into memory, the in operating is corresponding to sql query existence from index database. The faster alternative way to do this is:

fa = pyfastx.Fasta(‘tests/data/test.fa.gz’)

load all sequence names into a set object

all_names = set(fa.keys())
for seqname in seqnames:
if seqname in all_names:
do something

Testing

The pyfaidx module was used to test pyfastx. To run the tests:

$ python setup.py test

Acknowledgements

kseq.h and zlib was used to parse FASTA format. Sqlite3 was used to store built indexes. pyfastx can randomly access to sequences from gzipped FASTA file mainly attributed to indexed_gzip.