Apply seqtk PR to improve kseq.h parsing performance #1674
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Doubles throughput; can now parse FASTA at 2GB/s.
|
Thanks both. A nice simple but effective change. Sad face over kstream using int everywhere, giving a 2GB limit on fasta sequence length, but that's a totally different PR. :/ |
jmarshall
pushed a commit
to pysam-developers/pysam
that referenced
this pull request
Sep 28, 2023
Apply the main part of PR samtools/htslib#1674. Pysam does use kseq itself, so it is worth applying this now without waiting for the next HTSlib release to be imported. Co-Authored-By: Fabian Klötzl <fabian@kloetzl.info>
|
This is missing a cast to support C++ with its stricter typing rules. lh3/seqtk@7fe58c8 |
jkbonfield
added a commit
to jkbonfield/htslib
that referenced
this pull request
Oct 3, 2023
Klist.h and kseq.h use calloc, realloc and memchr in static inline code, but if we wish to permit our external headers to be used from a C++ include then these all need explicit casts as "void *" doesn't work the same in C++ as in C. It's tempting to use `extern "C"` in an `#ifdef __cplusplus` guard, but the nature of these pseudo-template klib headers is such that the code will be expanded up later inside a C++ file so the extern "C" doesn't solve it. See samtools#1674 and samtools#1682
jkbonfield
added a commit
to jkbonfield/htslib
that referenced
this pull request
Oct 4, 2023
Klist.h and kseq.h use calloc, realloc and memchr in static inline code, but if we wish to permit our external headers to be used from a C++ include then these all need explicit casts as "void *" doesn't work the same in C++ as in C. It's tempting to use `extern "C"` in an `#ifdef __cplusplus` guard, but the nature of these pseudo-template klib headers is such that the code will be expanded up later inside a C++ file so the extern "C" doesn't solve it. See samtools#1674 and samtools#1682
daviesrob
pushed a commit
that referenced
this pull request
Oct 6, 2023
Klist.h and kseq.h use calloc, realloc and memchr in static inline code, but if we wish to permit our external headers to be used from a C++ include then these all need explicit casts as "void *" doesn't work the same in C++ as in C. It's tempting to use `extern "C"` in an `#ifdef __cplusplus` guard, but the nature of these pseudo-template klib headers is such that the code will be expanded up later inside a C++ file so the extern "C" doesn't solve it. See #1674 and #1682
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Port of @kloetzl's lh3/seqtk#123 and attractivechaos/klib#173 to HTSlib's copy of kseq.h.
Per the original PR: “Doubles throughput; can now parse FASTA at 2GB/s”.
For one large data file on my machine, this reduces klib's kseq_bench2's
kstreamtime from ~1.65s to ~0.92s.(I have not heavily tested it otherwise.)
As noted here, HTSlib doesn't use
kseqitself though a few samtools commands do use it to read a few small BED files etc.