Kraken is a taxonomic sequence classifier that assigns taxonomic labels to DNA sequences. Kraken examines the k-mers within a query sequence and uses the information within those k-mers to query a database. That database maps k-mers to the lowest common ancestor (LCA) of all genomes known to contain a given k-mer.
The first version of Kraken used a large indexed and sorted list of k-mer/LCA pairs as its database. While fast, the large memory requirements posed some problems for users, and so Kraken 2 was created to provide a solution to those problems.
Kraken 2 differs from Kraken 1 in several important ways:
Only minimizers of the k-mers in the query sequences are used as database queries. Similarly, only minimizers of the k-mers in the reference sequences in the database’s genomic library are stored in the database. We will also refer to the minimizers as ℓ-mers, where ℓ ≤ k. All k-mers are considered to have the same LCA as their minimizer’s database LCA value.
Kraken 2 uses a compact hash table that is a probabilistic data structure. This means that occasionally, database queries will fail by either returning the wrong LCA, or by not resulting in a search failure when a queried minimizer was never actually stored in the database. By incurring the risk of these false positives in the data structure, Kraken 2 is able to achieve faster speeds and lower memory requirements. Users should be aware that database false positive errors occur in less than 1% of queries, and can be compensated for by use of confidence scoring thresholds.
Kraken 2 has the ability to build a database from amino acid sequences and perform a translated search of the query sequences against that database.
Kraken 2 utilizes spaced seeds in the storage and querying of minimizers to improve classification accuracy.
Kraken 2 provides support for “special†databases that are not based on NCBI’s taxonomy. These are currently limited to three popular 16S databases.
Because Kraken 2 only stores minimizers in its hash table, and k can be much larger than ℓ, only a small percentage of the possible ℓ-mers in a genomic library are actually deposited in the database. This creates a situation similar to the Kraken 1 “MiniKraken†databases; however, preliminary testing has shown the accuracy of a reduced Kraken 2 database to be quite similar to the full-sized Kraken 2 database, while Kraken 1’s MiniKraken databases often resulted in a substantial loss of per-read sensitivity.
If you use Kraken 2 in your own work, please cite either the Kraken 2 paper and/or the original Kraken paper as appropriate. Thank you!
System Requirements
Disk space: Construction of a Kraken 2 standard database requires approximately 100 GB of disk space. A test on 01 Jan 2018 of the default installation showed 42 GB of disk space was used to store the genomic library files, 26 GB was used to store the taxonomy information from NCBI, and 29 GB was used to store the Kraken 2 compact hash table.
Like in Kraken 1, we strongly suggest against using NFS storage to store the Kraken 2 database if at all possible.
Memory: To run efficiently, Kraken 2 requires enough free memory to hold the database (primarily the hash table) in RAM. While this can be accomplished with a ramdisk, Kraken 2 will by default load the database into process-local RAM; the --memory-mapping switch to kraken2 will avoid doing so. The default database size is 29 GB (as of Jan. 2018), and you will need slightly more than that in RAM if you want to build the default database.
Dependencies: Kraken 2 currently makes extensive use of Linux utilities such as sed, find, and wget. Many scripts are written using the Bash shell, and the main scripts are written using Perl. Core programs needed to build the database and run the classifier are written in C++11, and need to be compiled using a somewhat recent version of g++ that will support C++11. Multithreading is handled using OpenMP. Downloads of NCBI data are performed by wget and rsync. Most Linux systems will have all of the above listed programs and development libraries available either by default or via package download.
Unlike Kraken 1, Kraken 2 does not use an external k-mer counter. However, by default, Kraken 2 will attempt to use the dustmasker or segmasker programs provided as part of NCBI’s BLAST suite to mask low-complexity regions (see Masking of Low-complexity Sequences).
MacOS NOTE: MacOS and other non-Linux operating systems are not explicitly supported by the developers, and MacOS users should refer to the Kraken-users group for support in installing the appropriate utilities to allow for full operation of Kraken 2. We will attempt to use MacOS-compliant code when possible, but development and testing time is at a premium and we cannot guarantee that Kraken 2 will install and work to its full potential on a default installation of MacOS.
In particular, we note that the default MacOS X installation of GCC does not have support for OpenMP. Without OpenMP, Kraken 2 is limited to single-threaded operation, resulting in slower build and classification runtimes.
Network connectivity: Kraken 2’s standard database build and download commands expect unfettered FTP and rsync access to the NCBI FTP server. If you’re working behind a proxy, you may need to set certain environment variables (such as ftp_proxy or RSYNC_PROXY) in order to get these commands to work properly.
Kraken 2’s scripts default to using rsync for most downloads; however, you may find that your network situation prevents use of rsync. In such cases, you can try the --use-ftp option to kraken2-build to force the downloads to occur via FTP.
MiniKraken: At present, users with low-memory computing environments can replicate the “MiniKraken†functionality of Kraken 1 in two ways: first, by increasing the value of k with respect to ℓ (using the --kmer-len and --minimizer-len options to kraken2-build); and secondly, through downsampling of minimizers (from both the database and query sequences) using a hash function. This second option is performed if the --max-db-size option to kraken2-build is used; however, the two options are not mutually exclusive. In a difference from Kraken 1, Kraken 2 does not require building a full database and then shrinking it to obtain a reduced database.
Installation
To begin using Kraken 2, you will first need to install it, and then either download or create a database.
Kraken 2 consists of two main scripts (kraken2 and kraken2-build), along with several programs and smaller scripts. As part of the installation process, all scripts and programs are installed in the same directory. After installation, you can move the main scripts elsewhere, but moving the other scripts and programs requires editing the scripts and changing the $KRAKEN2_DIR variables in the main scripts.
Once an install directory is selected, you need to run the following command in the directory where you extracted the Kraken 2 source:
./install_kraken2.sh $KRAKEN2_DIR
(Replace $KRAKEN2_DIR above with the directory where you want to install Kraken 2’s programs/scripts.)
The install_kraken2.sh script should compile all of Kraken 2’s code and setup your Kraken 2 program directory. Installation is successful if you see the message “Kraken 2 installation complete.â€
Once installation is complete, you may want to copy the main Kraken 2 scripts into a directory found in your PATH variable (e.g., “$HOME/binâ€):
After installation, you’re ready to either create or download a database.
Kraken 2 Databases
A Kraken 2 database is a directory containing at least 3 files:
hash.k2d: Contains the minimizer to taxon mappings
opts.k2d: Contains information about the options used to build the database
taxo.k2d: Contains taxonomy information used to build the database
None of these three files are in a human-readable format. Other files may also be present as part of the database build process, and can, if desired, be removed after a successful build of the database.
In interacting with Kraken 2, you should not have to directly reference any of these files, but rather simply provide the name of the directory in which they are stored. Kraken 2 allows both the use of a standard database as well as custom databases; these are described in the sections Standard Kraken 2 Database and Custom Databases below, respectively.
Standard Kraken 2 Database
To create the standard Kraken 2 database, you can use the following command:
kraken2-build --standard --db $DBNAME
(Replace “$DBNAME†above with your preferred database name/location. Please note that the database will use approximately 100 GB of disk space during creation, with the majority of that being reference sequences or taxonomy mapping information that can be removed after the build.)
This will download NCBI taxonomic information, as well as the complete genomes in RefSeq for the bacterial, archaeal, and viral domains, along with the human genome and a collection of known vectors (UniVec_Core). After downloading all this data, the build process begins; this can be the most time-consuming step. If you have multiple processing cores, you can run this process with multiple threads, e.g.:
Using 32 threads on an AWS EC2 r4.8xlarge instance with 16 dual-core hyperthreaded 2.30 GHz CPUs and 244 GB of RAM, the build process took approximately 35 minutes in Jan. 2018.
The build process itself has two main steps, each of which requires passing over the contents of the reference library:
Estimation of the capacity needed in the Kraken 2 compact hash table. This uses a low-memory method to reliably estimate the number of minimizers present in the reference library given the selected parameters k and â„“.
Population of the hash table (and conversion of the taxonomy to an internal format). This step is a second pass over the reference library to find minimizers and then place them in the database.
(There is one other preliminary step where sequence IDs are mapped to taxonomy IDs, but this is usually a rather quick process and is mostly handled during library downloading.)
Unlike Kraken 1’s build process, Kraken 2 does not perform checkpointing after the estimation step. This is because the estimation step is dependent on the selected k and ℓ values, and if the population step fails, it is likely because k needs to be increased (reducing the overall memory requirements).
Classification
To classify a set of sequences, use the kraken2 command:
kraken2 --db $DBNAME seqs.fa
Output will be sent to standard output by default. The files containing the sequences to be classified should be specified on the command line. Sequences can also be provided through standard input using the special filename /dev/fd/0.
The kraken2 program allows several different options:
Multithreading: Use the --threads NUM switch to use multiple threads.
Quick operation: Rather than searching all â„“-mers in a sequence, stop classification after the first database hit; use --quick to enable this mode.
Hit group threshold: The option --minimum-hit-groups will allow you to require multiple hit groups (a group of overlapping k-mers that share a common minimizer that is found in the hash table) be found before declaring a sequence classified, which can be especially useful with custom databases when testing to see if sequences either do or do not belong to a particular genome.
Sequence filtering: Classified or unclassified sequences can be sent to a file for later processing, using the --classified-out and --unclassified-out switches, respectively.
Output redirection: Output can be directed using standard shell redirection (| or >), or using the --output switch.
Compressed input: Kraken 2 can handle gzip and bzip2 compressed files as input by specifying the proper switch of --gzip-compressed or --bzip2-compressed.
Input format auto-detection: If regular files (i.e., not pipes or device files) are specified on the command line as input, Kraken 2 will attempt to determine the format of your input prior to classification. You can disable this by explicitly specifying --gzip-compressed or --bzip2-compressed as appropriate. Note that use of the character device file /dev/fd/0 to read from standard input (aka stdin) will not allow auto-detection.
Paired reads: Kraken 2 provides an enhancement over Kraken 1 in its handling of paired read data. Rather than needing to concatenate the pairs together with an N character between the reads, Kraken 2 is able to process the mates individually while still recognizing the pairing information. Using the --paired option to kraken2 will indicate to kraken2 that the input files provided are paired read data, and data will be read from the pairs of files concurrently.
Usage of --paired also affects the --classified-out and --unclassified-out options; users should provide a # character in the filenames provided to those options, which will be replaced by kraken2 with “_1†and “_2†with mates spread across the two files appropriately. For example:
will put the first reads from classified pairs in cseqs_1.fq, and the second reads from those pairs in cseqs_2.fq.
To get a full list of options, use kraken2 --help.
Output Formats
Standard Kraken Output Format
Each sequence (or sequence pair, in the case of paired reads) classified by Kraken 2 results in a single line of output. Kraken 2’s output lines contain five tab-delimited fields; from left to right, they are:
“Câ€/“Uâ€: a one letter code indicating that the sequence was either classified or unclassified.
The sequence ID, obtained from the FASTA/FASTQ header.
The taxonomy ID Kraken 2 used to label the sequence; this is 0 if the sequence is unclassified.
The length of the sequence in bp. In the case of paired read data, this will be a string containing the lengths of the two sequences in bp, separated by a pipe character, e.g. “98|94â€.
A space-delimited list indicating the LCA mapping of each k-mer in the sequence(s). For example, “562:13 561:4 A:31 0:1 562:3†would indicate that:
the first 13 k-mers mapped to taxonomy ID #562
the next 4 k-mers mapped to taxonomy ID #561
the next 31 k-mers contained an ambiguous nucleotide
the next k-mer was not in the database
the last 3 k-mers mapped to taxonomy ID #562
Note that paired read data will contain a “|:|†token in this list to indicate the end of one read and the beginning of another.
When Kraken 2 is run against a protein database (see Translated Search), the LCA hitlist will contain the results of querying all six frames of each sequence. Reading frame data is separated by a “-:-†token.
Kraken 1 offered a kraken-translate and kraken-report script to change the output into different formats. Through the use of kraken2 --use-names, Kraken 2 will replace the taxonomy ID column with the scientific name and the taxonomy ID in parenthesis (e.g., “Bacteria (taxid 2)†instead of “2â€), yielding similar functionality to Kraken 1’s kraken-translate script. The sample report functionality now exists as part of the kraken2 script, with the use of the --report option; the sample report formats are described below.
Sample Report Output Format
Like Kraken 1, Kraken 2 offers two formats of sample-wide results. Kraken 2’s standard sample report format is tab-delimited with one line per taxon. The fields of the output, from left-to-right, are as follows:
Percentage of fragments covered by the clade rooted at this taxon
Number of fragments covered by the clade rooted at this taxon
Number of fragments assigned directly to this taxon
A rank code, indicating (U)nclassified, (R)oot, (D)omain, (K)ingdom, (P)hylum, (C)lass, (O)rder, (F)amily, (G)enus, or (S)pecies. Taxa that are not at any of these 10 ranks have a rank code that is formed by using the rank code of the closest ancestor rank with a number indicating the distance from that rank. E.g., “G2†is a rank code indicating a taxon is between genus and species and the grandparent taxon is at the genus rank.
NCBI taxonomic ID number
Indented scientific name
The scientific names are indented using space, according to the tree structure specified by the taxonomy.
By default, taxa with no reads assigned to (or under) them will not have any output produced. However, if you wish to have all taxa displayed, you can use the --report-zero-counts switch to do so. This can be useful if you are looking to do further downstream analysis of the reports, and want to compare samples. Sorting by the taxonomy ID (using sort -k5,5n) can provide a consistent line ordering between reports.
In addition, we also provide the option --use-mpa-style that can be used in conjunction with --report. This option provides output in a format similar to MetaPhlAn’s output. The output with this option provides one taxon per line, with a lowercase version of the rank codes in Kraken 2’s standard sample report format (except for ‘U’ and ‘R’), two underscores, and the scientific name of the taxon (e.g., "d__Viruses“). The full taxonomy of each taxon (at the eight ranks considered) is given, with each rank’s name separated by a pipe character (e.g.,â€d__Viruses|o_Caudovirales"). Following this version of the taxon’s scientific name is a tab and the number of fragments assigned to the clade rooted at that taxon.
Translated Search
Kraken 2 allows users to perform a six-frame translated search, similar to the well-known BLASTX program. To do this, Kraken 2 uses a reduced 15 amino acid alphabet and stores amino acid minimizers in its database. LCA results from all 6 frames are combined to yield a set of LCA hits, which is then resolved in the same manner as in Kraken’s normal operation.
To build a protein database, the --protein option should be given to kraken2-build (either along with --standard, or with all steps if building a custom database).
Custom Databases
We realize the standard database may not suit everyone’s needs. Kraken 2 also allows creation of customized databases.
To build a custom database:
Install a taxonomy. Usually, you will just use the NCBI taxonomy, which you can easily download using:
kraken2-build --download-taxonomy --db $DBNAME
This will download the accession number to taxon maps, as well as the taxonomic name and tree information from NCBI. These files can be found in $DBNAME/taxonomy/ . If you need to modify the taxonomy, edits can be made to the names.dmp and nodes.dmp files in this directory; you may also need to modify the *.accession2taxid files appropriately.
Some of the standard sets of genomic libraries have taxonomic information associated with them, and don’t need the accession number to taxon maps to build the database successfully. These libraries include all those available through the --download-library option (see next point), except for the plasmid and non-redundant databases. If you are not using custom sequences (see the --add-to-library option) and are not using one of the plasmid or non-redundant database libraries, you may want to skip downloading of the accession number to taxon maps. This can be done by passing --skip-maps to the kraken2-build --download-taxonomy command.
Install one or more reference libraries. Several sets of standard genomes/proteins are made easily available through kraken2-build:
The above commands would prepare a database that would contain archaeal and viral genomes; the --build option (see below) will still need to be used after downloading these libraries to actually build the database, however.
(Note that downloading nr requires use of the --protein option, and that UniVec and UniVec_Core are incompatible with the --protein option.)
Other genomes can also be added, but such genomes must meet certain requirements:
Sequences must be in a FASTA file (multi-FASTA is allowed)
Each sequence’s ID (the string between the > and the first whitespace character on the header line) must contain either an NCBI accession number to allow Kraken 2 to lookup the correct taxa, or an explicit assignment of the taxonomy ID using kraken:taxid (see below).
Sequences not downloaded from NCBI may need their taxonomy information assigned explicitly. This can be done using the string kraken:taxid|XXX in the sequence ID, with XXX replaced by the desired taxon ID. For example, to put a known adapter sequence in taxon 32630 (“synthetic constructâ€), you could use the following:
The kraken:taxid string must begin the sequence ID or be immediately preceded by a pipe character (|). Explicit assignment of taxonomy IDs in this manner will override the accession number mapping provided by NCBI.
If your genomes meet the requirements above, then you can add each sequence to your database’s genomic library using the --add-to-library switch, e.g.:
(You may also find the -P option to xargs useful to add many files in parallel if you have multiple processors.)
Once your library is finalized, you need to build the database. This can be done with the command:
kraken2-build --build --db $DBNAME
The --threads option is also helpful here to reduce build time.
By default, the values of k and â„“ are 35 and 31, respectively (or 15 and 12 for protein databases). These values can be explicitly set with the --kmer-len and --minimizer-len options, however. Note that the minimizer length must be no more than 31 for nucleotide databases, and 15 for protein databases. Additionally, the minimizer length â„“ must be no more than the k-mer length. There is no upper bound on the value of k, but sequences less than k bp in length cannot be classified.
Kraken 2 also utilizes a simple spaced seed approach to increase accuracy. A number s < â„“/4 can be chosen, and s positions in the minimizer will be masked out during all comparisons. Masked positions are chosen to alternate from the second-to-last position in the minimizer; e.g., s = 5 and â„“ = 31 will result in masking out the 0 positions shown here:
111 1111 1111 1111 1111 1101 0101 0101
By default, s = 7 for nucleotide databases, and s = 0 for protein databases. This can be changed using the --minimizer-spaces option along with the --build task of kraken2-build.
A full list of options for kraken2-build can be obtained using kraken2-build --help.
After building a database, if you want to reduce the disk usage of the database, you can use the --clean option for kraken2-build to remove intermediate files from the database directory.
Masking of Low-complexity Sequences
Low-complexity sequences, e.g. “ACACACACACACACACACACACACACâ€, are known to occur in many different organisms and are typically less informative for use in alignments; the BLAST programs often mask these sequences by default. Using this masking can help prevent false positives in Kraken 2’s results, and so we have added this functionality as a default option to Kraken 2’s library download/addition process.
Kraken 2 uses two programs to perform low-complexity sequence masking, both available from NCBI: dustmasker, for nucleotide sequences, and segmasker, for amino acid sequences. These programs are available as part of the NCBI BLAST+ suite. If these programs are not installed on the local system and in the user’s PATH when trying to use kraken2-build, the database build will fail. Users who do not wish to install these programs can use the --no-masking option to kraken2-build in conjunction with any of the --download-library, --add-to-library, or --standard options; use of the --no-masking option will skip masking of low-complexity sequences during the build of the Kraken 2 database.
Special Databases
To support some common use cases, we provide the ability to build Kraken 2 databases using data from various external databases. These external databases may not follow the NCBI taxonomy, and so we’ve provided mechanisms to automatically create a taxonomy that will work with Kraken 2 (although such taxonomies may not be identical to NCBI’s).
To build one of these “special†Kraken 2 databases, use the following command:
kraken2-build --db $DBNAME --special TYPE
where the TYPE string is one of the database names listed below.
At present, the “special†Kraken 2 database support we provide is limited to pre-packaged solutions for some public 16S sequence databases, but this may grow in the future.
16S Databases
For targeted 16S sequencing projects, a normal Kraken 2 database using whole genome data may use more resources than necessary. A Kraken 2 database created from a well-curated genomic library of just 16S data can provide both a more efficient solution as well as a more accurate set of predictions for such projects. We provide support for building Kraken 2 databases from three publicly available 16S databases:
Greengenes (Kraken 2 database name: greengenes), using all available 16S data.
RDP (Kraken 2 database name: rdp), using the bacterial and archaeal 16S data.
SILVA (Kraken 2 database name: silva), using the Small subunit NR99 sequence set.
Note that these databases may have licensing restrictions regarding their data, and it is your responsibility to ensure you are in compliance with those restrictions; please visit the databases’ websites for further details. The kraken2-build script only uses publicly available URLs to download data and then converts that data into a form compatible for use with Kraken 2.
Furthermore, if you use one of these databases in your research, please visit the corresponding database’s website to determine the appropriate and up-to-date citation.
Confidence Scoring
At present, we have not yet developed a confidence score with a probabilistic interpretation for Kraken 2. However, we have developed a simple scoring scheme that has yielded good results for us, and we’ve made that available in Kraken 2 through use of the --confidence option to kraken2. The approach we use allows a user to specify a threshold score in the [0,1] interval; the classifier then will adjust labels up the tree until the label’s score (described below) meets or exceeds that threshold. If a label at the root of the taxonomic tree would not have a score exceeding the threshold, the sequence is called unclassified by Kraken 2 when this threshold is applied.
A sequence label’s score is a fraction C/Q, where C is the number of k-mers mapped to LCA values in the clade rooted at the label, and Q is the number of k-mers in the sequence that lack an ambiguous nucleotide (i.e., they were queried against the database). Consider the example of the LCA mappings in Kraken 2’s output given earlier:
“562:13 561:4 A:31 0:1 562:3†would indicate that:
the first 13 k-mers mapped to taxonomy ID #562
the next 4 k-mers mapped to taxonomy ID #561
the next 31 k-mers contained an ambiguous nucleotide
the next k-mer was not in the database
the last 3 k-mers mapped to taxonomy ID #562
In this case, ID #561 is the parent node of #562. Here, a label of #562 for this sequence would have a score of C/Q = (13+3)/(13+4+1+3) = 16/21. A label of #561 would have a score of C/Q = (13+4+3)/(13+4+1+3) = 20/21. If a user specified a --confidence threshold over 16/21, the classifier would adjust the original label from #562 to #561; if the threshold was greater than 20/21, the sequence would become unclassified.
Inspecting a Kraken 2 Database’s Contents
The kraken2-inspect script allows users to gain information about the content of a Kraken 2 database. The output format of kraken2-inspect is identical to the reports generated with the --report option to kraken2. Instead of reporting how many reads in input data classified to a given taxon or clade, as kraken2’s --report option would, the kraken2-inspect script will report the number of minimizers in the database that are mapped to the various taxa/clades. For example, the first five lines of kraken2-inspect’s output on an example database might look like this:
$ kraken2-inspect --db EXAMPLE_DB | head -5
100.00% 1770368409 1581179 R 1 root
96.50% 1708407622 58003 R1 131567 cellular organisms
91.28% 1615910070 985309 D 2 Bacteria
43.89% 777062062 1312736 P 1224 Proteobacteria
18.62% 329590216 555667 C 1236 Gammaproteobacteria
This output indicates that 555667 of the minimizers in the database map directly to the Gammaproteobacteria class (taxid #1236), and 329590216 (18.62%) of the database’s minimizers map to a taxon in the clade rooted at Gammaproteobacteria. For more information on kraken2-inspect’s options, use its --help option.
Distinct minimizer count information
The KrakenUniq project extended Kraken 1 by, among other things, reporting an estimate of the number of distinct k-mers associated with each taxon in the input sequencing data. This allows users to better determine if Kraken’s classifications are due to reads distributed throughout a reference genome, or due to only a small segment of a reference genome (and therefore likely false positive).
Thanks to the generosity of KrakenUniq’s developer Florian Breitwieser in allowing parts of the KrakenUniq source code to be licensed under Kraken 2’s MIT license, this distinct counting estimation is now available in Kraken 2. Development work by Martin Steinegger and Ben Langmead helped bring this functionality to Kraken 2.
At present, this functionality is an optional experimental feature – meaning that we may later alter it in a way that is not backwards compatible with previous versions of the feature.
To use this functionality, simply run the kraken2 script with the additional --report-minimizer-data flag along with --report, e.g.:
This will put the standard Kraken 2 output (formatted as described in Standard Kraken Output Format) in k2_output.txt and the report information in k2_report.txt. Within the report file, two additional columns will be present, e.g.:
normal report format:
36.40 182 182 S2 211044 Influenza A virus (A/Puerto Rico/8/1934(H1N1))
In this modified report format, the two new columns are the fourth and fifth, respectively representing the number of minimizers found to be associated with a taxon in the read sequences (1688), and the estimate of the number of distinct minimizers associated with a taxon in the read sequence data (18). This would indicate that although 182 reads were classified as belonging to H1N1 influenza, only 18 distinct minimizers led to those 182 classifications.
The format with the --report-minimizer-data flag, then, is similar to that described in Sample Report Output Format, but slightly different. The fields in this new format, from left-to-right, are:
Percentage of fragments covered by the clade rooted at this taxon
Number of fragments covered by the clade rooted at this taxon
Number of fragments assigned directly to this taxon
Number of minimizers in read data associated with this taxon (new)
An estimate of the number of distinct minimizers in read data associated with this taxon (new)
A rank code, indicating (U)nclassified, (R)oot, (D)omain, (K)ingdom, (P)hylum, (C)lass, (O)rder, (F)amily, (G)enus, or (S)pecies. Taxa that are not at any of these 10 ranks have a rank code that is formed by using the rank code of the closest ancestor rank with a number indicating the distance from that rank. E.g., “G2†is a rank code indicating a taxon is between genus and species and the grandparent taxon is at the genus rank.
NCBI taxonomic ID number
Indented scientific name
We decided to make this an optional feature so as not to break existing software that processes Kraken 2’s standard report format. However, this new format can be converted to the standard report format with the command:
As noted above, this is an experimental feature. We intend to continue development on this feature, and may change the new format and/or its information if we determine it to be necessary.
For background on the data structures used in this feature and their interaction with Kraken, please read the KrakenUniq paper, and please cite that paper if you use this functionality as part of your work.
Kraken 2 Environment Variables
The kraken2 and kraken2-inspect scripts supports the use of some environment variables to help in reducing command line lengths:
KRAKEN2_NUM_THREADS: if the --threads option is not supplied to kraken2, then the value of this variable (if it is set) will be used as the number of threads to run kraken2. (This variable does not affect kraken2-inspect.)
KRAKEN2_DB_PATH: much like the PATH variable is used for executables by your shell, KRAKEN2_DB_PATH is a colon-separated list of directories that will be searched for the database you name if the named database does not have a slash (/) character. By default, Kraken 2 assumes the value of this variable is “.†(i.e., the current working directory). This variable can be used to create one (or more) central repositories of Kraken databases in a multi-user system. Example usage in bash:
This will cause three directories to be searched, in this order:
/home/user/my_kraken2_dbs
/data/kraken2_dbs
the current working directory (caused by the empty string as the third colon-separated field in the KRAKEN2_DB_PATH string)
The search for a database will stop when a name match is found; if two directories in the KRAKEN2_DB_PATH have databases with the same name, the directory of the two that is searched first will have its database selected.
If the above variable and value are used, and the databases /data/kraken2_dbs/mainDB and ./mainDB are present, then
kraken2 --db mainDB sequences.fa
will classify sequences.fa using /data/kraken_dbs/mainDB; if instead you wanted to use the mainDB present in the current directory, you would need to specify a directory path to that database in order to circumvent searching, e.g.:
kraken2 --db ./mainDB sequences.fa
Note that the KRAKEN2_DB_PATH directory list can be skipped by the use of any absolute (beginning with /) or relative pathname (including at least one /) as the database name.
KRAKEN2_DEFAULT_DB: if no database is supplied with the --db option, the database named in this variable will be used instead. Using this variable, you can avoid using --db if you only have a single database that you usually use, e.g. in bash:
This will classify sequences.fa using the /home/user/kraken2db database.
Note that the value of KRAKEN2_DEFAULT_DB will also be interpreted in the context of the value of KRAKEN2_DB_PATH if you don’t set KRAKEN2_DEFAULT_DB to an absolute or relative pathname. Given the earlier example in this section, the following:
will use /data/kraken_dbs/mainDB to classify sequences.fa.
kraken2-2.1.3/docs/MANUAL.markdown 0000664 0000000 0000000 00000116265 14437747201 0016547 0 ustar 00root root 0000000 0000000 Introduction
============
Kraken is a taxonomic sequence classifier that assigns taxonomic
labels to DNA sequences. Kraken examines the $k$-mers within
a query sequence and uses the information within those $k$-mers
to query a database. That database maps $k$-mers to the lowest
common ancestor (LCA) of all genomes known to contain a given $k$-mer.
The first version of [Kraken] used a large indexed and sorted list of
$k$-mer/LCA pairs as its database. While fast, the large memory
requirements posed some problems for users, and so Kraken 2 was
created to provide a solution to those problems.
[Kraken 2] differs from Kraken 1 in several important ways:
1. Only minimizers of the $k$-mers in the query sequences are used
as database queries. Similarly, only minimizers of the $k$-mers in
the reference sequences in the database's genomic library are stored
in the database. We will also refer to the minimizers as $\ell$-mers,
where $\ell \leq k$. All $k$-mers are considered to have the same LCA
as their minimizer's database LCA value.
2. Kraken 2 uses a compact hash table that is a probabilistic data
structure. This means that occasionally, database queries will fail
by either returning the wrong LCA, or by not resulting in a search
failure when a queried minimizer was never actually stored in the
database. By incurring the risk of these false positives in the data
structure, Kraken 2 is able to achieve faster speeds and lower memory
requirements. Users should be aware that database false positive
errors occur in less than 1% of queries, and can be compensated for
by use of confidence scoring thresholds.
3. Kraken 2 has the ability to build a database from amino acid
sequences and perform a translated search of the query sequences
against that database.
4. Kraken 2 utilizes spaced seeds in the storage and querying of
minimizers to improve classification accuracy.
5. Kraken 2 provides support for "special" databases that are
not based on NCBI's taxonomy. These are currently limited to
three popular 16S databases.
Because Kraken 2 only stores minimizers in its hash table, and $k$ can be
much larger than $\ell$, only a small percentage
of the possible $\ell$-mers in a genomic library are actually deposited in
the database. This creates a situation similar to the Kraken 1 "MiniKraken"
databases; however, preliminary testing has shown the accuracy of a reduced
Kraken 2 database to be quite similar to the full-sized Kraken 2 database,
while Kraken 1's MiniKraken databases often resulted in a substantial loss
of per-read sensitivity.
If you use Kraken 2 in your own work, please cite either the
[Kraken 2 paper] and/or the original [Kraken paper] as appropriate. Thank you!
[Kraken]: https://ccb.jhu.edu/software/kraken/
[Kraken 2]: https://ccb.jhu.edu/software/kraken2/
[Kraken paper]: https://genomebiology.biomedcentral.com/articles/10.1186/gb-2014-15-3-r46
[Kraken 2 paper]: https://genomebiology.biomedcentral.com/articles/10.1186/s13059-019-1891-0
System Requirements
===================
* **Disk space**: Construction of a Kraken 2 standard database requires
approximately 100 GB of disk space. A test on 01 Jan 2018 of the
default installation showed 42 GB of disk space was used to store
the genomic library files, 26 GB was used to store the taxonomy
information from NCBI, and 29 GB was used to store the Kraken 2
compact hash table.
Like in Kraken 1, we strongly suggest against using NFS storage
to store the Kraken 2 database if at all possible.
* **Memory**: To run efficiently, Kraken 2 requires enough free memory
to hold the database (primarily the hash table) in RAM. While this
can be accomplished with a ramdisk, Kraken 2 will by default load
the database into process-local RAM; the `--memory-mapping` switch
to `kraken2` will avoid doing so. The default database size is 29 GB
(as of Jan. 2018), and you will need slightly more than that in
RAM if you want to build the default database.
* **Dependencies**: Kraken 2 currently makes extensive use of Linux
utilities such as sed, find, and wget. Many scripts are written
using the Bash shell, and the main scripts are written using Perl.
Core programs needed to build the database and run the classifier
are written in C++11, and need to be compiled using a somewhat
recent version of g++ that will support C++11. Multithreading is
handled using OpenMP. Downloads of NCBI data are performed by wget
and rsync. Most Linux systems will have all of the above listed
programs and development libraries available either by default or
via package download.
Unlike Kraken 1, Kraken 2 does not use an external $k$-mer counter.
However, by default, Kraken 2 will attempt to use the `dustmasker` or
`segmasker` programs provided as part of NCBI's BLAST suite to mask
low-complexity regions (see [Masking of Low-complexity Sequences]).
**MacOS NOTE:** MacOS and other non-Linux operating systems are *not*
explicitly supported by the developers, and MacOS users should refer to
the Kraken-users group for support in installing the appropriate utilities
to allow for full operation of Kraken 2. We will attempt to use
MacOS-compliant code when possible, but development and testing time
is at a premium and we cannot guarantee that Kraken 2 will install
and work to its full potential on a default installation of MacOS.
In particular, we note that the default MacOS X installation of GCC
does not have support for OpenMP. Without OpenMP, Kraken 2 is
limited to single-threaded operation, resulting in slower build and
classification runtimes.
* **Network connectivity**: Kraken 2's standard database build and download
commands expect unfettered FTP and rsync access to the NCBI FTP
server. If you're working behind a proxy, you may need to set
certain environment variables (such as `ftp_proxy` or `RSYNC_PROXY`)
in order to get these commands to work properly.
Kraken 2's scripts default to using rsync for most downloads; however, you
may find that your network situation prevents use of rsync. In such cases,
you can try the `--use-ftp` option to `kraken2-build` to force the
downloads to occur via FTP.
* **MiniKraken**: At present, users with low-memory computing environments
can replicate the "MiniKraken" functionality of Kraken 1 in two ways:
first, by increasing
the value of $k$ with respect to $\ell$ (using the `--kmer-len` and
`--minimizer-len` options to `kraken2-build`); and secondly, through
downsampling of minimizers (from both the database and query sequences)
using a hash function. This second option is performed if
the `--max-db-size` option to `kraken2-build` is used; however, the two
options are not mutually exclusive.
In a difference from Kraken 1, Kraken 2 does not require building a full
database and then shrinking it to obtain a reduced database.
Installation
============
To begin using Kraken 2, you will first need to install it, and then
either download or create a database.
Kraken 2 consists of two main scripts (`kraken2` and `kraken2-build`),
along with several programs and smaller scripts. As part of the installation
process, all scripts and programs are installed in the same directory.
After installation, you can move the main scripts elsewhere, but moving
the other scripts and programs requires editing the scripts and changing
the `$KRAKEN2_DIR` variables in the main scripts.
Once an install directory is selected, you need to run the following
command in the directory where you extracted the Kraken 2 source:
./install_kraken2.sh $KRAKEN2_DIR
(Replace `$KRAKEN2_DIR` above with the directory where you want to install
Kraken 2's programs/scripts.)
The `install_kraken2.sh` script should compile all of Kraken 2's code
and setup your Kraken 2 program directory. Installation is successful if
you see the message "`Kraken 2 installation complete.`"
Once installation is complete, you may want to copy the main Kraken 2
scripts into a directory found in your `PATH` variable (e.g., "`$HOME/bin`"):
cp $KRAKEN2_DIR/kraken2{,-build,-inspect} $HOME/bin
After installation, you're ready to either create or download a database.
Kraken 2 Databases
==================
A Kraken 2 database is a directory containing at least 3 files:
* `hash.k2d`: Contains the minimizer to taxon mappings
* `opts.k2d`: Contains information about the options used to build the database
* `taxo.k2d`: Contains taxonomy information used to build the database
None of these three files are in a human-readable format. Other files
may also be present as part of the database build process, and can, if
desired, be removed after a successful build of the database.
In interacting with Kraken 2, you should not have to directly reference
any of these files, but rather simply provide the name of the directory
in which they are stored. Kraken 2 allows both the use of a standard
database as well as custom databases; these are described in the
sections [Standard Kraken 2 Database] and [Custom Databases] below,
respectively.
Standard Kraken 2 Database
==========================
To create the standard Kraken 2 database, you can use the following command:
kraken2-build --standard --db $DBNAME
(Replace "`$DBNAME`" above with your preferred database name/location.
Please note that the database will use approximately 100 GB of
disk space during creation, with the majority of that being reference
sequences or taxonomy mapping information that can be removed after the
build.)
This will download NCBI taxonomic information, as well as the
complete genomes in RefSeq for the bacterial, archaeal, and
viral domains, along with the human genome and a collection of
known vectors (UniVec_Core). After downloading all this data, the build
process begins; this can be the most time-consuming step. If you
have multiple processing cores, you can run this process with
multiple threads, e.g.:
kraken2-build --standard --threads 24 --db $DBNAME
Using 32 threads on an AWS EC2 r4.8xlarge instance with 16 dual-core
hyperthreaded 2.30 GHz CPUs and 244 GB of RAM, the build process took
approximately 35 minutes in Jan. 2018.
The build process itself has two main steps, each of which requires passing
over the contents of the reference library:
1. **Estimation** of the capacity needed in the Kraken 2 compact hash table.
This uses a low-memory method to reliably estimate the number of
minimizers present in the reference library given the selected parameters
$k$ and $\ell$.
2. **Population** of the hash table (and conversion of the taxonomy to an
internal format). This step is a second pass over the reference library
to find minimizers and then place them in the database.
(There is one other preliminary step where sequence IDs are mapped to
taxonomy IDs, but this is usually a rather quick process and is mostly handled
during library downloading.)
Unlike Kraken 1's build process, Kraken 2 does not perform checkpointing
after the estimation step. This is because the estimation step is dependent
on the selected $k$ and $\ell$ values, and if the population step fails, it is
likely because $k$ needs to be increased (reducing the overall memory
requirements).
Classification
==============
To classify a set of sequences, use the `kraken2` command:
kraken2 --db $DBNAME seqs.fa
Output will be sent to standard output by default. The files
containing the sequences to be classified should be specified
on the command line. Sequences can also be provided through
standard input using the special filename `/dev/fd/0`.
The `kraken2` program allows several different options:
* **Multithreading**: Use the `--threads NUM` switch to use multiple
threads.
* **Quick operation**: Rather than searching all $\ell$-mers in a sequence,
stop classification after the first database hit; use `--quick`
to enable this mode.
* **Hit group threshold**: The option `--minimum-hit-groups` will allow
you to require multiple hit groups (a group of overlapping k-mers that
share a common minimizer that is found in the hash table) be found
before declaring a sequence classified,
which can be especially useful with custom databases when testing
to see if sequences either do or do not belong to a particular
genome.
* **Sequence filtering**: Classified or unclassified sequences can be
sent to a file for later processing, using the `--classified-out`
and `--unclassified-out` switches, respectively.
* **Output redirection**: Output can be directed using standard shell
redirection (`|` or `>`), or using the `--output` switch.
* **Compressed input**: Kraken 2 can handle gzip and bzip2 compressed
files as input by specifying the proper switch of `--gzip-compressed`
or `--bzip2-compressed`.
* **Input format auto-detection**: If regular files (i.e., not pipes or device files)
are specified on the command line as input, Kraken 2 will attempt to
determine the format of your input prior to classification.
You can disable this by explicitly specifying
`--gzip-compressed` or `--bzip2-compressed` as appropriate.
Note that use of the character device file `/dev/fd/0` to read
from standard input (aka `stdin`) will **not** allow auto-detection.
* **Paired reads**: Kraken 2 provides an enhancement over Kraken 1 in its
handling of paired read data. Rather than needing to concatenate the
pairs together with an `N` character between the reads, Kraken 2 is
able to process the mates individually while still recognizing the
pairing information. Using the `--paired` option to `kraken2` will
indicate to `kraken2` that the input files provided are paired read
data, and data will be read from the pairs of files concurrently.
Usage of `--paired` also affects the `--classified-out` and
`--unclassified-out` options; users should provide a `#` character
in the filenames provided to those options, which will be replaced
by `kraken2` with "`_1`" and "`_2`" with mates spread across the two
files appropriately. For example:
kraken2 --paired --classified-out cseqs#.fq seqs_1.fq seqs_2.fq
will put the first reads from classified pairs in `cseqs_1.fq`, and
the second reads from those pairs in `cseqs_2.fq`.
To get a full list of options, use `kraken2 --help`.
Output Formats
==============
Standard Kraken Output Format
-----------------------------
Each sequence (or sequence pair, in the case of paired reads) classified
by Kraken 2 results in a single line of output. Kraken 2's output lines
contain five tab-delimited fields; from left to right, they are:
1. "C"/"U": a one letter code indicating that the sequence was either
classified or unclassified.
2. The sequence ID, obtained from the FASTA/FASTQ header.
3. The taxonomy ID Kraken 2 used to label the sequence; this is 0 if
the sequence is unclassified.
4. The length of the sequence in bp. In the case of paired read data,
this will be a string containing the lengths of the two sequences in
bp, separated by a pipe character, e.g. "98|94".
5. A space-delimited list indicating the LCA mapping of each $k$-mer in
the sequence(s). For example, "562:13 561:4 A:31 0:1 562:3" would
indicate that:
- the first 13 $k$-mers mapped to taxonomy ID #562
- the next 4 $k$-mers mapped to taxonomy ID #561
- the next 31 $k$-mers contained an ambiguous nucleotide
- the next $k$-mer was not in the database
- the last 3 $k$-mers mapped to taxonomy ID #562
Note that paired read data will contain a "`|:|`" token in this list
to indicate the end of one read and the beginning of another.
When Kraken 2 is run against a protein database (see [Translated Search]),
the LCA hitlist will contain the results of querying all six frames of
each sequence. Reading frame data is separated by a "`-:-`" token.
Kraken 1 offered a `kraken-translate` and `kraken-report` script to change
the output into different formats. Through the use of `kraken2 --use-names`,
Kraken 2 will replace the taxonomy ID column with the scientific name and
the taxonomy ID in parenthesis (e.g., "Bacteria (taxid 2)" instead of "2"),
yielding similar functionality to Kraken 1's `kraken-translate` script.
The sample report functionality now exists as part of the `kraken2` script,
with the use of the `--report` option; the sample report formats are
described below.
Sample Report Output Format
---------------------------
Like Kraken 1, Kraken 2 offers two formats of sample-wide results.
Kraken 2's standard sample report format is tab-delimited with one
line per taxon. The fields of the output, from left-to-right, are
as follows:
1. Percentage of fragments covered by the clade rooted at this taxon
2. Number of fragments covered by the clade rooted at this taxon
3. Number of fragments assigned directly to this taxon
4. A rank code, indicating (U)nclassified, (R)oot, (D)omain, (K)ingdom,
(P)hylum, (C)lass, (O)rder, (F)amily, (G)enus, or (S)pecies.
Taxa that are not at any of these 10 ranks have a rank code that is
formed by using the rank code of the closest ancestor rank with
a number indicating the distance from that rank. E.g., "G2" is a
rank code indicating a taxon is between genus and species and the
grandparent taxon is at the genus rank.
5. NCBI taxonomic ID number
6. Indented scientific name
The scientific names are indented using space, according to the tree
structure specified by the taxonomy.
By default, taxa with no reads assigned to (or under) them will not have
any output produced. However, if you wish to have all taxa displayed, you
can use the `--report-zero-counts` switch to do so. This can be useful if
you are looking to do further downstream analysis of the reports, and want
to compare samples. Sorting by the taxonomy ID (using `sort -k5,5n`) can
provide a consistent line ordering between reports.
In addition, we also provide the option `--use-mpa-style` that can be used
in conjunction with `--report`. This option provides output in a format
similar to MetaPhlAn's output. The output with this option provides one
taxon per line, with a lowercase version of the rank codes in Kraken 2's
standard sample report format (except for 'U' and 'R'), two underscores,
and the scientific name of the taxon (e.g., "d__Viruses"). The full
taxonomy of each taxon (at the eight ranks considered) is given, with each
rank's name separated by a pipe character (e.g., "d__Viruses|o_Caudovirales").
Following this version of the taxon's scientific name is a tab and the
number of fragments assigned to the clade rooted at that taxon.
Translated Search
=================
Kraken 2 allows users to perform a six-frame translated search, similar
to the well-known BLASTX program. To do this, Kraken 2 uses a reduced
15 amino acid alphabet and stores amino acid minimizers in its database.
LCA results from all 6 frames are combined to yield a set of LCA hits,
which is then resolved in the same manner as in Kraken's normal operation.
To build a protein database, the `--protein` option should be given to
`kraken2-build` (either along with `--standard`, or with all steps if
building a custom database).
Custom Databases
================
We realize the standard database may not suit everyone's needs. Kraken 2
also allows creation of customized databases.
To build a custom database:
1. Install a taxonomy. Usually, you will just use the NCBI taxonomy,
which you can easily download using:
kraken2-build --download-taxonomy --db $DBNAME
This will download the accession number to taxon maps, as well as the
taxonomic name and tree information from NCBI. These files can
be found in `$DBNAME/taxonomy/` . If you need to modify the taxonomy,
edits can be made to the `names.dmp` and `nodes.dmp` files in this
directory; you may also need to modify the `*.accession2taxid` files
appropriately.
Some of the standard sets of genomic libraries have taxonomic information
associated with them, and don't need the accession number to taxon maps
to build the database successfully. These libraries include all those
available through the `--download-library` option (see next point), except
for the `plasmid` and non-redundant databases. If you are not using
custom sequences (see the `--add-to-library` option) and are not using
one of the `plasmid` or non-redundant database libraries, you may want to
skip downloading of the accession number to taxon maps. This can be done
by passing `--skip-maps` to the `kraken2-build --download-taxonomy` command.
2. Install one or more reference libraries. Several sets of standard
genomes/proteins are made easily available through `kraken2-build`:
- `archaea`: RefSeq complete archaeal genomes/proteins
- `bacteria`: RefSeq complete bacterial genomes/proteins
- `plasmid`: RefSeq plasmid nucleotide/protein sequences
- `viral`: RefSeq complete viral genomes/proteins
- `human`: GRCh38 human genome/proteins
- `fungi`: RefSeq complete fungal genomes/proteins
- `plant`: RefSeq complete plant genomes/proteins
- `protozoa`: RefSeq complete protozoan genomes/proteins
- `nr`: NCBI non-redundant protein database
- `nt`: NCBI non-redundant nucleotide database
- `UniVec`: NCBI-supplied database of vector, adapter, linker, and
primer sequences that may be contaminating sequencing projects and/or
assemblies
- `UniVec_Core`: A subset of UniVec chosen to minimize false positive
hits to the vector database
To download and install any one of these, use the `--download-library`
switch, e.g.:
kraken2-build --download-library bacteria --db $DBNAME
Multiple libraries can be downloaded into a database prior to building
by issuing multiple `kraken2-build --download-library` commands, e.g.:
kraken2-build --download-library archaea --db $DBNAME
kraken2-build --download-library viral --db $DBNAME
The above commands would prepare a database that would contain archaeal
and viral genomes; the `--build` option (see below) will still need to
be used after downloading these libraries to actually build the database,
however.
(Note that downloading `nr` requires use of the `--protein`
option, and that `UniVec` and `UniVec_Core` are incompatible with
the `--protein` option.)
Other genomes can also be added, but such genomes must meet certain
requirements:
- Sequences must be in a FASTA file (multi-FASTA is allowed)
- Each sequence's ID (the string between the `>` and the first
whitespace character on the header line) must contain either
an NCBI accession number to allow Kraken 2 to lookup the correct taxa,
or an explicit assignment of the taxonomy ID using `kraken:taxid`
(see below).
Sequences not downloaded from NCBI may need their taxonomy information
assigned explicitly. This can be done using the string `kraken:taxid|XXX`
in the sequence ID, with `XXX` replaced by the desired taxon ID. For
example, to put a known adapter sequence in taxon 32630 ("synthetic
construct"), you could use the following:
>sequence16|kraken:taxid|32630 Adapter sequence
CAAGCAGAAGACGGCATACGAGATCTTCGAGTGACTGGAGTTCCTTGGCACCCGAGAATTCCA
The `kraken:taxid` string must begin the sequence ID or be immediately
preceded by a pipe character (`|`). Explicit assignment of taxonomy IDs
in this manner will override the accession number mapping provided by NCBI.
If your genomes meet the requirements above, then you can add each
sequence to your database's genomic library using the `--add-to-library`
switch, e.g.:
kraken2-build --add-to-library chr1.fa --db $DBNAME
kraken2-build --add-to-library chr2.fa --db $DBNAME
Note that if you have a list of files to add, you can do something like
this in `bash`:
for file in chr*.fa
do
kraken2-build --add-to-library $file --db $DBNAME
done
Or even add all `*.fa` files found in the directory `genomes`:
`find genomes/ -name '*.fa' -print0 | xargs -0 -I{} -n1 kraken2-build --add-to-library {} --db $DBNAME`
(You may also find the `-P` option to `xargs` useful to add many files in
parallel if you have multiple processors.)
3. Once your library is finalized, you need to build the database. This
can be done with the command:
kraken2-build --build --db $DBNAME
The `--threads` option is also helpful here to reduce build time.
By default, the values of $k$ and $\ell$ are 35 and 31, respectively (or
15 and 12 for protein databases). These values can be explicitly set
with the `--kmer-len` and `--minimizer-len` options, however. Note that
the minimizer length must be no more than 31 for nucleotide databases,
and 15 for protein databases. Additionally, the minimizer length $\ell$
must be no more than the $k$-mer length. There is no upper bound on
the value of $k$, but sequences less than $k$ bp in length cannot be
classified.
Kraken 2 also utilizes a simple spaced seed approach to increase
accuracy. A number $s$ < $\ell$/4 can be chosen, and $s$ positions
in the minimizer will be masked out during all comparisons.
Masked positions are chosen to alternate from the second-to-last
position in the minimizer; e.g., $s$ = 5 and $\ell$ = 31 will result
in masking out the 0 positions shown here:
111 1111 1111 1111 1111 1101 0101 0101
By default, $s$ = 7 for nucleotide databases, and $s$ = 0 for
protein databases. This can be changed using the `--minimizer-spaces`
option along with the `--build` task of `kraken2-build`.
A full list of options for `kraken2-build` can be obtained using
`kraken2-build --help`.
After building a database, if you want to reduce the disk usage of
the database, you can use the `--clean` option for `kraken2-build`
to remove intermediate files from the database directory.
Masking of Low-complexity Sequences
===================================
Low-complexity sequences, e.g. "ACACACACACACACACACACACACAC", are known
to occur in many different organisms and are typically less informative
for use in alignments; the BLAST programs often mask these sequences by
default. Using this masking can help prevent false positives in Kraken 2's
results, and so we have added this functionality as a default option to
Kraken 2's library download/addition process.
Kraken 2 uses two programs to perform low-complexity sequence masking,
both available from NCBI: `dustmasker`, for nucleotide sequences, and
`segmasker`, for amino acid sequences. These programs are available
as part of the NCBI BLAST+ suite. If these programs are not installed
on the local system and in the user's PATH when trying to use
`kraken2-build`, the database build will fail. Users who do not wish to
install these programs can use the `--no-masking` option to `kraken2-build`
in conjunction with any of the `--download-library`, `--add-to-library`, or
`--standard` options; use of the `--no-masking` option will skip masking of
low-complexity sequences during the build of the Kraken 2 database.
Special Databases
=================
To support some common use cases, we provide the ability to build Kraken 2
databases using data from various external databases. These external
databases may not follow the NCBI taxonomy, and so we've provided
mechanisms to automatically create a taxonomy that will work with Kraken 2
(although such taxonomies may not be identical to NCBI's).
To build one of these "special" Kraken 2 databases, use the following command:
kraken2-build --db $DBNAME --special TYPE
where the `TYPE` string is one of the database names listed below.
At present, the "special" Kraken 2 database support we provide is limited
to pre-packaged solutions for some public 16S sequence databases, but this may
grow in the future.
16S Databases
-------------
For targeted 16S sequencing projects, a normal Kraken 2 database using whole
genome data may use more resources than necessary. A Kraken 2 database created
from a well-curated genomic library of just 16S data can provide both a more
efficient solution as well as a more accurate set of predictions for such
projects. We provide support for building Kraken 2 databases from three
publicly available 16S databases:
* [Greengenes] (Kraken 2 database name: `greengenes`), using all available 16S data.
* [RDP] (Kraken 2 database name: `rdp`), using the bacterial and archaeal 16S data.
* [SILVA] (Kraken 2 database name: `silva`), using the Small subunit NR99 sequence set.
Note that these databases may have licensing restrictions regarding their data,
and it is your responsibility to ensure you are in compliance with those
restrictions; please visit the databases' websites for further details. The
`kraken2-build` script only uses publicly available URLs to download data and
then converts that data into a form compatible for use with Kraken 2.
Furthermore, if you use one of these databases in your research, please
visit the corresponding database's website to determine the appropriate and
up-to-date citation.
[Greengenes]: http://greengenes.lbl.gov/
[RDP]: http://rdp.cme.msu.edu/
[SILVA]: http://www.arb-silva.de/
Confidence Scoring
==================
At present, we have not yet developed a confidence score with a
probabilistic interpretation for Kraken 2. However, we have developed a
simple scoring scheme that has yielded good results for us, and we've
made that available in Kraken 2 through use of the `--confidence` option
to `kraken2`. The approach we use allows a user to specify a threshold
score in the [0,1] interval; the classifier then will adjust labels up
the tree until the label's score (described below) meets or exceeds that
threshold. If a label at the root of the taxonomic tree would not have
a score exceeding the threshold, the sequence is called unclassified by
Kraken 2 when this threshold is applied.
A sequence label's score is a fraction $C$/$Q$, where $C$ is the number of
$k$-mers mapped to LCA values in the clade rooted at the label, and $Q$ is the
number of $k$-mers in the sequence that lack an ambiguous nucleotide (i.e.,
they were queried against the database). Consider the example of the
LCA mappings in Kraken 2's output given earlier:
"562:13 561:4 A:31 0:1 562:3" would indicate that:
* the first 13 $k$-mers mapped to taxonomy ID #562
* the next 4 $k$-mers mapped to taxonomy ID #561
* the next 31 $k$-mers contained an ambiguous nucleotide
* the next $k$-mer was not in the database
* the last 3 $k$-mers mapped to taxonomy ID #562
In this case, ID #561 is the parent node of #562. Here, a label of #562
for this sequence would have a score of $C$/$Q$ = (13+3)/(13+4+1+3) = 16/21.
A label of #561 would have a score of $C$/$Q$ = (13+4+3)/(13+4+1+3) = 20/21.
If a user specified a `--confidence` threshold over 16/21, the classifier
would adjust the original label from #562 to #561; if the threshold was
greater than 20/21, the sequence would become unclassified.
Inspecting a Kraken 2 Database's Contents
=========================================
The `kraken2-inspect` script allows users to gain information about the content
of a Kraken 2 database. The output format of `kraken2-inspect`
is identical to the reports generated with the `--report` option to `kraken2`.
Instead of reporting how many reads in input data classified to a given taxon
or clade, as `kraken2`'s `--report` option would, the `kraken2-inspect` script
will report the number of minimizers in the database that are mapped to the
various taxa/clades. For example, the first five lines of `kraken2-inspect`'s
output on an example database might look like this:
$ kraken2-inspect --db EXAMPLE_DB | head -5
100.00% 1770368409 1581179 R 1 root
96.50% 1708407622 58003 R1 131567 cellular organisms
91.28% 1615910070 985309 D 2 Bacteria
43.89% 777062062 1312736 P 1224 Proteobacteria
18.62% 329590216 555667 C 1236 Gammaproteobacteria
This output indicates that 555667 of the minimizers in the database map
directly to the Gammaproteobacteria class (taxid #1236), and 329590216 (18.62%)
of the database's minimizers map to a taxon in the clade rooted at
Gammaproteobacteria. For more information on `kraken2-inspect`'s options,
use its `--help` option.
Distinct minimizer count information
====================================
The [KrakenUniq] project extended Kraken 1 by, among other things, reporting
an estimate of the number of distinct k-mers associated with each taxon in the
input sequencing data. This allows users to better determine if Kraken's
classifications are due to reads distributed throughout a reference genome,
or due to only a small segment of a reference genome (and therefore likely
false positive).
Thanks to the generosity of KrakenUniq's developer Florian Breitwieser in
allowing parts of the KrakenUniq source code to be licensed under Kraken 2's
MIT license, this distinct counting estimation is now available in Kraken 2.
Development work by Martin Steinegger and Ben Langmead helped bring this
functionality to Kraken 2.
At present, this functionality is an optional *experimental feature* -- meaning
that we may later alter it in a way that is not backwards compatible with
previous versions of the feature.
To use this functionality, simply run the `kraken2` script with the additional
`--report-minimizer-data` flag along with `--report`, e.g.:
kraken2 --db $DBNAME --report k2_report.txt --report-minimizer-data \
--output k2_output.txt sequence_data.fq
This will put the standard Kraken 2 output (formatted as described in
[Standard Kraken Output Format]) in `k2_output.txt` and the report information
in `k2_report.txt`. Within the report file, two additional columns will be
present, e.g.:
**normal report format**:
36.40 182 182 S2 211044 Influenza A virus (A/Puerto Rico/8/1934(H1N1))
**modified report format**:
36.40 182 182 1688 18 S2 211044 Influenza A virus (A/Puerto Rico/8/1934(H1N1))
In this modified report format, the two new columns are the fourth and fifth,
respectively representing the number of minimizers found to be associated with
a taxon in the read sequences (1688), and the estimate of the number of distinct
minimizers associated with a taxon in the read sequence data (18). This would
indicate that although 182 reads were classified as belonging to H1N1 influenza,
only 18 distinct minimizers led to those 182 classifications.
The format with the `--report-minimizer-data` flag, then, is similar to that
described in [Sample Report Output Format], but slightly different. The fields
in this new format, from left-to-right, are:
1. Percentage of fragments covered by the clade rooted at this taxon
2. Number of fragments covered by the clade rooted at this taxon
3. Number of fragments assigned directly to this taxon
4. Number of minimizers in read data associated with this taxon (**new**)
5. An estimate of the number of distinct minimizers in read data associated
with this taxon (**new**)
6. A rank code, indicating (U)nclassified, (R)oot, (D)omain, (K)ingdom,
(P)hylum, (C)lass, (O)rder, (F)amily, (G)enus, or (S)pecies.
Taxa that are not at any of these 10 ranks have a rank code that is
formed by using the rank code of the closest ancestor rank with
a number indicating the distance from that rank. E.g., "G2" is a
rank code indicating a taxon is between genus and species and the
grandparent taxon is at the genus rank.
7. NCBI taxonomic ID number
8. Indented scientific name
We decided to make this an optional feature so as not to break existing
software that processes Kraken 2's standard report format. However, this
new format can be converted to the standard report format with the command:
cut -f1-3,6-8 k2_new_report.txt > k2_std_report.txt
As noted above, this is an *experimental feature*. We intend to continue
development on this feature, and may change the new format and/or its
information if we determine it to be necessary.
For background on the data structures used in this feature and their
interaction with Kraken, please read the [KrakenUniq paper], and please
cite that paper if you use this functionality as part of your work.
[KrakenUniq]: https://github.com/fbreitwieser/krakenuniq
[KrakenUniq paper]: https://genomebiology.biomedcentral.com/articles/10.1186/s13059-018-1568-0
Kraken 2 Environment Variables
==============================
The `kraken2` and `kraken2-inspect` scripts supports the use of some
environment variables to help in reducing command line lengths:
* **`KRAKEN2_NUM_THREADS`**: if the
`--threads` option is not supplied to `kraken2`, then the value of this
variable (if it is set) will be used as the number of threads to run
`kraken2`. (This variable does not affect `kraken2-inspect`.)
* **`KRAKEN2_DB_PATH`**: much like the `PATH` variable is used for executables
by your shell, `KRAKEN2_DB_PATH` is a colon-separated list of directories
that will be searched for the database you name if the named database
does not have a slash (`/`) character. By default, Kraken 2 assumes the
value of this variable is "`.`" (i.e., the current working directory).
This variable can be used to create one (or more) central repositories
of Kraken databases in a multi-user system. Example usage in bash:
export KRAKEN2_DB_PATH="/home/user/my_kraken2_dbs:/data/kraken2_dbs:"
This will cause three directories to be searched, in this order:
1) `/home/user/my_kraken2_dbs`
2) `/data/kraken2_dbs`
3) the current working directory (caused by the empty string as
the third colon-separated field in the `KRAKEN2_DB_PATH` string)
The search for a database will stop when a name match is found; if
two directories in the `KRAKEN2_DB_PATH` have databases with the same
name, the directory of the two that is searched first will have its
database selected.
If the above variable and value are used, and the databases
`/data/kraken2_dbs/mainDB` and `./mainDB` are present, then
kraken2 --db mainDB sequences.fa
will classify `sequences.fa` using `/data/kraken_dbs/mainDB`; if instead
you wanted to use the `mainDB` present in the current directory,
you would need to specify a directory path to that database in order
to circumvent searching, e.g.:
kraken2 --db ./mainDB sequences.fa
Note that the `KRAKEN2_DB_PATH` directory list can be skipped by the use
of any absolute (beginning with `/`) or relative pathname (including
at least one `/`) as the database name.
* **`KRAKEN2_DEFAULT_DB`**: if no database is supplied with the `--db` option,
the database named in this variable will be used instead. Using this
variable, you can avoid using `--db` if you only have a single database
that you usually use, e.g. in bash:
export KRAKEN2_DEFAULT_DB="/home/user/kraken2db"
kraken2 sequences.fa > kraken2.output
This will classify `sequences.fa` using the `/home/user/kraken2db`
database.
Note that the value of `KRAKEN2_DEFAULT_DB` will also be interpreted in
the context of the value of `KRAKEN2_DB_PATH` if you don't set
`KRAKEN2_DEFAULT_DB` to an absolute or relative pathname. Given the earlier
example in this section, the following:
export KRAKEN2_DEFAULT_DB="mainDB"
kraken2 sequences.fa
will use `/data/kraken_dbs/mainDB` to classify `sequences.fa`.
kraken2-2.1.3/docs/Makefile 0000664 0000000 0000000 00000000505 14437747201 0015453 0 ustar 00root root 0000000 0000000 all:
pandoc -s \
-V "pagetitle:Kraken 2 Manual" \
--include-in-header head.html \
--include-before-body top.html \
--from markdown --to html \
--table-of-contents \
--css kraken.css \
--output MANUAL.html \
< MANUAL.markdown
#-V "pagetitle:Kraken 2 Manual" \
kraken2-2.1.3/docs/bar-bg.png 0000664 0000000 0000000 00000000274 14437747201 0015656 0 ustar 00root root 0000000 0000000 ‰PNG
IHDR š«Ä pHYs šœ tIMEß # %z¤ tEXtComment Created with GIMPW 6IDATHÇcdØðá?à¿??#ŒÍ¸ñã||b à Q‹G- µxÔâQ‹G- µxÔbú á % Ë" IEND®B`‚ kraken2-2.1.3/docs/head.html 0000664 0000000 0000000 00000000157 14437747201 0015605 0 ustar 00root root 0000000 0000000
kraken2-2.1.3/docs/kraken.css 0000664 0000000 0000000 00000001327 14437747201 0016003 0 ustar 00root root 0000000 0000000 body {
background: #f0f0f0 url("bar-bg.png") top left repeat-y;
color: #333;
margin: 10px;
margin-left: 50px;
margin-bottom: 20px;
font-family: 'Ubuntu', sans-serif;
}
a { color: #00b0f0; }
a:visited { color: #0090f0; }
a:hover { color: #00d0ff; }
a:active { color: #00f0ff; }
.pretoc {
text-align: center;
font-size: 1.2em;
}
.title {
font-size: 2em;
font-weight: bold;
margin-bottom: 0;
}
.version {
font-size: 0.9em;
}
h1 {
color: #0090f0;
border-bottom: 1px #0090f0 solid;
margin-left: -10px;
margin-bottom: 3px;
}
h1 a {
color: #0090f0;
text-decoration: none;
}
div#confidence-score-table table th {
width: 7em;
}
pre {
margin-left: 4em;
}
code {
font-size: 1.2em;
}
kraken2-2.1.3/docs/top.html 0000664 0000000 0000000 00000000302 14437747201 0015476 0 ustar 00root root 0000000 0000000
Kraken taxonomic sequence classification system
Version 2.1.3
Operating Manual
Table of Contents
kraken2-2.1.3/install_kraken2.sh 0000775 0000000 0000000 00000002361 14437747201 0016507 0 ustar 00root root 0000000 0000000 #!/bin/bash
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
set -e
VERSION="2.1.3"
if [ -z "$1" ] || [ -n "$2" ]
then
echo "Usage: $(basename $0) KRAKEN2_DIR"
exit 64
fi
if [ "$1" = "KRAKEN2_DIR" ]
then
echo "Please replace \"KRAKEN2_DIR\" with the name of the directory"
echo "that you want to install Kraken 2 in."
exit 1
fi
# Perl cmd used to canonicalize dirname - "readlink -f" doesn't work
# on OS X.
export KRAKEN2_DIR=$(perl -MCwd=abs_path -le 'print abs_path(shift)' "$1")
mkdir -p "$KRAKEN2_DIR"
make -C src install
for file in scripts/*
do
perl -pl -e 'BEGIN { while (@ARGV) { $_ = shift; ($k,$v) = split /=/, $_, 2; $H{$k} = $v } }'\
-e 's/#####=(\w+)=#####/$H{$1}/g' \
"KRAKEN2_DIR=$KRAKEN2_DIR" "VERSION=$VERSION" \
< "$file" > "$KRAKEN2_DIR/$(basename $file)"
if [ -x "$file" ]
then
chmod +x "$KRAKEN2_DIR/$(basename $file)"
fi
done
echo
echo "Kraken 2 installation complete."
echo
echo "To make things easier for you, you may want to copy/symlink the following"
echo "files into a directory in your PATH:"
for file in $KRAKEN2_DIR/kraken2*
do
if [ -x "$file" ]
then
echo " $file"
fi
done
kraken2-2.1.3/scripts/ 0000775 0000000 0000000 00000000000 14437747201 0014552 5 ustar 00root root 0000000 0000000 kraken2-2.1.3/scripts/16S_gg_installation.sh 0000775 0000000 0000000 00000001677 14437747201 0020733 0 ustar 00root root 0000000 0000000 #!/bin/bash
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Build a 16S database from Greengenes data
set -u # Protect against uninitialized vars.
set -e # Stop on error
set -o pipefail # Stop on failures in non-final pipeline commands
FTP_SERVER="ftp://greengenes.microbio.me/"
GG_VERSION="gg_13_5"
REMOTE_DIR="$FTP_SERVER/greengenes_release/$GG_VERSION"
mkdir -p "$KRAKEN2_DB_NAME"
pushd "$KRAKEN2_DB_NAME"
mkdir -p data taxonomy library
pushd data
wget "$REMOTE_DIR/${GG_VERSION}.fasta.gz"
gunzip "${GG_VERSION}.fasta.gz"
wget "$REMOTE_DIR/${GG_VERSION}_taxonomy.txt.gz"
gunzip "${GG_VERSION}_taxonomy.txt.gz"
build_gg_taxonomy.pl "${GG_VERSION}_taxonomy.txt"
popd
mv data/names.dmp data/nodes.dmp taxonomy/
mv data/seqid2taxid.map .
mv "data/${GG_VERSION}.fasta" library/gg.fna
popd
kraken2-build --db $KRAKEN2_DB_NAME --build --threads $KRAKEN2_THREAD_CT
kraken2-2.1.3/scripts/16S_rdp_installation.sh 0000775 0000000 0000000 00000001674 14437747201 0021120 0 ustar 00root root 0000000 0000000 #!/bin/bash
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Build a 16S database from RDP data
set -u # Protect against uninitialized vars.
set -e # Stop on error
set -o pipefail # Stop on failures in non-final pipeline commands
HTTP_SERVER="http://rdp.cme.msu.edu/"
REMOTE_DIR="$HTTP_SERVER/download/"
mkdir -p "$KRAKEN2_DB_NAME"
pushd "$KRAKEN2_DB_NAME"
mkdir -p data taxonomy library
pushd data
wget "$REMOTE_DIR/current_Bacteria_unaligned.fa.gz"
gunzip "current_Bacteria_unaligned.fa.gz"
wget "$REMOTE_DIR/current_Archaea_unaligned.fa.gz"
gunzip "current_Archaea_unaligned.fa.gz"
build_rdp_taxonomy.pl current_*_unaligned.fa
popd
mv data/names.dmp data/nodes.dmp taxonomy/
mv data/seqid2taxid.map .
for file in data/*.fa; do
mv $file library/$(basename $file .fa).fna
done
popd
kraken2-build --db $KRAKEN2_DB_NAME --build --threads $KRAKEN2_THREAD_CT
kraken2-2.1.3/scripts/16S_silva_installation.sh 0000775 0000000 0000000 00000002251 14437747201 0021441 0 ustar 00root root 0000000 0000000 #!/bin/bash
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Build a 16S database from Silva data
set -u # Protect against uninitialized vars.
set -e # Stop on error
set -o pipefail # Stop on failures in non-final pipeline commands
HTTPS_SERVER="https://ftp.arb-silva.de/"
SILVA_VERSION="138_1"
REMOTE_DIR="$HTTPS_SERVER/release_$SILVA_VERSION/Exports"
FASTA_FILENAME="SILVA_${SILVA_VERSION/_/.}_SSURef_NR99_tax_silva.fasta"
TAXO_PREFIX="tax_slv_ssu_${SILVA_VERSION/_/.}"
mkdir -p "$KRAKEN2_DB_NAME"
pushd "$KRAKEN2_DB_NAME"
mkdir -p data taxonomy library
pushd data
wget "$REMOTE_DIR/${FASTA_FILENAME}.gz"
gunzip "${FASTA_FILENAME}.gz"
wget "$REMOTE_DIR/taxonomy/${TAXO_PREFIX}.acc_taxid.gz"
gunzip "${TAXO_PREFIX}.acc_taxid.gz"
wget "$REMOTE_DIR/taxonomy/${TAXO_PREFIX}.txt.gz"
gunzip "${TAXO_PREFIX}.txt.gz"
build_silva_taxonomy.pl "${TAXO_PREFIX}.txt"
popd
mv data/names.dmp data/nodes.dmp taxonomy/
mv data/${TAXO_PREFIX}.acc_taxid seqid2taxid.map
sed -e '/^>/!y/U/T/' "data/$FASTA_FILENAME" > library/silva.fna
popd
kraken2-build --db $KRAKEN2_DB_NAME --build --threads $KRAKEN2_THREAD_CT
kraken2-2.1.3/scripts/add_to_library.sh 0000775 0000000 0000000 00000001663 14437747201 0020075 0 ustar 00root root 0000000 0000000 #!/bin/bash
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Copy specified file into a Kraken library
set -u # Protect against uninitialized vars.
set -e # Stop on error
LIBRARY_DIR="$KRAKEN2_DB_NAME/library"
if [ ! -e "$1" ]
then
1>&2 echo "Can't add \"$1\": file does not exist"
exit 1
fi
if [ ! -f "$1" ]
then
1>&2 echo "Can't add \"$1\": not a regular file"
exit 1
fi
add_dir="$LIBRARY_DIR/added"
mkdir -p "$add_dir"
prelim_map=$(cp_into_tempfile.pl -t "prelim_map_XXXXXXXXXX" -d "$add_dir" -s txt /dev/null)
scan_fasta_file.pl "$1" > "$prelim_map"
filename=$(cp_into_tempfile.pl -t "XXXXXXXXXX" -d "$add_dir" -s fna "$1")
if [ -n "$KRAKEN2_MASK_LC" ]; then
1>&2 echo -n "Masking low-complexity regions of new file..."
mask_low_complexity.sh $filename
1>&2 echo " done."
fi
1>&2 echo "Added \"$1\" to library ($KRAKEN2_DB_NAME)"
kraken2-2.1.3/scripts/build_gg_taxonomy.pl 0000775 0000000 0000000 00000004606 14437747201 0020632 0 ustar 00root root 0000000 0000000 #!/usr/bin/env perl
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Parses Greengenes taxonomy file to create Kraken taxonomy
# and sequence ID -> taxonomy ID mapping
# Input (as <>): gg_13_5_taxonomy.txt
use strict;
use warnings;
use File::Basename;
my $PROG = basename $0;
my %RANK_CODES = (
k => "superkingdom",
p => "phylum",
c => "class",
o => "order",
f => "family",
g => "genus",
s => "species"
);
my %seqid_map;
my %seen_it;
my %child_data = ("root" => {});
LINE: while (<>) {
chomp;
my ($seqid, $taxo_str) = split /\t/;
$taxo_str =~ s/(; [a-z]__)+$//; # Remove empty data
$seqid_map{$seqid} = $taxo_str;
next if $seen_it{$taxo_str}++;
while ($taxo_str =~ s/(; [a-z]__[^;]+$)//) {
my $level = $1;
my $parent = $taxo_str;
$child_data{$parent} ||= {};
$child_data{$parent}->{"$taxo_str$level"}++;
next LINE if $seen_it{$taxo_str}++;
}
$child_data{"root"}->{$taxo_str}++;
}
# Assign IDs through BFS of tree, report names/nodes info in
# NCBI format
my %id_map;
my $next_node_id = 1;
open NAMES, ">", "names.dmp" or die "$PROG: can't write names.dmp: $!\n";
open NODES, ">", "nodes.dmp" or die "$PROG: can't write nodes.dmp: $!\n";
my @bfs_queue = (["root", 1]);
while (@bfs_queue) {
my $arg_ref = shift @bfs_queue;
my ($node, $parent_id) = @$arg_ref;
my $display_name = $node;
my $rank;
# special handling for species
if ($node =~ /g__([^;]+); s__([^;]+)$/) {
my ($genus, $species) = ($1, $2);
$rank = "species";
if ($species =~ / endosymbiont /) {
$display_name = $species;
}
else {
$display_name = "$genus $species";
}
}
elsif ($node =~ /([a-z])__([^;]+)$/) {
$rank = $RANK_CODES{$1};
$display_name = $2;
}
$rank ||= "no rank";
my $node_id = $next_node_id++;
$id_map{$node} = $node_id;
print NAMES "$node_id\t|\t$display_name\t|\t-\t|\tscientific name\t|\n";
print NODES "$node_id\t|\t$parent_id\t|\t$rank\t|\t-\t|\n";
my @children = sort keys %{ $child_data{$node} };
push @bfs_queue, [$_, $node_id] for @children;
}
close NAMES;
close NODES;
open SEQID_TAXID_MAP, ">", "seqid2taxid.map" or die "$PROG: can't write seqid2taxid.map: $!\n";
for my $seqid (sort { $a <=> $b } keys %seqid_map) {
my $taxid = $id_map{ $seqid_map{$seqid} };
print SEQID_TAXID_MAP "$seqid\t$taxid\n";
}
close SEQID_TAXID_MAP;
kraken2-2.1.3/scripts/build_kraken2_db.sh 0000775 0000000 0000000 00000011056 14437747201 0020275 0 ustar 00root root 0000000 0000000 #!/bin/bash
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Build a Kraken 2 database
# Designed to be called by kraken2-build
set -u # Protect against uninitialized vars.
set -e # Stop on error
set -o pipefail # Stop on failures in non-final pipeline commands
function finalize_file() {
mv $1.tmp $1
}
function get_current_time() {
date "+%s.%N"
}
function report_time_elapsed() {
curr_time=$(get_current_time)
perl -e '$time = $ARGV[1] - $ARGV[0];' \
-e '$sec = int($time); $nsec = $time - $sec;' \
-e '$min = int($sec/60); $sec %= 60;' \
-e '$hr = int($min/60); $min %= 60;' \
-e 'print "${hr}h" if $hr;' \
-e 'print "${min}m" if $min || $hr;' \
-e 'printf "%.3fs", $sec + $nsec;' \
$1 $curr_time
}
function list_sequence_files() {
find library/ '(' -name '*.fna' -o -name '*.faa' ')' -print0
}
start_time=$(get_current_time)
DATABASE_DIR="$KRAKEN2_DB_NAME"
if [ ! -d "$DATABASE_DIR" ]
then
echo "Can't find Kraken 2 DB directory \"$KRAKEN2_DB_NAME\""
exit 1
fi
cd "$DATABASE_DIR"
if [ ! -d "taxonomy/" ]
then
echo "Can't find taxonomy/ subdirectory in database directory, exiting."
exit 1
fi
if [ ! -d "library/" ]
then
echo "Can't find library/ subdirectory in database directory, exiting."
exit 1
fi
KRAKEN2XFLAG=""
if [ -n "$KRAKEN2_PROTEIN_DB" ]
then
KRAKEN2XFLAG="-X"
fi
echo "Creating sequence ID to taxonomy ID map (step 1)..."
if [ -d "library/added" ]; then
find library/added/ -name 'prelim_map_*.txt' | xargs cat > library/added/prelim_map.txt
fi
seqid2taxid_map_file=seqid2taxid.map
if [ -e "$seqid2taxid_map_file" ]; then
echo "Sequence ID to taxonomy ID map already present, skipping map creation."
else
step_time=$(get_current_time)
find library/ -maxdepth 2 -name prelim_map.txt | xargs cat > taxonomy/prelim_map.txt
if [ ! -s "taxonomy/prelim_map.txt" ]; then
echo "No preliminary seqid/taxid mapping files found, aborting."
exit 1
fi
grep "^TAXID" taxonomy/prelim_map.txt | cut -f 2- > $seqid2taxid_map_file.tmp || true
if grep "^ACCNUM" taxonomy/prelim_map.txt | cut -f 2- > accmap_file.tmp; then
if compgen -G "taxonomy/*.accession2taxid" > /dev/null; then
lookup_accession_numbers accmap_file.tmp taxonomy/*.accession2taxid > seqid2taxid_acc.tmp
cat seqid2taxid_acc.tmp >> $seqid2taxid_map_file.tmp
rm seqid2taxid_acc.tmp
else
echo "Accession to taxid map files are required to build this DB."
echo "Run 'kraken2-build --db $KRAKEN2_DB_NAME --download-taxonomy' again?"
exit 1
fi
fi
rm -f accmap_file.tmp
finalize_file $seqid2taxid_map_file
echo "Sequence ID to taxonomy ID map complete. [$(report_time_elapsed $step_time)]"
fi
echo "Estimating required capacity (step 2)..."
step_time=$(get_current_time)
estimate=$(list_sequence_files | xargs -0 cat | estimate_capacity -k $KRAKEN2_KMER_LEN -l $KRAKEN2_MINIMIZER_LEN -S $KRAKEN2_SEED_TEMPLATE -p $KRAKEN2_THREAD_CT $KRAKEN2XFLAG )
# Slight upward adjustment of distinct minimizer estimate to protect
# against crash w/ small reference sets
estimate=$(( estimate + 8192 ))
required_capacity=$(perl -le 'print int(shift() / shift())' $estimate $KRAKEN2_LOAD_FACTOR);
echo "Estimated hash table requirement: $(( required_capacity * 4 )) bytes"
max_db_flag=""
if [ -n "$KRAKEN2_MAX_DB_SIZE" ]
then
if (( KRAKEN2_MAX_DB_SIZE < (required_capacity * 4) ))
then
max_db_flag="-M $(perl -le 'print int(shift() / 4)' $KRAKEN2_MAX_DB_SIZE)"
echo "Specifying lower maximum hash table size of $KRAKEN2_MAX_DB_SIZE bytes"
fi
fi
echo "Capacity estimation complete. [$(report_time_elapsed $step_time)]"
echo "Building database files (step 3)..."
fast_build_flag=""
if [ -n "$KRAKEN2_FAST_BUILD" ]
then
fast_build_flag="-F"
fi
if [ -e "hash.k2d" ]
then
echo "Hash table already present, skipping database file build."
else
step_time=$(get_current_time)
list_sequence_files | xargs -0 cat | \
build_db -k $KRAKEN2_KMER_LEN -l $KRAKEN2_MINIMIZER_LEN -S $KRAKEN2_SEED_TEMPLATE $KRAKEN2XFLAG \
-H hash.k2d.tmp -t taxo.k2d.tmp -o opts.k2d.tmp -n taxonomy/ -m $seqid2taxid_map_file \
-c $required_capacity -p $KRAKEN2_THREAD_CT $max_db_flag -B $KRAKEN2_BLOCK_SIZE -b $KRAKEN2_SUBBLOCK_SIZE \
-r $KRAKEN2_MIN_TAXID_BITS $fast_build_flag
finalize_file taxo.k2d
finalize_file opts.k2d
finalize_file hash.k2d
echo "Database files completed. [$(report_time_elapsed $step_time)]"
fi
echo "Database construction complete. [Total: $(report_time_elapsed $start_time)]"
kraken2-2.1.3/scripts/build_rdp_taxonomy.pl 0000775 0000000 0000000 00000004217 14437747201 0021020 0 ustar 00root root 0000000 0000000 #!/usr/bin/env perl
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Parses RDP sequence data to create Kraken taxonomy
# and sequence ID -> taxonomy ID mapping
# Input (as <>): current_{Archaea,Bacteria}_unaligned.fa
use strict;
use warnings;
use File::Basename;
my $PROG = basename $0;
my %seqid_map;
my %seen_it;
my %child_data = ("root;no rank" => {});
LINE: while (<>) {
next unless s/^>//;
chomp;
my ($seq_label, $taxo_str) = split /\t/;
my ($seqid) = split " ", $seq_label;
$taxo_str =~ s/^Lineage=Root;rootrank;/root;no rank;/;
$taxo_str =~ s/;$/;no rank/; # adjust for unclassified things
$seqid_map{$seqid} = $taxo_str;
next if $seen_it{$taxo_str}++;
while ($taxo_str =~ s/(;[^;]+;[^;]+)$//) {
my $level = $1;
my $parent = $taxo_str;
$child_data{$parent} ||= {};
$child_data{$parent}->{"$taxo_str$level"}++;
next LINE if $seen_it{$taxo_str}++;
}
}
# Assign IDs through BFS of tree, report names/nodes info in
# NCBI format
my %id_map;
my $next_node_id = 1;
open NAMES, ">", "names.dmp" or die "$PROG: can't write names.dmp: $!\n";
open NODES, ">", "nodes.dmp" or die "$PROG: can't write nodes.dmp: $!\n";
my @bfs_queue = (["root;no rank", 1]);
while (@bfs_queue) {
my $arg_ref = shift @bfs_queue;
my ($node, $parent_id) = @$arg_ref;
if ($node !~ /([^;]+);([^;]+)$/) {
die "$PROG: BFS processing encountered formatting error, \"$node\"\n";
}
my ($display_name, $rank) = ($1, $2);
$rank = "superkingdom" if $rank eq "domain"; # conform w/ NCBI taxonomy
my $node_id = $next_node_id++;
$id_map{$node} = $node_id;
print NAMES "$node_id\t|\t$display_name\t|\t-\t|\tscientific name\t|\n";
print NODES "$node_id\t|\t$parent_id\t|\t$rank\t|\t-\t|\n";
my @children = sort keys %{ $child_data{$node} };
push @bfs_queue, [$_, $node_id] for @children;
}
close NAMES;
close NODES;
open SEQID_TAXID_MAP, ">", "seqid2taxid.map" or die "$PROG: can't write seqid2taxid.map: $!\n";
for my $seqid (sort keys %seqid_map) {
my $taxid = $id_map{ $seqid_map{$seqid} };
print SEQID_TAXID_MAP "$seqid\t$taxid\n";
}
close SEQID_TAXID_MAP;
kraken2-2.1.3/scripts/build_silva_taxonomy.pl 0000775 0000000 0000000 00000002365 14437747201 0021353 0 ustar 00root root 0000000 0000000 #!/usr/bin/env perl
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Parses Silva taxonomy file to create Kraken taxonomy
# Input (as <>): tax_slv_ssu_nr_119.txt
use strict;
use warnings;
use File::Basename;
my $PROG = basename $0;
my %id_map = ("root" => 1);
open NAMES, ">", "names.dmp" or die "$PROG: can't write names.dmp: $!\n";
open NODES, ">", "nodes.dmp" or die "$PROG: can't write nodes.dmp: $!\n";
print NAMES "1\t|\troot\t|\t-\t|\tscientific name\t|\n";
print NODES "1\t|\t1\t|\tno rank\t|\t-\t|\n";
while (<>) {
chomp;
my ($taxo_str, $node_id, $rank) = split /\t/;
$id_map{$taxo_str} = $node_id;
if ($taxo_str =~ /^(.+;|)([^;]+);$/) {
my $parent_name = $1;
my $display_name = $2;
if ($parent_name eq "") {
$parent_name = "root";
}
my $parent_id = $id_map{$parent_name};
if (! defined $parent_id) {
die "$PROG: orphan error, line $.\n";
}
$rank = "superkingdom" if $rank eq "domain";
print NAMES "$node_id\t|\t$display_name\t|\t-\t|\tscientific name\t|\n";
print NODES "$node_id\t|\t$parent_id\t|\t$rank\t|\t-\t|\n";
}
else {
die "$PROG: strange input, line $.\n";
}
}
close NAMES;
close NODES;
kraken2-2.1.3/scripts/clean_db.sh 0000775 0000000 0000000 00000001107 14437747201 0016637 0 ustar 00root root 0000000 0000000 #!/bin/bash
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Removes intermediate files from a database directory,
# such as reference library FASTA files and taxonomy data from NCBI.
set -u # Protect against uninitialized vars.
set -e # Stop on error
cd $KRAKEN2_DB_NAME
previous_usage=$(du -sh | cut -f1)
1>&2 echo "Database disk usage: $previous_usage"
rm -rf library/ taxonomy/ seqid2taxid.map
current_usage=$(du -sh | cut -f1)
1>&2 echo "After cleaning, database uses $current_usage"
kraken2-2.1.3/scripts/cp_into_tempfile.pl 0000775 0000000 0000000 00000002375 14437747201 0020441 0 ustar 00root root 0000000 0000000 #!/usr/bin/env perl
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Create a file in a specified directory, then copy an
# existing file's contents into the new file. Write name of
# new file to standard output.
#
# This exists because the mktemp program doesn't act consistently across
# operating systems/distros/versions.
use strict;
use warnings;
use File::Basename;
use File::Temp 'tempfile';
use Getopt::Std;
my $PROG = basename $0;
getopts('d:t:s:', \my %opts) or usage();
$opts{$_} or usage() for qw/d t s/; # all switches mandatory
my ($directory, $template, $suffix) = @opts{qw/d t s/};
die "$PROG: '$directory' not a directory!\n" unless -d $directory;
die "$PROG: must specify a single filename\n" unless @ARGV == 1;
$suffix =~ s/^\.//;
my $old_filename = shift @ARGV;
open FILE, "<", $old_filename
or die "$PROG: can't read $old_filename: $!\n";
my ($fh, $new_filename) = tempfile($template, DIR => $directory,
UNLINK => 0, SUFFIX => ".$suffix");
# copy loop
while () {
print {$fh} $_;
}
close FILE;
close $fh;
print "$new_filename\n";
sub usage {
die "$PROG: <-d directory> <-t template> <-s suffix> \n";
}
kraken2-2.1.3/scripts/download_genomic_library.sh 0000775 0000000 0000000 00000010621 14437747201 0022145 0 ustar 00root root 0000000 0000000 #!/bin/bash
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Download specific genomic libraries for use with Kraken 2.
# Supported libraries were chosen based on support from NCBI's FTP site
# in easily obtaining a good collection of genomic data. Others may
# be added upon popular demand.
set -u # Protect against uninitialized vars.
set -e # Stop on error
LIBRARY_DIR="$KRAKEN2_DB_NAME/library"
NCBI_SERVER="ftp.ncbi.nlm.nih.gov"
FTP_SERVER="ftp://$NCBI_SERVER"
RSYNC_SERVER="rsync://$NCBI_SERVER"
THIS_DIR=$PWD
library_name="$1"
ftp_subdir=$library_name
library_file="library.fna"
if [ -n "$KRAKEN2_PROTEIN_DB" ]; then
library_file="library.faa"
fi
function download_file() {
file="$1"
if [ -n "$KRAKEN2_USE_FTP" ]
then
wget -q ${FTP_SERVER}${file}
else
rsync --no-motd ${RSYNC_SERVER}${file} .
fi
}
case $library_name in
"archaea" | "bacteria" | "viral" | "fungi" | "plant" | "human" | "protozoa")
mkdir -p $LIBRARY_DIR/$library_name
cd $LIBRARY_DIR/$library_name
rm -f assembly_summary.txt
remote_dir_name=$library_name
if [ "$library_name" = "human" ]; then
remote_dir_name="vertebrate_mammalian/Homo_sapiens"
fi
if ! download_file "/genomes/refseq/$remote_dir_name/assembly_summary.txt"; then
1>&2 echo "Error downloading assembly summary file for $library_name, exiting."
exit 1
fi
if [ "$library_name" = "human" ]; then
grep "Genome Reference Consortium" assembly_summary.txt > x
mv x assembly_summary.txt
fi
rm -rf all/ library.f* manifest.txt rsync.err
rsync_from_ncbi.pl assembly_summary.txt
scan_fasta_file.pl $library_file >> prelim_map.txt
;;
"plasmid")
mkdir -p $LIBRARY_DIR/plasmid
cd $LIBRARY_DIR/plasmid
rm -f library.f* plasmid.*
## This is staying FTP only D/L for now
1>&2 echo -n "Downloading plasmid files from FTP..."
wget -q --no-remove-listing --spider $FTP_SERVER/genomes/refseq/plasmid/
if [ -n "$KRAKEN2_PROTEIN_DB" ]; then
awk '{ print $NF }' .listing | perl -ple 'tr/\r//d' | grep '\.faa\.gz' > manifest.txt
else
awk '{ print $NF }' .listing | perl -ple 'tr/\r//d' | grep '\.fna\.gz' > manifest.txt
fi
cat manifest.txt | xargs -n1 -I{} wget -q $FTP_SERVER/genomes/refseq/plasmid/{}
cat manifest.txt | xargs -n1 -I{} gunzip -c {} > $library_file
rm -f plasmid.* .listing
scan_fasta_file.pl $library_file > prelim_map.txt
1>&2 echo " done."
;;
"nr" | "nt")
protein_lib=0
if [ "$library_name" = "nr" ]; then
protein_lib=1
fi
if (( protein_lib == 1 )) && [ -z "$KRAKEN2_PROTEIN_DB" ]; then
1>&2 echo "$library_name is a protein database, and the Kraken DB specified is nucleotide"
exit 1
fi
mkdir -p $LIBRARY_DIR/$library_name
cd $LIBRARY_DIR/$library_name
rm -f $library_name.gz
1>&2 echo -n "Downloading $library_name database from server... "
download_file "/blast/db/FASTA/$library_name.gz"
1>&2 echo "done."
1>&2 echo -n "Uncompressing $library_name database..."
gunzip $library_name.gz
mv $library_name $library_file
1>&2 echo "done."
1>&2 echo -n "Parsing $library_name FASTA file..."
# The nr/nt files tend to have non-standard sequence IDs, so
# --lenient is used here.
scan_fasta_file.pl --lenient $library_file >> prelim_map.txt
1>&2 echo "done."
;;
"UniVec" | "UniVec_Core")
if [ -n "$KRAKEN2_PROTEIN_DB" ]; then
1>&2 echo "$library_name is for nucleotide databases only"
exit 1
fi
mkdir -p $LIBRARY_DIR/$library_name
cd $LIBRARY_DIR/$library_name
1>&2 echo -n "Downloading $library_name data from server... "
download_file "/pub/UniVec/$library_name"
1>&2 echo "done."
# 28384: "other sequences"
special_taxid=28384
1>&2 echo -n "Adding taxonomy ID of $special_taxid to all sequences... "
sed -e "s/^>/>kraken:taxid|$special_taxid|/" $library_name > library.fna
scan_fasta_file.pl library.fna > prelim_map.txt
1>&2 echo "done."
;;
*)
1>&2 echo "Unsupported library. Valid options are: "
1>&2 echo " archaea bacteria viral fungi plant protozoa human plasmid"
1>&2 echo " nr nt UniVec UniVec_Core"
exit 1
;;
esac
if [ -n "$KRAKEN2_MASK_LC" ]; then
1>&2 echo -n "Masking low-complexity regions of downloaded library..."
mask_low_complexity.sh .
1>&2 echo " done."
fi
kraken2-2.1.3/scripts/download_taxonomy.sh 0000775 0000000 0000000 00000003340 14437747201 0020656 0 ustar 00root root 0000000 0000000 #!/bin/bash
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Download NCBI taxonomy information for Kraken 2.
# Designed to be called by kraken2-build
set -u # Protect against uninitialized vars.
set -e # Stop on error
TAXONOMY_DIR="$KRAKEN2_DB_NAME/taxonomy"
NCBI_SERVER="ftp.ncbi.nlm.nih.gov"
RSYNC_SERVER="rsync://$NCBI_SERVER"
FTP_SERVER="ftp://$NCBI_SERVER"
mkdir -p "$TAXONOMY_DIR"
cd "$TAXONOMY_DIR"
function download_file() {
file="$1"
if [ -n "$KRAKEN2_USE_FTP" ]
then
wget -q ${FTP_SERVER}${file}
else
rsync --no-motd ${RSYNC_SERVER}${file} .
fi
}
if [ ! -e "accmap.dlflag" ] && [ -z "$KRAKEN2_SKIP_MAPS" ]
then
if [ -z "$KRAKEN2_PROTEIN_DB" ]
then
for subsection in gb wgs
do
1>&2 echo -n "Downloading nucleotide ${subsection} accession to taxon map..."
download_file "/pub/taxonomy/accession2taxid/nucl_${subsection}.accession2taxid.gz"
1>&2 echo " done."
done
else
1>&2 echo -n "Downloading protein accession to taxon map..."
download_file "/pub/taxonomy/accession2taxid/prot.accession2taxid.gz"
1>&2 echo " done."
fi
touch accmap.dlflag
1>&2 echo "Downloaded accession to taxon map(s)"
fi
if [ ! -e "taxdump.dlflag" ]
then
1>&2 echo -n "Downloading taxonomy tree data..."
download_file "/pub/taxonomy/taxdump.tar.gz"
touch taxdump.dlflag
1>&2 echo " done."
fi
if ls | grep -q 'accession2taxid\.gz$'
then
1>&2 echo -n "Uncompressing taxonomy data..."
gunzip *accession2taxid.gz
1>&2 echo " done."
fi
if [ ! -e "taxdump.untarflag" ]
then
1>&2 echo -n "Untarring taxonomy tree data..."
tar zxf taxdump.tar.gz
touch taxdump.untarflag
1>&2 echo " done."
fi
kraken2-2.1.3/scripts/k2 0000775 0000000 0000000 00000226603 14437747201 0015025 0 ustar 00root root 0000000 0000000 #!/usr/bin/env python3
import argparse
import bz2
import ftplib
import glob
import gzip
import hashlib
import inspect
import itertools
import logging
import lzma
import math
import os
import re
import shutil
import subprocess
import sys
import tarfile
import tempfile
import threading
import urllib
import urllib.error
import urllib.parse
import urllib.request
LOG = None
SCRIPT_PATHNAME = None
NCBI_SERVER = "ftp.ncbi.nlm.nih.gov"
GREENGENES_SERVER = "greengenes.microbio.me"
SILVA_SERVER = "ftp.arb-silva.de"
WRAPPER_ARGS_TO_BIN_ARGS = {
"block_size": "-B",
"classified_out": "-C",
"confidence": "-T",
"fast_build": "-F",
"interleaved": "-S",
"kmer_len": "-k",
"memory_mapping": "-M",
"minimizer_len": "-l",
"minimum_bits_for_taxid": "-r",
"mininum_base_quality": "-Q",
"mininum_hit_groups": "-g",
"output": "-O",
"paired": "-P",
"protein": "-X",
"quick": "-q",
"report": "-R",
"report_minimizer_data": "-K",
"report_zero_counts": "-z",
"skip_counts": "-s",
"sub_block_size": "-b",
"threads": "-p",
"unclassified_out": "-U",
"use_mpa_style": "-m",
"use_names": "-n",
}
class FTP:
def __init__(self, server):
self.ftp = ftplib.FTP(server, timeout=600)
self.ftp.login()
self.ftp.sendcmd("TYPE I")
self.pwd = "/"
self.server = server
def _progress_bar(self, f, remote_size):
pb = ProgressBar(remote_size, f.tell())
def inner(block):
nonlocal f, remote_size, pb
f.write(block)
size_on_disk = f.tell()
# if size_on_disk > remote_size:
# size_on_disk = remote_size
pb.progress(size_on_disk)
LOG.debug(
"{:s} {: >10s}\r".format(
pb.get_bar(), format_bytes(size_on_disk)
)
)
return inner
def download(self, remote_dir, filepaths):
if isinstance(filepaths, str):
filepaths = [filepaths]
number_of_files = len(filepaths)
self.cwd(remote_dir)
for index, filepath in enumerate(filepaths):
mode = "ab"
local_size = 0
remote_size = self.size(filepath)
if os.path.exists(filepath):
local_size = os.stat(filepath).st_size
else:
if os.path.basename(filepath) != filepath:
os.makedirs(os.path.dirname(filepath), exist_ok=True)
if local_size == remote_size:
LOG.info(
"Already downloaded {:s}\n".format(get_abs_path(filepath))
)
continue
if local_size > remote_size:
mode = "wb"
url_components = urllib.parse.SplitResult(
"ftp", self.server, os.path.join(remote_dir, filepath), "", ""
)
url = urllib.parse.urlunsplit(url_components)
if number_of_files == 1:
LOG.info("Downloading {:s}\n".format(url))
else:
LOG.info(
"[{:d}/{:d}] Downloading {:s}\n".format(
index + 1, number_of_files, url
)
)
with open(filepath, mode) as f:
while True:
try:
cb = self._progress_bar(f, remote_size)
self.ftp.retrbinary(
"RETR " + filepath, cb, rest=f.tell())
break
except KeyboardInterrupt:
f.flush()
self.close()
sys.exit(1)
except Exception:
f.flush()
self.reconnect()
self.cwd(remote_dir)
continue
absolute_path = get_abs_path(filepath)
local_filename, local_dirname = os.path.basename(
absolute_path
), os.path.dirname(absolute_path)
clear_console_line()
LOG.info(
"Saved {:s} to {:s}\n".format(local_filename, local_dirname)
)
def cwd(self, remote_pathname):
self.ftp.cwd(remote_pathname)
self.pwd = remote_pathname
def size(self, filepath):
size = 0
while True:
try:
size = self.ftp.size(filepath)
break
except ftplib.error_temp:
self.reconnect()
continue
return size
def exists(self, filepath):
while True:
try:
self.size(filepath)
break
except ftplib.error_perm as e:
if e.args[0].find("No such file or directory"):
return False
raise
return True
def connect(self, server):
self.ftp = ftplib.FTP(server)
self.ftp.login()
self.ftp.sendcmd("TYPE I")
def reconnect(self):
host = self.ftp.host
self.ftp.close()
self.connect(host)
self.ftp.cwd(self.pwd)
def host(self):
return self.ftp.host
def close(self):
self.ftp.quit()
class ProgressBar:
def __init__(self, stop, current=0, width=30):
self.stop = stop
self.width = width
self.current = current
self.bar = list("-" * self.width)
self.step = stop / self.width
self.last_index = self._calculate_index()
if self.current > 0:
self.progress()
def progress(self, amount=0, relative=False):
if relative:
self.current += amount
else:
self.current = amount
if self.current > self.stop:
self.current = self.stop
index = self._calculate_index()
for i in range(self.last_index, index):
if i == 0:
self.bar[i] = ">"
else:
self.bar[i - 1], self.bar[i] = "=", ">"
self.last_index = index
def get_bar(self):
percentage = int(self.current / self.stop * 100)
return "{:3d}% {:s}".format(percentage, "[" + "".join(self.bar) + "]")
def _calculate_index(self):
return math.floor(self.current / self.step)
def clear_console_line():
LOG.debug("\33[2K\r")
def count_lines(*filenames):
lines = 0
for fname in filenames:
with open(fname, "r") as f:
for line in f:
lines += 1
return lines
def dwk2():
estimate_capacity = find_kraken2_binary("estimate_capacity")
output = subprocess.check_output(
[estimate_capacity, "-h"], stderr=subprocess.STDOUT
)
for line in output.split(b"\n"):
if line.startswith(b"Usage:"):
return True if line.strip().endswith(b"") else False
return False
def get_binary_options(binary_pathname):
options = []
proc = subprocess.Popen(binary_pathname, stderr=subprocess.PIPE)
lines = proc.stderr.readlines()
for line in lines:
match = re.search(br"\s(-\w)\s", line)
if not match:
continue
options.append(match.group(1).decode())
return options
def construct_seed_template(args):
if int(args.minimizer_len / 4) < args.minimizer_spaces:
LOG.error(
"Number of minimizer spaces, {}, exceeds max for minimizer length, {}; max: {}\n".format(
args.minimizer_spaces,
args.minimizer_len,
int(args.minimizer_len / 4),
)
)
sys.exit(1)
return (
"1" * (args.minimizer_len - 2 * args.minimizer_spaces)
+ "01" * args.minimizer_spaces
)
def wrapper_args_to_binary_args(opts, argv, binary_args):
for k, v in vars(opts).items():
if k not in WRAPPER_ARGS_TO_BIN_ARGS:
continue
if WRAPPER_ARGS_TO_BIN_ARGS[k] not in binary_args:
continue
if v is False:
continue
if v is True:
argv.append(WRAPPER_ARGS_TO_BIN_ARGS[k])
else:
argv.extend([WRAPPER_ARGS_TO_BIN_ARGS[k], str(v)])
def find_kraken2_binary(name):
# search the OS PATH
if "PATH" in os.environ:
for dir in os.environ["PATH"].split(":"):
if os.path.exists(os.path.join(dir, name)):
return os.path.join(dir, name)
# search for binary in the same directory as wrapper
script_parent_directory = get_parent_directory(SCRIPT_PATHNAME)
if os.path.exists(os.path.join(script_parent_directory, name)):
return os.path.join(script_parent_directory, name)
# if called from within kraken2 project root, search the src dir
project_root = get_parent_directory(script_parent_directory)
if "src" in os.listdir(project_root) and name in os.listdir(
os.path.join(project_root, "src")
):
return os.path.join(project_root, os.path.join("src", name))
# not found in these likely places, exit
LOG.error("Unable to find {:s}, exiting\n".format(name))
sys.exit(1)
def get_parent_directory(pathname):
if len(pathname) == 0:
return None
pathname = os.path.abspath(pathname)
if len(pathname) > 1 and pathname[-1] == os.path.sep:
return os.path.dirname(pathname[:-1])
return os.path.dirname(pathname)
def find_database(database_name):
database_path = None
if database_name.find(os.path.sep) < 0:
if "KRAKEN2_DB_PATH" in os.environ:
for directory in os.environ["KRAKEN2_DB_PATH"].split(":"):
if os.path.exists(os.path.join(directory, database_name)):
database_path = os.path.join(directory, database_name)
break
else:
if database_name in os.listdir(os.getcwd()):
database_path = database_name
elif os.path.exists(database_name):
database_path = database_name
if database_path:
for db_file in ["taxo.k2d", "hash.k2d", "opts.k2d"]:
if not os.path.exists(os.path.join(database_path, db_file)):
return None
return database_path
def remove_files(*filepaths):
for fname in filepaths:
if not os.path.exists(fname):
continue
elif os.path.isdir(fname):
shutil.rmtree(fname)
else:
os.remove(fname)
def check_seqid(seqid):
taxid = None
match = re.match(r"(?:^|\|)kraken:taxid\|(\d+)", seqid)
if match:
taxid = match.group(1)
elif re.match(r"^\d+$", seqid):
taxid = seqid
if not taxid:
match = re.match(r"(?:^|\|)([A-Z]+_?[A-Z0-9]+)(?:\||\b|\.)", seqid)
if match:
taxid = match.group(1)
return taxid
def hash_file(filename, buf_size=8192):
md5 = hashlib.md5()
with open(filename, "rb") as in_file:
while True:
data = in_file.read(buf_size)
if not data:
break
md5.update(data)
return md5.hexdigest()
# This function is part of the Kraken 2 taxonomic sequence
# classification system.
#
# Reads multi-FASTA input and examines each sequence header. Headers are
# OK if a taxonomy ID is found (as either the entire sequence ID or as part
# of a "kraken:taxid" token), or if something looking like an accession
# number is found. Not "OK" headers will are fatal errors unless "lenient"
# is used.
#
# Each sequence header results in a line with three tab-separated values;
# the first indicating whether third column is the taxonomy ID ("TAXID") or
# an accession number ("ACCNUM") for the sequence ID listed in the second
# column.
#
def scan_fasta_file(in_file, out_file, lenient=False, hash=False):
for line in in_file:
if not line.startswith(">"):
continue
for match in re.finditer(r"(?:^>|\x01)(\S+)", line):
seqid = match.group(1)
taxid = check_seqid(seqid)
if not taxid:
if lenient:
continue
else:
sys.exit(1)
if re.match(r"^\d+$", taxid):
out_file.write("TAXID\t{:s}\t{:s}\n".format(seqid, taxid))
else:
out_file.write("ACCNUM\t{:s}\t{:s}\n".format(seqid, taxid))
# This function is part of the Kraken 2 taxonomic sequence
# classification system.
#
# Looks up accession numbers and reports associated taxonomy IDs
#
# `lookup_list_file` is 1 2-column TSV file w/ sequence IDs and
# accession numbers, and `accession_map_files` is a list of
# accession2taxid files from NCBI. Output is tab-delimited lines,
# with sequence IDs in first column and taxonomy IDs in second.
#
def lookup_accession_numbers(
lookup_list_filename, out_filename, *accession_map_files
):
target_lists = {}
with open(lookup_list_filename, "r") as f:
for line in f:
line = line.strip()
seqid, acc_num = line.split("\t")
if acc_num in target_lists:
target_lists[acc_num].append(seqid)
else:
target_lists[acc_num] = []
initial_target_count = len(target_lists)
with open(out_filename, "a") as out_file:
for filename in accession_map_files:
with open(filename, "r") as in_file:
in_file.readline() # discard header line
for line in in_file:
line = line.strip()
accession, with_version, taxid, gi = line.split("\t")
if accession in target_lists:
lst = target_lists[accession]
del target_lists[accession]
for seqid in lst:
out_file.write(seqid + "\t" + taxid + "\n")
if len(target_lists) == 0:
break
if len(target_lists) == 0:
break
if target_lists:
LOG.warning(
"{}/{} accession numbers remain unmapped, see unmapped.txt in DB directory\n".format(
len(target_lists),
initial_target_count),
)
with open("unmapped.txt", "w") as f:
for k in target_lists:
f.write(k + "\n")
def spawn_masking_subprocess(output_file, protein=False):
masking_binary = "segmasker" if protein else "k2mask"
if "MASKER" in os.environ:
masking_binary = os.environ["MASKER"]
masking_binary = find_kraken2_binary(masking_binary)
argv = masking_binary + " -outfmt fasta | sed -e '/^>/!s/[a-z]/x/g'"
if masking_binary.find("k2mask") >= 0:
# k2mask can run multithreaded
argv = masking_binary + " -outfmt fasta -threads 4 -r x"
p = subprocess.Popen(
argv, shell=True, stdin=subprocess.PIPE, stdout=output_file
)
def masker(input_file, final=False):
shutil.copyfileobj(input_file, p.stdin)
if final:
p.stdin.close()
p.wait()
return masker
# Mask low complexity sequences in the database
def mask_files(input_filenames, output_filename, protein=False):
with open(output_filename, "wb") as fout:
masker = spawn_masking_subprocess(fout, protein)
number_of_files = len(input_filenames)
for i, input_filename in enumerate(input_filenames):
library_name = os.path.basename(os.getcwd())
if library_name == "added":
LOG.info("Masking low-complexity regions of added library.")
else:
LOG.info(
"Masking low-complexity regions of downloaded library {:s}\n".format(
library_name
)
)
with open(input_filename, "rb") as fin:
masker(fin, i + 1 == number_of_files)
def add_to_library(args):
if not os.path.isdir(args.db):
LOG.error("Invalid database: {:s}\n".format(args.db))
sys.exit(1)
library_pathname = os.path.join(args.db, "library")
added_pathname = os.path.join(library_pathname, "added")
os.makedirs(added_pathname, exist_ok=True)
args.files = [os.path.abspath(f) for f in args.files]
os.chdir(added_pathname)
hashes = []
if os.path.exists("added.md5"):
with open("added.md5", "r") as in_file:
hashes = [line.split()[0] for line in in_file.readlines()]
for filename in args.files:
filehash = hash_file(filename)
if filehash in hashes:
LOG.info(filename + " already added to library.\n")
LOG.info("If not the case, remove the entry from added.md5\n")
return
destination = os.path.basename(filename)
ext = ".faa" if args.protein else "fna"
if not destination.endswith(ext):
base = destination.rsplit(".", 1)[0]
destination = base + "." + ext
prelim_map_filename = "prelim_map_" + filehash + ".txt"
with open(prelim_map_filename, mode="a") as out_file:
with open(filename, "r") as in_file:
scan_fasta_file(in_file, out_file, lenient=True)
shutil.copyfile(filename, destination)
if not args.no_masking:
mask_files(
[destination], destination + ".masked", protein=args.protein
)
shutil.move(destination + ".masked", destination)
with open("added.md5", "a") as out_file:
out_file.write(filehash + "\t" + destination + "\n")
LOG.info("Added " + filename + " to library " + args.db + "\n")
def make_manifest_from_assembly_summary(
assembly_summary_file, is_protein=False
):
suffix = "_protein.faa.gz" if is_protein else "_genomic.fna.gz"
manifest_to_taxid = {}
for line in assembly_summary_file:
if line.startswith("#"):
continue
fields = line.strip().split("\t")
taxid, asm_level, ftp_path = fields[5], fields[11], fields[19]
if not re.match("Complete Genome|Chromosome", asm_level):
continue
if ftp_path == "na":
continue
remote_path = ftp_path + "/" + os.path.basename(ftp_path) + suffix
url_components = urllib.parse.urlsplit(remote_path)
local_path = url_components.path.replace("/genomes/", "")
manifest_to_taxid[local_path] = taxid
with open("manifest.txt", "w") as f:
for k in manifest_to_taxid:
f.write(k + "\n")
return manifest_to_taxid
def assign_taxid_to_sequences(manifest_to_taxid, is_protein):
LOG.info("Assigning taxonomic IDs to sequences\n")
out_filename = "library.faa" if is_protein else "library.fna"
with open(out_filename, "wb") as f:
projects_added = 0
total_projects = len(manifest_to_taxid)
sequences_added = 0
ch_added = 0
ch = "aa" if is_protein else "bp"
max_out_chars = 0
for filepath in sorted(manifest_to_taxid):
taxid = manifest_to_taxid[filepath]
with gzip.open(filepath) as in_file:
while True:
line = in_file.readline()
if line == b"":
break
if line.startswith(b">"):
line = line.replace(
b">", str.encode(">kraken:taxid|" + taxid + "|"), 1
)
sequences_added += 1
else:
ch_added += len(line) - 1
f.write(line)
projects_added += 1
out_line = progress_line(
projects_added, total_projects, sequences_added, ch_added, ch
)
max_out_chars = max(len(out_line), max_out_chars)
space_line = " " * max_out_chars
LOG.debug("\r{:s}\r{:s}".format(space_line, out_line))
LOG.info("\nFinished assigning taxonomic IDs to sequences\n")
def progress_line(projects, total_projects, seqs, chars, ch):
line = "Processed "
if projects == total_projects:
line += str(projects)
else:
line += "{:d}/{:d}".format(projects, total_projects)
line += " project(s), {:d} sequence(s), ".format(seqs)
prefix = None
for p in ["k", "M", "G", "T", "P", "E"]:
if chars >= 1000:
prefix = p
chars /= 1000
else:
break
if prefix:
line += "{:.2f} {:s}{:s}".format(chars, prefix, ch)
else:
line += "{:.2f} {:s}".format(chars, ch)
return line
def decompress_files(compressed_filenames, out_filename=None, buf_size=8192):
if out_filename:
if os.path.exists(out_filename + ".tmp"):
os.remove(out_filename + ".tmp")
with open(out_filename + ".tmp", "ab") as out_file:
for filename in compressed_filenames:
with gzip.open(filename) as gz:
decompress_file(gz, out_file)
os.rename(out_filename + ".tmp", out_filename)
else:
for filename in compressed_filenames:
out_filename, ext = os.path.splitext(filename)
if os.path.exists(out_filename + ".tmp"):
os.remove(out_filename + ".tmp")
with gzip.open(filename) as gz:
with open(out_filename + ".tmp", "wb") as out:
decompress_file(gz, out, buf_size)
os.rename(out_filename + ".tmp", out_filename)
remove_files(*compressed_filenames)
def decompress_file(in_file, out_file, buf_size=8129):
LOG.info(
"Decompressing {:s}\n".format(os.path.join(os.getcwd(), in_file.name))
)
while True:
data = in_file.read(buf_size)
out_file.write(data)
if data == b"":
break
LOG.info(
"Finished decompressing {:s}\n".format(
os.path.join(os.getcwd(), in_file.name)
)
)
def download_log(filename):
pb = None
current_size = 0
def inner(block_number, read_size, total_size):
nonlocal pb, current_size
if not pb:
pb = ProgressBar(total_size)
current_size += read_size
pb.progress(current_size)
LOG.debug(
"{:s} {: >10s}\r".format(pb.get_bar(), format_bytes(current_size))
)
return inner
def download_file(url, local_name=None):
if not local_name:
local_name = urllib.parse.urlparse(url).path.split("/")[-1]
else:
os.makedirs(os.path.dirname(local_name), exist_ok=True)
LOG.info("Beginning download of {:s}\n".format(local_name))
urllib.request.urlretrieve(
url, local_name, reporthook=download_log(local_name)
)
clear_console_line()
LOG.info("Saved {:s} to {:s}\n".format(local_name, os.getcwd()))
def make_manifest_filter(file, regex):
def inner(listing):
nonlocal file, regex
path = listing.split()[-1]
if path.endswith(regex):
file.write(path + "\n")
return inner
def move(src, dst):
src = os.path.abspath(src)
dst = os.path.abspath(dst)
if os.path.isfile(src) and os.path.isdir(dst):
dst = os.path.join(dst, os.path.basename(src))
shutil.move(src, dst)
def get_manifest(server, remote_path, regex):
with open("manifest.txt", "w") as f:
ftp = ftplib.FTP(server)
ftp.login()
ftp.cwd(remote_path)
ftp.retrlines("LIST", callback=make_manifest_filter(f, regex))
ftp.close()
def download_files_from_manifest(
server,
remote_dir,
manifest_filename="manifest.txt",
decompress=False,
out_filename=None,
filepath_to_taxid_table=None,
):
with open(manifest_filename, "r") as f:
filepaths = []
ftp = FTP(server)
spinner = ["|", "/", "—", "\\"]
i = 0
for filepath in f:
LOG.info(
"Checking if manifest files exist on server {:s}\r".format(
spinner[i % 4]
)
)
i += 1
filepath = filepath.strip()
if not ftp.exists(urllib.parse.urljoin(remote_dir, filepath)):
if filepath_to_taxid_table:
del filepath_to_taxid_table[filepath]
LOG.warning(
"{:s} does not exist on server, skipping\n".format(
remote_dir + filepath
)
)
continue
filepaths.append(filepath)
ftp.download(remote_dir, filepaths)
ftp.close()
if decompress:
decompress_files(filepaths, out_filename)
def download_taxonomy(args):
taxonomy_path = os.path.join(args.db, "taxonomy")
os.makedirs(taxonomy_path, exist_ok=True)
os.chdir(taxonomy_path)
ftp = FTP(NCBI_SERVER)
if not args.skip_maps:
if not args.protein:
for subsection in ["gb", "wgs"]:
LOG.info(
"Downloading nucleotide {:s} accession to taxon map\n".format(
subsection
)
)
filename = "nucl_" + subsection + ".accession2taxid.gz"
ftp.download("/pub/taxonomy/accession2taxid/", filename)
else:
LOG.info("Downloading protein accession to taxon map\n")
ftp.download(
"/pub/taxonomy/accession2taxid", "prot.accession2taxid.gz"
)
LOG.info("Downloaded accession to taxon map(s)")
LOG.info("and taxonomy tree data\n")
ftp.download("/pub/taxonomy", "taxdump.tar.gz")
ftp.close()
LOG.info("Decompressing taxonomy data\n")
decompress_files(glob.glob("*accession2taxid.gz"))
LOG.info("Untarring taxonomy tree data\n")
with tarfile.open("taxdump.tar.gz", "r:gz") as tar:
tar.extractall()
remove_files(*glob.glob("*.gz"))
def download_genomic_library(args):
library_filename = "library.faa" if args.protein else "library.fna"
library_pathname = os.path.join(args.db, "library")
LOG.info("Adding {:s} to {:s}\n".format(args.library, args.db))
files_to_remove = None
if args.library in [
"archaea",
"bacteria",
"viral",
"fungi",
"plant",
"human",
"protozoa",
]:
library_pathname = os.path.join(library_pathname, args.library)
os.makedirs(library_pathname, exist_ok=True)
os.chdir(library_pathname)
try:
os.remove("assembly_summary.txt")
except FileNotFoundError:
pass
remote_dir_name = args.library
if args.library == "human":
remote_dir_name = "vertebrate_mammalian/Homo_sapiens"
try:
url = "ftp://{:s}/genomes/refseq/{:s}/assembly_summary.txt".format(
NCBI_SERVER, remote_dir_name
)
download_file(url)
except urllib.error.URLError:
LOG.error(
"Error downloading assembly summary file for {:s}, exiting\n".format(
args.library
)
)
sys.exit(1)
if args.library == "human":
with open("assembly_summary.txt", "r") as f1:
with open("grc.txt", "w") as f2:
for line in f1:
if line.find("Genome Reference Consortium"):
f2.write(line)
os.rename("grc.txt", "assembly_summary.txt")
with open("assembly_summary.txt", "r") as f:
filepath_to_taxid_table = make_manifest_from_assembly_summary(
f, args.protein
)
download_files_from_manifest(
NCBI_SERVER,
"/genomes/",
filepath_to_taxid_table=filepath_to_taxid_table,
)
assign_taxid_to_sequences(filepath_to_taxid_table, args.protein)
with open(library_filename, "r") as in_file:
with open("prelim_map.txt", "w") as out_file:
scan_fasta_file(in_file, out_file)
files_to_remove = ["all", "manifest.txt"]
elif args.library == "plasmid":
library_pathname = os.path.join(library_pathname, args.library)
os.makedirs(library_pathname, exist_ok=True)
os.chdir(library_pathname)
pat = ".faa.gz" if args.protein else ".fna.gz"
get_manifest(NCBI_SERVER, "genomes/refseq/plasmid/", pat)
download_files_from_manifest(
NCBI_SERVER,
"/genomes/refseq/plasmid/",
decompress=True,
out_filename=library_filename,
)
with open(library_filename, "r") as in_file:
with open("prelim_map.txt", "w") as out_file:
scan_fasta_file(in_file, out_file)
files_to_remove = glob.glob("plasmid.*")
files_to_remove.append("manifest.txt")
elif args.library in ["nr", "nt"]:
protein_lib = True if args.library == "nr" else False
if protein_lib and not args.protein:
LOG.error(
"{:s} is a protein database, and the Kraken2 database specified is nucleotide".format(
args.library
)
)
sys.exit(1)
library_pathname = os.path.join(library_pathname, args.library)
os.makedirs(library_pathname, exist_ok=True)
os.chdir(library_pathname)
ftp = FTP(NCBI_SERVER)
ftp.download("blast/db/FASTA/", args.library + ".gz")
with gzip.open(args.library + ".gz", mode="rt") as in_file:
with open("prelim_map.txt", "w") as out_file:
scan_fasta_file(in_file, out_file, lenient=True)
files_to_remove = [glob.glob("*.gz")]
elif args.library in ["UniVec", "UniVec_Core"]:
if args.protein:
LOG.error(
"{:s} is for nucleotide databases only\n".format(args.library)
)
sys.exit(1)
library_pathname = os.path.join(library_pathname, args.library)
os.makedirs(library_pathname, exist_ok=True)
os.chdir(library_pathname)
ftp = FTP(NCBI_SERVER)
ftp.download("pub/UniVec", args.library)
special_taxid = 28384
LOG.info(
"Adding taxonomy ID of {:d} to all sequences\n".format(
special_taxid
)
)
with open(args.library, "r") as in_file:
with open("library.fna", "w") as out_file:
for line in in_file:
if line.startswith(">"):
line = re.sub(
">",
">kraken:taxid|" + str(special_taxid) + "|",
line,
)
out_file.write(line)
with open("library.fna", "r") as in_file:
with open("prelim_map.txt", "w") as out_file:
scan_fasta_file(in_file, out_file)
files_to_remove = [args.library]
if not args.no_masking:
mask_files(
[library_filename], library_filename + ".masked", args.protein
)
shutil.move(library_filename + ".masked", library_filename)
LOG.info("Added {:s} to {:s}\n".format(args.library, args.db))
if files_to_remove:
clean_up(files_to_remove)
def get_abs_path(filename):
return os.path.abspath(filename)
def is_compressed(filename):
bzip_magic = b"\x42\x5A\x68"
gzip_magic = b"\x1F\x8B"
xz_magic = b"\xFD\x37\x7A\x58\x5A\x00"
nbytes = len(xz_magic)
with open(filename, "rb") as f:
data = f.read(nbytes)
if data.startswith((bzip_magic, gzip_magic, xz_magic)):
return True
return False
def get_reader(filename):
bzip_magic = b"\x42\x5A\x68"
gzip_magic = b"\x1F\x8B"
xz_magic = b"\xFD\x37\x7A\x58\x5A\x00"
nbytes = len(xz_magic)
with open(filename, "rb") as f:
data = f.read(nbytes)
if data.startswith(bzip_magic):
return bz2.open
elif data.startswith(gzip_magic):
return gzip.open
elif data.startswith(xz_magic):
return lzma.open
else:
return open
def read_from_files(filename1, filename2=None):
reader1 = get_reader(filename1)
reader2 = None
if filename2 is not None:
reader2 = get_reader(filename2)
if reader2 is None:
with reader1(filename1, "rb") as f:
for seq in f:
yield seq
else:
with reader1(filename1, "rb") as f1, reader2(filename2, "rb") as f2:
for seq1, seq2 in itertools.zip_longest(f1, f2):
if seq1 is None:
LOG.error(
"{} contains more sequences than {}".format(filename1, filename2)
)
sys.exit(1)
if seq2 is None:
LOG.error(
"{} contains more sequences than {}".format(filename2, filename1)
)
sys.exit(1)
yield (seq1, seq2)
def write_to_fifo(filenames, fifo1=None, fifo2=None):
if fifo2 is not None:
with open(fifo1, "wb") as file1, open(fifo2, "wb") as file2:
for fn1, fn2 in zip(filenames[0::2], filenames[1::2]):
for seq1, seq2 in read_from_files(fn1, fn2):
file1.write(seq1)
file2.write(seq2)
else:
with open(fifo1, "wb") as file1:
for fn in filenames:
for seq in read_from_files(fn):
file1.write(seq)
def build_kraken2_db(args):
if not os.path.isdir(get_abs_path(args.db)):
LOG.error('Cannot find Kraken 2 database: "{:s}\n'.format(args.db))
sys.exit(1)
os.chdir(args.db)
if not os.path.isdir("taxonomy"):
LOG.error("Cannot find taxonomy subdirectory in database\n")
sys.exit(1)
if not os.path.isdir("library"):
LOG.error("Cannot find library subdirectory in database\n")
sys.exit(1)
if os.path.isdir(os.path.join("library", "added")):
added_dirpath = os.path.join("library", "added")
prelim_map_filenames = glob.glob(
os.path.join(added_dirpath, "prelim_map_*.txt")
)
if prelim_map_filenames:
with open(
os.path.join(added_dirpath, "prelim_map.txt"), "w"
) as out_file:
for filename in prelim_map_filenames:
with open(filename, "r") as in_file:
shutil.copyfileobj(in_file, out_file)
if os.path.isfile("seqid2taxid.map"):
LOG.info("Sequence ID to taxonomy ID map already present, skipping\n")
else:
LOG.info("Creating sequence ID to taxonomy ID map\n")
with open(os.path.join("taxonomy", "prelim_map.txt"), "w") as out_file:
for dirpath, dirnames, filenames in os.walk("library"):
if "prelim_map.txt" in filenames:
for line in open(
os.path.join(
os.path.abspath(dirpath), "prelim_map.txt"
),
"r",
):
out_file.write(line)
if os.path.getsize(os.path.join("taxonomy", "prelim_map.txt")) == 0:
os.remove(os.path.join("taxonomy", "prelim_map.txt"))
LOG.error(
"No preliminary seqid/taxid mapping files found, aborting\n"
)
sys.exit(1)
with open(os.path.join("taxonomy", "prelim_map.txt"), "r") as in_file:
with open("seqid2taxid.map.tmp", "w") as seqid2taxid_file:
with open("accmap.tmp", "w") as accmap_file:
for line in in_file:
new_line = "\t".join(line.split("\t")[1:])
if line.startswith("TAXID"):
seqid2taxid_file.write(new_line)
elif line.startswith("ACCNUM"):
accmap_file.write(new_line)
if os.path.getsize("accmap.tmp") > 0:
accession2taxid_filenames = glob.glob("taxonomy/*.accession2taxid")
if accession2taxid_filenames:
lookup_accession_numbers(
"accmap.tmp",
"seqid2taxid.map.tmp",
*accession2taxid_filenames
)
else:
LOG.error(
"Accession to taxid map files are required to build this database.\n"
)
LOG.error(
"Run k2 download-taxonomy --db {:s} again".format(args.db)
)
sys.exit(1)
os.remove("accmap.tmp")
move("seqid2taxid.map.tmp", "seqid2taxid.map")
LOG.info("Created sequence ID to taxonomy ID map\n")
estimate_capacity_binary = find_kraken2_binary("estimate_capacity")
argv = [estimate_capacity_binary, "-S", construct_seed_template(args)]
if args.protein:
argv.append("-X")
wrapper_args_to_binary_args(
args, argv, get_binary_options(estimate_capacity_binary)
)
fasta_filenames = glob.glob(
os.path.join("library", os.path.join("**", "*.f[an]a")), recursive=True
)
if not dwk2():
argv.extend(fasta_filenames)
LOG.info("Running: " + " ".join(argv) + "\n")
proc = subprocess.Popen(
argv, stdin=subprocess.PIPE, stdout=subprocess.PIPE
)
if dwk2():
for filename in fasta_filenames:
with open(filename, "rb") as in_file:
while True:
data = in_file.read(8192)
if not data:
break
proc.stdin.write(data)
estimate = proc.communicate()[0].decode()
required_capacity = (int(estimate.strip()) + 8192) / args.load_factor
LOG.info(
"Estimated hash table requirement: {:s}\n".format(
format_bytes(required_capacity * 4)
)
)
if args.max_db_size:
if args.max_db_size < required_capacity * 4:
args.max_db_size = int(args.max_db_size / 4)
LOG.warning(
"Specifiying lower maximum hash table size of {:f}\n".format(
args.max_db_size
)
)
if os.path.isfile("hash.k2d"):
LOG.info("Hash table already present, skipping build\n")
else:
LOG.info("Starting database build\n")
build_db_bin = find_kraken2_binary("build_db")
argv = [
build_db_bin,
"-H",
"hash.k2d.tmp",
"-t",
"taxo.k2d.tmp",
"-o",
"opts.k2d.tmp",
"-n",
"taxonomy",
"-m",
"seqid2taxid.map",
"-c",
str(required_capacity),
"-S",
construct_seed_template(args),
]
if args.protein:
argv.append("-X")
wrapper_args_to_binary_args(
args, argv, get_binary_options(build_db_bin)
)
LOG.info("Running: " + " ".join(argv) + "\n")
proc = subprocess.Popen(
argv, stdin=subprocess.PIPE, stdout=subprocess.PIPE
)
for filename in fasta_filenames:
with open(filename, "rb") as in_file:
while True:
data = in_file.read(8192)
if not data:
break
proc.stdin.write(data)
proc.stdin.close()
proc.wait()
move("hash.k2d.tmp", "hash.k2d")
move("taxo.k2d.tmp", "taxo.k2d")
move("opts.k2d.tmp", "opts.k2d")
LOG.info("Finished building database\n")
# Parses RDP sequence data to create Kraken taxonomy
# and sequence ID -> taxonomy ID mapping
# Input (as <>): current_{Archaea,Bacteria}_unaligned.fa
def build_rdp_taxonomy(f):
seqid_map = {}
seen_it = {}
child_data = {"root;no rank": {}}
for line in f:
if not line.startswith(">"):
continue
line = line.strip()
seq_label, taxonomy_string = line.split("\t")
seqid = seq_label.split(" ")[0]
taxonomy_string = re.sub(
"^Lineage=Root;rootrank;", "root;no rank;", taxonomy_string
)
taxonomy_string = re.sub(";$", ";no rank", taxonomy_string)
seqid_map[seqid] = taxonomy_string
seen_it.setdefault(taxonomy_string, 0)
seen_it[taxonomy_string] += 1
if seen_it[taxonomy_string] > 1:
continue
while True:
match = re.search("(;[^;]+;[^;]+)$", taxonomy_string)
if match is None:
break
level = match.group(1)
taxonomy_string = re.sub(";[^;]+;[^;]+$", "", taxonomy_string)
key = taxonomy_string + level
child_data.setdefault(taxonomy_string, {})
seen_it.setdefault(taxonomy_string, 0)
child_data[taxonomy_string].setdefault(key, 0)
child_data[taxonomy_string][key] += 1
seen_it[taxonomy_string] += 1
if seen_it[taxonomy_string] > 1:
break
id_map = {}
next_node_id = 1
with open("names.dmp", "w") as names_file:
with open("nodes.dmp", "w") as nodes_file:
bfs_queue = [["root;no rank", 1]]
while len(bfs_queue) > 0:
node, parent_id = bfs_queue.pop()
match = re.search("([^;]+);([^;]+)$", node)
if match is None:
LOG.error(
'BFS processing encountered formatting eror, "{:s}"\n'.format(
node
)
)
sys.exit(1)
display_name, rank = match.group(1), match.group(2)
if rank == "domain":
rank = "superkingdom"
node_id, next_node_id = next_node_id, next_node_id + 1
id_map[node] = node_id
names_file.write(
"{:d}\t|\t{:s}\t|\t-\t|\tscientific name\t|\n".format(
node_id, display_name
)
)
nodes_file.write(
"{:d}\t|\t{:d}\t|\t{:s}\t|\t-\t|\n".format(
node_id, parent_id, rank
)
)
children = (
sorted([key for key in child_data[node]])
if node in child_data
else []
)
for node in children:
bfs_queue.insert(0, [node, node_id])
with open("seqid2taxid.map", "w") as f:
for seqid in sorted([key for key in seqid_map]):
taxid = id_map[seqid_map[seqid]]
f.write("{:s}\t{:d}\n".format(seqid, taxid))
# Build the standard Kraken database
def build_standard_database(args):
download_taxonomy(args)
for library in [
"archaea",
"bacteria",
"viral",
"plasmid",
"human",
"UniVec_Core",
]:
if library == "UniVec_Core" and args.protein:
continue
args.library = library
download_genomic_library(args)
build_kraken2_db(args)
# Parses Silva taxonomy file to create Kraken taxonomy
# Input (as <>): tax_slv_ssu_nr_119.txt
def build_silva_taxonomy(in_file):
id_map = {"root": 1}
with open("names.dmp", "w") as names_file:
with open("nodes.dmp", "w") as nodes_file:
names_file.write("1\t|\troot\t|\t-\t|\tscientific name\t|\n")
nodes_file.write("1\t|\t1\t|\tno rank\t|\t-\t|\n")
for line in in_file:
line = line.strip()
taxonomy_string, node_id, rank = line.split("\t")[:3]
id_map[taxonomy_string] = node_id
match = re.search("^(.+;|)([^;]+);$", taxonomy_string)
if match:
parent_name = match.group(1)
display_name = match.group(2)
if parent_name == "":
parent_name = "root"
parent_id = id_map[parent_name] or None
if not parent_id:
LOG.error('orphan error: "{:s}"\n'.format(line))
sys.exit(1)
if rank == "domain":
rank = "superkingdom"
names_file.write(
"{:s}\t|\t{:s}\t|\t-\t|\tscientific name\t|\n".format(
node_id, display_name
)
)
nodes_file.write(
"{:s}\t|\t{:s}\t|\t{:s}\t|\t-\t|\n".format(
node_id, str(parent_id), rank
)
)
else:
LOG.error('strange input: "{:s}"\n'.format(line))
sys.exit(1)
# Build a 16S database from Silva data
def build_16S_silva(args):
os.makedirs(args.db, exist_ok=True)
os.chdir(args.db)
for directory in ["data", "taxonomy", "library"]:
os.makedirs(directory, exist_ok=True)
os.chdir("data")
remote_directory = "/release_138_1/Exports"
fasta_filename = "SILVA_138.1_SSURef_NR99_tax_silva.fasta.gz"
taxonomy_prefix = "tax_slv_ssu_138.1"
ftp = FTP(SILVA_SERVER)
ftp.download(remote_directory, fasta_filename)
ftp.download(
remote_directory + "/taxonomy", taxonomy_prefix + ".acc_taxid.gz"
)
decompress_files([taxonomy_prefix + ".acc_taxid.gz"])
ftp.download(remote_directory + "/taxonomy", taxonomy_prefix + ".txt.gz")
with gzip.open(taxonomy_prefix + ".txt.gz", "rt") as f:
build_silva_taxonomy(f)
os.chdir(os.path.pardir)
move(os.path.join("data", "names.dmp"), "taxonomy")
move(os.path.join("data", "nodes.dmp"), "taxonomy")
move(
os.path.join("data", taxonomy_prefix + ".acc_taxid"), "seqid2taxid.map"
)
with gzip.open(os.path.join("data", fasta_filename), "rt") as in_file:
with open(os.path.join("library", "silva.fna"), "w") as out_file:
for line in in_file:
if not line.startswith(">"):
line = line.replace("U", "T")
out_file.write(line)
if not args.no_masking:
filename = os.path.join("library", "silva.fna")
mask_files([filename], filename + ".masked")
shutil.move(filename + ".masked", filename)
os.chdir(os.path.pardir)
build_kraken2_db(args)
# Parses Greengenes taxonomy file to create Kraken taxonomy
# and sequence ID -> taxonomy ID mapping
# Input (as <>): gg_13_5_taxonomy.txt
def build_gg_taxonomy(in_file):
rank_codes = {
"k": "superkingdom",
"p": "phylum",
"c": "class",
"o": "order",
"f": "family",
"g": "genus",
"s": "species",
}
seqid_map = {}
seen_it = {}
child_data = {"root": {}}
for line in in_file:
line = line.strip()
seqid, taxonomy_string = line.split("\t")
taxonomy_string = re.sub("(; [a-z]__)+$", "", taxonomy_string)
seqid_map[seqid] = taxonomy_string
seen_it.setdefault(taxonomy_string, 0)
seen_it[taxonomy_string] += 1
if seen_it[taxonomy_string] > 1:
continue
while True:
match = re.search("(; [a-z]__[^;]+$)", taxonomy_string)
if not match:
break
level = match.group(1)
taxonomy_string = re.sub("(; [a-z]__[^;]+$)", "", taxonomy_string)
child_data.setdefault(taxonomy_string, {})
key = taxonomy_string + level
seen_it.setdefault(taxonomy_string, 0)
child_data[taxonomy_string].setdefault(key, 0)
child_data[taxonomy_string][key] += 1
seen_it[taxonomy_string] += 1
if seen_it[taxonomy_string] > 1:
break
if seen_it[taxonomy_string] == 1:
child_data["root"].setdefault(taxonomy_string, 0)
child_data["root"][taxonomy_string] += 1
id_map = {}
next_node_id = 1
with open("names.dmp", "w") as names_file:
with open("nodes.dmp", "w") as nodes_file:
bfs_queue = [["root", 1]]
while len(bfs_queue) > 0:
node, parent_id = bfs_queue.pop()
display_name = node
rank = None
match = re.search("g__([^;]+); s__([^;]+)$", node)
if match:
genus, species = match.group(1), match.group(2)
rank = "species"
if re.search(" endosymbiont ", species):
display_name = species
else:
display_name = genus + " " + species
else:
match = re.search("([a-z])__([^;]+)$", node)
if match:
rank = rank_codes[match.group(1)]
display_name = match.group(2)
rank = rank or "no rank"
node_id, next_node_id = next_node_id, next_node_id + 1
id_map[node] = node_id
names_file.write(
"{:d}\t|\t{:s}\t|\t-\t|\tscientific name\t|\n".format(
node_id, display_name
)
)
nodes_file.write(
"{:d}\t|\t{:d}\t|\t{:s}\t|\t-\t|\n".format(
node_id, parent_id, rank
)
)
children = (
sorted([key for key in child_data[node]])
if node in child_data
else []
)
for node in children:
bfs_queue.insert(0, [node, node_id])
with open("seqid2taxid.map", "w") as f:
for seqid in sorted([key for key in seqid_map], key=int):
taxid = id_map[seqid_map[seqid]]
f.write("{:s}\t{:d}\n".format(seqid, taxid))
# Build a 16S database from Greengenes data
def build_16S_gg(args):
os.makedirs(args.db, exist_ok=True)
gg_version = "gg_13_5"
remote_directory = "/greengenes_release/" + gg_version
os.chdir(args.db)
for directory in ["data", "taxonomy", "library"]:
os.makedirs(directory, exist_ok=True)
os.chdir("data")
ftp = FTP(GREENGENES_SERVER)
ftp.download(remote_directory, gg_version + ".fasta.gz")
decompress_files([gg_version + ".fasta.gz"])
ftp.download(remote_directory, gg_version + "_taxonomy.txt.gz")
decompress_files([gg_version + "_taxonomy.txt.gz"])
with open(gg_version + "_taxonomy.txt", "r") as f:
build_gg_taxonomy(f)
os.chdir(os.path.abspath(os.path.pardir))
move(os.path.join("data", "names.dmp"), "taxonomy")
move(os.path.join("data", "nodes.dmp"), "taxonomy")
move(os.path.join("data", "seqid2taxid.map"), os.getcwd())
move(
os.path.join("data", gg_version + ".fasta"),
os.path.join("library", "gg.fna"),
)
if not args.no_masking:
filename = os.path.join("library", "gg.fna")
mask_files([filename], filename + ".masked")
shutil.move(filename + ".masked", filename)
os.chdir(os.path.abspath(os.path.pardir))
build_kraken2_db(args)
# Build a 16S data from RDP data
def build_16S_rdp(args):
os.makedirs(args.db, exist_ok=True)
os.chdir(args.db)
for directory in ["data", "taxonomy", "library"]:
os.makedirs(directory, exist_ok=True)
os.chdir("data")
download_file(
"http://rdp.cme.msu.edu/download/current_Bacteria_unaligned.fa.gz"
)
download_file(
"http://rdp.cme.msu.edu/download/current_Archaea_unaligned.fa.gz"
)
decompress_files(glob.glob("*gz"))
for filename in glob.glob("current_*_unaligned.fa"):
with open(filename, "r") as f:
build_rdp_taxonomy(f)
os.chdir(os.pardir)
move(os.path.join("data", "names.dmp"), "taxonomy")
move(os.path.join("data", "nodes.dmp"), "taxonomy")
move(os.path.join("data", "seqid2taxid.map"), os.getcwd())
for filename in glob.glob(os.path.join("data", "*.fa")):
new_filename = os.path.basename(re.sub(r"\.fa$", ".fna", filename))
shutil.move(filename, os.path.join("library", new_filename))
if not args.no_masking:
new_filename = os.path.join("library", new_filename)
mask_files([new_filename], new_filename + ".masked")
shutil.move(new_filename + ".masked", new_filename)
build_kraken2_db(args)
# Reads multi-FASTA input and examines each sequence header. In quiet
# mode headers are OK if a taxonomy ID is found (as either the entire
# sequence ID or as part of a "kraken:taxid" token), or if something
# looking like a GI or accession number is found. In normal mode, the
# taxonomy ID will be looked up (if not explicitly specified in the
# sequence ID) and reported if it can be found. Output is
# tab-delimited lines, with sequence IDs in first column and taxonomy
# IDs in second.
# Sequence IDs with a kraken:taxid token will use that to assign taxonomy
# ID, e.g.:
# >gi|32499|ref|NC_021949.2|kraken:taxid|562|
#
# Sequence IDs that are completely numeric are assumed to be the taxonomy
# ID for that sequence.
#
# Otherwise, an accession number is searched for; if not found, a GI
# number is searched for. Failure to find any of the above is a fatal error.
# Without `quiet`, a comma-separated file list specified by -A (for both accession
# numbers and GI numbers) is examined; failure to find a
# taxonomy ID that maps to a provided accession/GI number is non-fatal and
# will emit a warning.
#
# With -q, does not print any output, and will die w/ nonzero exit instead
# of warning when unable to find a taxid, accession #, or GI #.
#
def make_seqid_to_taxid_map(
in_file, quiet, accession_map_filenames=False, library_map_filename=None
):
target_lists = {}
for line in in_file:
match = re.match(r">(\S+)", line)
if match is None:
continue
seqid = match.group(1)
output = None
regexes = [
r"(?:^|\|)kraken:taxid\|(\d+)",
r"^\d+$",
r"(?:^|\|)([A-Z]+_?[A-Z0-9]+)(?:\||\b|\.)",
r"(?:^|\|)gi\|(\d+)",
]
match = None
index = None
for i, regex in enumerate(regexes):
match = re.match(regex, seqid)
if match:
index = i
break
if index == 0:
output = seqid + "\t" + match.group(1) + "\n"
elif index == 1:
output = seqid + "\t" + seqid + "\n"
elif index in [2, 3]:
if not quiet:
capture = match.group(1)
target_lists.setdefault(capture, [])
target_lists[capture].insert(0, seqid)
else:
LOG.error(
"Unable to determine taxonomy ID for sequence {:s}\n".format(
seqid
)
)
sys.exit(1)
if output and not quiet:
print(output)
if quiet:
if len(target_lists) == 0:
LOG.error("External map required\n")
sys.exit(0)
if len(target_lists) == 0:
sys.exit(0)
if not accession_map_filenames and library_map_filename is None:
LOG.error(
"Found sequence ID without explicit taxonomy ID, but no map used\n"
)
sys.exit(1)
# Remove targets where we've already handled the mapping
if library_map_filename:
with open(library_map_filename, "r") as f:
for line in f:
line = line.strip()
seqid, taxid = line.split("\t")
if seqid in target_lists:
print("{:s}\t{:s}\n".format(seqid, taxid))
del target_lists[seqid]
if len(target_lists) == 0:
sys.exit(0)
for filename in accession_map_filenames:
with open(filename, "r") as f:
f.readline()
for line in f:
line = line.strip()
accession, with_version, taxid, gi = line.split("\t")
if accession in target_lists:
target_list = target_lists[accession]
del target_lists[accession]
for seqid in target_list:
print("{:s}\t{:s}".format(seqid, taxid))
if gi != "na" and gi in target_lists:
target_list = target_lists[gi]
del target_lists[gi]
for seqid in target_list:
print("{:s}\t{:s}\n".format(seqid, taxid))
def classify(args):
classify_bin = find_kraken2_binary("classify")
database_path = find_database(args.db)
if database_path is None:
LOG.error("{:s} is not a valid database... exiting".format(args.db))
sys.exit(1)
if "paired" in args and len(args.filenames) % 2 != 0:
LOG.error("--paired requires an even number of file names")
sys.exit(1)
if args.confidence < 0 or args.confidence > 1:
LOG.error(
"--confidence, {:f}, must be between 0 and 1 inclusive".format(
args.confidence
)
)
sys.exit(1)
argv = [
classify_bin,
"-H",
os.path.join(database_path, "hash.k2d"),
"-t",
os.path.join(database_path, "taxo.k2d"),
"-o",
os.path.join(database_path, "opts.k2d"),
]
wrapper_args_to_binary_args(args, argv, get_binary_options(classify_bin))
if any([is_compressed(filename) for filename in args.filenames]):
with tempfile.TemporaryDirectory() as temp_dir_name:
fifo1_pathname = os.path.join(temp_dir_name, "fifo1")
fifo2_pathname = None
try:
os.mkfifo(fifo1_pathname, 0o600)
except OSError:
LOG.error(
"Unable to create FIFO for processing compressed files"
)
sys.exit(1)
if "-P" in argv:
fifo2_pathname = os.path.join(temp_dir_name, "fifo2")
try:
os.mkfifo(fifo2_pathname, 0o600)
except OSError:
LOG.error(
"Unable to create FIFO for processing compressed files"
)
sys.exit(1)
argv.extend([fifo1_pathname, fifo2_pathname])
else:
argv.append(fifo1_pathname)
thread = threading.Thread(target=subprocess.call, args=(argv,))
thread.start()
write_to_fifo(args.filenames, fifo1_pathname, fifo2_pathname)
thread.join()
else:
argv.extend(args.filenames)
sys.exit(subprocess.call(argv))
def inspect_db(args):
database_pathname = find_database(args.db)
if not database_pathname:
LOG.error("{:s} database does not exist\n")
sys.exit(1)
for database_file in ["taxo.k2d", "hash.k2d", "opts.k2d"]:
if not os.path.isfile(os.path.join(database_pathname, database_file)):
LOG.error("{:s} does not exist\n".format(database_file))
dump_table_bin = find_kraken2_binary("dump_table")
argv = [
dump_table_bin,
"-H",
os.path.join(database_pathname, "hash.k2d"),
"-t",
os.path.join(database_pathname, "taxo.k2d"),
"-o",
os.path.join(database_pathname, "opts.k2d"),
]
wrapper_args_to_binary_args(args, argv, get_binary_options(dump_table_bin))
sys.exit(subprocess.call(argv))
def format_bytes(size):
current_suffix = "B"
for suffix in ["kB", "MB", "GB", "TB", "PB", "EB"]:
if size >= 1024:
current_suffix = suffix
size /= 1024
else:
break
return "{:.2f}{:s}".format(size, current_suffix)
def clean_up(filenames):
LOG.info("Removing extra files\n")
disk_usage_before = shutil.disk_usage(os.getcwd()).free
remove_files(*filenames)
disk_usage_after = shutil.disk_usage(os.getcwd()).free
freed = disk_usage_after - disk_usage_before
LOG.info("Cleaned up {:s} of space\n".format(format_bytes(freed)))
def clean_db(args):
os.chdir(args.db)
clean_up(["data", "library", "taxonomy", "seqid2taxid.map"])
def make_build_parser(subparsers):
parser = subparsers.add_parser(
"build",
help="Create DB from library\
(requires taxonomy downloaded and at least one file\
in library)",
)
parser.add_argument(
"--db",
type=str,
metavar="PATHNAME",
required=True,
help="Name of Kraken 2 database"
)
group = parser.add_argument_group("special")
mutex_group = group.add_mutually_exclusive_group()
mutex_group.add_argument(
"--standard", action="store_true", help="Make standard database"
)
mutex_group.add_argument(
"--special",
type=str,
choices=["greengenes", "rdp", "silva"],
help="Build special database",
)
group.add_argument(
"--no-masking",
action="store_true",
help="Avoid masking low-complexity sequences prior to\
building; masking requires dustmasker or segmasker to be\
installed",
)
parser.add_argument(
"--kmer-len",
type=int,
metavar="INT",
help="K-mer length in bp/aa"
)
parser.add_argument(
"--minimizer-len",
type=int,
metavar="INT",
help="Minimizer length in bp/aa"
)
parser.add_argument(
"--minimizer-spaces",
type=int,
metavar="INT",
help="Number of characters in minimizer that are\
ignored in comparisons",
)
parser.add_argument(
"--threads",
type=int,
metavar="INT",
default=os.environ.get("KRAKEN2_NUM_THREADS") or 1,
help="Number of threads",
)
parser.add_argument(
"--load-factor",
type=float,
metavar="FLOAT (0,1]",
default=0.7,
help="Proportion of the hash table to be populated",
)
parser.add_argument(
"--fast-build",
action="store_true",
help="Do not require database to be deterministically\
built when using multiple threads. This is faster, but\
does introduce variability in minimizer/LCA pairs.",
)
parser.add_argument(
"--max-db-size",
type=int,
metavar="INT",
help="Maximum number of bytes for Kraken 2 hash table;\
if the estimator determines more would normally be\
needed, the reference library will be downsampled to fit",
)
parser.add_argument(
"--skip-maps",
action="store_true",
help="Avoids downloading accession number to taxid maps",
)
parser.add_argument(
"--protein",
action="store_true",
help="Build a protein database for translated search",
)
parser.add_argument(
"--block-size",
type=int,
metavar="INT",
default=16384,
help="Read block size"
)
parser.add_argument(
"--sub-block-size",
type=int,
metavar="INT",
default=0,
help="Read subblock size"
)
parser.add_argument(
"--minimum-bits-for-taxid",
type=int,
metavar="INT",
default=0,
help="Bit storage requested for taxid",
)
parser.add_argument(
"--log",
type=str,
metavar="FILENAME",
default=None,
help="Specify a log file (default: stderr)",
)
def make_download_taxonomy_parser(subparsers):
parser = subparsers.add_parser(
"download-taxonomy", help="Download NCBI taxonomic information"
)
parser.add_argument(
"--db",
type=str,
metavar="PATHNAME",
required=True,
help="Name of Kraken 2 database"
)
parser.add_argument(
"--protein",
action="store_true",
help="Files being added are for a protein database",
)
parser.add_argument(
"--skip-maps",
action="store_true",
help="Avoids downloading accession number to taxid maps",
)
parser.add_argument(
"--log",
type=str,
metavar="FILENAME",
default=None,
help="Specify a log filename (default: stderr)",
)
def make_download_library_parser(subparsers):
parser = subparsers.add_parser(
"download-library", help="Download and build a special database"
)
parser.add_argument(
"--db",
type=str,
metavar="PATHNAME",
required=True,
help="Name of Kraken 2 database"
)
parser.add_argument(
"--library",
type=str,
required=True,
choices=[
"archaea",
"bacteria",
"plasmid",
"viral",
"human",
"fungi",
"plant",
"protozoa",
"nr",
"nt",
"UniVec",
"UniVec_Core",
],
help="Name of library to download",
)
parser.add_argument(
"--protein",
action="store_true",
help="Files being added are for a protein database",
)
parser.add_argument(
"--no-masking",
action="store_true",
help="Avoid masking low-complexity sequences prior to\
building; masking requires dustmasker or segmasker to be\
installed",
)
parser.add_argument(
"--log",
type=str,
metavar="FILENAME",
default=None,
help="Specify a log filename (default stderr)",
)
def make_add_to_library_parser(subparsers):
parser = subparsers.add_parser(
"add-to-library", help="Add file(s) to library"
)
parser.add_argument(
"--db",
type=str,
metavar="PATHNAME",
required=True,
help="Name of Kraken 2 database"
)
parser.add_argument(
"--file",
"--files",
type=str,
nargs="+",
dest="files",
help="Pathname of file(s) to be added to library",
)
parser.add_argument(
"--protein",
action="store_true",
help="Files being added are for a protein database",
)
parser.add_argument(
"--no-masking",
action="store_true",
help="Avoid asking low-complexity sequences prior to\
building; masking requires dustmasker or segmasker to be\
installed",
)
parser.add_argument(
"--log",
type=str,
metavar="FILENAME",
default=None,
help="Specify a log filename (default: stderr)",
)
def make_classify_parser(subparsers):
parser = subparsers.add_parser(
"classify", help="Classify a set of sequences"
)
parser.add_argument(
"--db",
type=str,
metavar="PATHNAME",
required=True,
help="Name of Kraken2 DB"
)
parser.add_argument(
"--threads",
type=int,
metavar="INT",
default=os.environ.get("KRAKEN2_NUM_THREADS") or 1,
help="Number of threads",
)
parser.add_argument(
"--quick",
action="store_true",
default=argparse.SUPPRESS,
help="Quick operation (use first hit or hits)",
)
parser.add_argument(
"--unclassified-out",
type=str,
default=argparse.SUPPRESS,
metavar="FILENAME",
help="Print unclassified sequences to filename",
)
parser.add_argument(
"--classified-out",
type=str,
metavar="FILENAME",
default=argparse.SUPPRESS,
help="Print classified sequences to filename",
)
parser.add_argument(
"--output",
type=str,
metavar="FILENAME",
default=argparse.SUPPRESS,
help='Print output to file (default: stdout) "-" will \
suppress normal output',
)
parser.add_argument(
"--confidence",
type=float,
default=0.0,
help="confidence score threshold (default: 0.0); must be in [0,1]",
)
parser.add_argument(
"--mininum-base-quality",
type=int,
metavar="INT",
default=0,
help="Mininum base quality used in classification",
)
parser.add_argument(
"--report",
type=str,
default=argparse.SUPPRESS,
help="Print a report with aggregate counts/clade to file",
)
parser.add_argument(
"--use-mpa-style",
action="store_true",
default=argparse.SUPPRESS,
help="With --report, format report output like Kraken 1's\
kraken-mpa-report",
)
parser.add_argument(
"--report-zero-counts",
action="store_true",
default=argparse.SUPPRESS,
help="With --report, report counts for ALL taxa, even if\
counts are zero",
)
parser.add_argument(
"--report-minimizer-data",
action="store_true",
default=argparse.SUPPRESS,
help="With --report, report minimizer and distinct minimizer\
count information in addition to normal Kraken report",
)
parser.add_argument(
"--memory-mapping",
action="store_true",
default=argparse.SUPPRESS,
help="Avoids loading database into RAM",
)
paired_group = parser.add_mutually_exclusive_group()
paired_group.add_argument(
"--paired",
action="store_true",
default=argparse.SUPPRESS,
help="The filenames provided have paired-end reads",
)
paired_group.add_argument(
"--interleaved",
action="store_true",
default=argparse.SUPPRESS,
help="The filenames provided have paired-end reads",
)
parser.add_argument(
"--use-names",
action="store_true",
default=argparse.SUPPRESS,
help="Print scientific names instead of just taxids",
)
parser.add_argument(
"--mininum-hit-groups",
type=int,
metavar="INT",
default=2,
help="Minimum number of hit groups (overlapping k-mers\
sharing the same minimizer) needed to make a call",
)
parser.add_argument(
"--log",
type=str,
metavar="FILENAME",
default=None,
help="Specify a log filename (default: stderr)",
)
parser.add_argument(
"filenames",
nargs="+",
type=str,
help="Filenames to be classified, supports bz2, gzip, and xz"
)
def make_inspect_parser(subparsers):
parser = subparsers.add_parser("inspect", help="Inspect Kraken 2 database")
parser.add_argument(
"--db",
type=str,
metavar="PATHNAME",
required=True,
help="Name of Kraken2 DB"
)
parser.add_argument(
"--threads",
type=int,
default=os.environ.get("KRAKEN2_NUM_THREADS") or 1,
help="Number of threads",
)
parser.add_argument(
"--skip-counts",
action="store_true",
help="Only print database summary statistics",
)
parser.add_argument(
"--use-mpa-style",
action="store_true",
help="Format output like Kraken 1's kraken-mpa-report",
)
parser.add_argument(
"--report-zero-counts",
action="store_true",
help="Report counts for ALL taxa, even if counts are zero",
)
parser.add_argument(
"--log",
type=str,
metavar="FILENAME",
default=None,
help="Specify a log filename (default: stderr)",
)
def make_clean_parser(subparsers):
parser = subparsers.add_parser(
"clean", help="Remove unneeded files from database"
)
parser.add_argument(
"--db",
type=str,
metavar="PATHNAME",
required=True,
help="Name of Kraken2 DB"
)
parser.add_argument(
"--log",
type=str,
metavar="FILENAME",
default=None,
help="Specify a log filename (default: stderr)",
)
class HelpAction(argparse._HelpAction):
def __call__(self, parser, namespace, values, option_string=None):
parser.print_help()
subparsers = parser._actions[1].choices
for action, arg_parser in subparsers.items():
sys.stderr.write("\n\n" + action + "\n" + "-" * len(action) + "\n")
arg_parser.print_help()
sys.exit(0)
def make_cmdline_parser():
parser = argparse.ArgumentParser("kraken2", add_help=False)
parser.add_argument("-h", "--help", action=HelpAction)
subparsers = parser.add_subparsers()
make_add_to_library_parser(subparsers)
make_download_library_parser(subparsers)
make_download_taxonomy_parser(subparsers)
make_build_parser(subparsers)
make_classify_parser(subparsers)
make_inspect_parser(subparsers)
make_clean_parser(subparsers)
return parser
def setup_logger(filename=None):
logging.StreamHandler.terminator = ""
logger = logging.getLogger("kraken2")
if filename:
logger.setLevel(logging.INFO)
handler = logging.FileHandler(filename)
formatter = logging.Formatter("%(asctime)s: %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
else:
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler()
formatter = logging.Formatter("%(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
return logger
def k2_main():
global SCRIPT_PATHNAME
global LOG
SCRIPT_PATHNAME = os.path.realpath(inspect.getsourcefile(k2_main))
parser = make_cmdline_parser()
if len(sys.argv) == 1:
parser.print_help()
sys.exit(1)
args = parser.parse_args(sys.argv[1:])
LOG = setup_logger(args.log)
task = sys.argv[1]
if task not in ["classify", "inspect"]:
args.db = os.path.abspath(args.db)
if task == "download-taxonomy":
download_taxonomy(args)
elif task == "classify":
classify(args)
elif task == "download-library":
download_genomic_library(args)
elif task == "add-to-library":
add_to_library(args)
elif task == "inspect":
inspect_db(args)
elif task == "clean":
clean_db(args)
elif task == "build":
# Protein defaults
default_aa_minimizer_length = 12
default_aa_kmer_length = 15
default_aa_minimizer_spaces = 0
# Nucleotide defaults
default_nt_minimizer_length = 31
default_nt_kmer_length = 35
default_nt_minimizer_spaces = 7
if args.sub_block_size == 0:
args.sub_block_size = math.ceil(args.block_size / args.threads)
if not args.kmer_len:
args.kmer_len = (
default_aa_kmer_length
if args.protein
else default_nt_kmer_length
)
if not args.minimizer_len:
args.minimizer_len = (
default_aa_minimizer_length
if args.protein
else default_nt_minimizer_length
)
if not args.minimizer_spaces:
args.minimizer_spaces = (
default_aa_minimizer_spaces
if args.protein
else default_nt_minimizer_spaces
)
if args.minimizer_len > args.kmer_len:
LOG.error(
"Minimizer length ({}) must not be greater than kmer length {}\n".format(
args.minimizer_len, args.kmer_len
)
)
sys.exit(1)
if args.load_factor <= 0 or args.load_factor > 1:
LOG.error(
"Load factor must be greater than 0 but no more than 1\n"
)
sys.exit(1)
if args.minimizer_len <= 0 or args.minimizer_len > 31:
LOG.error(
"Minimizer length must be a positive integer and cannot exceed 31\n"
)
sys.exit(1)
if args.standard:
build_standard_database(args)
elif args.special:
if args.special == "greengenes":
build_16S_gg(args)
elif args.special == "silva":
build_16S_silva(args)
else:
build_16S_rdp(args)
else:
if args.no_masking:
LOG.warning(
"--no-masking only affects the `--standard` and"
"`--special` flags. Its effect will be ignored.\n"
)
build_kraken2_db(args)
if __name__ == "__main__":
k2_main()
kraken2-2.1.3/scripts/kraken2 0000775 0000000 0000000 00000017741 14437747201 0016047 0 ustar 00root root 0000000 0000000 #!/usr/bin/env perl
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Wrapper for Kraken's classifier
use strict;
use warnings;
use Fcntl;
use File::Basename;
use Getopt::Long;
my $PROG = basename $0;
my $KRAKEN2_DIR = '#####=KRAKEN2_DIR=#####';
# Test to see if the executables got moved, try to recover if we can
if (! -e "$KRAKEN2_DIR/classify") {
use Cwd 'abs_path';
$KRAKEN2_DIR = dirname abs_path($0);
}
require "$KRAKEN2_DIR/kraken2lib.pm";
$ENV{"KRAKEN2_DIR"} = $KRAKEN2_DIR;
$ENV{"PATH"} = "$KRAKEN2_DIR:$ENV{PATH}";
my $CLASSIFY = "$KRAKEN2_DIR/classify";
my $GZIP_MAGIC = chr(hex "1f") . chr(hex "8b");
my $BZIP2_MAGIC = "BZ";
my $quick = 0;
my $min_hits = 1;
my $db_prefix;
my $threads;
my $memory_mapping = 0;
my $gunzip = 0;
my $bunzip2 = 0;
my $paired = 0;
my $names_in_output = 0;
my $only_classified_output = 0;
my $unclassified_out;
my $classified_out;
my $outfile;
my $confidence_threshold = 0.0;
my $minimum_base_quality = 0;
my $report_filename;
my $use_mpa_style = 0;
my $report_zero_counts = 0;
my $minimum_hit_groups = 2;
my $report_minimizer_data = 0;
GetOptions(
"help" => \&display_help,
"version" => \&display_version,
"db=s" => \$db_prefix,
"threads=i" => \$threads,
"quick" => \$quick,
"unclassified-out=s" => \$unclassified_out,
"classified-out=s" => \$classified_out,
"output=s" => \$outfile,
"confidence=f" => \$confidence_threshold,
"memory-mapping" => \$memory_mapping,
"paired" => \$paired,
"use-names" => \$names_in_output,
"gzip-compressed" => \$gunzip,
"bzip2-compressed" => \$bunzip2,
"only-classified-output" => \$only_classified_output,
"minimum-base-quality=i" => \$minimum_base_quality,
"report=s" => \$report_filename,
"use-mpa-style" => \$use_mpa_style,
"report-zero-counts" => \$report_zero_counts,
"minimum-hit-groups=i" => \$minimum_hit_groups,
"report-minimizer-data" => \$report_minimizer_data,
);
if (! defined $threads) {
$threads = $ENV{"KRAKEN2_NUM_THREADS"} || 1;
}
if (! @ARGV) {
print STDERR "Need to specify input filenames!\n";
usage();
}
eval { $db_prefix = kraken2lib::find_db($db_prefix); };
if ($@) {
die "$PROG: $@";
}
my $taxonomy = "$db_prefix/taxo.k2d";
my $kht_file = "$db_prefix/hash.k2d";
my $opt_file = "$db_prefix/opts.k2d";
for my $file ($taxonomy, $kht_file, $opt_file) {
if (! -e $file) {
die "$PROG: $file does not exist!\n";
}
}
if ($paired && ((@ARGV % 2) != 0 || @ARGV == 0)) {
die "$PROG: --paired requires positive and even number filenames\n";
}
my $compressed = $gunzip || $bunzip2;
if ($gunzip && $bunzip2) {
die "$PROG: can't use both gzip and bzip2 compression flags\n";
}
if ($confidence_threshold < 0) {
die "$PROG: confidence threshold must be nonnegative\n";
}
if ($confidence_threshold > 1) {
die "$PROG: confidence threshold must be no greater than 1\n";
}
if ($minimum_hit_groups < 0) {
die "$PROG: minimum number of hit groups must be nonnegative\n";
}
my $auto_detect = ! $compressed;
if ($auto_detect) {
auto_detect_file_format();
}
# set flags for classifier
my @flags;
push @flags, "-H", $kht_file;
push @flags, "-t", $taxonomy;
push @flags, "-o", $opt_file;
push @flags, "-p", $threads;
push @flags, "-q" if $quick;
push @flags, "-P" if $paired;
push @flags, "-n" if $names_in_output;
push @flags, "-T", $confidence_threshold;
push @flags, "-U", $unclassified_out if defined $unclassified_out;
push @flags, "-C", $classified_out if defined $classified_out;
push @flags, "-O", $outfile if defined $outfile;
push @flags, "-Q", $minimum_base_quality;
push @flags, "-R", $report_filename if defined $report_filename;
push @flags, "-m" if $use_mpa_style;
push @flags, "-z" if $report_zero_counts;
push @flags, "-M" if $memory_mapping;
push @flags, "-g", $minimum_hit_groups;
push @flags, "-K" if $report_minimizer_data;
# Stupid hack to keep filehandles from closing before exec
# filehandles opened inside for loop below go out of scope
# and are closed at end of loop without this
my @persistent_fhs;
# handle compressed files by opening pipes from decompression programs
if ($compressed) {
my @replacement_ARGV;
my $compression_program;
if ($gunzip) {
$compression_program = "gzip";
}
elsif ($bunzip2) {
$compression_program = "bzip2";
}
else {
die "$PROG: unrecognized compression program! This is a Kraken bug.\n";
}
for my $file (@ARGV) {
my $qm_file = quotemeta $file;
open my $fh, "$compression_program -dc $qm_file |"
or die "$PROG: error opening pipe from $compression_program for $file: $!\n";
# Have to unset close-on-exec flags to make these pipes stay open across
# exec call
my $flags = fcntl $fh, F_GETFD, 0 or die "$PROG: fcntl GETFD error: $!\n";
fcntl $fh, F_SETFD, ($flags & ~FD_CLOEXEC) or die "$PROG: fcntl SETFD error: $!\n";
push @persistent_fhs, $fh;
my $fd = fileno $fh;
push @replacement_ARGV, "/dev/fd/$fd";
}
@ARGV = @replacement_ARGV;
}
exec $CLASSIFY, @flags, @ARGV;
die "$PROG: exec error: $!\n";
sub usage {
my $exit_code = @_ ? shift : 64;
my $default_db = "none";
eval { $default_db = '"' . kraken2lib::find_db() . '"'; };
my $def_thread_ct = exists $ENV{"KRAKEN2_NUM_THREADS"} ? (0 + $ENV{"KRAKEN2_NUM_THREADS"}) : 1;
print STDERR <
Options:
--db NAME Name for Kraken 2 DB
(default: $default_db)
--threads NUM Number of threads (default: $def_thread_ct)
--quick Quick operation (use first hit or hits)
--unclassified-out FILENAME
Print unclassified sequences to filename
--classified-out FILENAME
Print classified sequences to filename
--output FILENAME Print output to filename (default: stdout); "-" will
suppress normal output
--confidence FLOAT Confidence score threshold (default: 0.0); must be
in [0, 1].
--minimum-base-quality NUM
Minimum base quality used in classification (def: 0,
only effective with FASTQ input).
--report FILENAME Print a report with aggregrate counts/clade to file
--use-mpa-style With --report, format report output like Kraken 1's
kraken-mpa-report
--report-zero-counts With --report, report counts for ALL taxa, even if
counts are zero
--report-minimizer-data With --report, report minimizer and distinct minimizer
count information in addition to normal Kraken report
--memory-mapping Avoids loading database into RAM
--paired The filenames provided have paired-end reads
--use-names Print scientific names instead of just taxids
--gzip-compressed Input files are compressed with gzip
--bzip2-compressed Input files are compressed with bzip2
--minimum-hit-groups NUM
Minimum number of hit groups (overlapping k-mers
sharing the same minimizer) needed to make a call
(default: $minimum_hit_groups)
--help Print this message
--version Print version information
If none of the *-compressed flags are specified, and the filename provided
is a regular file, automatic format detection is attempted.
EOF
exit $exit_code;
}
sub display_help {
usage(0);
}
sub display_version {
print "Kraken version #####=VERSION=#####\n";
print "Copyright 2013-2023, Derrick Wood (dwood\@cs.jhu.edu)\n";
exit 0;
}
sub auto_detect_file_format {
my $magic;
my $filename = $ARGV[0];
# Don't try to auto-detect when you can't unread data
if (! -f $filename) {
return;
}
# read 2-byte magic number to determine type of compression (if any)
open FILE, "<", $filename;
read FILE, $magic, 2;
close FILE;
if ($magic eq $GZIP_MAGIC) {
$compressed = 1;
$gunzip = 1;
}
elsif ($magic eq $BZIP2_MAGIC) {
$compressed = 1;
$bunzip2 = 1;
}
}
kraken2-2.1.3/scripts/kraken2-build 0000775 0000000 0000000 00000025313 14437747201 0017136 0 ustar 00root root 0000000 0000000 #!/usr/bin/env perl
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# General build process wrapper for Kraken 2.
use strict;
use warnings;
use File::Basename;
use Getopt::Long;
use POSIX;
my $PROG = basename $0;
my $KRAKEN2_DIR = '#####=KRAKEN2_DIR=#####';
# Test to see if the executables got moved, try to recover if we can
if (! -e "$KRAKEN2_DIR/classify") {
use Cwd 'abs_path';
$KRAKEN2_DIR = dirname abs_path($0);
}
$ENV{"KRAKEN2_DIR"} = $KRAKEN2_DIR;
$ENV{"PATH"} = "$KRAKEN2_DIR:$ENV{PATH}";
my $DEF_AA_MINIMIZER_LEN = 12;
my $DEF_AA_KMER_LEN = 15;
my $DEF_AA_MINIMIZER_SPACES = 0;
my $DEF_NT_MINIMIZER_LEN = 31;
my $DEF_NT_KMER_LEN = 35;
my $DEF_NT_MINIMIZER_SPACES = 7;
my $DEF_THREAD_CT = 1;
my $DEF_LOAD_FACTOR = 0.7;
my $DEF_BLOCK_SIZE = 16384;
my $DEF_SUBBLOCK_SIZE = 0;
my @VALID_LIBRARY_TYPES = qw/archaea bacteria plasmid viral plant
protozoa fungi human nr nt
UniVec UniVec_Core/;
my @VALID_SPECIAL_DB_TYPES = qw/greengenes silva rdp/;
# Option/task option variables
my (
$db,
$threads,
$minimizer_len,
$kmer_len,
$minimizer_spaces,
$is_protein,
$masking,
$max_db_size,
$use_ftp,
$skip_maps,
$load_factor,
$fast_build,
$block_size,
$subblock_size,
$minimum_bits_for_taxid,
$dl_taxonomy,
$dl_library,
$add_to_library,
$build,
$standard,
$clean,
$special,
);
# Initialization of defaults and env. var checks
$db = $ENV{"KRAKEN2_DB_NAME"};
$threads = $ENV{"KRAKEN2_THREAD_CT"} || $DEF_THREAD_CT;
$load_factor = $ENV{"KRAKEN2_LOAD_FACTOR"} || $DEF_LOAD_FACTOR;
$kmer_len = $ENV{"KRAKEN2_KMER_LEN"};
$minimizer_len = $ENV{"KRAKEN2_MINIMIZER_LEN"};
$minimizer_spaces = $ENV{"KRAKEN2_MINIMIZER_SPACES"};
$is_protein = $ENV{"KRAKEN2_PROTEIN_DB"} || 0;
$use_ftp = $ENV{"KRAKEN2_USE_FTP"} || 0;
$skip_maps = $ENV{"KRAKEN2_SKIP_MAPS"} || 0;
$masking = exists $ENV{"KRAKEN2_MASK_LC"} ? $ENV{"KRAKEN2_MASK_LC"} : 1;
$fast_build = $ENV{"KRAKEN2_FAST_BUILD"} || 0;
$block_size = $ENV{"KRAKEN2_BLOCK_SIZE"} || $DEF_BLOCK_SIZE;
$subblock_size = $ENV{"KRAKEN2_SUBBLOCK_SIZE"} || $DEF_SUBBLOCK_SIZE;
$minimum_bits_for_taxid = $ENV{"KRAKEN2_MIN_TAXID_BITS"} || 0;
# variables corresponding to task options
my @TASK_LIST = (
\$dl_taxonomy,
\$dl_library,
\$add_to_library,
\$build,
\$standard,
\$clean,
\$special,
);
GetOptions(
"help" => \&display_help,
"version" => \&display_version,
"db=s" => \$db,
"threads=i" => \$threads,
"minimizer-len=i" => \$minimizer_len,
"kmer-len=i" => \$kmer_len,
"minimizer-spaces=i", \$minimizer_spaces,
"protein" => \$is_protein,
"masking!" => \$masking,
"max-db-size=i" => \$max_db_size,
"use-ftp" => \$use_ftp,
"skip-maps" => \$skip_maps,
"load-factor=f" => \$load_factor,
"fast-build" => \$fast_build,
"block-size=i" => \$block_size,
"subblock-size=i" => \$subblock_size,
"minimum-bits-for-taxid=i" => \$minimum_bits_for_taxid,
"download-taxonomy" => \$dl_taxonomy,
"download-library=s" => \$dl_library,
"add-to-library=s" => \$add_to_library,
"build" => \$build,
"standard" => \$standard,
"clean" => \$clean,
"special=s" => \$special,
) or usage();
if ($is_protein) {
$kmer_len = $DEF_AA_KMER_LEN if ! defined $kmer_len;
$minimizer_len = $DEF_AA_MINIMIZER_LEN if ! defined $minimizer_len;
$minimizer_spaces = $DEF_AA_MINIMIZER_SPACES if ! defined $minimizer_spaces;
}
else {
$kmer_len = $DEF_NT_KMER_LEN if ! defined $kmer_len;
$minimizer_len = $DEF_NT_MINIMIZER_LEN if ! defined $minimizer_len;
$minimizer_spaces = $DEF_NT_MINIMIZER_SPACES if ! defined $minimizer_spaces;
}
if ($subblock_size == 0) {
$subblock_size = ceil($block_size / $threads);
}
if (@ARGV) {
warn "Extra arguments on command line.\n";
usage();
}
my $task_options = scalar grep defined $$_, @TASK_LIST;
if ($task_options > 1) {
warn "More than one task option selected.\n";
usage();
}
if ($task_options == 0) {
warn "Must select a task option.\n";
usage();
}
if (! defined $db) {
die "Must specify a database name\n";
}
if ($threads <= 0) {
die "Can't use nonpositive thread count of $threads\n";
}
if ($minimizer_len > $kmer_len) {
die "Minimizer length ($minimizer_len) must not be greater than k ($kmer_len)\n";
}
if ($minimizer_len <= 0) {
die "Can't use nonpositive minimizer length of $minimizer_len\n";
}
if ($minimizer_len > 31) {
die "Can't use minimizer len of $minimizer_len (must be <= 31)\n";
}
if ($load_factor <= 0) {
die "Can't have nonnegative load factor of $load_factor\n";
}
if ($load_factor > 1) {
die "Can't have load factor of $load_factor (must be no more than 1.0).\n";
}
$ENV{"KRAKEN2_DB_NAME"} = $db;
$ENV{"KRAKEN2_THREAD_CT"} = $threads;
$ENV{"KRAKEN2_MINIMIZER_LEN"} = $minimizer_len;
$ENV{"KRAKEN2_KMER_LEN"} = $kmer_len;
$ENV{"KRAKEN2_MINIMIZER_SPACES"} = $minimizer_spaces;
$ENV{"KRAKEN2_SEED_TEMPLATE"} = construct_seed_template();
$ENV{"KRAKEN2_PROTEIN_DB"} = $is_protein ? 1 : "";
$ENV{"KRAKEN2_MASK_LC"} = $masking ? 1 : "";
$ENV{"KRAKEN2_MAX_DB_SIZE"} = defined($max_db_size) ? $max_db_size : "";
$ENV{"KRAKEN2_USE_FTP"} = $use_ftp ? 1 : "";
$ENV{"KRAKEN2_SKIP_MAPS"} = $skip_maps ? 1 : "";
$ENV{"KRAKEN2_LOAD_FACTOR"} = $load_factor;
$ENV{"KRAKEN2_FAST_BUILD"} = $fast_build ? 1 : "";
$ENV{"KRAKEN2_BLOCK_SIZE"} = $block_size;
$ENV{"KRAKEN2_SUBBLOCK_SIZE"} = $subblock_size;
$ENV{"KRAKEN2_MIN_TAXID_BITS"} = $minimum_bits_for_taxid;
if ($dl_taxonomy) {
download_taxonomy();
}
elsif (defined($dl_library)) {
download_library($dl_library);
}
elsif (defined($add_to_library)) {
add_to_library($add_to_library);
}
elsif ($standard) {
standard_installation();
}
elsif ($build) {
build_database();
}
elsif ($clean) {
clean_database();
}
elsif ($special) {
build_special_database($special);
}
else {
usage();
}
exit -1;
# END OF MAIN CODE.
sub usage {
my $exit_code = @_ ? shift : 64;
print STDERR <
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Wrapper for the dump_table program to report minimizer counts per taxa
use strict;
use warnings;
use Fcntl;
use File::Basename;
use Getopt::Long;
my $PROG = basename $0;
my $KRAKEN2_DIR = '#####=KRAKEN2_DIR=#####';
# Test to see if the executables got moved, try to recover if we can
if (! -e "$KRAKEN2_DIR/classify") {
use Cwd 'abs_path';
$KRAKEN2_DIR = dirname abs_path($0);
}
require "$KRAKEN2_DIR/kraken2lib.pm";
$ENV{"KRAKEN2_DIR"} = $KRAKEN2_DIR;
$ENV{"PATH"} = "$KRAKEN2_DIR:$ENV{PATH}";
my $DUMP_TABLE = "$KRAKEN2_DIR/dump_table";
my $db_prefix;
my $names_in_output = 0;
my $use_mpa_style = 0;
my $report_zero_counts = 0;
my $skip_counts = 0;
my $threads;
GetOptions(
"help" => \&display_help,
"version" => \&display_version,
"db=s" => \$db_prefix,
"skip-counts" => \$skip_counts,
"use-mpa-style" => \$use_mpa_style,
"report-zero-counts" => \$report_zero_counts,
"threads=i" => \$threads,
);
if (@ARGV) {
usage();
}
eval { $db_prefix = kraken2lib::find_db($db_prefix); };
if ($@) {
die "$PROG: $@";
}
if (! defined $threads) {
$threads = $ENV{"KRAKEN2_NUM_THREADS"} || 1;
}
my $taxonomy = "$db_prefix/taxo.k2d";
my $kht_file = "$db_prefix/hash.k2d";
my $opt_file = "$db_prefix/opts.k2d";
for my $file ($taxonomy, $kht_file, $opt_file) {
if (! -e $file) {
die "$PROG: $file does not exist!\n";
}
}
# set flags for dump_table
my @flags;
push @flags, "-H", $kht_file;
push @flags, "-t", $taxonomy;
push @flags, "-o", $opt_file;
push @flags, "-s" if $skip_counts;
push @flags, "-m" if $use_mpa_style;
push @flags, "-z" if $report_zero_counts;
push @flags, "-p", $threads;
exec $DUMP_TABLE, @flags;
die "$PROG: exec error: $!\n";
sub usage {
my $exit_code = @_ ? shift : 64;
my $default_db = "none";
eval { $default_db = '"' . kraken2lib::find_db() . '"'; };
print STDERR <
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Common subroutines for other Kraken scripts
use strict;
use warnings;
# Input: the argument for a --db option (possibly undefined)
# Returns: the DB to use, taking KRAKEN2_DEFAULT_DB and KRAKEN2_DB_PATH
# into account.
sub find_db {
my $supplied_db_prefix = shift;
my $db_prefix;
if (! defined $supplied_db_prefix) {
if (! exists $ENV{"KRAKEN2_DEFAULT_DB"}) {
die "Must specify DB with either --db or \$KRAKEN2_DEFAULT_DB\n";
}
$supplied_db_prefix = $ENV{"KRAKEN2_DEFAULT_DB"};
}
my @db_path = (".");
if (exists $ENV{"KRAKEN2_DB_PATH"}) {
my $path_str = $ENV{"KRAKEN2_DB_PATH"};
# Allow zero-length path to be current dir
$path_str =~ s/^:/.:/;
$path_str =~ s/:$/:./;
$path_str =~ s/::/:.:/;
@db_path = split /:/, $path_str;
}
# Use supplied DB if abs. or rel. path is given
if ($supplied_db_prefix =~ m|/|) {
$db_prefix = $supplied_db_prefix;
}
else {
# Check all dirs in KRAKEN2_DB_PATH
for my $dir (@db_path) {
my $checked_db = "$dir/$supplied_db_prefix";
if (-e $checked_db && -d _) {
$db_prefix = $checked_db;
last;
}
}
if (! defined $db_prefix) {
my $printed_path = exists $ENV{"KRAKEN2_DB_PATH"} ? qq|"$ENV{'KRAKEN2_DB_PATH'}"| : "undefined";
die "unable to find $supplied_db_prefix in \$KRAKEN2_DB_PATH ($printed_path)\n";
}
}
for my $file (qw/taxo.k2d hash.k2d opts.k2d/) {
if (! -e "$db_prefix/$file") {
die "database (\"$db_prefix\") does not contain necessary file $file\n";
}
}
return $db_prefix;
}
# Input: a FASTA sequence ID
# Output: either (a) a taxonomy ID number found in the sequence ID,
# (b) an NCBI accession number found in the sequence ID, or undef
sub check_seqid {
my $seqid = shift;
my $taxid = undef;
# Note all regexes here use ?: to avoid capturing the ^ or | character
if ($seqid =~ /(?:^|\|)kraken:taxid\|(\d+)/) {
$taxid = $1; # OK, has explicit taxid
}
elsif ($seqid =~ /^(\d+)$/) {
$taxid = $1; # OK, has explicit taxid (w/o token)
}
# Accession number check
elsif ($seqid =~ /(?:^|\|) # Begins seqid or immediately follows pipe
([A-Z]+ # Starts with one or more UC alphas
_? # Might have an underscore next
[A-Z0-9]+) # Ends with UC alphas or digits
(?:\||\b|\.)/x # Followed by pipe, word boundary, or period
)
{
$taxid = $1; # A bit misleading - failure to pass /^\d+$/ means this is
# OK, but requires accession -> taxid mapping
}
return $taxid;
}
1;
kraken2-2.1.3/scripts/lookup_accession_numbers.pl 0000775 0000000 0000000 00000003364 14437747201 0022213 0 ustar 00root root 0000000 0000000 #!/usr/bin/env perl
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Looks up accession numbers and reports associated taxonomy IDs
#
# Input is (a) 1 2-column TSV file w/ sequence IDs and accession numbers,
# and (b) a list of accession2taxid files from NCBI.
# Output is tab-delimited lines, with sequence IDs in first
# column and taxonomy IDs in second.
use strict;
use warnings;
use File::Basename;
use Getopt::Std;
my $PROG = basename $0;
my $lookup_list_file = shift @ARGV;
my %target_lists;
open FILE, "<", $lookup_list_file
or die "$PROG: can't open $lookup_list_file: $!\n";
while () {
chomp;
my ($seqid, $accnum) = split /\t/;
$target_lists{$accnum} ||= [];
push @{ $target_lists{$accnum} }, $seqid;
}
close FILE;
my $initial_target_count = scalar keys %target_lists;
my @accession_map_files = @ARGV;
for my $file (@accession_map_files) {
open FILE, "<", $file
or die "$PROG: can't open $file: $!\n";
scalar(); # discard header line
while () {
chomp;
my ($accession, $with_version, $taxid, $gi) = split /\t/;
if ($target_lists{$accession}) {
my @list = @{ $target_lists{$accession} };
delete $target_lists{$accession};
for my $seqid (@list) {
print "$seqid\t$taxid\n";
}
last if ! %target_lists;
}
}
close FILE;
last if ! %target_lists;
}
if (%target_lists) {
warn "$PROG: @{[scalar keys %target_lists]}/$initial_target_count accession numbers remain unmapped, see unmapped.txt in DB directory\n";
open LIST, ">", "unmapped.txt"
or die "$PROG: can't write unmapped.txt: $!\n";
for my $target (keys %target_lists) {
print LIST "$target\n";
}
close LIST;
}
kraken2-2.1.3/scripts/make_seqid2taxid_map.pl 0000775 0000000 0000000 00000007453 14437747201 0021176 0 ustar 00root root 0000000 0000000 #!/usr/bin/env perl
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Reads multi-FASTA input and examines each sequence header. In quiet mode
# (-q, used as an initial check on file validity), headers are OK if a
# taxonomy ID is found (as either the entire sequence ID or as part of a
# "kraken:taxid" token), or if something looking like a GI or accession
# number is found. In normal mode, the taxonomy ID will be looked up (if
# not explicitly specified in the sequence ID) and reported if it can be
# found. Output is tab-delimited lines, with sequence IDs in first
# column and taxonomy IDs in second.
# Sequence IDs with a kraken:taxid token will use that to assign taxonomy
# ID, e.g.:
# >gi|32499|ref|NC_021949.2|kraken:taxid|562|
#
# Sequence IDs that are completely numeric are assumed to be the taxonomy
# ID for that sequence.
#
# Otherwise, an accession number is searched for; if not found, a GI
# number is searched for. Failure to find any of the above is a fatal error.
# Without -q, a comma-separated file list specified by -A (for both accession
# numbers and GI numbers) is examined; failure to find a
# taxonomy ID that maps to a provided accession/GI number is non-fatal and
# will emit a warning.
#
# With -q, does not print any output, and will die w/ nonzero exit instead
# of warning when unable to find a taxid, accession #, or GI #.
use strict;
use warnings;
use File::Basename;
use Getopt::Std;
my $PROG = basename $0;
getopts('qA:L:', \my %opts);
if ($opts{'q'} && (defined $opts{"A"} || defined $opts{"L"})) {
die "$PROG: -q doesn't allow for -A/-L\n";
}
my %target_lists;
while (<>) {
next unless /^>(\S+)/;
my $seq_id = $1;
my $output;
# Note all regexes here use ?: to avoid capturing the ^ or | character
if ($seq_id =~ /(?:^|\|)kraken:taxid\|(\d+)/) {
$output = "$seq_id\t$1\n";
}
elsif ($seq_id =~ /^\d+$/) {
$output = "$seq_id\t$seq_id\n";
}
# Accession regex is first, only puts number (not version) into $1
elsif ($seq_id =~ /(?:^|\|)([A-Z]+_?[A-Z0-9]+)(?:\||\b|\.)/ ||
$seq_id =~ /(?:^|\|)gi\|(\d+)/)
{
if (! $opts{"q"}) {
$target_lists{$1} ||= [];
push @{ $target_lists{$1} }, $seq_id;
}
}
else {
die "$PROG: unable to determine taxonomy ID for sequence $seq_id\n";
}
if (defined $output && ! $opts{"q"}) {
print $output;
}
}
if ($opts{"q"}) {
if (keys %target_lists) {
print "$PROG: requires external map\n";
}
exit 0;
}
exit 0 if ! keys %target_lists;
if (! (defined $opts{"A"} || defined $opts{"L"})) {
die "$PROG: found sequence ID without explicit taxonomy ID, but no map used\n";
}
# Remove targets where we've already handled the mapping
if (defined $opts{"L"}) {
my $library_map_file = $opts{"L"};
open FILE, "<", $library_map_file
or die "$PROG: can't open $library_map_file: $!\n";
while () {
chomp;
my ($seqid, $taxid) = split /\t/;
if (exists $target_lists{$seqid}) {
print "$seqid\t$taxid\n";
delete $target_lists{$seqid};
}
}
close FILE;
}
exit 0 if ! keys %target_lists;
my @accession_map_files = split /,/, $opts{"A"};
for my $file (@accession_map_files) {
open FILE, "<", $file
or die "$PROG: can't open $file: $!\n";
scalar(); # discard header line
while () {
chomp;
my ($accession, $with_version, $taxid, $gi) = split /\t/;
if ($target_lists{$accession}) {
my @list = @{ $target_lists{$accession} };
delete $target_lists{$accession};
for my $seqid (@list) {
print "$seqid\t$taxid\n";
}
}
if ($gi ne "na" && $target_lists{$gi}) {
my @list = @{ $target_lists{$gi} };
delete $target_lists{$gi};
for my $seqid (@list) {
print "$seqid\t$taxid\n";
}
}
}
close FILE;
}
kraken2-2.1.3/scripts/mask_low_complexity.sh 0000775 0000000 0000000 00000002271 14437747201 0021204 0 ustar 00root root 0000000 0000000 #!/bin/bash
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Masks low complexity sequences in the database using the dustmasker (nucl.)
# or segmasker (prot.) programs from NCBI.
set -u # Protect against uninitialized vars.
set -e # Stop on error
set -o pipefail # Stop on failures in non-final pipeline commands
target="$1"
MASKER="k2mask"
if [ -n "$KRAKEN2_PROTEIN_DB" ]; then
MASKER="segmasker"
fi
if ! which $MASKER > /dev/null; then
echo "Unable to find $MASKER in path, can't mask low-complexity sequences"
exit 1
fi
if [ -d $target ]; then
for file in $(find $target '(' -name '*.fna' -o -name '*.faa' ')'); do
if [ ! -e "$file.masked" ]; then
$MASKER -in $file -outfmt fasta | sed -e '/^>/!s/[a-z]/x/g' > "$file.tmp"
mv "$file.tmp" $file
touch "$file.masked"
fi
done
elif [ -f $target ]; then
if [ ! -e "$target.masked" ]; then
$MASKER -in $target -outfmt fasta | sed -e '/^>/!s/[a-z]/x/g' > "$target.tmp"
mv "$target.tmp" $target
touch "$target.masked"
fi
else
echo "Target $target must be directory or regular file, aborting masking"
exit 1
fi
kraken2-2.1.3/scripts/rsync_from_ncbi.pl 0000775 0000000 0000000 00000014412 14437747201 0020270 0 ustar 00root root 0000000 0000000 #!/usr/bin/env perl
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Reads an assembly_summary.txt file, which indicates taxids and FTP paths for
# genome/protein data. Performs the download of the complete genomes from
# that file, decompresses, and explicitly assigns taxonomy as needed.
use strict;
use warnings;
use File::Basename;
use Getopt::Std;
use Net::FTP;
use List::Util qw/max/;
my $PROG = basename $0;
my $SERVER = "ftp.ncbi.nlm.nih.gov";
my $SERVER_PATH = "/genomes";
my $FTP_USER = "anonymous";
my $FTP_PASS = "kraken2download";
my $qm_server = quotemeta $SERVER;
my $qm_server_path = quotemeta $SERVER_PATH;
my $is_protein = $ENV{"KRAKEN2_PROTEIN_DB"};
my $use_ftp = $ENV{"KRAKEN2_USE_FTP"};
my $suffix = $is_protein ? "_protein.faa.gz" : "_genomic.fna.gz";
# Manifest hash maps filenames (keys) to taxids (values)
my %manifest;
while (<>) {
next if /^#/;
chomp;
my @fields = split /\t/;
my ($taxid, $asm_level, $ftp_path) = @fields[5, 11, 19];
# Possible TODO - make the list here configurable by user-supplied flags
next unless grep {$asm_level eq $_} ("Complete Genome", "Chromosome");
next if $ftp_path eq "na"; # Skip if no provided path
my $full_path = $ftp_path . "/" . basename($ftp_path) . $suffix;
# strip off server/leading dir name to allow --files-from= to work w/ rsync
# also allows filenames to just start with "all/", which is nice
if (! ($full_path =~ s#^(?:ftp|https)://${qm_server}${qm_server_path}/##)) {
die "$PROG: unexpected FTP path (new server?) for $ftp_path\n";
}
$manifest{$full_path} = $taxid;
}
open MANIFEST, ">", "manifest.txt"
or die "$PROG: can't write manifest: $!\n";
print MANIFEST "$_\n" for keys %manifest;
close MANIFEST;
if ($is_protein && ! $use_ftp) {
print STDERR "Step 0/2: performing rsync dry run (only protein d/l requires this)...\n";
# Protein files aren't always present, so we have to do this two-rsync run hack
# First, do a dry run to find non-existent files, then delete them from the
# manifest; after this, execution can proceed as usual.
system("rsync --dry-run --no-motd --files-from=manifest.txt rsync://${SERVER}${SERVER_PATH} . 2> rsync.err");
open ERR_FILE, "<", "rsync.err"
or die "$PROG: can't read rsync.err file: $!\n";
while () {
chomp;
# I really doubt this will work across every version of rsync. :(
if (/failed: No such file or directory/ && /^rsync: link_stat "\/([^"]+)"/) {
delete $manifest{$1};
}
}
close ERR_FILE;
print STDERR "Rsync dry run complete, removing any non-existent files from manifest.\n";
# Rewrite manifest
open MANIFEST, ">", "manifest.txt"
or die "$PROG: can't write manifest: $!\n";
print MANIFEST "$_\n" for keys %manifest;
close MANIFEST;
}
sub ftp_connection {
my $ftp = Net::FTP->new($SERVER, Passive => 1)
or die "$PROG: FTP connection error: $@\n";
$ftp->login($FTP_USER, $FTP_PASS)
or die "$PROG: FTP login error: " . $ftp->message() . "\n";
$ftp->binary()
or die "$PROG: FTP binary mode error: " . $ftp->message() . "\n";
$ftp->cwd($SERVER_PATH)
or die "$PROG: FTP CD error: " . $ftp->message() . "\n";
return $ftp;
}
if ($use_ftp) {
print STDERR "Step 1/2: Performing ftp file transfer of requested files\n";
open MANIFEST, "<", "manifest.txt"
or die "$PROG: can't open manifest: $!\n";
mkdir "all" or die "$PROG: can't create 'all' directory: $!\n";
chdir "all" or die "$PROG: can't chdir into 'all' directory: $!\n";
while () {
chomp;
my $ftp = ftp_connection();
my $try = 0;
my $ntries = 5;
my $sleepsecs = 3;
while($try < $ntries) {
$try++;
last if $ftp->get($_);
warn "$PROG: unable to download $_ on try $try of $ntries: ".$ftp->message()."\n";
last if $try == $ntries;
sleep $sleepsecs;
$sleepsecs *= 3;
}
die "$PROG: unable to download ftp://${SERVER}${SERVER_PATH}/$_\n" if $try == $ntries;
$ftp->quit;
}
close MANIFEST;
chdir ".." or die "$PROG: can't return to correct directory: $!\n";
}
else {
print STDERR "Step 1/2: Performing rsync file transfer of requested files\n";
system("rsync --no-motd --files-from=manifest.txt rsync://${SERVER}${SERVER_PATH}/ .") == 0
or die "$PROG: rsync error, exiting: $?\n";
print STDERR "Rsync file transfer complete.\n";
}
print STDERR "Step 2/2: Assigning taxonomic IDs to sequences\n";
my $output_file = $is_protein ? "library.faa" : "library.fna";
open OUT, ">", $output_file
or die "$PROG: can't write $output_file: $!\n";
my $projects_added = 0;
my $sequences_added = 0;
my $ch_added = 0;
my $ch = $is_protein ? "aa" : "bp";
my $max_out_chars = 0;
for my $in_filename (sort keys %manifest) {
my $taxid = $manifest{$in_filename};
if ($use_ftp) { # FTP downloading doesn't create full path locally
$in_filename = "all/" . basename($in_filename);
}
open IN, "gunzip -c $in_filename |" or die "$PROG: can't read $in_filename: $!\n";
while () {
if (/^>/) {
s/^>/>kraken:taxid|$taxid|/;
$sequences_added++;
}
else {
$ch_added += length($_) - 1;
}
print OUT;
}
close IN;
unlink $in_filename;
$projects_added++;
my $out_line = progress_line($projects_added, scalar keys %manifest, $sequences_added, $ch_added) . "...";
$max_out_chars = max(length($out_line), $max_out_chars);
my $space_line = " " x $max_out_chars;
print STDERR "\r$space_line\r$out_line" if -t STDERR;
}
close OUT;
print STDERR " done.\n" if -t STDERR;
print STDERR "All files processed, cleaning up extra sequence files...";
system("rm -rf all/") == 0
or die "$PROG: can't clean up all/ directory: $?\n";
print STDERR " done, library complete.\n";
sub progress_line {
my ($projs, $total_projs, $seqs, $chs) = @_;
my $line = "Processed ";
$line .= ($projs == $total_projs) ? "$projs" : "$projs/$total_projs";
$line .= " project" . ($total_projs > 1 ? 's' : '') . " ";
$line .= "($seqs sequence" . ($seqs > 1 ? 's' : '') . ", ";
my $prefix;
my @prefixes = qw/k M G T P E/;
while (@prefixes && $chs >= 1000) {
$prefix = shift @prefixes;
$chs /= 1000;
}
if (defined $prefix) {
$line .= sprintf '%.2f %s%s)', $chs, $prefix, $ch;
}
else {
$line .= "$chs $ch)";
}
return $line;
}
kraken2-2.1.3/scripts/scan_fasta_file.pl 0000775 0000000 0000000 00000003137 14437747201 0020217 0 ustar 00root root 0000000 0000000 #!/usr/bin/env perl
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Reads multi-FASTA input and examines each sequence header. Headers are
# OK if a taxonomy ID is found (as either the entire sequence ID or as part
# of a # "kraken:taxid" token), or if something looking like an accession
# number is found. Not "OK" headers will are fatal errors unless "--lenient"
# is used.
#
# Each sequence header results in a line with three tab-separated values;
# the first indicating whether third column is the taxonomy ID ("TAXID") or
# an accession number ("ACCNUM") for the sequence ID listed in the second
# column.
use strict;
use warnings;
use File::Basename;
use Getopt::Long;
require "$ENV{KRAKEN2_DIR}/kraken2lib.pm";
my $PROG = basename $0;
my $lenient = 0;
GetOptions("lenient" => \$lenient)
or die "Usage: $PROG [--lenient] \n";
while (<>) {
next unless /^>/;
# while (/.../g) needed because non-redundant DBs sometimes have multiple
# sequence IDs in the header; extra sequence IDs are prefixed by
# '\x01' characters (if downloaded in FASTA format from NCBI FTP directly).
while (/(?:^>|\x01)(\S+)/g) {
my $seqid = $1;
my $taxid = kraken2lib::check_seqid($seqid);
if (! defined $taxid) {
next if $lenient;
die "$PROG: unable to determine taxonomy ID for sequence $seqid\n";
}
# "$taxid" may actually be an accession number
if ($taxid =~ /^\d+$/) {
print "TAXID\t$seqid\t$taxid\n";
}
else {
print "ACCNUM\t$seqid\t$taxid\n";
}
}
}
kraken2-2.1.3/scripts/standard_installation.sh 0000775 0000000 0000000 00000003074 14437747201 0021476 0 ustar 00root root 0000000 0000000 #!/bin/bash
# Copyright 2013-2023, Derrick Wood
#
# This file is part of the Kraken 2 taxonomic sequence classification system.
# Build the standard Kraken database
# Designed to be called by kraken_build
set -u # Protect against uninitialized vars.
set -e # Stop on error
set -o pipefail # Stop on failures in non-final pipeline commands
protein_flag=""
if [ -n "$KRAKEN2_PROTEIN_DB" ]; then
protein_flag="--protein"
fi
masking_flag=""
if [ -z "$KRAKEN2_MASK_LC" ]; then
masking_flag="--no-mask"
fi
ftp_flag=""
if [ -n "$KRAKEN2_USE_FTP" ]; then
ftp_flag="--use-ftp"
fi
kraken2-build --db $KRAKEN2_DB_NAME --download-taxonomy $masking_flag $protein_flag $ftp_flag
kraken2-build --db $KRAKEN2_DB_NAME --download-library archaea $masking_flag $protein_flag $ftp_flag
kraken2-build --db $KRAKEN2_DB_NAME --download-library bacteria $masking_flag $protein_flag $ftp_flag
kraken2-build --db $KRAKEN2_DB_NAME --download-library viral $masking_flag $protein_flag $ftp_flag
kraken2-build --db $KRAKEN2_DB_NAME --download-library plasmid $masking_flag $protein_flag $ftp_flag
kraken2-build --db $KRAKEN2_DB_NAME --download-library human --no-mask $protein_flag $ftp_flag
if [ -z "$KRAKEN2_PROTEIN_DB" ]; then
kraken2-build --db $KRAKEN2_DB_NAME --download-library UniVec_Core $masking_flag $ftp_flag
fi
kraken2-build --db $KRAKEN2_DB_NAME --build --threads $KRAKEN2_THREAD_CT \
--minimizer-len $KRAKEN2_MINIMIZER_LEN \
--kmer-len $KRAKEN2_KMER_LEN \
--minimizer-spaces $KRAKEN2_MINIMIZER_SPACES \
$protein_flag
kraken2-2.1.3/src/ 0000775 0000000 0000000 00000000000 14437747201 0013652 5 ustar 00root root 0000000 0000000 kraken2-2.1.3/src/CMakeLists.txt 0000664 0000000 0000000 00000002042 14437747201 0016410 0 ustar 00root root 0000000 0000000 add_executable(build_db
build_db.cc
mmap_file.cc
compact_hash.cc
taxonomy.cc
seqreader.cc
mmscanner.cc
omp_hack.cc
utilities.cc)
add_executable(classify
classify.cc
reports.cc
mmap_file.cc
compact_hash.cc
taxonomy.cc
seqreader.cc
mmscanner.cc
omp_hack.cc
aa_translate.cc
utilities.cc
hyperloglogplus.cc)
add_executable(estimate_capacity
estimate_capacity.cc
seqreader.cc
mmscanner.cc
omp_hack.cc
utilities.cc)
add_executable(dump_table
dump_table.cc
mmap_file.cc
compact_hash.cc
omp_hack.cc
taxonomy.cc
reports.cc
hyperloglogplus.cc)
add_executable(lookup_accession_numbers
lookup_accession_numbers.cc
mmap_file.cc
omp_hack.cc
utilities.cc)
add_executable(k2mask
k2mask.cc
seqreader.cc)
target_link_libraries(dustmasker ${ZLIB_LIBRARIES} Threads::Threads)
kraken2-2.1.3/src/Makefile 0000664 0000000 0000000 00000004116 14437747201 0015314 0 ustar 00root root 0000000 0000000 CXX = g++
KRAKEN2_SKIP_FOPENMP ?= -fopenmp
CXXFLAGS = $(KRAKEN2_SKIP_FOPENMP) -Wall -std=c++11 -O3
CXXFLAGS += -DLINEAR_PROBING
.PHONY: all clean install
PROGS = estimate_capacity build_db classify dump_table lookup_accession_numbers k2mask
all: $(PROGS)
install: $(PROGS)
cp $(PROGS) "$(KRAKEN2_DIR)/"
clean:
rm -f *.o $(PROGS)
taxonomy.o: taxonomy.cc taxonomy.h mmap_file.h
hyperloglogplus.o: hyperloglogplus.cc hyperloglogplus.h
mmap_file.o: mmap_file.cc mmap_file.h
compact_hash.o: compact_hash.cc compact_hash.h kv_store.h mmap_file.h kraken2_data.h
mmscanner.o: mmscanner.cc mmscanner.h
seqreader.o: seqreader.cc seqreader.h
omp_hack.o: omp_hack.cc omp_hack.h
reports.o: reports.cc reports.h kraken2_data.h
aa_translate.o: aa_translate.cc aa_translate.h
utilities.o: utilities.cc utilities.h
classify.o: classify.cc kraken2_data.h kv_store.h taxonomy.h seqreader.h mmscanner.h compact_hash.h aa_translate.h reports.h utilities.h readcounts.h
dump_table.o: dump_table.cc compact_hash.h taxonomy.h mmscanner.h kraken2_data.h reports.h
estimate_capacity.o: estimate_capacity.cc kv_store.h mmscanner.h seqreader.h utilities.h
build_db.o: build_db.cc taxonomy.h mmscanner.h seqreader.h compact_hash.h kv_store.h kraken2_data.h utilities.h
lookup_accession_numbers.o: lookup_accession_numbers.cc mmap_file.h utilities.h
k2mask.o: k2mask.cc gzstream.h threadpool.h
build_db: build_db.o mmap_file.o compact_hash.o taxonomy.o seqreader.o mmscanner.o omp_hack.o utilities.o
$(CXX) $(CXXFLAGS) -o $@ $^
classify: classify.o reports.o hyperloglogplus.o mmap_file.o compact_hash.o taxonomy.o seqreader.o mmscanner.o omp_hack.o aa_translate.o utilities.o
$(CXX) $(CXXFLAGS) -o $@ $^
estimate_capacity: estimate_capacity.o seqreader.o mmscanner.o omp_hack.o utilities.o
$(CXX) $(CXXFLAGS) -o $@ $^
dump_table: dump_table.o mmap_file.o compact_hash.o omp_hack.o taxonomy.o reports.o hyperloglogplus.o
$(CXX) $(CXXFLAGS) -o $@ $^
lookup_accession_numbers: lookup_accession_numbers.o mmap_file.o omp_hack.o utilities.o
$(CXX) $(CXXFLAGS) -o $@ $^
k2mask: k2mask.o seqreader.o
$(CXX) $(CXXFLAGS) -o $@ $^ -lz -pthread
kraken2-2.1.3/src/aa_translate.cc 0000664 0000000 0000000 00000007520 14437747201 0016623 0 ustar 00root root 0000000 0000000 /*
* Copyright 2013-2023, Derrick Wood
*
* This file is part of the Kraken 2 taxonomic sequence classification system.
*/
#include "aa_translate.h"
using std::string;
using std::vector;
namespace kraken2 {
/*
This *was* the map, which was ridiculously slow. Note that ordering is
AGCT, not ACGT.
static map old_translation_map = {
{ "AAA", 'K' }, { "AAG", 'K' }, { "AAC", 'N' }, { "AAT", 'N' },
{ "AGA", 'R' }, { "AGG", 'R' }, { "AGC", 'S' }, { "AGT", 'S' },
{ "ACA", 'T' }, { "ACG", 'T' }, { "ACC", 'T' }, { "ACT", 'T' },
{ "ATA", 'I' }, { "ATG", 'M' }, { "ATC", 'I' }, { "ATT", 'I' },
{ "GAA", 'E' }, { "GAG", 'E' }, { "GAC", 'D' }, { "GAT", 'D' },
{ "GGA", 'G' }, { "GGG", 'G' }, { "GGC", 'G' }, { "GGT", 'G' },
{ "GCA", 'A' }, { "GCG", 'A' }, { "GCC", 'A' }, { "GCT", 'A' },
{ "GTA", 'V' }, { "GTG", 'V' }, { "GTC", 'V' }, { "GTT", 'V' },
{ "CAA", 'Q' }, { "CAG", 'Q' }, { "CAC", 'H' }, { "CAT", 'H' },
{ "CGA", 'R' }, { "CGG", 'R' }, { "CGC", 'R' }, { "CGT", 'R' },
{ "CCA", 'P' }, { "CCG", 'P' }, { "CCC", 'P' }, { "CCT", 'P' },
{ "CTA", 'L' }, { "CTG", 'L' }, { "CTC", 'L' }, { "CTT", 'L' },
{ "TAA", '*' }, { "TAG", '*' }, { "TAC", 'Y' }, { "TAT", 'Y' },
{ "TGA", '*' }, { "TGG", 'W' }, { "TGC", 'C' }, { "TGT", 'C' },
{ "TCA", 'S' }, { "TCG", 'S' }, { "TCC", 'S' }, { "TCT", 'S' },
{ "TTA", 'L' }, { "TTG", 'L' }, { "TTC", 'F' }, { "TTT", 'F' }
};
*/
static char translation_map[] = "KKNNRRSSTTTTIMIIEEDDGGGGAAAAVVVVQQHHRRRRPPPPLLLL**YY*WCCSSSSLLFF";
static uint8_t fwd_lookup_table[UINT8_MAX + 1] = {0};
static uint8_t rev_lookup_table[UINT8_MAX + 1] = {0};
void TranslateToAllFrames(string &dna_seq, vector &aa_seqs) {
auto max_size = (dna_seq.size() / 3) + 1;
for (auto i = 0; i < 6; i++)
aa_seqs[i].assign(max_size, ' ');
if (dna_seq.size() < 3)
return;
if (fwd_lookup_table[0] == 0) {
for (size_t i = 0; i <= UINT8_MAX; i++)
fwd_lookup_table[i] = rev_lookup_table[i] = UINT8_MAX;
// Map is based on AGCT coding, not ACGT
fwd_lookup_table[(int) 'A'] = fwd_lookup_table[(int) 'a'] = 0x00;
fwd_lookup_table[(int) 'G'] = fwd_lookup_table[(int) 'g'] = 0x01;
fwd_lookup_table[(int) 'C'] = fwd_lookup_table[(int) 'c'] = 0x02;
fwd_lookup_table[(int) 'T'] = fwd_lookup_table[(int) 't'] = 0x03;
rev_lookup_table[(int) 'A'] = rev_lookup_table[(int) 'a'] = 0x30;
rev_lookup_table[(int) 'G'] = rev_lookup_table[(int) 'g'] = 0x20;
rev_lookup_table[(int) 'C'] = rev_lookup_table[(int) 'c'] = 0x10;
rev_lookup_table[(int) 'T'] = rev_lookup_table[(int) 't'] = 0x00;
}
uint8_t fwd_codon = 0, rev_codon = 0;
int ambig_nt_countdown = 0; // if positive, bases to go until N leaves codon
size_t frame_len[6] = {0};
for (auto i = 0u; i < dna_seq.size(); i++) {
auto frame = i % 3;
fwd_codon <<= 2;
fwd_codon &= 0x3f;
rev_codon >>= 2;
if (ambig_nt_countdown)
ambig_nt_countdown--;
auto fwd_lookup_code = fwd_lookup_table[(int) dna_seq[i]];
auto rev_lookup_code = rev_lookup_table[(int) dna_seq[i]];
if (fwd_lookup_code == UINT8_MAX)
ambig_nt_countdown = 3;
else {
fwd_codon |= fwd_lookup_code;
rev_codon |= rev_lookup_code;
}
if (i >= 2) { // we've got a full codon
char ch;
// Translate and append to forward frame
ch = ambig_nt_countdown == 0 ? translation_map[fwd_codon] : 'X';
aa_seqs[frame][frame_len[frame]++] = ch;
// Translate and prepend to reverse frame
ch = ambig_nt_countdown == 0 ? translation_map[rev_codon] : 'X';
aa_seqs[frame + 3][max_size - 1 - frame_len[frame + 3]++] = ch;
}
}
for (auto i = 0; i < 3; i++)
if (aa_seqs[i].size() != frame_len[i])
aa_seqs[i].resize(frame_len[i]);
for (auto i = 3; i < 6; i++)
if (aa_seqs[i].size() != frame_len[i])
aa_seqs[i].erase(0, aa_seqs[i].size() - frame_len[i]);
}
}
kraken2-2.1.3/src/aa_translate.h 0000664 0000000 0000000 00000000545 14437747201 0016465 0 ustar 00root root 0000000 0000000 /*
* Copyright 2013-2023, Derrick Wood
*
* This file is part of the Kraken 2 taxonomic sequence classification system.
*/
#ifndef KRAKEN2_AA_TRANSLATE_H_
#define KRAKEN2_AA_TRANSLATE_H_
#include "kraken2_headers.h"
namespace kraken2 {
void TranslateToAllFrames(std::string &dna_seq, std::vector &aa_seqs);
}
#endif
kraken2-2.1.3/src/build_db.cc 0000664 0000000 0000000 00000051001 14437747201 0015722 0 ustar 00root root 0000000 0000000 /*
* Copyright 2013-2023, Derrick Wood
*
* This file is part of the Kraken 2 taxonomic sequence classification system.
*/
#include "kraken2_headers.h"
#include "taxonomy.h"
#include "mmscanner.h"
#include "seqreader.h"
#include "compact_hash.h"
#include "kv_store.h"
#include "kraken2_data.h"
#include "utilities.h"
using std::string;
using std::map;
using std::cout;
using std::queue;
using std::cerr;
using std::endl;
using std::vector;
using std::istringstream;
using std::ifstream;
using std::ofstream;
using namespace kraken2;
// These will likely be overridden by wrapper script,
// just keeping sane defaults in case they aren't
#define DEFAULT_BLOCK_SIZE (10 * 1024 * 1024) // 10 MB
#define DEFAULT_SUBBLOCK_SIZE (1024)
struct Options {
string ID_to_taxon_map_filename;
string ncbi_taxonomy_directory;
string hashtable_filename;
string options_filename;
string taxonomy_filename;
size_t block_size;
size_t subblock_size;
size_t requested_bits_for_taxid;
int num_threads;
bool input_is_protein;
ssize_t k, l;
size_t capacity;
size_t maximum_capacity;
uint64_t spaced_seed_mask;
uint64_t toggle_mask;
uint64_t min_clear_hash_value;
bool deterministic_build;
};
void ParseCommandLine(int argc, char **argv, Options &opts);
void usage(int exit_code = EX_USAGE);
vector ExtractNCBISequenceIDs(const string &header);
void ProcessSequenceFast(const string &seq, taxid_t taxid,
CompactHashTable &hash, const Taxonomy &tax, MinimizerScanner &scanner,
uint64_t min_clear_hash_value);
void ProcessSequence(const Options &opts, const string &seq, taxid_t taxid,
CompactHashTable &hash, const Taxonomy &tax);
void ProcessSequencesFast(const Options &opts,
const map &ID_to_taxon_map,
CompactHashTable &kraken_index, const Taxonomy &taxonomy);
void ProcessSequences(const Options &opts,
const map &ID_to_taxon_map,
CompactHashTable &kraken_index, const Taxonomy &taxonomy);
void SetMinimizerLCA(CompactHashTable &hash, uint64_t minimizer, taxid_t taxid,
const Taxonomy &tax);
void ReadIDToTaxonMap(map &id_map, string &filename);
void GenerateTaxonomy(Options &opts, map &id_map);
struct TaxonSeqPair {
taxid_t taxon;
string seq;
};
int main(int argc, char **argv) {
Options opts;
opts.spaced_seed_mask = DEFAULT_SPACED_SEED_MASK;
opts.toggle_mask = DEFAULT_TOGGLE_MASK;
opts.input_is_protein = false;
opts.num_threads = 1;
opts.block_size = DEFAULT_BLOCK_SIZE;
opts.subblock_size = DEFAULT_SUBBLOCK_SIZE;
opts.requested_bits_for_taxid = 0;
opts.min_clear_hash_value = 0;
opts.maximum_capacity = 0;
opts.deterministic_build = true;
ParseCommandLine(argc, argv, opts);
omp_set_num_threads( opts.num_threads );
map ID_to_taxon_map;
ReadIDToTaxonMap(ID_to_taxon_map, opts.ID_to_taxon_map_filename);
GenerateTaxonomy(opts, ID_to_taxon_map);
std::cerr << "Taxonomy parsed and converted." << std::endl;
Taxonomy taxonomy(opts.taxonomy_filename.c_str());
taxonomy.GenerateExternalToInternalIDMap();
size_t bits_needed_for_value = 1;
while ((1 << bits_needed_for_value) < (ssize_t) taxonomy.node_count())
bits_needed_for_value++;
if (opts.requested_bits_for_taxid > 0 &&
bits_needed_for_value > opts.requested_bits_for_taxid)
errx(EX_DATAERR, "more bits required for storing taxid");
size_t bits_for_taxid = bits_needed_for_value;
if (bits_for_taxid < opts.requested_bits_for_taxid)
bits_for_taxid = opts.requested_bits_for_taxid;
auto actual_capacity = opts.capacity;
if (opts.maximum_capacity) {
double frac = opts.maximum_capacity * 1.0 / opts.capacity;
if (frac > 1)
errx(EX_DATAERR, "maximum capacity larger than requested capacity");
opts.min_clear_hash_value = (uint64_t) ((1 - frac) * UINT64_MAX);
actual_capacity = opts.maximum_capacity;
}
CompactHashTable kraken_index(actual_capacity, 32 - bits_for_taxid,
bits_for_taxid);
std::cerr << "CHT created with " << bits_for_taxid << " bits reserved for taxid." << std::endl;
if (opts.deterministic_build)
ProcessSequences(opts, ID_to_taxon_map, kraken_index, taxonomy);
else
ProcessSequencesFast(opts, ID_to_taxon_map, kraken_index, taxonomy);
std::cerr << "Writing data to disk... " << std::flush;
kraken_index.WriteTable(opts.hashtable_filename.c_str());
IndexOptions index_opts;
index_opts.k = opts.k;
index_opts.l = opts.l;
index_opts.spaced_seed_mask = opts.spaced_seed_mask;
index_opts.toggle_mask = opts.toggle_mask;
index_opts.dna_db = ! opts.input_is_protein;
index_opts.minimum_acceptable_hash_value = opts.min_clear_hash_value;
index_opts.revcom_version = CURRENT_REVCOM_VERSION;
index_opts.db_version = 0;
index_opts.db_type = 0;
ofstream opts_fs(opts.options_filename);
opts_fs.write((char *) &index_opts, sizeof(index_opts));
if (! opts_fs.good())
errx(EX_OSERR, "Unable to write options file %s",
opts.options_filename.c_str());
opts_fs.close();
std::cerr << " complete." << std::endl;
return 0;
}
// A quick but nondeterministic build
void ProcessSequencesFast(const Options &opts,
const map &ID_to_taxon_map,
CompactHashTable &kraken_index, const Taxonomy &taxonomy)
{
size_t processed_seq_ct = 0;
size_t processed_ch_ct = 0;
#pragma omp parallel
{
Sequence sequence;
MinimizerScanner scanner(opts.k, opts.l, opts.spaced_seed_mask,
! opts.input_is_protein, opts.toggle_mask);
BatchSequenceReader reader;
while (true) {
// Declaration of "ok" and break need to be done outside of critical
// section to conform with OpenMP spec.
bool ok;
#pragma omp critical(reader)
ok = reader.LoadBlock(std::cin, opts.block_size);
if (! ok)
break;
while (reader.NextSequence(sequence)) {
auto all_sequence_ids = ExtractNCBISequenceIDs(sequence.header);
taxid_t taxid = 0;
for (auto &seqid : all_sequence_ids) {
if (ID_to_taxon_map.count(seqid) == 0 || ID_to_taxon_map.at(seqid) == 0) continue;
auto ext_taxid = ID_to_taxon_map.at(seqid);
taxid = taxonomy.LowestCommonAncestor(taxid, taxonomy.GetInternalID(ext_taxid));
}
if (taxid) {
// Add terminator for protein sequences if not already there
if (opts.input_is_protein && sequence.seq.back() != '*')
sequence.seq.push_back('*');
ProcessSequenceFast(sequence.seq, taxid, kraken_index, taxonomy, scanner,
opts.min_clear_hash_value);
#pragma omp atomic
processed_seq_ct++;
#pragma omp atomic
processed_ch_ct += sequence.seq.size();
}
}
if (isatty(fileno(stderr))) {
#pragma omp critical(status_update)
std::cerr << "\rProcessed " << processed_seq_ct << " sequences (" << processed_ch_ct << " " << (opts.input_is_protein ? "aa" : "bp") << ")...";
}
}
}
if (isatty(fileno(stderr)))
std::cerr << "\r";
std::cerr << "Completed processing of " << processed_seq_ct << " sequences, " << processed_ch_ct << " " << (opts.input_is_protein ? "aa" : "bp") << std::endl;
}
// Slightly slower but deterministic when multithreaded
void ProcessSequences(const Options &opts,
const map &ID_to_taxon_map,
CompactHashTable &kraken_index, const Taxonomy &taxonomy)
{
size_t processed_seq_ct = 0;
size_t processed_ch_ct = 0;
Sequence sequence;
BatchSequenceReader reader;
while (reader.LoadBlock(std::cin, DEFAULT_BLOCK_SIZE)) {
while (reader.NextSequence(sequence)) {
auto all_sequence_ids = ExtractNCBISequenceIDs(sequence.header);
taxid_t taxid = 0;
for (auto &seqid : all_sequence_ids) {
if (ID_to_taxon_map.count(seqid) == 0 || ID_to_taxon_map.at(seqid) == 0) continue;
auto ext_taxid = ID_to_taxon_map.at(seqid);
taxid = taxonomy.LowestCommonAncestor(taxid, taxonomy.GetInternalID(ext_taxid));
}
if (taxid) {
// Add terminator for protein sequences if not already there
if (opts.input_is_protein && sequence.seq.back() != '*')
sequence.seq.push_back('*');
ProcessSequence(opts, sequence.seq, taxid, kraken_index, taxonomy);
processed_seq_ct++;
processed_ch_ct += sequence.seq.size();
}
}
if (isatty(fileno(stderr))) {
std::cerr << "\rProcessed " << processed_seq_ct << " sequences (" << processed_ch_ct << " " << (opts.input_is_protein ? "aa" : "bp") << ")...";
}
}
if (isatty(fileno(stderr)))
std::cerr << "\r";
std::cerr << "Completed processing of " << processed_seq_ct << " sequences, " << processed_ch_ct << " " << (opts.input_is_protein ? "aa" : "bp") << std::endl;
}
// This function exists to deal with NCBI's use of \x01 characters to denote
// the start of a new FASTA header in the same line (for non-redundant DBs).
// We return all sequence IDs in a header line, not just the first.
vector ExtractNCBISequenceIDs(const string &header) {
vector list;
string current_str;
bool in_id = true;
// start loop at first char after '>'
for (size_t i = 1; i < header.size(); ++i) {
if (header[i] == 0x01) {
// 0x01 starts new ID at next char
if (! current_str.empty())
list.push_back(current_str);
current_str.clear();
in_id = true;
}
else if (in_id && isspace(header[i])) {
// spaces end ID
if (! current_str.empty())
list.push_back(current_str);
current_str.clear();
in_id = false;
}
else if (in_id) {
// Build ID string char by char
current_str.push_back(header[i]);
}
}
if (! current_str.empty())
list.push_back(current_str);
return list;
}
void SetMinimizerLCA(CompactHashTable &hash, uint64_t minimizer, taxid_t taxid,
const Taxonomy &tax)
{
hvalue_t old_value = 0;
hvalue_t new_value = taxid;
while (! hash.CompareAndSet(minimizer, new_value, &old_value))
new_value = tax.LowestCommonAncestor(old_value, taxid);
}
void ProcessSequenceFast(const string &seq, taxid_t taxid,
CompactHashTable &hash, const Taxonomy &tax, MinimizerScanner &scanner,
uint64_t min_clear_hash_value)
{
scanner.LoadSequence(seq);
uint64_t *minimizer_ptr;
while ((minimizer_ptr = scanner.NextMinimizer())) {
if (scanner.is_ambiguous())
continue;
if (min_clear_hash_value && MurmurHash3(*minimizer_ptr) < min_clear_hash_value)
continue;
hvalue_t existing_taxid = 0;
hvalue_t new_taxid = taxid;
while (! hash.CompareAndSet(*minimizer_ptr, new_taxid, &existing_taxid)) {
new_taxid = tax.LowestCommonAncestor(new_taxid, existing_taxid);
}
}
}
void ProcessSequence(const Options &opts, const string &seq, taxid_t taxid,
CompactHashTable &hash, const Taxonomy &tax)
{
const int set_ct = 256;
omp_lock_t locks[set_ct];
for (int i = 0; i < set_ct; i++)
omp_init_lock(&locks[i]);
// for each block in the sequence
for (size_t j = 0; j < seq.size(); j += opts.block_size) {
// block: fixed length subsequence of DNA/protein, handled in series,
// consecutive blocks overlap by k-1 characters
size_t block_start = j;
size_t block_finish = j + opts.block_size + opts.k - 1;
if (block_finish > seq.size())
block_finish = seq.size();
std::set minimizer_sets[set_ct];
// for each subblock in the block, gather minimizers in parallel
#pragma omp parallel for schedule(dynamic)
for (size_t i = block_start; i < block_finish; i += opts.subblock_size) {
// subblock: fixed length subsequence of block, handled in parallel,
// consecutive subblocks overlap by k-1 characters
MinimizerScanner scanner(opts.k, opts.l, opts.spaced_seed_mask,
! opts.input_is_protein, opts.toggle_mask);
size_t subblock_finish = i + opts.subblock_size + opts.k - 1;
if (subblock_finish > block_finish)
subblock_finish = block_finish;
scanner.LoadSequence(seq, i, subblock_finish);
uint64_t *minimizer_ptr;
while ((minimizer_ptr = scanner.NextMinimizer())) {
if (scanner.is_ambiguous())
continue;
auto hc = MurmurHash3(*minimizer_ptr);
// Hash-based subsampling
if (opts.min_clear_hash_value && hc < opts.min_clear_hash_value)
continue;
auto zone = hc % set_ct;
omp_set_lock(&locks[zone]);
minimizer_sets[zone].insert(*minimizer_ptr);
omp_unset_lock(&locks[zone]);
}
} // end subblock for loop
// combine sets into sorted list
std::vector minimizer_lists[set_ct];
size_t minimizer_list_prefix_sizes[set_ct+1] = {0};
#pragma omp parallel for
for (int i = 0; i < set_ct; i++) {
minimizer_lists[i].reserve(minimizer_sets[i].size());
for (auto &m : minimizer_sets[i])
minimizer_lists[i].push_back(m);
sort(minimizer_lists[i].begin(), minimizer_lists[i].end());
minimizer_list_prefix_sizes[i+1] = minimizer_lists[i].size();
}
for (int i = 2; i <= set_ct; i++)
minimizer_list_prefix_sizes[i] += minimizer_list_prefix_sizes[i-1];
std::vector minimizer_list(minimizer_list_prefix_sizes[set_ct]);
#pragma omp parallel for
for (int i = 0; i < set_ct; i++) {
std::copy(minimizer_lists[i].begin(), minimizer_lists[i].end(),
minimizer_list.begin() + minimizer_list_prefix_sizes[i]);
}
size_t mm_ct = minimizer_list.size();
std::vector index_list(mm_ct);
std::vector insertion_list(mm_ct);
for (size_t i = 0; i < mm_ct; i++)
insertion_list[i] = true;
// Loop to enforce deterministic order
while (! minimizer_list.empty()) {
// Gather insertion point information for all remaining minimizers
#pragma omp parallel for
for (size_t i = 0; i < mm_ct; i++) {
// once we've determined that a minimizer won't be an insertion,
// don't bother calling FindIndex() again
if (insertion_list[i]) {
size_t idx;
insertion_list[i] = ! hash.FindIndex(minimizer_list[i], &idx);
index_list[i] = idx;
}
}
// Determine safe prefix of sorted set to insert in parallel
std::set novel_insertion_points;
size_t safe_ct = 0;
for (safe_ct = 0; safe_ct < mm_ct; safe_ct++) {
if (insertion_list[safe_ct]) {
if (novel_insertion_points.count(index_list[safe_ct]) > 0)
break;
novel_insertion_points.insert(index_list[safe_ct]);
}
}
// Adjust CHT values for all keys in the safe zone in parallel
#pragma omp parallel for
for (size_t i = 0; i < safe_ct; i++) {
SetMinimizerLCA(hash, minimizer_list[i], taxid, tax);
}
// Remove safe prefix and re-iterate to process remainder
minimizer_list.erase(minimizer_list.begin(), minimizer_list.begin() + safe_ct);
index_list.erase(index_list.begin(), index_list.begin() + safe_ct);
insertion_list.erase(insertion_list.begin(), insertion_list.begin() + safe_ct);
mm_ct -= safe_ct;
} // end deterministic order while loop
} // end block for loop
for (int i = 0; i < set_ct; i++)
omp_destroy_lock(&locks[i]);
}
void ReadIDToTaxonMap(map &id_map, string &filename) {
ifstream map_file(filename.c_str());
if (! map_file.good())
err(EX_NOINPUT, "unable to read from '%s'", filename.c_str());
string line;
while (getline(map_file, line)) {
string seq_id;
taxid_t taxid;
istringstream iss(line);
iss >> seq_id;
iss >> taxid;
if (taxid)
id_map[seq_id] = taxid;
}
}
void GenerateTaxonomy(Options &opts, map &id_map) {
NCBITaxonomy ncbi_taxonomy(
opts.ncbi_taxonomy_directory + "/nodes.dmp",
opts.ncbi_taxonomy_directory + "/names.dmp"
);
for (auto &kv_pair : id_map) {
if (kv_pair.second != 0) {
ncbi_taxonomy.MarkNode(kv_pair.second);
}
}
ncbi_taxonomy.ConvertToKrakenTaxonomy(opts.taxonomy_filename.c_str());
}
void ParseCommandLine(int argc, char **argv, Options &opts) {
int opt;
long long sig;
while ((opt = getopt(argc, argv, "?hB:b:c:FH:m:n:o:t:k:l:M:p:r:s:S:T:X")) != -1) {
switch (opt) {
case 'h' : case '?' :
usage(0);
break;
case 'B' :
sig = atoll(optarg);
if (sig < 1)
errx(EX_USAGE, "must have positive block size");
opts.block_size = sig;
break;
case 'b' :
sig = atoll(optarg);
if (sig < 1)
errx(EX_USAGE, "must have positive subblock size");
opts.subblock_size = sig;
break;
case 'r' :
sig = atoll(optarg);
if (sig < 0)
errx(EX_USAGE, "can't have negative bit storage");
if (sig > 31)
errx(EX_USAGE, "can't have more than 31 bits of storage for taxid");
opts.requested_bits_for_taxid = sig;
break;
case 'p' :
opts.num_threads = atoll(optarg);
if (opts.num_threads < 1)
errx(EX_USAGE, "can't have negative number of threads");
if (opts.num_threads > omp_get_max_threads())
errx(EX_USAGE, "OMP only wants you to use %d threads", omp_get_max_threads());
break;
case 'H' :
opts.hashtable_filename = optarg;
break;
case 'm' :
opts.ID_to_taxon_map_filename = optarg;
break;
case 'n' :
opts.ncbi_taxonomy_directory = optarg;
break;
case 'o' :
opts.options_filename = optarg;
break;
case 't' :
opts.taxonomy_filename = optarg;
break;
case 'S' :
opts.spaced_seed_mask = strtol(optarg, nullptr, 2);
break;
case 'T' :
opts.toggle_mask = strtol(optarg, nullptr, 2);
break;
case 'k' :
sig = atoll(optarg);
if (sig < 1)
errx(EX_USAGE, "k must be positive integer");
opts.k = sig;
break;
case 'l' :
sig = atoll(optarg);
if (sig < 1)
errx(EX_USAGE, "l must be positive integer");
if (sig > 31)
errx(EX_USAGE, "l must be no more than 31");
opts.l = sig;
break;
case 'c' :
sig = atoll(optarg);
if (sig < 1)
errx(EX_USAGE, "capacity must be positive integer");
opts.capacity = sig;
break;
case 'M' :
sig = atoll(optarg);
if (sig < 1)
errx(EX_USAGE, "max capacity must be positive integer");
opts.maximum_capacity = sig;
break;
case 'F' :
opts.deterministic_build = false;
break;
case 'X' :
opts.input_is_protein = true;
break;
}
}
if (opts.spaced_seed_mask != DEFAULT_SPACED_SEED_MASK)
ExpandSpacedSeedMask(opts.spaced_seed_mask,
opts.input_is_protein ? BITS_PER_CHAR_PRO : BITS_PER_CHAR_DNA);
if (opts.hashtable_filename.empty() ||
opts.ID_to_taxon_map_filename.empty() ||
opts.ncbi_taxonomy_directory.empty() ||
opts.options_filename.empty() ||
opts.taxonomy_filename.empty())
{
cerr << "missing mandatory filename parameter" << endl;
usage();
}
if (opts.k == 0 || opts.l == 0 || opts.capacity == 0) {
cerr << "missing mandatory integer parameter" << endl;
usage();
}
if (opts.k < opts.l) {
cerr << "k cannot be less than l" << endl;
usage();
}
if (opts.block_size < opts.subblock_size) {
cerr << "block size cannot be less than subblock size\n";
usage();
}
if (opts.maximum_capacity > opts.capacity) {
cerr << "maximum capacity option shouldn't specify larger capacity than normal" << endl;
usage();
}
}
void usage(int exit_code) {
cerr << "Usage: build_db \n"
<< "\n"
<< "Options (*mandatory):\n"
<< "* -H FILENAME Kraken 2 hash table filename\n"
<< "* -m FILENAME Sequence ID to taxon map filename\n"
<< "* -t FILENAME Kraken 2 taxonomy filename\n"
<< "* -n DIR NCBI taxonomy directory name\n"
<< "* -o FILENAME Kraken 2 options filename\n"
<< "* -k INT Set length of k-mers\n"
<< "* -l INT Set length of minimizers\n"
<< "* -c INT Set capacity of hash table\n"
<< " -M INT Set maximum capacity of hash table (MiniKraken)\n"
<< " -S BITSTRING Spaced seed mask\n"
<< " -T BITSTRING Minimizer toggle mask\n"
<< " -X Input seqs. are proteins\n"
<< " -p INT Number of threads\n"
<< " -F Use fast, nondeterministic building method\n"
<< " -B INT Read block size\n"
<< " -b INT Read subblock size\n"
<< " -r INT Bit storage requested for taxid" << endl;
exit(exit_code);
}
kraken2-2.1.3/src/classify.cc 0000664 0000000 0000000 00000067675 14437747201 0016023 0 ustar 00root root 0000000 0000000 /*
* Copyright 2013-2023, Derrick Wood
*
* This file is part of the Kraken 2 taxonomic sequence classification system.
*/
#include "kraken2_headers.h"
#include "kv_store.h"
#include "taxonomy.h"
#include "seqreader.h"
#include "mmscanner.h"
#include "compact_hash.h"
#include "kraken2_data.h"
#include "aa_translate.h"
#include "reports.h"
#include "utilities.h"
#include "readcounts.h"
using namespace kraken2;
using std::cout;
using std::cerr;
using std::endl;
using std::ifstream;
using std::map;
using std::ostringstream;
using std::ofstream;
using std::set;
using std::string;
using std::vector;
using namespace kraken2;
static const size_t NUM_FRAGMENTS_PER_THREAD = 10000;
static const taxid_t MATE_PAIR_BORDER_TAXON = TAXID_MAX;
static const taxid_t READING_FRAME_BORDER_TAXON = TAXID_MAX - 1;
static const taxid_t AMBIGUOUS_SPAN_TAXON = TAXID_MAX - 2;
struct Options {
string index_filename;
string taxonomy_filename;
string options_filename;
string report_filename;
string classified_output_filename;
string unclassified_output_filename;
string kraken_output_filename;
bool mpa_style_report;
bool report_kmer_data;
bool quick_mode;
bool report_zero_counts;
bool use_translated_search;
bool print_scientific_name;
double confidence_threshold;
int num_threads;
bool paired_end_processing;
bool single_file_pairs;
int minimum_quality_score;
int minimum_hit_groups;
bool use_memory_mapping;
bool match_input_order;
};
struct ClassificationStats {
uint64_t total_sequences;
uint64_t total_bases;
uint64_t total_classified;
};
struct OutputStreamData {
bool initialized;
bool printing_sequences;
std::ostream *classified_output1;
std::ostream *classified_output2;
std::ostream *unclassified_output1;
std::ostream *unclassified_output2;
std::ostream *kraken_output;
};
struct OutputData {
uint64_t block_id;
string kraken_str;
string classified_out1_str;
string classified_out2_str;
string unclassified_out1_str;
string unclassified_out2_str;
};
void ParseCommandLine(int argc, char **argv, Options &opts);
void usage(int exit_code=EX_USAGE);
void ProcessFiles(const char *filename1, const char *filename2,
KeyValueStore *hash, Taxonomy &tax,
IndexOptions &idx_opts, Options &opts, ClassificationStats &stats,
OutputStreamData &outputs, taxon_counters_t &total_taxon_counters);
taxid_t ClassifySequence(Sequence &dna, Sequence &dna2, ostringstream &koss,
KeyValueStore *hash, Taxonomy &tax, IndexOptions &idx_opts,
Options &opts, ClassificationStats &stats, MinimizerScanner &scanner,
vector &taxa, taxon_counts_t &hit_counts,
vector &tx_frames, taxon_counters_t &my_taxon_counts);
void AddHitlistString(ostringstream &oss, vector &taxa,
Taxonomy &taxonomy);
taxid_t ResolveTree(taxon_counts_t &hit_counts,
Taxonomy &tax, size_t total_minimizers, Options &opts);
void ReportStats(struct timeval time1, struct timeval time2,
ClassificationStats &stats);
void InitializeOutputs(Options &opts, OutputStreamData &outputs, SequenceFormat format);
void MaskLowQualityBases(Sequence &dna, int minimum_quality_score);
int main(int argc, char **argv) {
Options opts;
opts.quick_mode = false;
opts.confidence_threshold = 0;
opts.paired_end_processing = false;
opts.single_file_pairs = false;
opts.num_threads = 1;
opts.mpa_style_report = false;
opts.report_kmer_data = false;
opts.report_zero_counts = false;
opts.use_translated_search = false;
opts.print_scientific_name = false;
opts.minimum_quality_score = 0;
opts.minimum_hit_groups = 0;
opts.use_memory_mapping = false;
taxon_counters_t taxon_counters; // stats per taxon
ParseCommandLine(argc, argv, opts);
omp_set_num_threads(opts.num_threads);
cerr << "Loading database information...";
IndexOptions idx_opts = {0};
ifstream idx_opt_fs(opts.options_filename);
struct stat sb;
if (stat(opts.options_filename.c_str(), &sb) < 0)
errx(EX_OSERR, "unable to get filesize of %s", opts.options_filename.c_str());
auto opts_filesize = sb.st_size;
idx_opt_fs.read((char *) &idx_opts, opts_filesize);
opts.use_translated_search = ! idx_opts.dna_db;
Taxonomy taxonomy(opts.taxonomy_filename, opts.use_memory_mapping);
KeyValueStore *hash_ptr = new CompactHashTable(opts.index_filename, opts.use_memory_mapping);
cerr << " done." << endl;
ClassificationStats stats = {0, 0, 0};
OutputStreamData outputs = { false, false, nullptr, nullptr, nullptr, nullptr, &std::cout };
struct timeval tv1, tv2;
gettimeofday(&tv1, nullptr);
if (optind == argc) {
if (opts.paired_end_processing && ! opts.single_file_pairs)
errx(EX_USAGE, "paired end processing used with no files specified");
ProcessFiles(nullptr, nullptr, hash_ptr, taxonomy, idx_opts, opts, stats, outputs, taxon_counters);
}
else {
for (int i = optind; i < argc; i++) {
if (opts.paired_end_processing && ! opts.single_file_pairs) {
if (i + 1 == argc) {
errx(EX_USAGE, "paired end processing used with unpaired file");
}
ProcessFiles(argv[i], argv[i+1], hash_ptr, taxonomy, idx_opts, opts, stats, outputs, taxon_counters);
i += 1;
}
else {
ProcessFiles(argv[i], nullptr, hash_ptr, taxonomy, idx_opts, opts, stats, outputs, taxon_counters);
}
}
}
gettimeofday(&tv2, nullptr);
delete hash_ptr;
ReportStats(tv1, tv2, stats);
if (! opts.report_filename.empty()) {
if (opts.mpa_style_report)
ReportMpaStyle(opts.report_filename, opts.report_zero_counts, taxonomy,
taxon_counters);
else {
auto total_unclassified = stats.total_sequences - stats.total_classified;
ReportKrakenStyle(opts.report_filename, opts.report_zero_counts,
opts.report_kmer_data, taxonomy,
taxon_counters, stats.total_sequences, total_unclassified);
}
}
return 0;
}
void ReportStats(struct timeval time1, struct timeval time2,
ClassificationStats &stats)
{
time2.tv_usec -= time1.tv_usec;
time2.tv_sec -= time1.tv_sec;
if (time2.tv_usec < 0) {
time2.tv_sec--;
time2.tv_usec += 1000000;
}
double seconds = time2.tv_usec;
seconds /= 1e6;
seconds += time2.tv_sec;
uint64_t total_unclassified = stats.total_sequences - stats.total_classified;
if (isatty(fileno(stderr)))
cerr << "\r";
fprintf(stderr,
"%llu sequences (%.2f Mbp) processed in %.3fs (%.1f Kseq/m, %.2f Mbp/m).\n",
(unsigned long long) stats.total_sequences,
stats.total_bases / 1.0e6,
seconds,
stats.total_sequences / 1.0e3 / (seconds / 60),
stats.total_bases / 1.0e6 / (seconds / 60) );
fprintf(stderr, " %llu sequences classified (%.2f%%)\n",
(unsigned long long) stats.total_classified,
stats.total_classified * 100.0 / stats.total_sequences);
fprintf(stderr, " %llu sequences unclassified (%.2f%%)\n",
(unsigned long long) total_unclassified,
total_unclassified * 100.0 / stats.total_sequences);
}
void ProcessFiles(const char *filename1, const char *filename2,
KeyValueStore *hash, Taxonomy &tax,
IndexOptions &idx_opts, Options &opts, ClassificationStats &stats,
OutputStreamData &outputs,
taxon_counters_t &total_taxon_counters)
{
std::istream *fptr1 = nullptr, *fptr2 = nullptr;
if (filename1 == nullptr)
fptr1 = &std::cin;
else {
fptr1 = new std::ifstream(filename1);
}
if (opts.paired_end_processing && ! opts.single_file_pairs) {
fptr2 = new std::ifstream(filename2);
}
// The priority queue for output is designed to ensure fragment data
// is output in the same order it was input
auto comparator = [](const OutputData &a, const OutputData &b) {
return a.block_id > b.block_id;
};
std::priority_queue, decltype(comparator)>
output_queue(comparator);
uint64_t next_input_block_id = 0;
uint64_t next_output_block_id = 0;
omp_lock_t output_lock;
omp_init_lock(&output_lock);
#pragma omp parallel
{
MinimizerScanner scanner(idx_opts.k, idx_opts.l, idx_opts.spaced_seed_mask,
idx_opts.dna_db, idx_opts.toggle_mask,
idx_opts.revcom_version);
vector taxa;
taxon_counts_t hit_counts;
ostringstream kraken_oss, c1_oss, c2_oss, u1_oss, u2_oss;
ClassificationStats thread_stats = {0, 0, 0};
vector translated_frames(6);
BatchSequenceReader reader1, reader2;
Sequence seq1, seq2;
uint64_t block_id;
OutputData out_data;
taxon_counters_t thread_taxon_counters;
while (true) {
thread_stats.total_sequences = 0;
thread_stats.total_bases = 0;
thread_stats.total_classified = 0;
auto ok_read = false;
#pragma omp critical(seqread)
{ // Input processing block
if (! opts.paired_end_processing) {
// Unpaired data? Just read in a sized block
ok_read = reader1.LoadBlock(*fptr1, (size_t)(3 * 1024 * 1024));
}
else if (! opts.single_file_pairs) {
// Paired data in 2 files? Read a line-counted batch from each file.
ok_read = reader1.LoadBatch(*fptr1, NUM_FRAGMENTS_PER_THREAD);
if (ok_read && opts.paired_end_processing)
ok_read = reader2.LoadBatch(*fptr2, NUM_FRAGMENTS_PER_THREAD);
}
else {
auto frags = NUM_FRAGMENTS_PER_THREAD * 2;
// Ensure frag count is even - just in case above line is changed
if (frags % 2 == 1)
frags++;
ok_read = reader1.LoadBatch(*fptr1, frags);
}
block_id = next_input_block_id++;
}
if (! ok_read)
break;
// Reset all dynamically-growing things
kraken_oss.str("");
c1_oss.str("");
c2_oss.str("");
u1_oss.str("");
u2_oss.str("");
thread_taxon_counters.clear();
while (true) {
auto valid_fragment = reader1.NextSequence(seq1);
if (opts.paired_end_processing && valid_fragment) {
if (opts.single_file_pairs)
valid_fragment = reader1.NextSequence(seq2);
else
valid_fragment = reader2.NextSequence(seq2);
}
if (! valid_fragment)
break;
thread_stats.total_sequences++;
if (opts.minimum_quality_score > 0) {
MaskLowQualityBases(seq1, opts.minimum_quality_score);
if (opts.paired_end_processing)
MaskLowQualityBases(seq2, opts.minimum_quality_score);
}
auto call = ClassifySequence(seq1, seq2,
kraken_oss, hash, tax, idx_opts, opts, thread_stats, scanner,
taxa, hit_counts, translated_frames, thread_taxon_counters);
if (call) {
char buffer[1024] = "";
sprintf(buffer, " kraken:taxid|%llu",
(unsigned long long) tax.nodes()[call].external_id);
seq1.header += buffer;
c1_oss << seq1.to_string();
if (opts.paired_end_processing) {
seq2.header += buffer;
c2_oss << seq2.to_string();
}
}
else {
u1_oss << seq1.to_string();
if (opts.paired_end_processing)
u2_oss << seq2.to_string();
}
thread_stats.total_bases += seq1.seq.size();
if (opts.paired_end_processing)
thread_stats.total_bases += seq2.seq.size();
}
#pragma omp atomic
stats.total_sequences += thread_stats.total_sequences;
#pragma omp atomic
stats.total_bases += thread_stats.total_bases;
#pragma omp atomic
stats.total_classified += thread_stats.total_classified;
#pragma omp critical(output_stats)
{
if (isatty(fileno(stderr)))
cerr << "\rProcessed " << stats.total_sequences
<< " sequences (" << stats.total_bases << " bp) ...";
}
if (! outputs.initialized) {
InitializeOutputs(opts, outputs, reader1.file_format());
}
out_data.block_id = block_id;
out_data.kraken_str.assign(kraken_oss.str());
out_data.classified_out1_str.assign(c1_oss.str());
out_data.classified_out2_str.assign(c2_oss.str());
out_data.unclassified_out1_str.assign(u1_oss.str());
out_data.unclassified_out2_str.assign(u2_oss.str());
#pragma omp critical(output_queue)
{
output_queue.push(out_data);
}
#pragma omp critical(update_taxon_counters)
{
for (auto &kv_pair : thread_taxon_counters) {
total_taxon_counters[kv_pair.first] += std::move(kv_pair.second);
}
}
bool output_loop = true;
while (output_loop) {
#pragma omp critical(output_queue)
{
output_loop = ! output_queue.empty();
if (output_loop) {
out_data = output_queue.top();
if (out_data.block_id == next_output_block_id) {
output_queue.pop();
// Acquiring output lock obligates thread to print out
// next output data block, contained in out_data
omp_set_lock(&output_lock);
next_output_block_id++;
}
else
output_loop = false;
}
}
if (! output_loop)
break;
if (outputs.kraken_output != nullptr)
(*outputs.kraken_output) << out_data.kraken_str;
if (outputs.classified_output1 != nullptr)
(*outputs.classified_output1) << out_data.classified_out1_str;
if (outputs.classified_output2 != nullptr)
(*outputs.classified_output2) << out_data.classified_out2_str;
if (outputs.unclassified_output1 != nullptr)
(*outputs.unclassified_output1) << out_data.unclassified_out1_str;
if (outputs.unclassified_output2 != nullptr)
(*outputs.unclassified_output2) << out_data.unclassified_out2_str;
omp_unset_lock(&output_lock);
} // end while output loop
} // end while
} // end parallel block
omp_destroy_lock(&output_lock);
if (fptr1 != nullptr)
delete fptr1;
if (fptr2 != nullptr)
delete fptr2;
if (outputs.kraken_output != nullptr)
(*outputs.kraken_output) << std::flush;
if (outputs.classified_output1 != nullptr)
(*outputs.classified_output1) << std::flush;
if (outputs.classified_output2 != nullptr)
(*outputs.classified_output2) << std::flush;
if (outputs.unclassified_output1 != nullptr)
(*outputs.unclassified_output1) << std::flush;
if (outputs.unclassified_output2 != nullptr)
(*outputs.unclassified_output2) << std::flush;
}
taxid_t ResolveTree(taxon_counts_t &hit_counts,
Taxonomy &taxonomy, size_t total_minimizers, Options &opts)
{
taxid_t max_taxon = 0;
uint32_t max_score = 0;
uint32_t required_score = ceil(opts.confidence_threshold * total_minimizers);
// Sum each taxon's LTR path, find taxon with highest LTR score
for (auto &kv_pair : hit_counts) {
taxid_t taxon = kv_pair.first;
uint32_t score = 0;
for (auto &kv_pair2 : hit_counts) {
taxid_t taxon2 = kv_pair2.first;
if (taxonomy.IsAAncestorOfB(taxon2, taxon)) {
score += kv_pair2.second;
}
}
if (score > max_score) {
max_score = score;
max_taxon = taxon;
}
else if (score == max_score) {
max_taxon = taxonomy.LowestCommonAncestor(max_taxon, taxon);
}
}
// Reset max. score to be only hits at the called taxon
max_score = hit_counts[max_taxon];
// We probably have a call w/o required support (unless LCA resolved tie)
while (max_taxon && max_score < required_score) {
max_score = 0;
for (auto &kv_pair : hit_counts) {
taxid_t taxon = kv_pair.first;
// Add to score if taxon in max_taxon's clade
if (taxonomy.IsAAncestorOfB(max_taxon, taxon)) {
max_score += kv_pair.second;
}
}
// Score is now sum of hits at max_taxon and w/in max_taxon clade
if (max_score >= required_score)
// Kill loop and return, we've got enough support here
return max_taxon;
else
// Run up tree until confidence threshold is met
// Run off tree if required score isn't met
max_taxon = taxonomy.nodes()[max_taxon].parent_id;
}
return max_taxon;
}
std::string TrimPairInfo(std::string &id) {
size_t sz = id.size();
if (sz <= 2)
return id;
if ( id[sz - 2] == '/' && (id[sz - 1] == '1' || id[sz - 1] == '2') )
return id.substr(0, sz - 2);
return id;
}
taxid_t ClassifySequence(Sequence &dna, Sequence &dna2, ostringstream &koss,
KeyValueStore *hash, Taxonomy &taxonomy, IndexOptions &idx_opts,
Options &opts, ClassificationStats &stats, MinimizerScanner &scanner,
vector &taxa, taxon_counts_t &hit_counts,
vector &tx_frames,
taxon_counters_t &curr_taxon_counts)
{
uint64_t *minimizer_ptr;
taxid_t call = 0;
taxa.clear();
hit_counts.clear();
auto frame_ct = opts.use_translated_search ? 6 : 1;
int64_t minimizer_hit_groups = 0;
for (int mate_num = 0; mate_num < 2; mate_num++) {
if (mate_num == 1 && ! opts.paired_end_processing)
break;
if (opts.use_translated_search) {
TranslateToAllFrames(mate_num == 0 ? dna.seq : dna2.seq, tx_frames);
}
// index of frame is 0 - 5 w/ tx search (or 0 if no tx search)
for (int frame_idx = 0; frame_idx < frame_ct; frame_idx++) {
if (opts.use_translated_search) {
scanner.LoadSequence(tx_frames[frame_idx]);
}
else {
scanner.LoadSequence(mate_num == 0 ? dna.seq : dna2.seq);
}
uint64_t last_minimizer = UINT64_MAX;
taxid_t last_taxon = TAXID_MAX;
while ((minimizer_ptr = scanner.NextMinimizer()) != nullptr) {
taxid_t taxon;
if (scanner.is_ambiguous()) {
taxon = AMBIGUOUS_SPAN_TAXON;
}
else {
if (*minimizer_ptr != last_minimizer) {
bool skip_lookup = false;
if (idx_opts.minimum_acceptable_hash_value) {
if (MurmurHash3(*minimizer_ptr) < idx_opts.minimum_acceptable_hash_value)
skip_lookup = true;
}
taxon = 0;
if (! skip_lookup)
taxon = hash->Get(*minimizer_ptr);
last_taxon = taxon;
last_minimizer = *minimizer_ptr;
// Increment this only if (a) we have DB hit and
// (b) minimizer != last minimizer
if (taxon) {
minimizer_hit_groups++;
// New minimizer should trigger registering minimizer in RC/HLL
curr_taxon_counts[taxon].add_kmer(scanner.last_minimizer());
}
}
else {
taxon = last_taxon;
}
if (taxon) {
if (opts.quick_mode && minimizer_hit_groups >= opts.minimum_hit_groups) {
call = taxon;
goto finished_searching; // need to break 3 loops here
}
hit_counts[taxon]++;
}
}
taxa.push_back(taxon);
}
if (opts.use_translated_search && frame_idx != 5)
taxa.push_back(READING_FRAME_BORDER_TAXON);
}
if (opts.paired_end_processing && mate_num == 0)
taxa.push_back(MATE_PAIR_BORDER_TAXON);
}
finished_searching:
auto total_kmers = taxa.size();
if (opts.paired_end_processing)
total_kmers--; // account for the mate pair marker
if (opts.use_translated_search) // account for reading frame markers
total_kmers -= opts.paired_end_processing ? 4 : 2;
call = ResolveTree(hit_counts, taxonomy, total_kmers, opts);
// Void a call made by too few minimizer groups
if (call && minimizer_hit_groups < opts.minimum_hit_groups)
call = 0;
if (call) {
stats.total_classified++;
curr_taxon_counts[call].incrementReadCount();
}
if (call)
koss << "C\t";
else
koss << "U\t";
if (! opts.paired_end_processing)
koss << dna.id << "\t";
else
koss << TrimPairInfo(dna.id) << "\t";
auto ext_call = taxonomy.nodes()[call].external_id;
if (opts.print_scientific_name) {
const char *name = nullptr;
if (call) {
name = taxonomy.name_data() + taxonomy.nodes()[call].name_offset;
}
koss << (name ? name : "unclassified") << " (taxid " << ext_call << ")";
}
else {
koss << ext_call;
}
koss << "\t";
if (! opts.paired_end_processing)
koss << dna.seq.size() << "\t";
else
koss << dna.seq.size() << "|" << dna2.seq.size() << "\t";
if (opts.quick_mode) {
koss << ext_call << ":Q";
}
else {
if (taxa.empty())
koss << "0:0";
else
AddHitlistString(koss, taxa, taxonomy);
}
koss << endl;
return call;
}
void AddHitlistString(ostringstream &oss, vector &taxa,
Taxonomy &taxonomy)
{
auto last_code = taxa[0];
auto code_count = 1;
for (size_t i = 1; i < taxa.size(); i++) {
auto code = taxa[i];
if (code == last_code) {
code_count += 1;
}
else {
if (last_code != MATE_PAIR_BORDER_TAXON && last_code != READING_FRAME_BORDER_TAXON) {
if (last_code == AMBIGUOUS_SPAN_TAXON) {
oss << "A:" << code_count << " ";
}
else {
auto ext_code = taxonomy.nodes()[last_code].external_id;
oss << ext_code << ":" << code_count << " ";
}
}
else { // mate pair/reading frame marker
oss << (last_code == MATE_PAIR_BORDER_TAXON ? "|:| " : "-:- ");
}
code_count = 1;
last_code = code;
}
}
if (last_code != MATE_PAIR_BORDER_TAXON && last_code != READING_FRAME_BORDER_TAXON) {
if (last_code == AMBIGUOUS_SPAN_TAXON) {
oss << "A:" << code_count << " ";
}
else {
auto ext_code = taxonomy.nodes()[last_code].external_id;
oss << ext_code << ":" << code_count;
}
}
else { // mate pair/reading frame marker
oss << (last_code == MATE_PAIR_BORDER_TAXON ? "|:|" : "-:-");
}
}
void InitializeOutputs(Options &opts, OutputStreamData &outputs, SequenceFormat format) {
#pragma omp critical(output_init)
{
if (! outputs.initialized) {
if (! opts.classified_output_filename.empty()) {
if (opts.paired_end_processing) {
vector fields = SplitString(opts.classified_output_filename, "#", 3);
if (fields.size() < 2) {
errx(EX_DATAERR, "Paired filename format missing # character: %s",
opts.classified_output_filename.c_str());
}
else if (fields.size() > 2) {
errx(EX_DATAERR, "Paired filename format has >1 # character: %s",
opts.classified_output_filename.c_str());
}
outputs.classified_output1 = new ofstream(fields[0] + "_1" + fields[1]);
outputs.classified_output2 = new ofstream(fields[0] + "_2" + fields[1]);
}
else
outputs.classified_output1 = new ofstream(opts.classified_output_filename);
outputs.printing_sequences = true;
}
if (! opts.unclassified_output_filename.empty()) {
if (opts.paired_end_processing) {
vector fields = SplitString(opts.unclassified_output_filename, "#", 3);
if (fields.size() < 2) {
errx(EX_DATAERR, "Paired filename format missing # character: %s",
opts.unclassified_output_filename.c_str());
}
else if (fields.size() > 2) {
errx(EX_DATAERR, "Paired filename format has >1 # character: %s",
opts.unclassified_output_filename.c_str());
}
outputs.unclassified_output1 = new ofstream(fields[0] + "_1" + fields[1]);
outputs.unclassified_output2 = new ofstream(fields[0] + "_2" + fields[1]);
}
else
outputs.unclassified_output1 = new ofstream(opts.unclassified_output_filename);
outputs.printing_sequences = true;
}
if (! opts.kraken_output_filename.empty()) {
if (opts.kraken_output_filename == "-") // Special filename to silence Kraken output
outputs.kraken_output = nullptr;
else
outputs.kraken_output = new ofstream(opts.kraken_output_filename);
}
outputs.initialized = true;
}
}
}
void MaskLowQualityBases(Sequence &dna, int minimum_quality_score) {
if (dna.format != FORMAT_FASTQ)
return;
if (dna.seq.size() != dna.quals.size())
errx(EX_DATAERR, "%s: Sequence length (%d) != Quality string length (%d)",
dna.id.c_str(), (int) dna.seq.size(), (int) dna.quals.size());
for (size_t i = 0; i < dna.seq.size(); i++) {
if ((dna.quals[i] - '!') < minimum_quality_score)
dna.seq[i] = 'x';
}
}
void ParseCommandLine(int argc, char **argv, Options &opts) {
int opt;
while ((opt = getopt(argc, argv, "h?H:t:o:T:p:R:C:U:O:Q:g:nmzqPSMK")) != -1) {
switch (opt) {
case 'h' : case '?' :
usage(0);
break;
case 'H' :
opts.index_filename = optarg;
break;
case 't' :
opts.taxonomy_filename = optarg;
break;
case 'T' :
opts.confidence_threshold = std::stod(optarg);
if (opts.confidence_threshold < 0 || opts.confidence_threshold > 1) {
errx(EX_USAGE, "confidence threshold must be in [0, 1]");
}
break;
case 'o' :
opts.options_filename = optarg;
break;
case 'q' :
opts.quick_mode = true;
break;
case 'p' :
opts.num_threads = atoi(optarg);
if (opts.num_threads < 1)
errx(EX_USAGE, "number of threads can't be less than 1");
break;
case 'g' :
opts.minimum_hit_groups = atoi(optarg);
break;
case 'P' :
opts.paired_end_processing = true;
break;
case 'S' :
opts.paired_end_processing = true;
opts.single_file_pairs = true;
break;
case 'm' :
opts.mpa_style_report = true;
break;
case 'K':
opts.report_kmer_data = true;
break;
case 'R' :
opts.report_filename = optarg;
break;
case 'z' :
opts.report_zero_counts = true;
break;
case 'C' :
opts.classified_output_filename = optarg;
break;
case 'U' :
opts.unclassified_output_filename = optarg;
break;
case 'O' :
opts.kraken_output_filename = optarg;
break;
case 'n' :
opts.print_scientific_name = true;
break;
case 'Q' :
opts.minimum_quality_score = atoi(optarg);
break;
case 'M' :
opts.use_memory_mapping = true;
break;
}
}
if (opts.index_filename.empty() ||
opts.taxonomy_filename.empty() ||
opts.options_filename.empty())
{
warnx("mandatory filename missing");
usage();
}
if (opts.mpa_style_report && opts.report_filename.empty()) {
warnx("-m requires -R be used");
usage();
}
}
void usage(int exit_code) {
cerr << "Usage: classify [options] " << endl
<< endl
<< "Options: (*mandatory)" << endl
<< "* -H filename Kraken 2 index filename" << endl
<< "* -t filename Kraken 2 taxonomy filename" << endl
<< "* -o filename Kraken 2 options filename" << endl
<< " -q Quick mode" << endl
<< " -M Use memory mapping to access hash & taxonomy" << endl
<< " -T NUM Confidence score threshold (def. 0)" << endl
<< " -p NUM Number of threads (def. 1)" << endl
<< " -Q NUM Minimum quality score (FASTQ only, def. 0)" << endl
<< " -P Process pairs of reads" << endl
<< " -S Process pairs with mates in same file" << endl
<< " -R filename Print report to filename" << endl
<< " -m In comb. w/ -R, use mpa-style report" << endl
<< " -z In comb. w/ -R, report taxa w/ 0 count" << endl
<< " -n Print scientific name instead of taxid in Kraken output" << endl
<< " -g NUM Minimum number of hit groups needed for call" << endl
<< " -C filename Filename/format to have classified sequences" << endl
<< " -U filename Filename/format to have unclassified sequences" << endl
<< " -O filename Output file for normal Kraken output" << endl
<< " -K In comb. w/ -R, provide minimizer information in report" << endl;
exit(exit_code);
}
kraken2-2.1.3/src/compact_hash.cc 0000664 0000000 0000000 00000020151 14437747201 0016611 0 ustar 00root root 0000000 0000000 /*
* Copyright 2013-2023, Derrick Wood
*
* This file is part of the Kraken 2 taxonomic sequence classification system.
*/
#include "compact_hash.h"
using std::string;
using std::ofstream;
using std::unordered_map;
namespace kraken2 {
CompactHashTable::CompactHashTable(size_t capacity, size_t key_bits, size_t value_bits)
: capacity_(capacity), size_(0), key_bits_(key_bits), value_bits_(value_bits),
file_backed_(false), locks_initialized_(true)
{
if (key_bits + value_bits != sizeof(*table_) * 8)
errx(EX_SOFTWARE, "sum of key bits and value bits must equal %u",
(unsigned int) (sizeof(*table_) * 8));
if (key_bits == 0)
errx(EX_SOFTWARE, "key bits cannot be zero");
if (value_bits == 0)
errx(EX_SOFTWARE, "value bits cannot be zero");
for (size_t i = 0; i < LOCK_ZONES; i++)
omp_init_lock(&zone_locks_[i]);
try {
table_ = new CompactHashCell[capacity_];
} catch (std::bad_alloc &ex) {
std::cerr << "Failed attempt to allocate " << (sizeof(*table_) * capacity_) << "bytes;\n"
<< "you may not have enough free memory to build this database.\n"
<< "Perhaps increasing the k-mer length, or reducing memory usage from\n"
<< "other programs could help you build this database?" << std::endl;
errx(EX_OSERR, "unable to allocate hash table memory");
}
memset(table_, 0, capacity_ * sizeof(*table_));
}
CompactHashTable::CompactHashTable(const string &filename, bool memory_mapping) {
LoadTable(filename.c_str(), memory_mapping);
}
CompactHashTable::CompactHashTable(const char *filename, bool memory_mapping) {
LoadTable(filename, memory_mapping);
}
CompactHashTable::~CompactHashTable() {
if (! file_backed_)
delete[] table_;
if (locks_initialized_)
for (size_t i = 0; i < LOCK_ZONES; i++)
omp_destroy_lock(&zone_locks_[i]);
}
void CompactHashTable::LoadTable(const char *filename, bool memory_mapping) {
locks_initialized_ = false;
if (memory_mapping) {
backing_file_.OpenFile(filename);
char *ptr = backing_file_.fptr();
memcpy((char *) &capacity_, ptr, sizeof(capacity_));
ptr += sizeof(capacity_);
memcpy((char *) &size_, ptr, sizeof(size_));
ptr += sizeof(size_);
memcpy((char *) &key_bits_, ptr, sizeof(key_bits_));
ptr += sizeof(key_bits_);
memcpy((char *) &value_bits_, ptr, sizeof(value_bits_));
ptr += sizeof(value_bits_);
table_ = (CompactHashCell *) ptr;
if (backing_file_.filesize() - (ptr - backing_file_.fptr()) !=
sizeof(*table_) * capacity_)
{
errx(EX_DATAERR, "Capacity mismatch in %s, aborting", filename);
}
file_backed_ = true;
}
else {
std::ifstream ifs(filename);
ifs.read((char *) &capacity_, sizeof(capacity_));
ifs.read((char *) &size_, sizeof(size_));
ifs.read((char *) &key_bits_, sizeof(key_bits_));
ifs.read((char *) &value_bits_, sizeof(value_bits_));
try {
table_ = new CompactHashCell[capacity_];
} catch (std::bad_alloc &ex) {
std::cerr << "Failed attempt to allocate " << (sizeof(*table_) * capacity_) << "bytes;\n"
<< "you may not have enough free memory to load this database.\n"
<< "If your computer has enough RAM, perhaps reducing memory usage from\n"
<< "other programs could help you load this database?" << std::endl;
errx(EX_OSERR, "unable to allocate hash table memory");
}
ifs.read((char *) table_, capacity_ * sizeof(*table_));
if (! ifs)
errx(EX_OSERR, "Error reading in hash table");
file_backed_ = false;
}
}
void CompactHashTable::WriteTable(const char *filename) {
ofstream ofs(filename, ofstream::binary);
ofs.write((char *) &capacity_, sizeof(capacity_));
ofs.write((char *) &size_, sizeof(size_));
ofs.write((char *) &key_bits_, sizeof(key_bits_));
ofs.write((char *) &value_bits_, sizeof(value_bits_));
ofs.write((char *) table_, sizeof(*table_) * capacity_);
ofs.close();
}
hvalue_t CompactHashTable::Get(hkey_t key) const {
uint64_t hc = MurmurHash3(key);
uint64_t compacted_key = hc >> (32 + value_bits_);
size_t idx = hc % capacity_;
size_t first_idx = idx;
size_t step = 0;
while (true) {
if (! table_[idx].value(value_bits_)) // value of 0 means data is 0, saves work
break; // search over, empty cell encountered in probe
if (table_[idx].hashed_key(value_bits_) == compacted_key)
return table_[idx].value(value_bits_);
if (step == 0)
step = second_hash(hc);
idx += step;
idx %= capacity_;
if (idx == first_idx)
break; // search over, we've exhausted the table
}
return 0;
}
bool CompactHashTable::FindIndex(hkey_t key, size_t *idx) const {
uint64_t hc = MurmurHash3(key);
uint64_t compacted_key = hc >> (32 + value_bits_);
*idx = hc % capacity_;
size_t first_idx = *idx;
size_t step = 0;
while (true) {
if (! table_[*idx].value(value_bits_)) // value of 0 means data is 0, saves work
return false; // search over, empty cell encountered in probe
if (table_[*idx].hashed_key(value_bits_) == compacted_key)
return true;
if (step == 0)
step = second_hash(hc);
*idx += step;
*idx %= capacity_;
if (*idx == first_idx)
break; // search over, we've exhausted the table
}
return false;
}
bool CompactHashTable::CompareAndSet
(hkey_t key, hvalue_t new_value, hvalue_t *old_value)
{
if (file_backed_)
return false;
if (new_value == 0)
return false;
uint64_t hc = MurmurHash3(key);
hkey_t compacted_key = hc >> (32 + value_bits_);
size_t idx, first_idx;
bool set_successful = false;
bool search_successful = false;
idx = first_idx = hc % capacity_;
size_t step = 0;
while (! search_successful) {
size_t zone = idx % LOCK_ZONES;
omp_set_lock(&zone_locks_[zone]);
if (! table_[idx].value(value_bits_)
|| table_[idx].hashed_key(value_bits_) == compacted_key)
{
search_successful = true;
#pragma omp flush
if (*old_value == table_[idx].value(value_bits_)) {
table_[idx].populate(compacted_key, new_value, key_bits_, value_bits_);
if (! *old_value) {
#pragma omp atomic
size_++;
}
set_successful = true;
}
else {
*old_value = table_[idx].value(value_bits_);
}
}
omp_unset_lock(&zone_locks_[zone]);
if (step == 0)
step = second_hash(hc);
idx += step;
idx %= capacity_;
if (idx == first_idx)
errx(EX_SOFTWARE, "compact hash table capacity exceeded");
}
return set_successful;
}
bool CompactHashTable::DirectCompareAndSet
(size_t idx, hkey_t key, hvalue_t new_value, hvalue_t *old_value)
{
uint64_t hc = MurmurHash3(key);
hkey_t compacted_key = hc >> (32 + value_bits_);
bool set_successful = false;
size_t zone = idx % LOCK_ZONES;
omp_set_lock(&zone_locks_[zone]);
#pragma omp flush
if (*old_value == table_[idx].value(value_bits_)) {
table_[idx].populate(compacted_key, new_value, key_bits_, value_bits_);
if (! *old_value) {
#pragma omp atomic
size_++;
}
set_successful = true;
}
else {
*old_value = table_[idx].value(value_bits_);
}
omp_unset_lock(&zone_locks_[zone]);
return set_successful;
}
// Linear probing may be ok for accuracy, as long as occupancy is < 95%
// Linear probing leads to more clustering, longer probing paths, and
// higher probability of a false answer
// Double hashing can have shorter probing paths, but less cache efficiency
inline uint64_t CompactHashTable::second_hash(uint64_t first_hash) const {
#ifdef LINEAR_PROBING
return 1;
#else // Double hashing
return (first_hash >> 8) | 1;
#endif
}
taxon_counts_t CompactHashTable::GetValueCounts() const {
taxon_counts_t value_counts;
int thread_ct = omp_get_max_threads();
taxon_counts_t thread_value_counts[thread_ct];
#pragma omp parallel for
for (size_t i = 0; i < capacity_; i++) {
auto val = table_[i].value(value_bits_);
if (val)
thread_value_counts[omp_get_thread_num()][val]++;
}
for (auto i = 0; i < thread_ct; i++) {
for (auto &kv_pair : thread_value_counts[i])
value_counts[kv_pair.first] += kv_pair.second;
}
return value_counts;
}
} // end namespace
kraken2-2.1.3/src/compact_hash.h 0000664 0000000 0000000 00000006554 14437747201 0016466 0 ustar 00root root 0000000 0000000 /*
* Copyright 2013-2023, Derrick Wood
*
* This file is part of the Kraken 2 taxonomic sequence classification system.
*/
#ifndef KRAKEN2_COMPACT_HASH_H_
#define KRAKEN2_COMPACT_HASH_H_
#include "kv_store.h"
#include "mmap_file.h"
#include "kraken2_headers.h"
#include "kraken2_data.h"
namespace kraken2 {
struct CompactHashCell {
inline hkey_t hashed_key(size_t value_bits) {
return (hkey_t) (data >> value_bits);
}
inline hvalue_t value(size_t value_bits) {
return (hvalue_t) (data & ((1 << value_bits) - 1));
}
void populate(hkey_t compacted_key, hvalue_t val, size_t key_bits, size_t value_bits) {
if (key_bits + value_bits != 32)
errx(EX_SOFTWARE, "key len of %u and value len of %u don't sum to 32",
(unsigned int) key_bits, (unsigned int) value_bits);
if (! key_bits || ! value_bits)
errx(EX_SOFTWARE, "key len and value len must be nonzero");
uint64_t max_value = (1llu << value_bits) - 1;
if (max_value < val)
errx(EX_SOFTWARE, "value len of %u too small for value of %llu",
(unsigned int) value_bits, (unsigned long long int) val);
data = compacted_key << value_bits;
data |= val;
}
// value in the low bits
// hash of the key in the high bits
uint32_t data;
};
/**
Implementation of a compact, fixed-size probabilistic hash table.
Operations supported:
- search
- compare-and-set
Does NOT support:
- values of 0
Keys must be 64-bit unsigned integers, < 2 ** 63.
Values must be 32-bit unsigned integers, > 0.
Cells are 32-bit unsigned integers, with truncated hashed keys stored in
the most significant bits, and truncated values stored in the least
significant bits. Compaction level is set when the table is created
by the WriteCompactedTable() method of the HashTable class.
**/
class CompactHashTable : public KeyValueStore {
public:
CompactHashTable(size_t capacity, size_t key_bits, size_t value_bits);
CompactHashTable(const std::string &filename, bool memory_mapping=false);
CompactHashTable(const char *filename, bool memory_mapping=false);
~CompactHashTable();
hvalue_t Get(hkey_t key) const;
bool FindIndex(hkey_t key, size_t *idx) const;
// How CompareAndSet works:
// if *old_value == CHT[key]
// CHT[key] = new_value
// return true
// else
// *old_value = CHT[key]
// return false
bool CompareAndSet(hkey_t key, hvalue_t new_value, hvalue_t *old_value);
bool DirectCompareAndSet(size_t idx, hkey_t key, hvalue_t new_value, hvalue_t *old_value);
void WriteTable(const char *filename);
taxon_counts_t GetValueCounts() const;
size_t capacity() const { return capacity_; }
size_t size() const { return size_; }
size_t key_bits() const { return key_bits_; }
size_t value_bits() const { return value_bits_; }
double occupancy() const { return size_ * 1.0 / capacity_; }
private:
static const size_t LOCK_ZONES = 256;
size_t capacity_;
size_t size_;
//size_t index_mask_;
size_t key_bits_;
size_t value_bits_;
CompactHashCell *table_;
bool file_backed_;
bool locks_initialized_;
MMapFile backing_file_;
omp_lock_t zone_locks_[LOCK_ZONES];
CompactHashTable(const CompactHashTable &rhs);
CompactHashTable& operator=(const CompactHashTable &rhs);
void LoadTable(const char *filename, bool memory_mapping);
uint64_t second_hash(uint64_t first_hash) const;
};
} // end namespace
#endif
kraken2-2.1.3/src/dump_table.cc 0000664 0000000 0000000 00000010541 14437747201 0016276 0 ustar 00root root 0000000 0000000 /*
* Copyright 2013-2023, Derrick Wood
*
* This file is part of the Kraken 2 taxonomic sequence classification system.
*/
#include "kraken2_headers.h"
#include "compact_hash.h"
#include "taxonomy.h"
#include "mmscanner.h"
#include "kraken2_data.h"
#include "reports.h"
using std::string;
using std::unordered_map;
using std::cout;
using std::cerr;
using std::endl;
using namespace kraken2;
struct Options {
string hashtable_filename;
string taxonomy_filename;
string options_filename;
string output_filename;
bool use_mpa_style;
bool report_zeros;
bool skip_counts;
unsigned int num_threads;
};
void ParseCommandLine(int argc, char **argv, Options &opts);
void usage(int exit_code = EX_USAGE);
std::string mask2str(uint64_t mask, int digits) {
std::string str;
for (int i = digits - 1; i >= 0; i--)
str.push_back((mask >> i) & 1 ? '1' : '0');
return str;
}
int main(int argc, char **argv) {
Options opts;
opts.report_zeros = false;
opts.output_filename = "/dev/fd/1";
opts.use_mpa_style = false;
opts.skip_counts = false;
opts.num_threads = 1;
ParseCommandLine(argc, argv, opts);
omp_set_num_threads(opts.num_threads);
CompactHashTable kraken_index(opts.hashtable_filename);
Taxonomy taxonomy(opts.taxonomy_filename);
IndexOptions idx_opts = {0};
std::ifstream idx_opt_fs(opts.options_filename);
idx_opt_fs.read((char *) &idx_opts, sizeof(idx_opts));
std::cout << "# Database options: "
<< (idx_opts.dna_db ? "nucleotide" : "protein") << " db, "
<< "k = " << idx_opts.k << ", "
<< "l = " << idx_opts.l << "\n"
<< "# Spaced mask = "
<< mask2str(idx_opts.spaced_seed_mask,
idx_opts.l * (idx_opts.dna_db ? BITS_PER_CHAR_DNA : BITS_PER_CHAR_PRO))
<< "\n"
<< "# Toggle mask = " << mask2str(idx_opts.toggle_mask, 64) << "\n"
<< "# Total taxonomy nodes: " << taxonomy.node_count() << "\n"
<< "# Table size: " << kraken_index.size() << "\n"
<< "# Table capacity: " << kraken_index.capacity() << "\n"
<< "# Min clear hash value = " << idx_opts.minimum_acceptable_hash_value << "\n";
if (idx_opts.revcom_version != CURRENT_REVCOM_VERSION)
std::cout << "# Built with outdated revcom version\n";
std::cout << std::flush;
if (! opts.skip_counts) {
taxon_counts_t taxid_counts = kraken_index.GetValueCounts();
uint64_t total_seqs = 0;
taxon_counters_t taxid_counters;
for (auto &kv_pair : taxid_counts) {
total_seqs += kv_pair.second;
READCOUNTER rc(kv_pair.second, 0);
taxid_counters[kv_pair.first] = rc;
}
if (opts.use_mpa_style)
ReportMpaStyle(opts.output_filename, opts.report_zeros, taxonomy, taxid_counters);
else
ReportKrakenStyle(opts.output_filename, opts.report_zeros, false, taxonomy,
taxid_counters, total_seqs, 0);
}
return 0;
}
void ParseCommandLine(int argc, char **argv, Options &opts) {
int opt;
while ((opt = getopt(argc, argv, "?hH:t:o:O:p:zms")) != -1) {
switch (opt) {
case 'h' : case '?' :
usage(0);
break;
case 'H' :
opts.hashtable_filename = optarg;
break;
case 't' :
opts.taxonomy_filename = optarg;
break;
case 'o' :
opts.options_filename = optarg;
break;
case 'z' :
opts.report_zeros = true;
break;
case 'O' :
opts.output_filename = optarg;
break;
case 'm' :
opts.use_mpa_style = true;
break;
case 's' :
opts.skip_counts = true;
break;
case 'p' :
opts.num_threads = atoi(optarg);
break;
}
}
if (opts.hashtable_filename.empty() || opts.taxonomy_filename.empty() ||
opts.options_filename.empty())
{
cerr << "missing mandatory filename parameter" << endl;
usage();
}
}
void usage(int exit_code) {
cerr << "Usage: dump_table \n"
<< "\n"
<< "Options (*mandatory):\n"
<< "* -H FILENAME Kraken 2 hash table filename\n"
<< "* -t FILENAME Kraken 2 taxonomy filename\n"
<< "* -o FILENAME Kraken 2 database options filename\n"
<< " -O FILENAME Output filename (def: /dev/fd/1)\n"
<< " -m Use MPA style output instead of Kraken 2 output\n"
<< " -s Skip reporting minimizer counts, just show DB stats\n"
<< " -z Report taxa with zero counts\n";
exit(exit_code);
}
kraken2-2.1.3/src/estimate_capacity.cc 0000664 0000000 0000000 00000012330 14437747201 0017650 0 ustar 00root root 0000000 0000000 /*
* Copyright 2013-2023, Derrick Wood
*
* This file is part of the Kraken 2 taxonomic sequence classification system.
*/
#include "kraken2_headers.h"
#include "kv_store.h"
#include "mmscanner.h"
#include "seqreader.h"
#include "utilities.h"
using std::string;
using std::cout;
using std::queue;
using std::cerr;
using std::endl;
using std::ifstream;
using std::ofstream;
using std::unordered_set;
using std::vector;
using namespace kraken2;
#define RANGE_SECTIONS 1024 // must be power of 2
#define RANGE_MASK (RANGE_SECTIONS - 1)
#define MAX_N RANGE_SECTIONS
#define DEFAULT_N 4
#define DEFAULT_BLOCK_SIZE (30 * 1024 * 1024) // yes, 30 MB
struct Options {
size_t k, l, n;
bool input_is_protein;
int threads;
size_t block_size;
uint64_t spaced_seed_mask;
uint64_t toggle_mask;
};
void ParseCommandLine(int argc, char **argv, Options &opts);
void usage(int exit_code = EX_USAGE);
void ProcessSequence(string &seq, Options &opts,
vector> &sets);
void ProcessSequences(Options &opts);
int main(int argc, char **argv) {
Options opts;
opts.k = 0;
opts.l = 0;
opts.n = DEFAULT_N;
opts.threads = 1;
opts.input_is_protein = false;
opts.spaced_seed_mask = DEFAULT_SPACED_SEED_MASK;
opts.toggle_mask = DEFAULT_TOGGLE_MASK;
opts.block_size = DEFAULT_BLOCK_SIZE;
ParseCommandLine(argc, argv, opts);
omp_set_num_threads(opts.threads);
ProcessSequences(opts);
return 0;
}
void ProcessSequences(Options &opts)
{
vector> sets(opts.n);
#pragma omp parallel
{
bool have_work = true;
BatchSequenceReader reader;
Sequence sequence;
while (have_work) {
#pragma omp critical(batch_reading)
have_work = reader.LoadBlock(std::cin, opts.block_size);
if (have_work)
while (reader.NextSequence(sequence))
ProcessSequence(sequence.seq, opts, sets);
}
}
size_t sum_set_sizes = 0;
for (auto &s : sets) {
sum_set_sizes += s.size();
}
sum_set_sizes++; // ensure non-zero estimate
cout << (size_t) (sum_set_sizes * RANGE_SECTIONS * 1.0 / opts.n) << endl;
}
void ParseCommandLine(int argc, char **argv, Options &opts) {
int opt;
long long sig;
while ((opt = getopt(argc, argv, "?hk:l:n:S:T:B:p:X")) != -1) {
switch (opt) {
case 'h' : case '?' :
usage(0);
break;
case 'p' :
sig = atoll(optarg);
if (sig < 1)
errx(EX_USAGE, "must have at least 1 thread");
opts.threads = sig;
break;
case 'B' :
sig = atoll(optarg);
if (sig < 1)
errx(EX_USAGE, "block size must be positive");
opts.block_size = sig;
break;
case 'n' :
sig = atoll(optarg);
if (sig < 1)
errx(EX_USAGE, "n must be positive integer");
if (sig > MAX_N)
errx(EX_USAGE, "n must be no more than %d", MAX_N);
opts.n = sig;
break;
case 'k' :
sig = atoll(optarg);
if (sig < 1)
errx(EX_USAGE, "k must be positive integer");
opts.k = sig;
break;
case 'l' :
sig = atoll(optarg);
if (sig < 1)
errx(EX_USAGE, "l must be positive integer");
if (sig > 31)
errx(EX_USAGE, "l must be no more than 31");
opts.l = sig;
break;
case 'X' :
opts.input_is_protein = true;
break;
case 'S' :
opts.spaced_seed_mask = strtol(optarg, nullptr, 2);
break;
case 'T' :
opts.toggle_mask = strtol(optarg, nullptr, 2);
break;
}
}
if (opts.spaced_seed_mask != DEFAULT_SPACED_SEED_MASK)
ExpandSpacedSeedMask(opts.spaced_seed_mask,
opts.input_is_protein ? BITS_PER_CHAR_PRO : BITS_PER_CHAR_DNA);
if (opts.k == 0 || opts.l == 0) {
cerr << "missing mandatory integer parameter" << endl;
usage();
}
if (opts.k < opts.l) {
cerr << "k cannot be less than l" << endl;
usage();
}
}
void usage(int exit_code) {
cerr << "Usage: estimate_capacity " << endl
<< endl
<< "Options (*mandatory):" << endl
<< "* -k INT Set length of k-mers" << endl
<< "* -l INT Set length of minimizers" << endl
<< " -n INT Set maximum qualifying hash code" << endl
<< " -X Input sequences are proteins" << endl
<< " -S BITSTRING Spaced seed mask" << endl
<< " -T BITSTRING Minimizer ordering toggle mask" << endl
<< " -B INT Read block size" << endl
<< " -p INT Number of threads" << endl;
exit(exit_code);
}
void ProcessSequence(string &seq, Options &opts,
vector> &sets)
{
MinimizerScanner scanner(opts.k, opts.l, opts.spaced_seed_mask, ! opts.input_is_protein, opts.toggle_mask);
// Add terminator for protein sequences if not already there
if (opts.input_is_protein && seq.back() != '*')
seq.push_back('*');
scanner.LoadSequence(seq);
uint64_t *minimizer_ptr;
while ((minimizer_ptr = scanner.NextMinimizer())) {
if (scanner.is_ambiguous())
continue;
uint64_t hash_code = MurmurHash3(*minimizer_ptr);
if ((hash_code & RANGE_MASK) < opts.n) {
#pragma omp critical(set_insert)
sets[hash_code & RANGE_MASK].insert(*minimizer_ptr);
}
}
}
kraken2-2.1.3/src/gzstream.h 0000664 0000000 0000000 00000007646 14437747201 0015674 0 ustar 00root root 0000000 0000000 #ifndef __GZSTREAM_H__
#define __GZSTREAM_H__
#include
#include
#include
#include
#include
#include
#include
#include
template>
class basic_gzbuf : public std::basic_streambuf {
typedef charT char_type;
typedef traits traits_type;
typedef typename traits::int_type int_type;
typedef typename traits::pos_type pos_type;
typedef typename traits::off_type off_type;
public:
basic_gzbuf(const char *filename);
basic_gzbuf(const std::vector *filenames);
void open(const char *filename);
void close();
virtual ~basic_gzbuf() {
close();
}
protected:
basic_gzbuf() {
delete[] buffer_;
}
virtual int_type underflow();
// virtual basic_gzbuf *setbuf(char_type *buf, std::streamsize size);
private:
char_type *buffer_;
std::streamsize size_;
const std::vector *filenames_;
size_t file_idx_;
static const int DEFAULT_BUFSIZ = 1024 * 8;
gzFile f;
};
template
basic_gzbuf::basic_gzbuf(const char *filename)
: buffer_(new char_type[DEFAULT_BUFSIZ]),
size_(DEFAULT_BUFSIZ),
filenames_(NULL)
{
this->open(filename);
this->setg(buffer_, buffer_ + size_, buffer_ + size_);
}
template
basic_gzbuf::basic_gzbuf(const std::vector *filenames)
: buffer_(new char_type[DEFAULT_BUFSIZ]),
size_(DEFAULT_BUFSIZ),
filenames_(filenames),
file_idx_(0)
{
assert(!filenames->empty());
this->open(filenames_->at(file_idx_).c_str());
file_idx_++;
this->setg(buffer_, buffer_ + size_, buffer_ + size_);
}
template
typename basic_gzbuf::int_type
basic_gzbuf::underflow() {
int read = gzread(f, buffer_, size_);
if (read == 0 && filenames_ != NULL && file_idx_ < filenames_->size()) {
close();
open(filenames_->at(file_idx_++).c_str());
read = gzread(f, buffer_, size_);
}
this->setg(buffer_, buffer_, buffer_ + read);
if (this->egptr() == this->gptr())
return traits::eof();
else
return traits::to_int_type(*this->gptr());
}
template
void basic_gzbuf::open(const char *filename) {
f = gzopen(filename, "rb");
if (f == NULL) {
std::cerr << "Unable to open \"" << filename << "\" for reading"
<< std::endl;
exit(1);
}
gzbuffer(f, DEFAULT_BUFSIZ);
}
template
void basic_gzbuf::close() {
if (f != NULL)
gzclose(f);
}
template>
class basic_gzistream : public std::basic_istream
{
public:
typedef charT char_type;
typedef traits traits_type;
typedef typename traits::int_type int_type;
typedef typename traits::pos_type pos_type;
typedef typename traits::off_type off_type;
basic_gzistream(const char *filename) : std::basic_istream(NULL),
sb(filename)
{
this->init(&sb);
}
basic_gzistream(const std::vector *filenames): std::basic_istream(NULL),
sb(filenames)
{
this->init(&sb);
}
private:
basic_gzbuf sb;
};
typedef basic_gzistream gzistream;
#endif
kraken2-2.1.3/src/hyperloglogplus-bias.h 0000664 0000000 0000000 00000215601 14437747201 0020203 0 ustar 00root root 0000000 0000000 /*
* hyperloglogbias.h
*
* Created on: Apr 25, 2015
* Author: fbreitwieser
*/
#ifndef HYPERLOGLOGBIAS_H_
#define HYPERLOGLOGBIAS_H_
// Bias data from Heule et al., 2015
// URL: https://docs.google.com/document/d/1gyjfMHy43U9OWBXxfaeG-3MjGzejW1dlpyMwEYAAWEI/view?fullscreen
// experimentally determined threshold values for p - 4
static const uint32_t threshold[] = {
10, // precision 4
20,
40,
80,
220,
400,
900,
1800,
3100,
6500,
11500,
20000,
50000,
120000,
350000 // precision 18
};
const std::vector rawEstimateData[] = {
// precision 4
{ 11, 11.717, 12.207, 12.7896, 13.2882, 13.8204, 14.3772, 14.9342, 15.5202, 16.161, 16.7722, 17.4636, 18.0396, 18.6766, 19.3566, 20.0454, 20.7936, 21.4856, 22.2666, 22.9946, 23.766, 24.4692, 25.3638, 26.0764, 26.7864, 27.7602, 28.4814, 29.433, 30.2926, 31.0664, 31.9996, 32.7956, 33.5366, 34.5894, 35.5738, 36.2698, 37.3682, 38.0544, 39.2342, 40.0108, 40.7966, 41.9298, 42.8704, 43.6358, 44.5194, 45.773, 46.6772, 47.6174, 48.4888, 49.3304, 50.2506, 51.4996, 52.3824, 53.3078, 54.3984, 55.5838, 56.6618, 57.2174, 58.3514, 59.0802, 60.1482, 61.0376, 62.3598, 62.8078, 63.9744, 64.914, 65.781, 67.1806, 68.0594, 68.8446, 69.7928, 70.8248, 71.8324, 72.8598, 73.6246, 74.7014, 75.393, 76.6708, 77.2394, },
// precision 5
{ 23, 23.1194, 23.8208, 24.2318, 24.77, 25.2436, 25.7774, 26.2848, 26.8224, 27.3742, 27.9336, 28.503, 29.0494, 29.6292, 30.2124, 30.798, 31.367, 31.9728, 32.5944, 33.217, 33.8438, 34.3696, 35.0956, 35.7044, 36.324, 37.0668, 37.6698, 38.3644, 39.049, 39.6918, 40.4146, 41.082, 41.687, 42.5398, 43.2462, 43.857, 44.6606, 45.4168, 46.1248, 46.9222, 47.6804, 48.447, 49.3454, 49.9594, 50.7636, 51.5776, 52.331, 53.19, 53.9676, 54.7564, 55.5314, 56.4442, 57.3708, 57.9774, 58.9624, 59.8796, 60.755, 61.472, 62.2076, 63.1024, 63.8908, 64.7338, 65.7728, 66.629, 67.413, 68.3266, 69.1524, 70.2642, 71.1806, 72.0566, 72.9192, 73.7598, 74.3516, 75.5802, 76.4386, 77.4916, 78.1524, 79.1892, 79.8414, 80.8798, 81.8376, 82.4698, 83.7656, 84.331, 85.5914, 86.6012, 87.7016, 88.5582, 89.3394, 90.3544, 91.4912, 92.308, 93.3552, 93.9746, 95.2052, 95.727, 97.1322, 98.3944, 98.7588, 100.242, 101.1914, 102.2538, 102.8776, 103.6292, 105.1932, 105.9152, 107.0868, 107.6728, 108.7144, 110.3114, 110.8716, 111.245, 112.7908, 113.7064, 114.636, 115.7464, 116.1788, 117.7464, 118.4896, 119.6166, 120.5082, 121.7798, 122.9028, 123.4426, 124.8854, 125.705, 126.4652, 128.3464, 128.3462, 130.0398, 131.0342, 131.0042, 132.4766, 133.511, 134.7252, 135.425, 136.5172, 138.0572, 138.6694, 139.3712, 140.8598, 141.4594, 142.554, 143.4006, 144.7374, 146.1634, 146.8994, 147.605, 147.9304, 149.1636, 150.2468, 151.5876, 152.2096, 153.7032, 154.7146, 155.807, 156.9228, 157.0372, 158.5852, },
// precision 6
{ 46, 46.1902, 47.271, 47.8358, 48.8142, 49.2854, 50.317, 51.354, 51.8924, 52.9436, 53.4596, 54.5262, 55.6248, 56.1574, 57.2822, 57.837, 58.9636, 60.074, 60.7042, 61.7976, 62.4772, 63.6564, 64.7942, 65.5004, 66.686, 67.291, 68.5672, 69.8556, 70.4982, 71.8204, 72.4252, 73.7744, 75.0786, 75.8344, 77.0294, 77.8098, 79.0794, 80.5732, 81.1878, 82.5648, 83.2902, 84.6784, 85.3352, 86.8946, 88.3712, 89.0852, 90.499, 91.2686, 92.6844, 94.2234, 94.9732, 96.3356, 97.2286, 98.7262, 100.3284, 101.1048, 102.5962, 103.3562, 105.1272, 106.4184, 107.4974, 109.0822, 109.856, 111.48, 113.2834, 114.0208, 115.637, 116.5174, 118.0576, 119.7476, 120.427, 122.1326, 123.2372, 125.2788, 126.6776, 127.7926, 129.1952, 129.9564, 131.6454, 133.87, 134.5428, 136.2, 137.0294, 138.6278, 139.6782, 141.792, 143.3516, 144.2832, 146.0394, 147.0748, 148.4912, 150.849, 151.696, 153.5404, 154.073, 156.3714, 157.7216, 158.7328, 160.4208, 161.4184, 163.9424, 165.2772, 166.411, 168.1308, 168.769, 170.9258, 172.6828, 173.7502, 175.706, 176.3886, 179.0186, 180.4518, 181.927, 183.4172, 184.4114, 186.033, 188.5124, 189.5564, 191.6008, 192.4172, 193.8044, 194.997, 197.4548, 198.8948, 200.2346, 202.3086, 203.1548, 204.8842, 206.6508, 206.6772, 209.7254, 210.4752, 212.7228, 214.6614, 215.1676, 217.793, 218.0006, 219.9052, 221.66, 223.5588, 225.1636, 225.6882, 227.7126, 229.4502, 231.1978, 232.9756, 233.1654, 236.727, 238.1974, 237.7474, 241.1346, 242.3048, 244.1948, 245.3134, 246.879, 249.1204, 249.853, 252.6792, 253.857, 254.4486, 257.2362, 257.9534, 260.0286, 260.5632, 262.663, 264.723, 265.7566, 267.2566, 267.1624, 270.62, 272.8216, 273.2166, 275.2056, 276.2202, 278.3726, 280.3344, 281.9284, 283.9728, 284.1924, 286.4872, 287.587, 289.807, 291.1206, 292.769, 294.8708, 296.665, 297.1182, 299.4012, 300.6352, 302.1354, 304.1756, 306.1606, 307.3462, 308.5214, 309.4134, 310.8352, 313.9684, 315.837, 316.7796, 318.9858, },
// precision 7
{ 92, 93.4934, 94.9758, 96.4574, 97.9718, 99.4954, 101.5302, 103.0756, 104.6374, 106.1782, 107.7888, 109.9522, 111.592, 113.2532, 114.9086, 116.5938, 118.9474, 120.6796, 122.4394, 124.2176, 125.9768, 128.4214, 130.2528, 132.0102, 133.8658, 135.7278, 138.3044, 140.1316, 142.093, 144.0032, 145.9092, 148.6306, 150.5294, 152.5756, 154.6508, 156.662, 159.552, 161.3724, 163.617, 165.5754, 167.7872, 169.8444, 172.7988, 174.8606, 177.2118, 179.3566, 181.4476, 184.5882, 186.6816, 189.0824, 191.0258, 193.6048, 196.4436, 198.7274, 200.957, 203.147, 205.4364, 208.7592, 211.3386, 213.781, 215.8028, 218.656, 221.6544, 223.996, 226.4718, 229.1544, 231.6098, 234.5956, 237.0616, 239.5758, 242.4878, 244.5244, 248.2146, 250.724, 252.8722, 255.5198, 258.0414, 261.941, 264.9048, 266.87, 269.4304, 272.028, 274.4708, 278.37, 281.0624, 283.4668, 286.5532, 289.4352, 293.2564, 295.2744, 298.2118, 300.7472, 304.1456, 307.2928, 309.7504, 312.5528, 315.979, 318.2102, 322.1834, 324.3494, 327.325, 330.6614, 332.903, 337.2544, 339.9042, 343.215, 345.2864, 348.0814, 352.6764, 355.301, 357.139, 360.658, 363.1732, 366.5902, 369.9538, 373.0828, 375.922, 378.9902, 382.7328, 386.4538, 388.1136, 391.2234, 394.0878, 396.708, 401.1556, 404.1852, 406.6372, 409.6822, 412.7796, 416.6078, 418.4916, 422.131, 424.5376, 428.1988, 432.211, 434.4502, 438.5282, 440.912, 444.0448, 447.7432, 450.8524, 453.7988, 456.7858, 458.8868, 463.9886, 466.5064, 468.9124, 472.6616, 475.4682, 478.582, 481.304, 485.2738, 488.6894, 490.329, 496.106, 497.6908, 501.1374, 504.5322, 506.8848, 510.3324, 513.4512, 516.179, 520.4412, 522.6066, 526.167, 528.7794, 533.379, 536.067, 538.46, 542.9116, 545.692, 547.9546, 552.493, 555.2722, 557.335, 562.449, 564.2014, 569.0738, 571.0974, 574.8564, 578.2996, 581.409, 583.9704, 585.8098, 589.6528, 594.5998, 595.958, 600.068, 603.3278, 608.2016, 609.9632, 612.864, 615.43, 620.7794, 621.272, 625.8644, 629.206, 633.219, 634.5154, 638.6102, },
// precision 8
{ 184.2152, 187.2454, 190.2096, 193.6652, 196.6312, 199.6822, 203.249, 206.3296, 210.0038, 213.2074, 216.4612, 220.27, 223.5178, 227.4412, 230.8032, 234.1634, 238.1688, 241.6074, 245.6946, 249.2664, 252.8228, 257.0432, 260.6824, 264.9464, 268.6268, 272.2626, 276.8376, 280.4034, 284.8956, 288.8522, 292.7638, 297.3552, 301.3556, 305.7526, 309.9292, 313.8954, 318.8198, 322.7668, 327.298, 331.6688, 335.9466, 340.9746, 345.1672, 349.3474, 354.3028, 358.8912, 364.114, 368.4646, 372.9744, 378.4092, 382.6022, 387.843, 392.5684, 397.1652, 402.5426, 407.4152, 412.5388, 417.3592, 422.1366, 427.486, 432.3918, 437.5076, 442.509, 447.3834, 453.3498, 458.0668, 463.7346, 469.1228, 473.4528, 479.7, 484.644, 491.0518, 495.5774, 500.9068, 506.432, 512.1666, 517.434, 522.6644, 527.4894, 533.6312, 538.3804, 544.292, 550.5496, 556.0234, 562.8206, 566.6146, 572.4188, 579.117, 583.6762, 590.6576, 595.7864, 601.509, 607.5334, 612.9204, 619.772, 624.2924, 630.8654, 636.1836, 642.745, 649.1316, 655.0386, 660.0136, 666.6342, 671.6196, 678.1866, 684.4282, 689.3324, 695.4794, 702.5038, 708.129, 713.528, 720.3204, 726.463, 732.7928, 739.123, 744.7418, 751.2192, 756.5102, 762.6066, 769.0184, 775.2224, 781.4014, 787.7618, 794.1436, 798.6506, 805.6378, 811.766, 819.7514, 824.5776, 828.7322, 837.8048, 843.6302, 849.9336, 854.4798, 861.3388, 867.9894, 873.8196, 880.3136, 886.2308, 892.4588, 899.0816, 905.4076, 912.0064, 917.3878, 923.619, 929.998, 937.3482, 943.9506, 947.991, 955.1144, 962.203, 968.8222, 975.7324, 981.7826, 988.7666, 994.2648, 1000.3128, 1007.4082, 1013.7536, 1020.3376, 1026.7156, 1031.7478, 1037.4292, 1045.393, 1051.2278, 1058.3434, 1062.8726, 1071.884, 1076.806, 1082.9176, 1089.1678, 1095.5032, 1102.525, 1107.2264, 1115.315, 1120.93, 1127.252, 1134.1496, 1139.0408, 1147.5448, 1153.3296, 1158.1974, 1166.5262, 1174.3328, 1175.657, 1184.4222, 1190.9172, 1197.1292, 1204.4606, 1210.4578, 1218.8728, 1225.3336, 1226.6592, 1236.5768, 1241.363, 1249.4074, 1254.6566, 1260.8014, 1266.5454, 1274.5192, },
// precision 9
{ 369, 374.8294, 381.2452, 387.6698, 394.1464, 400.2024, 406.8782, 413.6598, 420.462, 427.2826, 433.7102, 440.7416, 447.9366, 455.1046, 462.285, 469.0668, 476.306, 483.8448, 491.301, 498.9886, 506.2422, 513.8138, 521.7074, 529.7428, 537.8402, 545.1664, 553.3534, 561.594, 569.6886, 577.7876, 585.65, 594.228, 602.8036, 611.1666, 620.0818, 628.0824, 637.2574, 646.302, 655.1644, 664.0056, 672.3802, 681.7192, 690.5234, 700.2084, 708.831, 718.485, 728.1112, 737.4764, 746.76, 756.3368, 766.5538, 775.5058, 785.2646, 795.5902, 804.3818, 814.8998, 824.9532, 835.2062, 845.2798, 854.4728, 864.9582, 875.3292, 886.171, 896.781, 906.5716, 916.7048, 927.5322, 937.875, 949.3972, 958.3464, 969.7274, 980.2834, 992.1444, 1003.4264, 1013.0166, 1024.018, 1035.0438, 1046.34, 1057.6856, 1068.9836, 1079.0312, 1091.677, 1102.3188, 1113.4846, 1124.4424, 1135.739, 1147.1488, 1158.9202, 1169.406, 1181.5342, 1193.2834, 1203.8954, 1216.3286, 1226.2146, 1239.6684, 1251.9946, 1262.123, 1275.4338, 1285.7378, 1296.076, 1308.9692, 1320.4964, 1333.0998, 1343.9864, 1357.7754, 1368.3208, 1380.4838, 1392.7388, 1406.0758, 1416.9098, 1428.9728, 1440.9228, 1453.9292, 1462.617, 1476.05, 1490.2996, 1500.6128, 1513.7392, 1524.5174, 1536.6322, 1548.2584, 1562.3766, 1572.423, 1587.1232, 1596.5164, 1610.5938, 1622.5972, 1633.1222, 1647.7674, 1658.5044, 1671.57, 1683.7044, 1695.4142, 1708.7102, 1720.6094, 1732.6522, 1747.841, 1756.4072, 1769.9786, 1782.3276, 1797.5216, 1808.3186, 1819.0694, 1834.354, 1844.575, 1856.2808, 1871.1288, 1880.7852, 1893.9622, 1906.3418, 1920.6548, 1932.9302, 1945.8584, 1955.473, 1968.8248, 1980.6446, 1995.9598, 2008.349, 2019.8556, 2033.0334, 2044.0206, 2059.3956, 2069.9174, 2082.6084, 2093.7036, 2106.6108, 2118.9124, 2132.301, 2144.7628, 2159.8422, 2171.0212, 2183.101, 2193.5112, 2208.052, 2221.3194, 2233.3282, 2247.295, 2257.7222, 2273.342, 2286.5638, 2299.6786, 2310.8114, 2322.3312, 2335.516, 2349.874, 2363.5968, 2373.865, 2387.1918, 2401.8328, 2414.8496, 2424.544, 2436.7592, 2447.1682, 2464.1958, 2474.3438, 2489.0006, 2497.4526, 2513.6586, 2527.19, 2540.7028, 2553.768, },
// precision 10
{ 738.1256, 750.4234, 763.1064, 775.4732, 788.4636, 801.0644, 814.488, 827.9654, 841.0832, 854.7864, 868.1992, 882.2176, 896.5228, 910.1716, 924.7752, 938.899, 953.6126, 968.6492, 982.9474, 998.5214, 1013.1064, 1028.6364, 1044.2468, 1059.4588, 1075.3832, 1091.0584, 1106.8606, 1123.3868, 1139.5062, 1156.1862, 1172.463, 1189.339, 1206.1936, 1223.1292, 1240.1854, 1257.2908, 1275.3324, 1292.8518, 1310.5204, 1328.4854, 1345.9318, 1364.552, 1381.4658, 1400.4256, 1419.849, 1438.152, 1456.8956, 1474.8792, 1494.118, 1513.62, 1532.5132, 1551.9322, 1570.7726, 1590.6086, 1610.5332, 1630.5918, 1650.4294, 1669.7662, 1690.4106, 1710.7338, 1730.9012, 1750.4486, 1770.1556, 1791.6338, 1812.7312, 1833.6264, 1853.9526, 1874.8742, 1896.8326, 1918.1966, 1939.5594, 1961.07, 1983.037, 2003.1804, 2026.071, 2047.4884, 2070.0848, 2091.2944, 2114.333, 2135.9626, 2158.2902, 2181.0814, 2202.0334, 2224.4832, 2246.39, 2269.7202, 2292.1714, 2314.2358, 2338.9346, 2360.891, 2384.0264, 2408.3834, 2430.1544, 2454.8684, 2476.9896, 2501.4368, 2522.8702, 2548.0408, 2570.6738, 2593.5208, 2617.0158, 2640.2302, 2664.0962, 2687.4986, 2714.2588, 2735.3914, 2759.6244, 2781.8378, 2808.0072, 2830.6516, 2856.2454, 2877.2136, 2903.4546, 2926.785, 2951.2294, 2976.468, 3000.867, 3023.6508, 3049.91, 3073.5984, 3098.162, 3121.5564, 3146.2328, 3170.9484, 3195.5902, 3221.3346, 3242.7032, 3271.6112, 3296.5546, 3317.7376, 3345.072, 3369.9518, 3394.326, 3418.1818, 3444.6926, 3469.086, 3494.2754, 3517.8698, 3544.248, 3565.3768, 3588.7234, 3616.979, 3643.7504, 3668.6812, 3695.72, 3719.7392, 3742.6224, 3770.4456, 3795.6602, 3819.9058, 3844.002, 3869.517, 3895.6824, 3920.8622, 3947.1364, 3973.985, 3995.4772, 4021.62, 4046.628, 4074.65, 4096.2256, 4121.831, 4146.6406, 4173.276, 4195.0744, 4223.9696, 4251.3708, 4272.9966, 4300.8046, 4326.302, 4353.1248, 4374.312, 4403.0322, 4426.819, 4450.0598, 4478.5206, 4504.8116, 4528.8928, 4553.9584, 4578.8712, 4603.8384, 4632.3872, 4655.5128, 4675.821, 4704.6222, 4731.9862, 4755.4174, 4781.2628, 4804.332, 4832.3048, 4862.8752, 4883.4148, 4906.9544, 4935.3516, 4954.3532, 4984.0248, 5011.217, 5035.3258, 5057.3672, 5084.1828, },
// precision 11
{ 1477, 1501.6014, 1526.5802, 1551.7942, 1577.3042, 1603.2062, 1629.8402, 1656.2292, 1682.9462, 1709.9926, 1737.3026, 1765.4252, 1793.0578, 1821.6092, 1849.626, 1878.5568, 1908.527, 1937.5154, 1967.1874, 1997.3878, 2027.37, 2058.1972, 2089.5728, 2120.1012, 2151.9668, 2183.292, 2216.0772, 2247.8578, 2280.6562, 2313.041, 2345.714, 2380.3112, 2414.1806, 2447.9854, 2481.656, 2516.346, 2551.5154, 2586.8378, 2621.7448, 2656.6722, 2693.5722, 2729.1462, 2765.4124, 2802.8728, 2838.898, 2876.408, 2913.4926, 2951.4938, 2989.6776, 3026.282, 3065.7704, 3104.1012, 3143.7388, 3181.6876, 3221.1872, 3261.5048, 3300.0214, 3339.806, 3381.409, 3421.4144, 3461.4294, 3502.2286, 3544.651, 3586.6156, 3627.337, 3670.083, 3711.1538, 3753.5094, 3797.01, 3838.6686, 3882.1678, 3922.8116, 3967.9978, 4009.9204, 4054.3286, 4097.5706, 4140.6014, 4185.544, 4229.5976, 4274.583, 4316.9438, 4361.672, 4406.2786, 4451.8628, 4496.1834, 4543.505, 4589.1816, 4632.5188, 4678.2294, 4724.8908, 4769.0194, 4817.052, 4861.4588, 4910.1596, 4956.4344, 5002.5238, 5048.13, 5093.6374, 5142.8162, 5187.7894, 5237.3984, 5285.6078, 5331.0858, 5379.1036, 5428.6258, 5474.6018, 5522.7618, 5571.5822, 5618.59, 5667.9992, 5714.88, 5763.454, 5808.6982, 5860.3644, 5910.2914, 5953.571, 6005.9232, 6055.1914, 6104.5882, 6154.5702, 6199.7036, 6251.1764, 6298.7596, 6350.0302, 6398.061, 6448.4694, 6495.933, 6548.0474, 6597.7166, 6646.9416, 6695.9208, 6742.6328, 6793.5276, 6842.1934, 6894.2372, 6945.3864, 6996.9228, 7044.2372, 7094.1374, 7142.2272, 7192.2942, 7238.8338, 7288.9006, 7344.0908, 7394.8544, 7443.5176, 7490.4148, 7542.9314, 7595.6738, 7641.9878, 7694.3688, 7743.0448, 7797.522, 7845.53, 7899.594, 7950.3132, 7996.455, 8050.9442, 8092.9114, 8153.1374, 8197.4472, 8252.8278, 8301.8728, 8348.6776, 8401.4698, 8453.551, 8504.6598, 8553.8944, 8604.1276, 8657.6514, 8710.3062, 8758.908, 8807.8706, 8862.1702, 8910.4668, 8960.77, 9007.2766, 9063.164, 9121.0534, 9164.1354, 9218.1594, 9267.767, 9319.0594, 9372.155, 9419.7126, 9474.3722, 9520.1338, 9572.368, 9622.7702, 9675.8448, 9726.5396, 9778.7378, 9827.6554, 9878.1922, 9928.7782, 9978.3984, 10026.578, 10076.5626, 10137.1618, 10177.5244, 10229.9176, },
// precision 12
{ 2954, 3003.4782, 3053.3568, 3104.3666, 3155.324, 3206.9598, 3259.648, 3312.539, 3366.1474, 3420.2576, 3474.8376, 3530.6076, 3586.451, 3643.38, 3700.4104, 3757.5638, 3815.9676, 3875.193, 3934.838, 3994.8548, 4055.018, 4117.1742, 4178.4482, 4241.1294, 4304.4776, 4367.4044, 4431.8724, 4496.3732, 4561.4304, 4627.5326, 4693.949, 4761.5532, 4828.7256, 4897.6182, 4965.5186, 5034.4528, 5104.865, 5174.7164, 5244.6828, 5316.6708, 5387.8312, 5459.9036, 5532.476, 5604.8652, 5679.6718, 5753.757, 5830.2072, 5905.2828, 5980.0434, 6056.6264, 6134.3192, 6211.5746, 6290.0816, 6367.1176, 6447.9796, 6526.5576, 6606.1858, 6686.9144, 6766.1142, 6847.0818, 6927.9664, 7010.9096, 7091.0816, 7175.3962, 7260.3454, 7344.018, 7426.4214, 7511.3106, 7596.0686, 7679.8094, 7765.818, 7852.4248, 7936.834, 8022.363, 8109.5066, 8200.4554, 8288.5832, 8373.366, 8463.4808, 8549.7682, 8642.0522, 8728.3288, 8820.9528, 8907.727, 9001.0794, 9091.2522, 9179.988, 9269.852, 9362.6394, 9453.642, 9546.9024, 9640.6616, 9732.6622, 9824.3254, 9917.7484, 10007.9392, 10106.7508, 10196.2152, 10289.8114, 10383.5494, 10482.3064, 10576.8734, 10668.7872, 10764.7156, 10862.0196, 10952.793, 11049.9748, 11146.0702, 11241.4492, 11339.2772, 11434.2336, 11530.741, 11627.6136, 11726.311, 11821.5964, 11918.837, 12015.3724, 12113.0162, 12213.0424, 12306.9804, 12408.4518, 12504.8968, 12604.586, 12700.9332, 12798.705, 12898.5142, 12997.0488, 13094.788, 13198.475, 13292.7764, 13392.9698, 13486.8574, 13590.1616, 13686.5838, 13783.6264, 13887.2638, 13992.0978, 14081.0844, 14189.9956, 14280.0912, 14382.4956, 14486.4384, 14588.1082, 14686.2392, 14782.276, 14888.0284, 14985.1864, 15088.8596, 15187.0998, 15285.027, 15383.6694, 15495.8266, 15591.3736, 15694.2008, 15790.3246, 15898.4116, 15997.4522, 16095.5014, 16198.8514, 16291.7492, 16402.6424, 16499.1266, 16606.2436, 16697.7186, 16796.3946, 16902.3376, 17005.7672, 17100.814, 17206.8282, 17305.8262, 17416.0744, 17508.4092, 17617.0178, 17715.4554, 17816.758, 17920.1748, 18012.9236, 18119.7984, 18223.2248, 18324.2482, 18426.6276, 18525.0932, 18629.8976, 18733.2588, 18831.0466, 18940.1366, 19032.2696, 19131.729, 19243.4864, 19349.6932, 19442.866, 19547.9448, 19653.2798, 19754.4034, 19854.0692, 19965.1224, 20065.1774, 20158.2212, 20253.353, 20366.3264, 20463.22, },
// precision 13
{ 5908.5052, 6007.2672, 6107.347, 6208.5794, 6311.2622, 6414.5514, 6519.3376, 6625.6952, 6732.5988, 6841.3552, 6950.5972, 7061.3082, 7173.5646, 7287.109, 7401.8216, 7516.4344, 7633.3802, 7751.2962, 7870.3784, 7990.292, 8110.79, 8233.4574, 8356.6036, 8482.2712, 8607.7708, 8735.099, 8863.1858, 8993.4746, 9123.8496, 9255.6794, 9388.5448, 9522.7516, 9657.3106, 9792.6094, 9930.5642, 10068.794, 10206.7256, 10347.81, 10490.3196, 10632.0778, 10775.9916, 10920.4662, 11066.124, 11213.073, 11358.0362, 11508.1006, 11659.1716, 11808.7514, 11959.4884, 12112.1314, 12265.037, 12420.3756, 12578.933, 12734.311, 12890.0006, 13047.2144, 13207.3096, 13368.5144, 13528.024, 13689.847, 13852.7528, 14018.3168, 14180.5372, 14346.9668, 14513.5074, 14677.867, 14846.2186, 15017.4186, 15184.9716, 15356.339, 15529.2972, 15697.3578, 15871.8686, 16042.187, 16216.4094, 16389.4188, 16565.9126, 16742.3272, 16919.0042, 17094.7592, 17273.965, 17451.8342, 17634.4254, 17810.5984, 17988.9242, 18171.051, 18354.7938, 18539.466, 18721.0408, 18904.9972, 19081.867, 19271.9118, 19451.8694, 19637.9816, 19821.2922, 20013.1292, 20199.3858, 20387.8726, 20572.9514, 20770.7764, 20955.1714, 21144.751, 21329.9952, 21520.709, 21712.7016, 21906.3868, 22096.2626, 22286.0524, 22475.051, 22665.5098, 22862.8492, 23055.5294, 23249.6138, 23437.848, 23636.273, 23826.093, 24020.3296, 24213.3896, 24411.7392, 24602.9614, 24805.7952, 24998.1552, 25193.9588, 25389.0166, 25585.8392, 25780.6976, 25981.2728, 26175.977, 26376.5252, 26570.1964, 26773.387, 26962.9812, 27163.0586, 27368.164, 27565.0534, 27758.7428, 27961.1276, 28163.2324, 28362.3816, 28565.7668, 28758.644, 28956.9768, 29163.4722, 29354.7026, 29561.1186, 29767.9948, 29959.9986, 30164.0492, 30366.9818, 30562.5338, 30762.9928, 30976.1592, 31166.274, 31376.722, 31570.3734, 31770.809, 31974.8934, 32179.5286, 32387.5442, 32582.3504, 32794.076, 32989.9528, 33191.842, 33392.4684, 33595.659, 33801.8672, 34000.3414, 34200.0922, 34402.6792, 34610.0638, 34804.0084, 35011.13, 35218.669, 35418.6634, 35619.0792, 35830.6534, 36028.4966, 36229.7902, 36438.6422, 36630.7764, 36833.3102, 37048.6728, 37247.3916, 37453.5904, 37669.3614, 37854.5526, 38059.305, 38268.0936, 38470.2516, 38674.7064, 38876.167, 39068.3794, 39281.9144, 39492.8566, 39684.8628, 39898.4108, 40093.1836, 40297.6858, 40489.7086, 40717.2424, },
// precision 14
{ 11817.475, 12015.0046, 12215.3792, 12417.7504, 12623.1814, 12830.0086, 13040.0072, 13252.503, 13466.178, 13683.2738, 13902.0344, 14123.9798, 14347.394, 14573.7784, 14802.6894, 15033.6824, 15266.9134, 15502.8624, 15741.4944, 15980.7956, 16223.8916, 16468.6316, 16715.733, 16965.5726, 17217.204, 17470.666, 17727.8516, 17986.7886, 18247.6902, 18510.9632, 18775.304, 19044.7486, 19314.4408, 19587.202, 19862.2576, 20135.924, 20417.0324, 20697.9788, 20979.6112, 21265.0274, 21550.723, 21841.6906, 22132.162, 22428.1406, 22722.127, 23020.5606, 23319.7394, 23620.4014, 23925.2728, 24226.9224, 24535.581, 24845.505, 25155.9618, 25470.3828, 25785.9702, 26103.7764, 26420.4132, 26742.0186, 27062.8852, 27388.415, 27714.6024, 28042.296, 28365.4494, 28701.1526, 29031.8008, 29364.2156, 29704.497, 30037.1458, 30380.111, 30723.8168, 31059.5114, 31404.9498, 31751.6752, 32095.2686, 32444.7792, 32794.767, 33145.204, 33498.4226, 33847.6502, 34209.006, 34560.849, 34919.4838, 35274.9778, 35635.1322, 35996.3266, 36359.1394, 36722.8266, 37082.8516, 37447.7354, 37815.9606, 38191.0692, 38559.4106, 38924.8112, 39294.6726, 39663.973, 40042.261, 40416.2036, 40779.2036, 41161.6436, 41540.9014, 41921.1998, 42294.7698, 42678.5264, 43061.3464, 43432.375, 43818.432, 44198.6598, 44583.0138, 44970.4794, 45353.924, 45729.858, 46118.2224, 46511.5724, 46900.7386, 47280.6964, 47668.1472, 48055.6796, 48446.9436, 48838.7146, 49217.7296, 49613.7796, 50010.7508, 50410.0208, 50793.7886, 51190.2456, 51583.1882, 51971.0796, 52376.5338, 52763.319, 53165.5534, 53556.5594, 53948.2702, 54346.352, 54748.7914, 55138.577, 55543.4824, 55941.1748, 56333.7746, 56745.1552, 57142.7944, 57545.2236, 57935.9956, 58348.5268, 58737.5474, 59158.5962, 59542.6896, 59958.8004, 60349.3788, 60755.0212, 61147.6144, 61548.194, 61946.0696, 62348.6042, 62763.603, 63162.781, 63560.635, 63974.3482, 64366.4908, 64771.5876, 65176.7346, 65597.3916, 65995.915, 66394.0384, 66822.9396, 67203.6336, 67612.2032, 68019.0078, 68420.0388, 68821.22, 69235.8388, 69640.0724, 70055.155, 70466.357, 70863.4266, 71276.2482, 71677.0306, 72080.2006, 72493.0214, 72893.5952, 73314.5856, 73714.9852, 74125.3022, 74521.2122, 74933.6814, 75341.5904, 75743.0244, 76166.0278, 76572.1322, 76973.1028, 77381.6284, 77800.6092, 78189.328, 78607.0962, 79012.2508, 79407.8358, 79825.725, 80238.701, 80646.891, 81035.6436, 81460.0448, 81876.3884, },
// precision 15
{ 23635.0036, 24030.8034, 24431.4744, 24837.1524, 25246.7928, 25661.326, 26081.3532, 26505.2806, 26933.9892, 27367.7098, 27805.318, 28248.799, 28696.4382, 29148.8244, 29605.5138, 30066.8668, 30534.2344, 31006.32, 31480.778, 31962.2418, 32447.3324, 32938.0232, 33432.731, 33930.728, 34433.9896, 34944.1402, 35457.5588, 35974.5958, 36497.3296, 37021.9096, 37554.326, 38088.0826, 38628.8816, 39171.3192, 39723.2326, 40274.5554, 40832.3142, 41390.613, 41959.5908, 42532.5466, 43102.0344, 43683.5072, 44266.694, 44851.2822, 45440.7862, 46038.0586, 46640.3164, 47241.064, 47846.155, 48454.7396, 49076.9168, 49692.542, 50317.4778, 50939.65, 51572.5596, 52210.2906, 52843.7396, 53481.3996, 54127.236, 54770.406, 55422.6598, 56078.7958, 56736.7174, 57397.6784, 58064.5784, 58730.308, 59404.9784, 60077.0864, 60751.9158, 61444.1386, 62115.817, 62808.7742, 63501.4774, 64187.5454, 64883.6622, 65582.7468, 66274.5318, 66976.9276, 67688.7764, 68402.138, 69109.6274, 69822.9706, 70543.6108, 71265.5202, 71983.3848, 72708.4656, 73433.384, 74158.4664, 74896.4868, 75620.9564, 76362.1434, 77098.3204, 77835.7662, 78582.6114, 79323.9902, 80067.8658, 80814.9246, 81567.0136, 82310.8536, 83061.9952, 83821.4096, 84580.8608, 85335.547, 86092.5802, 86851.6506, 87612.311, 88381.2016, 89146.3296, 89907.8974, 90676.846, 91451.4152, 92224.5518, 92995.8686, 93763.5066, 94551.2796, 95315.1944, 96096.1806, 96881.0918, 97665.679, 98442.68, 99229.3002, 100011.0994, 100790.6386, 101580.1564, 102377.7484, 103152.1392, 103944.2712, 104730.216, 105528.6336, 106324.9398, 107117.6706, 107890.3988, 108695.2266, 109485.238, 110294.7876, 111075.0958, 111878.0496, 112695.2864, 113464.5486, 114270.0474, 115068.608, 115884.3626, 116673.2588, 117483.3716, 118275.097, 119085.4092, 119879.2808, 120687.5868, 121499.9944, 122284.916, 123095.9254, 123912.5038, 124709.0454, 125503.7182, 126323.259, 127138.9412, 127943.8294, 128755.646, 129556.5354, 130375.3298, 131161.4734, 131971.1962, 132787.5458, 133588.1056, 134431.351, 135220.2906, 136023.398, 136846.6558, 137667.0004, 138463.663, 139283.7154, 140074.6146, 140901.3072, 141721.8548, 142543.2322, 143356.1096, 144173.7412, 144973.0948, 145794.3162, 146609.5714, 147420.003, 148237.9784, 149050.5696, 149854.761, 150663.1966, 151494.0754, 152313.1416, 153112.6902, 153935.7206, 154746.9262, 155559.547, 156401.9746, 157228.7036, 158008.7254, 158820.75, 159646.9184, 160470.4458, 161279.5348, 162093.3114, 162918.542, 163729.2842, },
// precision 16
{ 47271, 48062.3584, 48862.7074, 49673.152, 50492.8416, 51322.9514, 52161.03, 53009.407, 53867.6348, 54734.206, 55610.5144, 56496.2096, 57390.795, 58297.268, 59210.6448, 60134.665, 61068.0248, 62010.4472, 62962.5204, 63923.5742, 64895.0194, 65876.4182, 66862.6136, 67862.6968, 68868.8908, 69882.8544, 70911.271, 71944.0924, 72990.0326, 74040.692, 75100.6336, 76174.7826, 77252.5998, 78340.2974, 79438.2572, 80545.4976, 81657.2796, 82784.6336, 83915.515, 85059.7362, 86205.9368, 87364.4424, 88530.3358, 89707.3744, 90885.9638, 92080.197, 93275.5738, 94479.391, 95695.918, 96919.2236, 98148.4602, 99382.3474, 100625.6974, 101878.0284, 103141.6278, 104409.4588, 105686.2882, 106967.5402, 108261.6032, 109548.1578, 110852.0728, 112162.231, 113479.0072, 114806.2626, 116137.9072, 117469.5048, 118813.5186, 120165.4876, 121516.2556, 122875.766, 124250.5444, 125621.2222, 127003.2352, 128387.848, 129775.2644, 131181.7776, 132577.3086, 133979.9458, 135394.1132, 136800.9078, 138233.217, 139668.5308, 141085.212, 142535.2122, 143969.0684, 145420.2872, 146878.1542, 148332.7572, 149800.3202, 151269.66, 152743.6104, 154213.0948, 155690.288, 157169.4246, 158672.1756, 160160.059, 161650.6854, 163145.7772, 164645.6726, 166159.1952, 167682.1578, 169177.3328, 170700.0118, 172228.8964, 173732.6664, 175265.5556, 176787.799, 178317.111, 179856.6914, 181400.865, 182943.4612, 184486.742, 186033.4698, 187583.7886, 189148.1868, 190688.4526, 192250.1926, 193810.9042, 195354.2972, 196938.7682, 198493.5898, 200079.2824, 201618.912, 203205.5492, 204765.5798, 206356.1124, 207929.3064, 209498.7196, 211086.229, 212675.1324, 214256.7892, 215826.2392, 217412.8474, 218995.6724, 220618.6038, 222207.1166, 223781.0364, 225387.4332, 227005.7928, 228590.4336, 230217.8738, 231805.1054, 233408.9, 234995.3432, 236601.4956, 238190.7904, 239817.2548, 241411.2832, 243002.4066, 244640.1884, 246255.3128, 247849.3508, 249479.9734, 251106.8822, 252705.027, 254332.9242, 255935.129, 257526.9014, 259154.772, 260777.625, 262390.253, 264004.4906, 265643.59, 267255.4076, 268873.426, 270470.7252, 272106.4804, 273722.4456, 275337.794, 276945.7038, 278592.9154, 280204.3726, 281841.1606, 283489.171, 285130.1716, 286735.3362, 288364.7164, 289961.1814, 291595.5524, 293285.683, 294899.6668, 296499.3434, 298128.0462, 299761.8946, 301394.2424, 302997.6748, 304615.1478, 306269.7724, 307886.114, 309543.1028, 311153.2862, 312782.8546, 314421.2008, 316033.2438, 317692.9636, 319305.2648, 320948.7406, 322566.3364, 324228.4224, 325847.1542, },
// precision 17
{ 94542, 96125.811, 97728.019, 99348.558, 100987.9705, 102646.7565, 104324.5125, 106021.7435, 107736.7865, 109469.272, 111223.9465, 112995.219, 114787.432, 116593.152, 118422.71, 120267.2345, 122134.6765, 124020.937, 125927.2705, 127851.255, 129788.9485, 131751.016, 133726.8225, 135722.592, 137736.789, 139770.568, 141821.518, 143891.343, 145982.1415, 148095.387, 150207.526, 152355.649, 154515.6415, 156696.05, 158887.7575, 161098.159, 163329.852, 165569.053, 167837.4005, 170121.6165, 172420.4595, 174732.6265, 177062.77, 179412.502, 181774.035, 184151.939, 186551.6895, 188965.691, 191402.8095, 193857.949, 196305.0775, 198774.6715, 201271.2585, 203764.78, 206299.3695, 208818.1365, 211373.115, 213946.7465, 216532.076, 219105.541, 221714.5375, 224337.5135, 226977.5125, 229613.0655, 232270.2685, 234952.2065, 237645.3555, 240331.1925, 243034.517, 245756.0725, 248517.6865, 251232.737, 254011.3955, 256785.995, 259556.44, 262368.335, 265156.911, 267965.266, 270785.583, 273616.0495, 276487.4835, 279346.639, 282202.509, 285074.3885, 287942.2855, 290856.018, 293774.0345, 296678.5145, 299603.6355, 302552.6575, 305492.9785, 308466.8605, 311392.581, 314347.538, 317319.4295, 320285.9785, 323301.7325, 326298.3235, 329301.3105, 332301.987, 335309.791, 338370.762, 341382.923, 344431.1265, 347464.1545, 350507.28, 353619.2345, 356631.2005, 359685.203, 362776.7845, 365886.488, 368958.2255, 372060.6825, 375165.4335, 378237.935, 381328.311, 384430.5225, 387576.425, 390683.242, 393839.648, 396977.8425, 400101.9805, 403271.296, 406409.8425, 409529.5485, 412678.7, 415847.423, 419020.8035, 422157.081, 425337.749, 428479.6165, 431700.902, 434893.1915, 438049.582, 441210.5415, 444379.2545, 447577.356, 450741.931, 453959.548, 457137.0935, 460329.846, 463537.4815, 466732.3345, 469960.5615, 473164.681, 476347.6345, 479496.173, 482813.1645, 486025.6995, 489249.4885, 492460.1945, 495675.8805, 498908.0075, 502131.802, 505374.3855, 508550.9915, 511806.7305, 515026.776, 518217.0005, 521523.9855, 524705.9855, 527950.997, 531210.0265, 534472.497, 537750.7315, 540926.922, 544207.094, 547429.4345, 550666.3745, 553975.3475, 557150.7185, 560399.6165, 563662.697, 566916.7395, 570146.1215, 573447.425, 576689.6245, 579874.5745, 583202.337, 586503.0255, 589715.635, 592910.161, 596214.3885, 599488.035, 602740.92, 605983.0685, 609248.67, 612491.3605, 615787.912, 619107.5245, 622307.9555, 625577.333, 628840.4385, 632085.2155, 635317.6135, 638691.7195, 641887.467, 645139.9405, 648441.546, 651666.252, 654941.845, },
// precision 18
{ 189084, 192250.913, 195456.774, 198696.946, 201977.762, 205294.444, 208651.754, 212042.099, 215472.269, 218941.91, 222443.912, 225996.845, 229568.199, 233193.568, 236844.457, 240543.233, 244279.475, 248044.27, 251854.588, 255693.2, 259583.619, 263494.621, 267445.385, 271454.061, 275468.769, 279549.456, 283646.446, 287788.198, 291966.099, 296181.164, 300431.469, 304718.618, 309024.004, 313393.508, 317760.803, 322209.731, 326675.061, 331160.627, 335654.47, 340241.442, 344841.833, 349467.132, 354130.629, 358819.432, 363574.626, 368296.587, 373118.482, 377914.93, 382782.301, 387680.669, 392601.981, 397544.323, 402529.115, 407546.018, 412593.658, 417638.657, 422762.865, 427886.169, 433017.167, 438213.273, 443441.254, 448692.421, 453937.533, 459239.049, 464529.569, 469910.083, 475274.03, 480684.473, 486070.26, 491515.237, 496995.651, 502476.617, 507973.609, 513497.19, 519083.233, 524726.509, 530305.505, 535945.728, 541584.404, 547274.055, 552967.236, 558667.862, 564360.216, 570128.148, 575965.08, 581701.952, 587532.523, 593361.144, 599246.128, 605033.418, 610958.779, 616837.117, 622772.818, 628672.04, 634675.369, 640574.831, 646585.739, 652574.547, 658611.217, 664642.684, 670713.914, 676737.681, 682797.313, 688837.897, 694917.874, 701009.882, 707173.648, 713257.254, 719415.392, 725636.761, 731710.697, 737906.209, 744103.074, 750313.39, 756504.185, 762712.579, 768876.985, 775167.859, 781359, 787615.959, 793863.597, 800245.477, 806464.582, 812785.294, 819005.925, 825403.057, 831676.197, 837936.284, 844266.968, 850642.711, 856959.756, 863322.774, 869699.931, 876102.478, 882355.787, 888694.463, 895159.952, 901536.143, 907872.631, 914293.672, 920615.14, 927130.974, 933409.404, 939922.178, 946331.47, 952745.93, 959209.264, 965590.224, 972077.284, 978501.961, 984953.19, 991413.271, 997817.479, 1004222.658, 1010725.676, 1017177.138, 1023612.529, 1030098.236, 1036493.719, 1043112.207, 1049537.036, 1056008.096, 1062476.184, 1068942.337, 1075524.95, 1081932.864, 1088426.025, 1094776.005, 1101327.448, 1107901.673, 1114423.639, 1120884.602, 1127324.923, 1133794.24, 1140328.886, 1146849.376, 1153346.682, 1159836.502, 1166478.703, 1172953.304, 1179391.502, 1185950.982, 1192544.052, 1198913.41, 1205430.994, 1212015.525, 1218674.042, 1225121.683, 1231551.101, 1238126.379, 1244673.795, 1251260.649, 1257697.86, 1264320.983, 1270736.319, 1277274.694, 1283804.95, 1290211.514, 1296858.568, 1303455.691, }
};
const std::vector biasData[] = {
// precision 4
{ 10, 9.717, 9.207, 8.7896, 8.2882, 7.8204, 7.3772, 6.9342, 6.5202, 6.161, 5.7722, 5.4636, 5.0396, 4.6766, 4.3566, 4.0454, 3.7936, 3.4856, 3.2666, 2.9946, 2.766, 2.4692, 2.3638, 2.0764, 1.7864, 1.7602, 1.4814, 1.433, 1.2926, 1.0664, 0.999600000000001, 0.7956, 0.5366, 0.589399999999998, 0.573799999999999, 0.269799999999996, 0.368200000000002, 0.0544000000000011, 0.234200000000001, 0.0108000000000033, -0.203400000000002, -0.0701999999999998, -0.129600000000003, -0.364199999999997, -0.480600000000003, -0.226999999999997, -0.322800000000001, -0.382599999999996, -0.511200000000002, -0.669600000000003, -0.749400000000001, -0.500399999999999, -0.617600000000003, -0.6922, -0.601599999999998, -0.416200000000003, -0.338200000000001, -0.782600000000002, -0.648600000000002, -0.919800000000002, -0.851799999999997, -0.962400000000002, -0.6402, -1.1922, -1.0256, -1.086, -1.21899999999999, -0.819400000000002, -0.940600000000003, -1.1554, -1.2072, -1.1752, -1.16759999999999, -1.14019999999999, -1.3754, -1.29859999999999, -1.607, -1.3292, -1.7606, },
// precision 5
{ 22, 21.1194, 20.8208, 20.2318, 19.77, 19.2436, 18.7774, 18.2848, 17.8224, 17.3742, 16.9336, 16.503, 16.0494, 15.6292, 15.2124, 14.798, 14.367, 13.9728, 13.5944, 13.217, 12.8438, 12.3696, 12.0956, 11.7044, 11.324, 11.0668, 10.6698, 10.3644, 10.049, 9.6918, 9.4146, 9.082, 8.687, 8.5398, 8.2462, 7.857, 7.6606, 7.4168, 7.1248, 6.9222, 6.6804, 6.447, 6.3454, 5.9594, 5.7636, 5.5776, 5.331, 5.19, 4.9676, 4.7564, 4.5314, 4.4442, 4.3708, 3.9774, 3.9624, 3.8796, 3.755, 3.472, 3.2076, 3.1024, 2.8908, 2.7338, 2.7728, 2.629, 2.413, 2.3266, 2.1524, 2.2642, 2.1806, 2.0566, 1.9192, 1.7598, 1.3516, 1.5802, 1.43859999999999, 1.49160000000001, 1.1524, 1.1892, 0.841399999999993, 0.879800000000003, 0.837599999999995, 0.469800000000006, 0.765600000000006, 0.331000000000003, 0.591399999999993, 0.601200000000006, 0.701599999999999, 0.558199999999999, 0.339399999999998, 0.354399999999998, 0.491200000000006, 0.308000000000007, 0.355199999999996, -0.0254000000000048, 0.205200000000005, -0.272999999999996, 0.132199999999997, 0.394400000000005, -0.241200000000006, 0.242000000000004, 0.191400000000002, 0.253799999999998, -0.122399999999999, -0.370800000000003, 0.193200000000004, -0.0848000000000013, 0.0867999999999967, -0.327200000000005, -0.285600000000002, 0.311400000000006, -0.128399999999999, -0.754999999999995, -0.209199999999996, -0.293599999999998, -0.364000000000004, -0.253600000000006, -0.821200000000005, -0.253600000000006, -0.510400000000004, -0.383399999999995, -0.491799999999998, -0.220200000000006, -0.0972000000000008, -0.557400000000001, -0.114599999999996, -0.295000000000002, -0.534800000000004, 0.346399999999988, -0.65379999999999, 0.0398000000000138, 0.0341999999999985, -0.995800000000003, -0.523400000000009, -0.489000000000004, -0.274799999999999, -0.574999999999989, -0.482799999999997, 0.0571999999999946, -0.330600000000004, -0.628800000000012, -0.140199999999993, -0.540600000000012, -0.445999999999998, -0.599400000000003, -0.262599999999992, 0.163399999999996, -0.100599999999986, -0.39500000000001, -1.06960000000001, -0.836399999999998, -0.753199999999993, -0.412399999999991, -0.790400000000005, -0.29679999999999, -0.28540000000001, -0.193000000000012, -0.0772000000000048, -0.962799999999987, -0.414800000000014, },
// precision 6
{ 45, 44.1902, 43.271, 42.8358, 41.8142, 41.2854, 40.317, 39.354, 38.8924, 37.9436, 37.4596, 36.5262, 35.6248, 35.1574, 34.2822, 33.837, 32.9636, 32.074, 31.7042, 30.7976, 30.4772, 29.6564, 28.7942, 28.5004, 27.686, 27.291, 26.5672, 25.8556, 25.4982, 24.8204, 24.4252, 23.7744, 23.0786, 22.8344, 22.0294, 21.8098, 21.0794, 20.5732, 20.1878, 19.5648, 19.2902, 18.6784, 18.3352, 17.8946, 17.3712, 17.0852, 16.499, 16.2686, 15.6844, 15.2234, 14.9732, 14.3356, 14.2286, 13.7262, 13.3284, 13.1048, 12.5962, 12.3562, 12.1272, 11.4184, 11.4974, 11.0822, 10.856, 10.48, 10.2834, 10.0208, 9.637, 9.51739999999999, 9.05759999999999, 8.74760000000001, 8.42700000000001, 8.1326, 8.2372, 8.2788, 7.6776, 7.79259999999999, 7.1952, 6.9564, 6.6454, 6.87, 6.5428, 6.19999999999999, 6.02940000000001, 5.62780000000001, 5.6782, 5.792, 5.35159999999999, 5.28319999999999, 5.0394, 5.07480000000001, 4.49119999999999, 4.84899999999999, 4.696, 4.54040000000001, 4.07300000000001, 4.37139999999999, 3.7216, 3.7328, 3.42080000000001, 3.41839999999999, 3.94239999999999, 3.27719999999999, 3.411, 3.13079999999999, 2.76900000000001, 2.92580000000001, 2.68279999999999, 2.75020000000001, 2.70599999999999, 2.3886, 3.01859999999999, 2.45179999999999, 2.92699999999999, 2.41720000000001, 2.41139999999999, 2.03299999999999, 2.51240000000001, 2.5564, 2.60079999999999, 2.41720000000001, 1.80439999999999, 1.99700000000001, 2.45480000000001, 1.8948, 2.2346, 2.30860000000001, 2.15479999999999, 1.88419999999999, 1.6508, 0.677199999999999, 1.72540000000001, 1.4752, 1.72280000000001, 1.66139999999999, 1.16759999999999, 1.79300000000001, 1.00059999999999, 0.905200000000008, 0.659999999999997, 1.55879999999999, 1.1636, 0.688199999999995, 0.712600000000009, 0.450199999999995, 1.1978, 0.975599999999986, 0.165400000000005, 1.727, 1.19739999999999, -0.252600000000001, 1.13460000000001, 1.3048, 1.19479999999999, 0.313400000000001, 0.878999999999991, 1.12039999999999, 0.853000000000009, 1.67920000000001, 0.856999999999999, 0.448599999999999, 1.2362, 0.953399999999988, 1.02859999999998, 0.563199999999995, 0.663000000000011, 0.723000000000013, 0.756599999999992, 0.256599999999992, -0.837600000000009, 0.620000000000005, 0.821599999999989, 0.216600000000028, 0.205600000000004, 0.220199999999977, 0.372599999999977, 0.334400000000016, 0.928400000000011, 0.972800000000007, 0.192400000000021, 0.487199999999973, -0.413000000000011, 0.807000000000016, 0.120600000000024, 0.769000000000005, 0.870799999999974, 0.66500000000002, 0.118200000000002, 0.401200000000017, 0.635199999999998, 0.135400000000004, 0.175599999999974, 1.16059999999999, 0.34620000000001, 0.521400000000028, -0.586599999999976, -1.16480000000001, 0.968399999999974, 0.836999999999989, 0.779600000000016, 0.985799999999983, },
// precision 7
{ 91, 89.4934, 87.9758, 86.4574, 84.9718, 83.4954, 81.5302, 80.0756, 78.6374, 77.1782, 75.7888, 73.9522, 72.592, 71.2532, 69.9086, 68.5938, 66.9474, 65.6796, 64.4394, 63.2176, 61.9768, 60.4214, 59.2528, 58.0102, 56.8658, 55.7278, 54.3044, 53.1316, 52.093, 51.0032, 49.9092, 48.6306, 47.5294, 46.5756, 45.6508, 44.662, 43.552, 42.3724, 41.617, 40.5754, 39.7872, 38.8444, 37.7988, 36.8606, 36.2118, 35.3566, 34.4476, 33.5882, 32.6816, 32.0824, 31.0258, 30.6048, 29.4436, 28.7274, 27.957, 27.147, 26.4364, 25.7592, 25.3386, 24.781, 23.8028, 23.656, 22.6544, 21.996, 21.4718, 21.1544, 20.6098, 19.5956, 19.0616, 18.5758, 18.4878, 17.5244, 17.2146, 16.724, 15.8722, 15.5198, 15.0414, 14.941, 14.9048, 13.87, 13.4304, 13.028, 12.4708, 12.37, 12.0624, 11.4668, 11.5532, 11.4352, 11.2564, 10.2744, 10.2118, 9.74720000000002, 10.1456, 9.2928, 8.75040000000001, 8.55279999999999, 8.97899999999998, 8.21019999999999, 8.18340000000001, 7.3494, 7.32499999999999, 7.66140000000001, 6.90300000000002, 7.25439999999998, 6.9042, 7.21499999999997, 6.28640000000001, 6.08139999999997, 6.6764, 6.30099999999999, 5.13900000000001, 5.65800000000002, 5.17320000000001, 4.59019999999998, 4.9538, 5.08280000000002, 4.92200000000003, 4.99020000000002, 4.7328, 5.4538, 4.11360000000002, 4.22340000000003, 4.08780000000002, 3.70800000000003, 4.15559999999999, 4.18520000000001, 3.63720000000001, 3.68220000000002, 3.77960000000002, 3.6078, 2.49160000000001, 3.13099999999997, 2.5376, 3.19880000000001, 3.21100000000001, 2.4502, 3.52820000000003, 2.91199999999998, 3.04480000000001, 2.7432, 2.85239999999999, 2.79880000000003, 2.78579999999999, 1.88679999999999, 2.98860000000002, 2.50639999999999, 1.91239999999999, 2.66160000000002, 2.46820000000002, 1.58199999999999, 1.30399999999997, 2.27379999999999, 2.68939999999998, 1.32900000000001, 3.10599999999999, 1.69080000000002, 2.13740000000001, 2.53219999999999, 1.88479999999998, 1.33240000000001, 1.45119999999997, 1.17899999999997, 2.44119999999998, 1.60659999999996, 2.16700000000003, 0.77940000000001, 2.37900000000002, 2.06700000000001, 1.46000000000004, 2.91160000000002, 1.69200000000001, 0.954600000000028, 2.49300000000005, 2.2722, 1.33500000000004, 2.44899999999996, 1.20140000000004, 3.07380000000001, 2.09739999999999, 2.85640000000001, 2.29960000000005, 2.40899999999999, 1.97040000000004, 0.809799999999996, 1.65279999999996, 2.59979999999996, 0.95799999999997, 2.06799999999998, 2.32780000000002, 4.20159999999998, 1.96320000000003, 1.86400000000003, 1.42999999999995, 3.77940000000001, 1.27200000000005, 1.86440000000005, 2.20600000000002, 3.21900000000005, 1.5154, 2.61019999999996, },
// precision 8
{ 183.2152, 180.2454, 177.2096, 173.6652, 170.6312, 167.6822, 164.249, 161.3296, 158.0038, 155.2074, 152.4612, 149.27, 146.5178, 143.4412, 140.8032, 138.1634, 135.1688, 132.6074, 129.6946, 127.2664, 124.8228, 122.0432, 119.6824, 116.9464, 114.6268, 112.2626, 109.8376, 107.4034, 104.8956, 102.8522, 100.7638, 98.3552, 96.3556, 93.7526, 91.9292, 89.8954, 87.8198, 85.7668, 83.298, 81.6688, 79.9466, 77.9746, 76.1672, 74.3474, 72.3028, 70.8912, 69.114, 67.4646, 65.9744, 64.4092, 62.6022, 60.843, 59.5684, 58.1652, 56.5426, 55.4152, 53.5388, 52.3592, 51.1366, 49.486, 48.3918, 46.5076, 45.509, 44.3834, 43.3498, 42.0668, 40.7346, 40.1228, 38.4528, 37.7, 36.644, 36.0518, 34.5774, 33.9068, 32.432, 32.1666, 30.434, 29.6644, 28.4894, 27.6312, 26.3804, 26.292, 25.5496000000001, 25.0234, 24.8206, 22.6146, 22.4188, 22.117, 20.6762, 20.6576, 19.7864, 19.509, 18.5334, 17.9204, 17.772, 16.2924, 16.8654, 15.1836, 15.745, 15.1316, 15.0386, 14.0136, 13.6342, 12.6196, 12.1866, 12.4281999999999, 11.3324, 10.4794000000001, 11.5038, 10.129, 9.52800000000002, 10.3203999999999, 9.46299999999997, 9.79280000000006, 9.12300000000005, 8.74180000000001, 9.2192, 7.51020000000005, 7.60659999999996, 7.01840000000004, 7.22239999999999, 7.40139999999997, 6.76179999999999, 7.14359999999999, 5.65060000000005, 5.63779999999997, 5.76599999999996, 6.75139999999999, 5.57759999999996, 3.73220000000003, 5.8048, 5.63019999999995, 4.93359999999996, 3.47979999999995, 4.33879999999999, 3.98940000000005, 3.81960000000004, 3.31359999999995, 3.23080000000004, 3.4588, 3.08159999999998, 3.4076, 3.00639999999999, 2.38779999999997, 2.61900000000003, 1.99800000000005, 3.34820000000002, 2.95060000000001, 0.990999999999985, 2.11440000000005, 2.20299999999997, 2.82219999999995, 2.73239999999998, 2.7826, 3.76660000000004, 2.26480000000004, 2.31280000000004, 2.40819999999997, 2.75360000000001, 3.33759999999995, 2.71559999999999, 1.7478000000001, 1.42920000000004, 2.39300000000003, 2.22779999999989, 2.34339999999997, 0.87259999999992, 3.88400000000001, 1.80600000000004, 1.91759999999999, 1.16779999999994, 1.50320000000011, 2.52500000000009, 0.226400000000012, 2.31500000000005, 0.930000000000064, 1.25199999999995, 2.14959999999996, 0.0407999999999902, 2.5447999999999, 1.32960000000003, 0.197400000000016, 2.52620000000002, 3.33279999999991, -1.34300000000007, 0.422199999999975, 0.917200000000093, 1.12920000000008, 1.46060000000011, 1.45779999999991, 2.8728000000001, 3.33359999999993, -1.34079999999994, 1.57680000000005, 0.363000000000056, 1.40740000000005, 0.656600000000026, 0.801400000000058, -0.454600000000028, 1.51919999999996, },
// precision 9
{ 368, 361.8294, 355.2452, 348.6698, 342.1464, 336.2024, 329.8782, 323.6598, 317.462, 311.2826, 305.7102, 299.7416, 293.9366, 288.1046, 282.285, 277.0668, 271.306, 265.8448, 260.301, 254.9886, 250.2422, 244.8138, 239.7074, 234.7428, 229.8402, 225.1664, 220.3534, 215.594, 210.6886, 205.7876, 201.65, 197.228, 192.8036, 188.1666, 184.0818, 180.0824, 176.2574, 172.302, 168.1644, 164.0056, 160.3802, 156.7192, 152.5234, 149.2084, 145.831, 142.485, 139.1112, 135.4764, 131.76, 129.3368, 126.5538, 122.5058, 119.2646, 116.5902, 113.3818, 110.8998, 107.9532, 105.2062, 102.2798, 99.4728, 96.9582, 94.3292, 92.171, 89.7809999999999, 87.5716, 84.7048, 82.5322, 79.875, 78.3972, 75.3464, 73.7274, 71.2834, 70.1444, 68.4263999999999, 66.0166, 64.018, 62.0437999999999, 60.3399999999999, 58.6856, 57.9836, 55.0311999999999, 54.6769999999999, 52.3188, 51.4846, 49.4423999999999, 47.739, 46.1487999999999, 44.9202, 43.4059999999999, 42.5342000000001, 41.2834, 38.8954000000001, 38.3286000000001, 36.2146, 36.6684, 35.9946, 33.123, 33.4338, 31.7378000000001, 29.076, 28.9692, 27.4964, 27.0998, 25.9864, 26.7754, 24.3208, 23.4838, 22.7388000000001, 24.0758000000001, 21.9097999999999, 20.9728, 19.9228000000001, 19.9292, 16.617, 17.05, 18.2996000000001, 15.6128000000001, 15.7392, 14.5174, 13.6322, 12.2583999999999, 13.3766000000001, 11.423, 13.1232, 9.51639999999998, 10.5938000000001, 9.59719999999993, 8.12220000000002, 9.76739999999995, 7.50440000000003, 7.56999999999994, 6.70440000000008, 6.41419999999994, 6.71019999999999, 5.60940000000005, 4.65219999999999, 6.84099999999989, 3.4072000000001, 3.97859999999991, 3.32760000000007, 5.52160000000003, 3.31860000000006, 2.06940000000009, 4.35400000000004, 1.57500000000005, 0.280799999999999, 2.12879999999996, -0.214799999999968, -0.0378000000000611, -0.658200000000079, 0.654800000000023, -0.0697999999999865, 0.858400000000074, -2.52700000000004, -2.1751999999999, -3.35539999999992, -1.04019999999991, -0.651000000000067, -2.14439999999991, -1.96659999999997, -3.97939999999994, -0.604400000000169, -3.08260000000018, -3.39159999999993, -5.29640000000018, -5.38920000000007, -5.08759999999984, -4.69900000000007, -5.23720000000003, -3.15779999999995, -4.97879999999986, -4.89899999999989, -7.48880000000008, -5.94799999999987, -5.68060000000014, -6.67180000000008, -4.70499999999993, -7.27779999999984, -4.6579999999999, -4.4362000000001, -4.32139999999981, -5.18859999999995, -6.66879999999992, -6.48399999999992, -5.1260000000002, -4.4032000000002, -6.13500000000022, -5.80819999999994, -4.16719999999987, -4.15039999999999, -7.45600000000013, -7.24080000000004, -9.83179999999993, -5.80420000000004, -8.6561999999999, -6.99940000000015, -10.5473999999999, -7.34139999999979, -6.80999999999995, -6.29719999999998, -6.23199999999997, },
// precision 10
{ 737.1256, 724.4234, 711.1064, 698.4732, 685.4636, 673.0644, 660.488, 647.9654, 636.0832, 623.7864, 612.1992, 600.2176, 588.5228, 577.1716, 565.7752, 554.899, 543.6126, 532.6492, 521.9474, 511.5214, 501.1064, 490.6364, 480.2468, 470.4588, 460.3832, 451.0584, 440.8606, 431.3868, 422.5062, 413.1862, 404.463, 395.339, 386.1936, 378.1292, 369.1854, 361.2908, 353.3324, 344.8518, 337.5204, 329.4854, 321.9318, 314.552, 306.4658, 299.4256, 292.849, 286.152, 278.8956, 271.8792, 265.118, 258.62, 252.5132, 245.9322, 239.7726, 233.6086, 227.5332, 222.5918, 216.4294, 210.7662, 205.4106, 199.7338, 194.9012, 188.4486, 183.1556, 178.6338, 173.7312, 169.6264, 163.9526, 159.8742, 155.8326, 151.1966, 147.5594, 143.07, 140.037, 134.1804, 131.071, 127.4884, 124.0848, 120.2944, 117.333, 112.9626, 110.2902, 107.0814, 103.0334, 99.4832000000001, 96.3899999999999, 93.7202000000002, 90.1714000000002, 87.2357999999999, 85.9346, 82.8910000000001, 80.0264000000002, 78.3834000000002, 75.1543999999999, 73.8683999999998, 70.9895999999999, 69.4367999999999, 64.8701999999998, 65.0408000000002, 61.6738, 59.5207999999998, 57.0158000000001, 54.2302, 53.0962, 50.4985999999999, 52.2588000000001, 47.3914, 45.6244000000002, 42.8377999999998, 43.0072, 40.6516000000001, 40.2453999999998, 35.2136, 36.4546, 33.7849999999999, 33.2294000000002, 32.4679999999998, 30.8670000000002, 28.6507999999999, 28.9099999999999, 27.5983999999999, 26.1619999999998, 24.5563999999999, 23.2328000000002, 21.9484000000002, 21.5902000000001, 21.3346000000001, 17.7031999999999, 20.6111999999998, 19.5545999999999, 15.7375999999999, 17.0720000000001, 16.9517999999998, 15.326, 13.1817999999998, 14.6925999999999, 13.0859999999998, 13.2754, 10.8697999999999, 11.248, 7.3768, 4.72339999999986, 7.97899999999981, 8.7503999999999, 7.68119999999999, 9.7199999999998, 7.73919999999998, 5.6224000000002, 7.44560000000001, 6.6601999999998, 5.9058, 4.00199999999995, 4.51699999999983, 4.68240000000014, 3.86220000000003, 5.13639999999987, 5.98500000000013, 2.47719999999981, 2.61999999999989, 1.62800000000016, 4.65000000000009, 0.225599999999758, 0.831000000000131, -0.359400000000278, 1.27599999999984, -2.92559999999958, -0.0303999999996449, 2.37079999999969, -2.0033999999996, 0.804600000000391, 0.30199999999968, 1.1247999999996, -2.6880000000001, 0.0321999999996478, -1.18099999999959, -3.9402, -1.47940000000017, -0.188400000000001, -2.10720000000038, -2.04159999999956, -3.12880000000041, -4.16160000000036, -0.612799999999879, -3.48719999999958, -8.17900000000009, -5.37780000000021, -4.01379999999972, -5.58259999999973, -5.73719999999958, -7.66799999999967, -5.69520000000011, -1.1247999999996, -5.58520000000044, -8.04560000000038, -4.64840000000004, -11.6468000000004, -7.97519999999986, -5.78300000000036, -7.67420000000038, -10.6328000000003, -9.81720000000041, },
// precision 11
{ 1476, 1449.6014, 1423.5802, 1397.7942, 1372.3042, 1347.2062, 1321.8402, 1297.2292, 1272.9462, 1248.9926, 1225.3026, 1201.4252, 1178.0578, 1155.6092, 1132.626, 1110.5568, 1088.527, 1066.5154, 1045.1874, 1024.3878, 1003.37, 982.1972, 962.5728, 942.1012, 922.9668, 903.292, 884.0772, 864.8578, 846.6562, 828.041, 809.714, 792.3112, 775.1806, 757.9854, 740.656, 724.346, 707.5154, 691.8378, 675.7448, 659.6722, 645.5722, 630.1462, 614.4124, 600.8728, 585.898, 572.408, 558.4926, 544.4938, 531.6776, 517.282, 505.7704, 493.1012, 480.7388, 467.6876, 456.1872, 445.5048, 433.0214, 420.806, 411.409, 400.4144, 389.4294, 379.2286, 369.651, 360.6156, 350.337, 342.083, 332.1538, 322.5094, 315.01, 305.6686, 298.1678, 287.8116, 280.9978, 271.9204, 265.3286, 257.5706, 249.6014, 242.544, 235.5976, 229.583, 220.9438, 214.672, 208.2786, 201.8628, 195.1834, 191.505, 186.1816, 178.5188, 172.2294, 167.8908, 161.0194, 158.052, 151.4588, 148.1596, 143.4344, 138.5238, 133.13, 127.6374, 124.8162, 118.7894, 117.3984, 114.6078, 109.0858, 105.1036, 103.6258, 98.6018000000004, 95.7618000000002, 93.5821999999998, 88.5900000000001, 86.9992000000002, 82.8800000000001, 80.4539999999997, 74.6981999999998, 74.3644000000004, 73.2914000000001, 65.5709999999999, 66.9232000000002, 65.1913999999997, 62.5882000000001, 61.5702000000001, 55.7035999999998, 56.1764000000003, 52.7596000000003, 53.0302000000001, 49.0609999999997, 48.4694, 44.933, 46.0474000000004, 44.7165999999997, 41.9416000000001, 39.9207999999999, 35.6328000000003, 35.5276000000003, 33.1934000000001, 33.2371999999996, 33.3864000000003, 33.9228000000003, 30.2371999999996, 29.1373999999996, 25.2272000000003, 24.2942000000003, 19.8338000000003, 18.9005999999999, 23.0907999999999, 21.8544000000002, 19.5176000000001, 15.4147999999996, 16.9314000000004, 18.6737999999996, 12.9877999999999, 14.3688000000002, 12.0447999999997, 15.5219999999999, 12.5299999999997, 14.5940000000001, 14.3131999999996, 9.45499999999993, 12.9441999999999, 3.91139999999996, 13.1373999999996, 5.44720000000052, 9.82779999999912, 7.87279999999919, 3.67760000000089, 5.46980000000076, 5.55099999999948, 5.65979999999945, 3.89439999999922, 3.1275999999998, 5.65140000000065, 6.3062000000009, 3.90799999999945, 1.87060000000019, 5.17020000000048, 2.46680000000015, 0.770000000000437, -3.72340000000077, 1.16400000000067, 8.05340000000069, 0.135399999999208, 2.15940000000046, 0.766999999999825, 1.0594000000001, 3.15500000000065, -0.287399999999252, 2.37219999999979, -2.86620000000039, -1.63199999999961, -2.22979999999916, -0.15519999999924, -1.46039999999994, -0.262199999999211, -2.34460000000036, -2.8078000000005, -3.22179999999935, -5.60159999999996, -8.42200000000048, -9.43740000000071, 0.161799999999857, -10.4755999999998, -10.0823999999993, },
// precision 12
{ 2953, 2900.4782, 2848.3568, 2796.3666, 2745.324, 2694.9598, 2644.648, 2595.539, 2546.1474, 2498.2576, 2450.8376, 2403.6076, 2357.451, 2311.38, 2266.4104, 2221.5638, 2176.9676, 2134.193, 2090.838, 2048.8548, 2007.018, 1966.1742, 1925.4482, 1885.1294, 1846.4776, 1807.4044, 1768.8724, 1731.3732, 1693.4304, 1657.5326, 1621.949, 1586.5532, 1551.7256, 1517.6182, 1483.5186, 1450.4528, 1417.865, 1385.7164, 1352.6828, 1322.6708, 1291.8312, 1260.9036, 1231.476, 1201.8652, 1173.6718, 1145.757, 1119.2072, 1092.2828, 1065.0434, 1038.6264, 1014.3192, 988.5746, 965.0816, 940.1176, 917.9796, 894.5576, 871.1858, 849.9144, 827.1142, 805.0818, 783.9664, 763.9096, 742.0816, 724.3962, 706.3454, 688.018, 667.4214, 650.3106, 633.0686, 613.8094, 597.818, 581.4248, 563.834, 547.363, 531.5066, 520.455400000001, 505.583199999999, 488.366, 476.480799999999, 459.7682, 450.0522, 434.328799999999, 423.952799999999, 408.727000000001, 399.079400000001, 387.252200000001, 373.987999999999, 360.852000000001, 351.6394, 339.642, 330.902400000001, 322.661599999999, 311.662200000001, 301.3254, 291.7484, 279.939200000001, 276.7508, 263.215200000001, 254.811400000001, 245.5494, 242.306399999999, 234.8734, 223.787200000001, 217.7156, 212.0196, 200.793, 195.9748, 189.0702, 182.449199999999, 177.2772, 170.2336, 164.741, 158.613600000001, 155.311, 147.5964, 142.837, 137.3724, 132.0162, 130.0424, 121.9804, 120.451800000001, 114.8968, 111.585999999999, 105.933199999999, 101.705, 98.5141999999996, 95.0488000000005, 89.7880000000005, 91.4750000000004, 83.7764000000006, 80.9698000000008, 72.8574000000008, 73.1615999999995, 67.5838000000003, 62.6263999999992, 63.2638000000006, 66.0977999999996, 52.0843999999997, 58.9956000000002, 47.0912000000008, 46.4956000000002, 48.4383999999991, 47.1082000000006, 43.2392, 37.2759999999998, 40.0283999999992, 35.1864000000005, 35.8595999999998, 32.0998, 28.027, 23.6694000000007, 33.8266000000003, 26.3736000000008, 27.2008000000005, 21.3245999999999, 26.4115999999995, 23.4521999999997, 19.5013999999992, 19.8513999999996, 10.7492000000002, 18.6424000000006, 13.1265999999996, 18.2436000000016, 6.71860000000015, 3.39459999999963, 6.33759999999893, 7.76719999999841, 0.813999999998487, 3.82819999999992, 0.826199999999517, 8.07440000000133, -1.59080000000176, 5.01780000000144, 0.455399999998917, -0.24199999999837, 0.174800000000687, -9.07640000000174, -4.20160000000033, -3.77520000000004, -4.75179999999818, -5.3724000000002, -8.90680000000066, -6.10239999999976, -5.74120000000039, -9.95339999999851, -3.86339999999836, -13.7304000000004, -16.2710000000006, -7.51359999999841, -3.30679999999847, -13.1339999999982, -10.0551999999989, -6.72019999999975, -8.59660000000076, -10.9307999999983, -1.8775999999998, -4.82259999999951, -13.7788, -21.6470000000008, -10.6735999999983, -15.7799999999988, },
// precision 13
{ 5907.5052, 5802.2672, 5697.347, 5593.5794, 5491.2622, 5390.5514, 5290.3376, 5191.6952, 5093.5988, 4997.3552, 4902.5972, 4808.3082, 4715.5646, 4624.109, 4533.8216, 4444.4344, 4356.3802, 4269.2962, 4183.3784, 4098.292, 4014.79, 3932.4574, 3850.6036, 3771.2712, 3691.7708, 3615.099, 3538.1858, 3463.4746, 3388.8496, 3315.6794, 3244.5448, 3173.7516, 3103.3106, 3033.6094, 2966.5642, 2900.794, 2833.7256, 2769.81, 2707.3196, 2644.0778, 2583.9916, 2523.4662, 2464.124, 2406.073, 2347.0362, 2292.1006, 2238.1716, 2182.7514, 2128.4884, 2077.1314, 2025.037, 1975.3756, 1928.933, 1879.311, 1831.0006, 1783.2144, 1738.3096, 1694.5144, 1649.024, 1606.847, 1564.7528, 1525.3168, 1482.5372, 1443.9668, 1406.5074, 1365.867, 1329.2186, 1295.4186, 1257.9716, 1225.339, 1193.2972, 1156.3578, 1125.8686, 1091.187, 1061.4094, 1029.4188, 1000.9126, 972.3272, 944.004199999999, 915.7592, 889.965, 862.834200000001, 840.4254, 812.598399999999, 785.924200000001, 763.050999999999, 741.793799999999, 721.466, 699.040799999999, 677.997200000002, 649.866999999998, 634.911800000002, 609.8694, 591.981599999999, 570.2922, 557.129199999999, 538.3858, 521.872599999999, 502.951400000002, 495.776399999999, 475.171399999999, 459.751, 439.995200000001, 426.708999999999, 413.7016, 402.3868, 387.262599999998, 372.0524, 357.050999999999, 342.5098, 334.849200000001, 322.529399999999, 311.613799999999, 295.848000000002, 289.273000000001, 274.093000000001, 263.329600000001, 251.389599999999, 245.7392, 231.9614, 229.7952, 217.155200000001, 208.9588, 199.016599999999, 190.839199999999, 180.6976, 176.272799999999, 166.976999999999, 162.5252, 151.196400000001, 149.386999999999, 133.981199999998, 130.0586, 130.164000000001, 122.053400000001, 110.7428, 108.1276, 106.232400000001, 100.381600000001, 98.7668000000012, 86.6440000000002, 79.9768000000004, 82.4722000000002, 68.7026000000005, 70.1186000000016, 71.9948000000004, 58.998599999999, 59.0492000000013, 56.9818000000014, 47.5338000000011, 42.9928, 51.1591999999982, 37.2740000000013, 42.7220000000016, 31.3734000000004, 26.8090000000011, 25.8934000000008, 26.5286000000015, 29.5442000000003, 19.3503999999994, 26.0760000000009, 17.9527999999991, 14.8419999999969, 10.4683999999979, 8.65899999999965, 9.86720000000059, 4.34139999999752, -0.907800000000861, -3.32080000000133, -0.936199999996461, -11.9916000000012, -8.87000000000262, -6.33099999999831, -11.3366000000024, -15.9207999999999, -9.34659999999712, -15.5034000000014, -19.2097999999969, -15.357799999998, -28.2235999999975, -30.6898000000001, -19.3271999999997, -25.6083999999973, -24.409599999999, -13.6385999999984, -33.4473999999973, -32.6949999999997, -28.9063999999998, -31.7483999999968, -32.2935999999972, -35.8329999999987, -47.620600000002, -39.0855999999985, -33.1434000000008, -46.1371999999974, -37.5892000000022, -46.8164000000033, -47.3142000000007, -60.2914000000019, -37.7575999999972, },
// precision 14
{ 11816.475, 11605.0046, 11395.3792, 11188.7504, 10984.1814, 10782.0086, 10582.0072, 10384.503, 10189.178, 9996.2738, 9806.0344, 9617.9798, 9431.394, 9248.7784, 9067.6894, 8889.6824, 8712.9134, 8538.8624, 8368.4944, 8197.7956, 8031.8916, 7866.6316, 7703.733, 7544.5726, 7386.204, 7230.666, 7077.8516, 6926.7886, 6778.6902, 6631.9632, 6487.304, 6346.7486, 6206.4408, 6070.202, 5935.2576, 5799.924, 5671.0324, 5541.9788, 5414.6112, 5290.0274, 5166.723, 5047.6906, 4929.162, 4815.1406, 4699.127, 4588.5606, 4477.7394, 4369.4014, 4264.2728, 4155.9224, 4055.581, 3955.505, 3856.9618, 3761.3828, 3666.9702, 3575.7764, 3482.4132, 3395.0186, 3305.8852, 3221.415, 3138.6024, 3056.296, 2970.4494, 2896.1526, 2816.8008, 2740.2156, 2670.497, 2594.1458, 2527.111, 2460.8168, 2387.5114, 2322.9498, 2260.6752, 2194.2686, 2133.7792, 2074.767, 2015.204, 1959.4226, 1898.6502, 1850.006, 1792.849, 1741.4838, 1687.9778, 1638.1322, 1589.3266, 1543.1394, 1496.8266, 1447.8516, 1402.7354, 1361.9606, 1327.0692, 1285.4106, 1241.8112, 1201.6726, 1161.973, 1130.261, 1094.2036, 1048.2036, 1020.6436, 990.901400000002, 961.199800000002, 924.769800000002, 899.526400000002, 872.346400000002, 834.375, 810.432000000001, 780.659800000001, 756.013800000001, 733.479399999997, 707.923999999999, 673.858, 652.222399999999, 636.572399999997, 615.738599999997, 586.696400000001, 564.147199999999, 541.679600000003, 523.943599999999, 505.714599999999, 475.729599999999, 461.779600000002, 449.750800000002, 439.020799999998, 412.7886, 400.245600000002, 383.188199999997, 362.079599999997, 357.533799999997, 334.319000000003, 327.553399999997, 308.559399999998, 291.270199999999, 279.351999999999, 271.791400000002, 252.576999999997, 247.482400000001, 236.174800000001, 218.774599999997, 220.155200000001, 208.794399999999, 201.223599999998, 182.995600000002, 185.5268, 164.547400000003, 176.5962, 150.689599999998, 157.8004, 138.378799999999, 134.021200000003, 117.614399999999, 108.194000000003, 97.0696000000025, 89.6042000000016, 95.6030000000028, 84.7810000000027, 72.635000000002, 77.3482000000004, 59.4907999999996, 55.5875999999989, 50.7346000000034, 61.3916000000027, 50.9149999999936, 39.0384000000049, 58.9395999999979, 29.633600000001, 28.2032000000036, 26.0078000000067, 17.0387999999948, 9.22000000000116, 13.8387999999977, 8.07240000000456, 14.1549999999988, 15.3570000000036, 3.42660000000615, 6.24820000000182, -2.96940000000177, -8.79940000000352, -5.97860000000219, -14.4048000000039, -3.4143999999942, -13.0148000000045, -11.6977999999945, -25.7878000000055, -22.3185999999987, -24.409599999999, -31.9756000000052, -18.9722000000038, -22.8678000000073, -30.8972000000067, -32.3715999999986, -22.3907999999938, -43.6720000000059, -35.9038, -39.7492000000057, -54.1641999999993, -45.2749999999942, -42.2989999999991, -44.1089999999967, -64.3564000000042, -49.9551999999967, -42.6116000000038, },
// precision 15
{ 23634.0036, 23210.8034, 22792.4744, 22379.1524, 21969.7928, 21565.326, 21165.3532, 20770.2806, 20379.9892, 19994.7098, 19613.318, 19236.799, 18865.4382, 18498.8244, 18136.5138, 17778.8668, 17426.2344, 17079.32, 16734.778, 16397.2418, 16063.3324, 15734.0232, 15409.731, 15088.728, 14772.9896, 14464.1402, 14157.5588, 13855.5958, 13559.3296, 13264.9096, 12978.326, 12692.0826, 12413.8816, 12137.3192, 11870.2326, 11602.5554, 11340.3142, 11079.613, 10829.5908, 10583.5466, 10334.0344, 10095.5072, 9859.694, 9625.2822, 9395.7862, 9174.0586, 8957.3164, 8738.064, 8524.155, 8313.7396, 8116.9168, 7913.542, 7718.4778, 7521.65, 7335.5596, 7154.2906, 6968.7396, 6786.3996, 6613.236, 6437.406, 6270.6598, 6107.7958, 5945.7174, 5787.6784, 5635.5784, 5482.308, 5337.9784, 5190.0864, 5045.9158, 4919.1386, 4771.817, 4645.7742, 4518.4774, 4385.5454, 4262.6622, 4142.74679999999, 4015.5318, 3897.9276, 3790.7764, 3685.13800000001, 3573.6274, 3467.9706, 3368.61079999999, 3271.5202, 3170.3848, 3076.4656, 2982.38400000001, 2888.4664, 2806.4868, 2711.9564, 2634.1434, 2551.3204, 2469.7662, 2396.61139999999, 2318.9902, 2243.8658, 2171.9246, 2105.01360000001, 2028.8536, 1960.9952, 1901.4096, 1841.86079999999, 1777.54700000001, 1714.5802, 1654.65059999999, 1596.311, 1546.2016, 1492.3296, 1433.8974, 1383.84600000001, 1339.4152, 1293.5518, 1245.8686, 1193.50659999999, 1162.27959999999, 1107.19439999999, 1069.18060000001, 1035.09179999999, 999.679000000004, 957.679999999993, 925.300199999998, 888.099400000006, 848.638600000006, 818.156400000007, 796.748399999997, 752.139200000005, 725.271200000003, 692.216, 671.633600000001, 647.939799999993, 621.670599999998, 575.398799999995, 561.226599999995, 532.237999999998, 521.787599999996, 483.095799999996, 467.049599999998, 465.286399999997, 415.548599999995, 401.047399999996, 380.607999999993, 377.362599999993, 347.258799999996, 338.371599999999, 310.096999999994, 301.409199999995, 276.280799999993, 265.586800000005, 258.994399999996, 223.915999999997, 215.925399999993, 213.503800000006, 191.045400000003, 166.718200000003, 166.259000000005, 162.941200000001, 148.829400000002, 141.645999999993, 123.535399999993, 122.329800000007, 89.473399999988, 80.1962000000058, 77.5457999999926, 59.1056000000099, 83.3509999999951, 52.2906000000075, 36.3979999999865, 40.6558000000077, 42.0003999999899, 19.6630000000005, 19.7153999999864, -8.38539999999921, -0.692799999989802, 0.854800000000978, 3.23219999999856, -3.89040000000386, -5.25880000001052, -24.9052000000083, -22.6837999999989, -26.4286000000138, -34.997000000003, -37.0216000000073, -43.430400000012, -58.2390000000014, -68.8034000000043, -56.9245999999985, -57.8583999999973, -77.3097999999882, -73.2793999999994, -81.0738000000129, -87.4530000000086, -65.0254000000132, -57.296399999992, -96.2746000000043, -103.25, -96.081600000005, -91.5542000000132, -102.465200000006, -107.688599999994, -101.458000000013, -109.715800000005, },
// precision 16
{ 47270, 46423.3584, 45585.7074, 44757.152, 43938.8416, 43130.9514, 42330.03, 41540.407, 40759.6348, 39988.206, 39226.5144, 38473.2096, 37729.795, 36997.268, 36272.6448, 35558.665, 34853.0248, 34157.4472, 33470.5204, 32793.5742, 32127.0194, 31469.4182, 30817.6136, 30178.6968, 29546.8908, 28922.8544, 28312.271, 27707.0924, 27114.0326, 26526.692, 25948.6336, 25383.7826, 24823.5998, 24272.2974, 23732.2572, 23201.4976, 22674.2796, 22163.6336, 21656.515, 21161.7362, 20669.9368, 20189.4424, 19717.3358, 19256.3744, 18795.9638, 18352.197, 17908.5738, 17474.391, 17052.918, 16637.2236, 16228.4602, 15823.3474, 15428.6974, 15043.0284, 14667.6278, 14297.4588, 13935.2882, 13578.5402, 13234.6032, 12882.1578, 12548.0728, 12219.231, 11898.0072, 11587.2626, 11279.9072, 10973.5048, 10678.5186, 10392.4876, 10105.2556, 9825.766, 9562.5444, 9294.2222, 9038.2352, 8784.848, 8533.2644, 8301.7776, 8058.30859999999, 7822.94579999999, 7599.11319999999, 7366.90779999999, 7161.217, 6957.53080000001, 6736.212, 6548.21220000001, 6343.06839999999, 6156.28719999999, 5975.15419999999, 5791.75719999999, 5621.32019999999, 5451.66, 5287.61040000001, 5118.09479999999, 4957.288, 4798.4246, 4662.17559999999, 4512.05900000001, 4364.68539999999, 4220.77720000001, 4082.67259999999, 3957.19519999999, 3842.15779999999, 3699.3328, 3583.01180000001, 3473.8964, 3338.66639999999, 3233.55559999999, 3117.799, 3008.111, 2909.69140000001, 2814.86499999999, 2719.46119999999, 2624.742, 2532.46979999999, 2444.7886, 2370.1868, 2272.45259999999, 2196.19260000001, 2117.90419999999, 2023.2972, 1969.76819999999, 1885.58979999999, 1833.2824, 1733.91200000001, 1682.54920000001, 1604.57980000001, 1556.11240000001, 1491.3064, 1421.71960000001, 1371.22899999999, 1322.1324, 1264.7892, 1196.23920000001, 1143.8474, 1088.67240000001, 1073.60380000001, 1023.11660000001, 959.036400000012, 927.433199999999, 906.792799999996, 853.433599999989, 841.873800000001, 791.1054, 756.899999999994, 704.343200000003, 672.495599999995, 622.790399999998, 611.254799999995, 567.283200000005, 519.406599999988, 519.188400000014, 495.312800000014, 451.350799999986, 443.973399999988, 431.882199999993, 392.027000000002, 380.924200000009, 345.128999999986, 298.901400000002, 287.771999999997, 272.625, 247.253000000026, 222.490600000019, 223.590000000026, 196.407599999977, 176.425999999978, 134.725199999986, 132.4804, 110.445599999977, 86.7939999999944, 56.7038000000175, 64.915399999998, 38.3726000000024, 37.1606000000029, 46.170999999973, 49.1716000000015, 15.3362000000197, 6.71639999997569, -34.8185999999987, -39.4476000000141, 12.6830000000191, -12.3331999999937, -50.6565999999875, -59.9538000000175, -65.1054000000004, -70.7576000000117, -106.325200000021, -126.852200000023, -110.227599999984, -132.885999999999, -113.897200000007, -142.713800000027, -151.145399999979, -150.799200000009, -177.756200000003, -156.036399999983, -182.735199999996, -177.259399999981, -198.663600000029, -174.577600000019, -193.84580000001, },
// precision 17
{ 94541, 92848.811, 91174.019, 89517.558, 87879.9705, 86262.7565, 84663.5125, 83083.7435, 81521.7865, 79977.272, 78455.9465, 76950.219, 75465.432, 73994.152, 72546.71, 71115.2345, 69705.6765, 68314.937, 66944.2705, 65591.255, 64252.9485, 62938.016, 61636.8225, 60355.592, 59092.789, 57850.568, 56624.518, 55417.343, 54231.1415, 53067.387, 51903.526, 50774.649, 49657.6415, 48561.05, 47475.7575, 46410.159, 45364.852, 44327.053, 43318.4005, 42325.6165, 41348.4595, 40383.6265, 39436.77, 38509.502, 37594.035, 36695.939, 35818.6895, 34955.691, 34115.8095, 33293.949, 32465.0775, 31657.6715, 30877.2585, 30093.78, 29351.3695, 28594.1365, 27872.115, 27168.7465, 26477.076, 25774.541, 25106.5375, 24452.5135, 23815.5125, 23174.0655, 22555.2685, 21960.2065, 21376.3555, 20785.1925, 20211.517, 19657.0725, 19141.6865, 18579.737, 18081.3955, 17578.995, 17073.44, 16608.335, 16119.911, 15651.266, 15194.583, 14749.0495, 14343.4835, 13925.639, 13504.509, 13099.3885, 12691.2855, 12328.018, 11969.0345, 11596.5145, 11245.6355, 10917.6575, 10580.9785, 10277.8605, 9926.58100000001, 9605.538, 9300.42950000003, 8989.97850000003, 8728.73249999998, 8448.3235, 8175.31050000002, 7898.98700000002, 7629.79100000003, 7413.76199999999, 7149.92300000001, 6921.12650000001, 6677.1545, 6443.28000000003, 6278.23450000002, 6014.20049999998, 5791.20299999998, 5605.78450000001, 5438.48800000001, 5234.2255, 5059.6825, 4887.43349999998, 4682.935, 4496.31099999999, 4322.52250000002, 4191.42499999999, 4021.24200000003, 3900.64799999999, 3762.84250000003, 3609.98050000001, 3502.29599999997, 3363.84250000003, 3206.54849999998, 3079.70000000001, 2971.42300000001, 2867.80349999998, 2727.08100000001, 2630.74900000001, 2496.6165, 2440.902, 2356.19150000002, 2235.58199999999, 2120.54149999999, 2012.25449999998, 1933.35600000003, 1820.93099999998, 1761.54800000001, 1663.09350000002, 1578.84600000002, 1509.48149999999, 1427.3345, 1379.56150000001, 1306.68099999998, 1212.63449999999, 1084.17300000001, 1124.16450000001, 1060.69949999999, 1007.48849999998, 941.194499999983, 879.880500000028, 836.007500000007, 782.802000000025, 748.385499999975, 647.991500000004, 626.730500000005, 570.776000000013, 484.000500000024, 513.98550000001, 418.985499999952, 386.996999999974, 370.026500000036, 355.496999999974, 356.731499999994, 255.92200000002, 259.094000000041, 205.434499999974, 165.374500000034, 197.347500000033, 95.718499999959, 67.6165000000037, 54.6970000000438, 31.7395000000251, -15.8784999999916, 8.42500000004657, -26.3754999999655, -118.425500000012, -66.6629999999423, -42.9745000000112, -107.364999999991, -189.839000000036, -162.611499999999, -164.964999999967, -189.079999999958, -223.931499999948, -235.329999999958, -269.639500000048, -249.087999999989, -206.475499999942, -283.04449999996, -290.667000000016, -304.561499999953, -336.784499999951, -380.386500000022, -283.280499999993, -364.533000000054, -389.059499999974, -364.454000000027, -415.748000000021, -417.155000000028, },
// precision 18
{ 189083, 185696.913, 182348.774, 179035.946, 175762.762, 172526.444, 169329.754, 166166.099, 163043.269, 159958.91, 156907.912, 153906.845, 150924.199, 147996.568, 145093.457, 142239.233, 139421.475, 136632.27, 133889.588, 131174.2, 128511.619, 125868.621, 123265.385, 120721.061, 118181.769, 115709.456, 113252.446, 110840.198, 108465.099, 106126.164, 103823.469, 101556.618, 99308.004, 97124.508, 94937.803, 92833.731, 90745.061, 88677.627, 86617.47, 84650.442, 82697.833, 80769.132, 78879.629, 77014.432, 75215.626, 73384.587, 71652.482, 69895.93, 68209.301, 66553.669, 64921.981, 63310.323, 61742.115, 60205.018, 58698.658, 57190.657, 55760.865, 54331.169, 52908.167, 51550.273, 50225.254, 48922.421, 47614.533, 46362.049, 45098.569, 43926.083, 42736.03, 41593.473, 40425.26, 39316.237, 38243.651, 37170.617, 36114.609, 35084.19, 34117.233, 33206.509, 32231.505, 31318.728, 30403.404, 29540.0550000001, 28679.236, 27825.862, 26965.216, 26179.148, 25462.08, 24645.952, 23922.523, 23198.144, 22529.128, 21762.4179999999, 21134.779, 20459.117, 19840.818, 19187.04, 18636.3689999999, 17982.831, 17439.7389999999, 16874.547, 16358.2169999999, 15835.684, 15352.914, 14823.681, 14329.313, 13816.897, 13342.874, 12880.882, 12491.648, 12021.254, 11625.392, 11293.7610000001, 10813.697, 10456.209, 10099.074, 9755.39000000001, 9393.18500000006, 9047.57900000003, 8657.98499999999, 8395.85900000005, 8033, 7736.95900000003, 7430.59699999995, 7258.47699999996, 6924.58200000005, 6691.29399999999, 6357.92500000005, 6202.05700000003, 5921.19700000004, 5628.28399999999, 5404.96799999999, 5226.71100000001, 4990.75600000005, 4799.77399999998, 4622.93099999998, 4472.478, 4171.78700000001, 3957.46299999999, 3868.95200000005, 3691.14300000004, 3474.63100000005, 3341.67200000002, 3109.14000000001, 3071.97400000005, 2796.40399999998, 2756.17799999996, 2611.46999999997, 2471.93000000005, 2382.26399999997, 2209.22400000005, 2142.28399999999, 2013.96100000001, 1911.18999999994, 1818.27099999995, 1668.47900000005, 1519.65800000005, 1469.67599999998, 1367.13800000004, 1248.52899999998, 1181.23600000003, 1022.71900000004, 1088.20700000005, 959.03600000008, 876.095999999903, 791.183999999892, 703.337000000058, 731.949999999953, 586.86400000006, 526.024999999907, 323.004999999888, 320.448000000091, 340.672999999952, 309.638999999966, 216.601999999955, 102.922999999952, 19.2399999999907, -0.114000000059605, -32.6240000000689, -89.3179999999702, -153.497999999905, -64.2970000000205, -143.695999999996, -259.497999999905, -253.017999999924, -213.948000000091, -397.590000000084, -434.006000000052, -403.475000000093, -297.958000000101, -404.317000000039, -528.898999999976, -506.621000000043, -513.205000000075, -479.351000000024, -596.139999999898, -527.016999999993, -664.681000000099, -680.306000000099, -704.050000000047, -850.486000000034, -757.43200000003, -713.308999999892, }
};
#endif /* HYPERLOGLOGBIAS_H_ */
kraken2-2.1.3/src/hyperloglogplus.cc 0000664 0000000 0000000 00000066421 14437747201 0017431 0 ustar 00root root 0000000 0000000 /*
* Copyright 2017-2018, Florian Breitwieser
*
* This file was originally developed for the KrakenUniq taxonomic classification system.
*/
#include "hyperloglogplus.h"
#include
#include
#include
#include
#include
#include //log and sqrt
#include //vector.count
#include
#include //accummulate
#include "hyperloglogplus-bias.h"
/////////////////////////////////////////////////////////////////////
// Helper methods for bit operations
// functions for counting the number of leading 0-bits (clz)
// and counting the number of trailing 0-bits (ctz)
//#ifdef __GNUC__
// TODO: switch between builtin clz and 64_clz based on architecture
//#define clz(x) __builtin_clz(x)
#if 0
static int clz_manual(uint64_t x)
{
// This uses a binary search (counting down) algorithm from Hacker's Delight.
uint64_t y;
int n = 64;
y = x >>32; if (y != 0) {n -= 32; x = y;}
y = x >>16; if (y != 0) {n -= 16; x = y;}
y = x >> 8; if (y != 0) {n -= 8; x = y;}
y = x >> 4; if (y != 0) {n -= 4; x = y;}
y = x >> 2; if (y != 0) {n -= 2; x = y;}
y = x >> 1; if (y != 0) return n - 2;
return n - x;
}
#endif
#ifdef WIN32
#include
#define __builtin_clz(x) __lzcnt(x)
#define __builtin_clzl(x) __lzcnt64(x)
#endif
inline uint8_t clz(const uint32_t x, const uint8_t max = 32) {
if (x == 0) { return max; }
return __builtin_clz(x);
}
inline uint8_t clz(const uint64_t x, const uint8_t max = 64) {
if (x == 0) { return max; }
return __builtin_clzl(x);
}
template
inline uint8_t clz_p(const T x, const uint8_t p = 1) {
return __builtin_clz(((x<<1)|1)<<(p-1));
}
/**
* Extract bits (from uint32_t or uint64_t) using LSB 0 numbering from hi to lo, including lo
*/
template
T extractBits(T value, uint8_t hi, uint8_t lo, bool shift_left = false) {
// create a bitmask:
// (T(1) << (hi - lo) a 1 at the position (hi - lo)
// ((T(1) << (hi - lo) - 1) 1's from position 0 to position (hi-lo-1)
// (((T(1) << (hi - lo)) - 1) << lo) 1's from position lo to position hi
// The T(1) is required to not cause overflow on 32bit machines
// TODO: consider creating a bitmask only once in the beginning
T bitmask = (((T(1) << (hi - lo)) - 1) << lo);
T result = value & bitmask;
if (!shift_left) {
// shift resulting bits to the right
result = result >> lo;
} else {
// shift resulting bits to the left
result = result << (sizeof(T)*8 - hi);
}
return result;
}
inline uint64_t extractHighBits(uint64_t bits, uint8_t hi) {
return bits >> (64u - hi);
}
inline uint32_t extractHighBits(uint32_t bits, uint8_t hi) {
return bits >> (32u - hi);
}
inline uint32_t getIndex(const uint64_t hash_value, const uint8_t p) {
// take first p bits as index {x63,...,x64-p}
return hash_value >> (64 - p);
}
inline uint32_t getIndex(const uint32_t hash_value, const uint8_t p) {
// take first p bits as index {x31,...,x32-p}
return hash_value >> (32 - p);
}
template inline
T trailingOnes(const uint8_t p) {
return (T(1) << p ) - 1;
}
uint8_t getRank(const uint32_t hash_value, const uint8_t p) {
// shift p values off, and count leading zeros of the remaining string {x31-p,...,x0}
uint32_t rank_bits (hash_value << p);
uint8_t rank_val = clz(rank_bits, 32-p) + 1;
//uint8_t rank_val = clz_p(hash_value, p) + 1;
assert(rank_val < 32-p+1);
return rank_val;
}
uint8_t getRank(const uint64_t hash_value, const uint8_t p) {
// shift p values off, and count leading zeros of the remaining string {x63-p,...,x0}
uint64_t rank_bits (hash_value << p);
uint8_t rank_val = clz(rank_bits, 64-p) + 1;
//uint8_t rank_val = clz_p(hash_value, p) + 1;
assert(rank_val <= 64-p+1);
return rank_val;
}
/////////////////////////////////////////////////////////////////////
// Methods for spare representation, Heule et al., 2015
uint8_t getEncodedRank(const uint32_t encoded_hash_value, const uint8_t pPrime, const uint8_t p) {
// check if the least significant bit is 1
if ( (encoded_hash_value & 1) == 1) {
// if yes: the hash was stored with higher precision, bits p to pPrime were 0
uint8_t additional_rank = pPrime - p;
return additional_rank + extractBits(encoded_hash_value, 7, 1);
} else {
return getRank(encoded_hash_value,p);
}
}
/**
* To decode hash from sparse representation:
res.idx = getIndex(encoded_hash_value, p);
res.rank = getEncodedRank(encoded_hash_value, p, Prime);
*/
/**
* Encode the 64-bit hash code x as an 32-bit integer, to be used in the sparse representation.
*
* Difference from the algorithm described in the paper:
* The index always is in the p most significant bits
*
* see section 5.3 in Heule et al.
* @param x the hash bits
* @return encoded hash value
*/
inline
uint32_t encodeHashIn32Bit(uint64_t hash_value, uint8_t pPrime, uint8_t p) {
// extract first pPrime bits as index
uint32_t idx = (uint32_t)(extractHighBits(hash_value,pPrime) << (32-pPrime));
#ifdef HLL_DEBUG2
cerr << "encoding hash: " << bitset<64>(hash_value) << endl;
cerr << "index': " << bitset<32>(idx) << " ( bits from 64 to " << 64-pPrime << "; " << idx << ")" << endl;
#endif
// are the bits after bit p in index' all 0?
if (idx << p == 0) {
// compute the additional rank (minimum rank is already p'-p)
// the maximal size will be below 2^6=64. We thus combine the 25 bits of the index with 6 bits for the rank, and one bit as flag
uint8_t additional_rank = getRank(hash_value, pPrime); // this is rank - (p'-p), as we know that positions p'...p are 0
#ifdef HLL_DEBUG2
cerr << "All zero " << endl;
#endif
return idx | uint32_t(additional_rank<<1) | 1;
} else {
// else, return the idx, only - it has enough length to calculate the rank (left-shifted, last bit = 0)
assert((idx & 1) == 0);
return idx;
}
}
inline
void addHashToSparseList(vector& vec, const uint32_t val, const uint8_t pPrime) {
//if (sparseList_is_sorted) {
auto it = std::lower_bound( vec.begin(), vec.end(), val); // find proper position in descending order
if (it == vec.end()) { // position at the end
vec.insert( it, val ); // insert before iterator it
} else if (*it != val) { // val not in the vector
if (extractHighBits(val,pPrime) == extractHighBits(*it,pPrime)) { //the values have the same index
// it's pretty unlikely to observe different hashes with the same index
// pPrime is 25, which means that the chance of another random number
// having the same index is 1/2^25 or about 1/33,500,000. Sparse representation
// doesn't go that high, though.
// Maybe a hash should be added even if it has the same index?
// (TODO: Check how often this occurs and the relative errors of the estimates)
if ((*it & 1) == (val & 1)) { // if both are in the same category
if ((val & 1) == 1) { // both have ones as lsb - replace if val is greater
if (val > *it) *it = val;
} else { // both have zeros as lsb - replace if val is smaller
if (val < *it) *it = val;
}
} else if ((val & 1) == 1) { // replace if lsb of val is 1
*it = val;
}
} else {
vec.insert( it, val ); // insert before iterator it
}
}
//} else {
// vec.push_back(val);
//}
}
template
inline
void addHashToSparseList(SET& uset, const uint32_t val, const uint8_t /*pPrime*/) {
// this implementation currently does not check if there is a value for an index of length pPrime
uset.insert(val);
}
////////////////////////////////////////////////////////////////////
// Other Flajolet/Heule HLL functions
/**
* Bias correction factors for specific m's
* @param m
* @return
*/
double alpha(uint32_t m) {
switch (m) {
case 16: return 0.673;
case 32: return 0.697;
case 64: return 0.709;
}
// m >= 128
return 0.7213 / (1 + 1.079/double(m));
}
/**
* Gives the estimated cardinality for m bins, v of which are non-zero
* using linear counting of Whang et al., 1990: n_hat = -m ln(v)
* @param m number of bins in the matrix
* @param v number of non-zero bins
* @return
*/
double linearCounting(const uint32_t m, const uint32_t v) {
if (v > m) {
throw std::invalid_argument("number of v should not be greater than m");
}
return double(m) * log(double(m)/double(v));
}
/**
* calculate the raw estimate as harmonic mean of the ranks in the register
*/
inline double calculateRawEstimate(const vector& M) {
double inverseSum = 0.0;
for (size_t i = 0; i < M.size(); ++i) {
inverseSum += 1. / (1ull << M[i]);
}
return alpha(M.size()) * double(M.size() * M.size()) * 1. / inverseSum;
}
uint32_t countZeros(vector s) {
return (uint32_t)count(s.begin(), s.end(), 0);
}
/**
* Estimate the bias of raw estimate using empirically determined values.
* Uses weighted average of the two cells between which the estimate falls.
* TODO: Check if nearest neighbor average gives better values, as proposed in the paper
* @param est
* @return correction value for
*/
double getEstimateBias(double estimate, uint8_t p) {
const vector& rawEstimateTable = rawEstimateData[p-4];
const vector& biasTable = biasData[p-4];
// check if estimate is lower than first entry, or larger than last
if (rawEstimateTable.front() >= estimate) { return biasTable.front(); }
if (rawEstimateTable.back() <= estimate) { return biasTable.back(); }
// get iterator to first element that is not smaller than estimate
vector::const_iterator it = lower_bound(rawEstimateTable.begin(),rawEstimateTable.end(),estimate);
size_t pos = it - rawEstimateTable.begin();
double e1 = rawEstimateTable[pos-1];
double e2 = rawEstimateTable[pos];
double c = (estimate - e1) / (e2 - e1);
D(cerr << "bias correction factor c: (estimate - e1)/ (e2 - e1) = (" << estimate << " - " << e1 << ")/(" << e2 << " - " << e1 << ")" << endl;)
D(cerr << "biasTable[" << pos-1 << "]*(1-c) + biasTable[" << pos << "]*c" << endl; )
return biasTable[pos-1]*(1-c) + biasTable[pos]*c;
}
/////////////////////////////////////////////////////////////////////
// Methods for improved estimator of Ertl et al.
/**
* returns a 'register histogram' vector C,
* where C[i] is the number of elements in M with value i
* it's size is q+1 = 64-p+1
* used in Ertl's improved estimator
*/
vector registerHistogram(const vector& M, uint8_t q) {
vector C(q+2, 0);
for (size_t i = 0; i < M.size(); ++i) {
if (M[i] >= q+1) {
cerr << "M["< sparseRegisterHistogram(const SparseListType& sparseList, uint8_t pPrime, uint8_t p, uint8_t q){
vector C(q+2, 0);
size_t m = 1 << pPrime;
for (const auto& encoded_hash_value : sparseList) {
uint8_t rank_val = getEncodedRank(encoded_hash_value, pPrime, p);
++C[rank_val];
--m;
}
C[0] = m;
return C;
}
/**
* Ertl - calculation of sigma correction for 0-registers in M
* x is the proportion of 0-registers, thus x \in [0, 1]
* sigma := x + sum[from k=1 to Inf] ( x^(2^k) * 2^(k-1) )
*/
double sigma(double x) {
assert(x >= 0.0 && x <= 1.0);
if (x == 1.0) { return std::numeric_limits::infinity(); }
double prev_sigma_x;
double sigma_x = x;
double y = 1.0;
do { // loop until sigma_x does not change anymore
prev_sigma_x = sigma_x;
x *= x; // gives x^(2^k)
sigma_x += x * y;
y += y; // gives 2^(k-1)
} while (sigma_x != prev_sigma_x);
return sigma_x;
}
// modification of the implementation - currently not used
double sigma_mod(double x) {
assert(x >= 0.0 && x <= 1.0);
if (x == 1.0) { return std::numeric_limits::infinity(); }
double sigma_x = x;
for (double x_sq = x*x, two_exp = 1.0;
x_sq > std::numeric_limits::epsilon();
x_sq *= x_sq, two_exp += two_exp) {
sigma_x += x_sq * two_exp;
}
return sigma_x;
}
/**
* Ertl - calculation of tau correction for values higher then q in M
* x is the proportion of registers with a value below q in M, thus x \in [0,1]
* tau := 1/3 (1 - x - sum[from k=1 to Inf](1-x^(2^(-k))^2 * 2^(-k) ))
*/
double tau(double x) {
assert(x >= 0.0 && x <= 1.0);
if (x == 0.0 || x == 1.0) { return 0.0; }
double prev_tau_x;
double y = 1.0;
double tau_x = 1 - x;
do { // loop until tau_x does not change anymore
prev_tau_x = tau_x;
x = std::sqrt(x); // gives x^(2^-k)
y /= 2.0; // gives 2^(-k)
tau_x -= std::pow(1-x, 2) * y;
} while (tau_x != prev_tau_x);
return tau_x / 3.0;
}
/////////////////////////////////////////////////////////////////////
// HyperLogLogPlusMinus class methods
template<>
HyperLogLogPlusMinus::HyperLogLogPlusMinus(uint8_t precision, bool sparse, uint64_t (*bit_mixer) (uint64_t)):
p(precision), m(1< 18 || precision < 4) {
throw std::invalid_argument("precision (number of register = 2^precision) must be between 4 and 18");
}
if (sparse) {
this->sparseList = SparseListType(); // TODO: if SparseListType is changed, initialize with appropriate size
this->sparseList.reserve(m/4);
} else {
this->M = vector(m);
}
}
template
HyperLogLogPlusMinus& HyperLogLogPlusMinus::operator= (HyperLogLogPlusMinus&& other) {
p = other.p;
m = other.m;
M = std::move(other.M);
n_observed = other.n_observed;
sparse = other.sparse;
sparseList = std::move(other.sparseList);
bit_mixer = other.bit_mixer;
return *this;
}
template
HyperLogLogPlusMinus& HyperLogLogPlusMinus::operator= (const HyperLogLogPlusMinus& other) {
p = other.p;
m = other.m;
M = other.M;
n_observed = other.n_observed;
sparse = other.sparse;
sparseList = other.sparseList;
bit_mixer = other.bit_mixer;
return *this;
}
template
HyperLogLogPlusMinus::HyperLogLogPlusMinus(const HyperLogLogPlusMinus& other):
p(other.p), m(other.m),
M(other.M), n_observed(other.n_observed), sparse(other.sparse),
sparseList(other.sparseList),
bit_mixer(other.bit_mixer) {
}
template
HyperLogLogPlusMinus::HyperLogLogPlusMinus(HyperLogLogPlusMinus&& other):
p(other.p), m(other.m),
M(std::move(other.M)),
n_observed(other.n_observed), sparse(other.sparse),
sparseList(std::move(other.sparseList)),
bit_mixer(other.bit_mixer) {
}
template<>
void HyperLogLogPlusMinus::insert(uint64_t item) {
++ n_observed;
// compute hash for item
uint64_t hash_value = bit_mixer(item);
#ifdef HLL_DEBUG2
cerr << "Value: " << item << "; hash(value): " << hash_value << endl;
cerr << bitset<64>(hash_value) << endl;
#endif
if (sparse && this->sparseList.size() + 1 > this->m/4) {
switchToNormalRepresentation();
}
if (sparse) {
// sparse mode: put the encoded hash into sparse list
uint32_t encoded_hash_value = encodeHashIn32Bit(hash_value, pPrime, p);
addHashToSparseList(sparseList, encoded_hash_value, pPrime);
#ifdef HLL_DEBUG2
cerr << "encoded hash: " << bitset<32>(encoded_hash_value) << endl;
assert(getIndex(encoded_hash_value,p) == getIndex(hash_value, p));
assert(getEncodedRank(encoded_hash_value,pPrime,p) == getRank(hash_value, p));
#endif
// if the sparseList is too large, switch to normal (register) representation
} else {
// normal mode
// take first p bits as index {x63,...,x64-p}
uint32_t idx = getIndex(hash_value, p);
// shift those p values off, and count leading zeros of the remaining string {x63-p,...,x0}
uint8_t rank = getRank(hash_value, p);
// update the register if current rank is bigger
if (rank > this->M[idx]) {
this->M[idx] = rank;
}
}
}
template
void HyperLogLogPlusMinus::insert(const vector& items) {
for(auto i : items) {
this->insert(i);
}
}
// reset to original state
template
void HyperLogLogPlusMinus::reset() {
this->sparse = true;
this->sparseList.clear(); //
this->M.clear();
}
// Convert from sparse representation (using sparseList) to normal (using register)
template
void HyperLogLogPlusMinus::switchToNormalRepresentation() {
if (!sparse)
return;
D(cerr << "switching to normal representation" << endl;)
#ifdef HLL_DEBUG
cerr << " est before: " << cardinality() << endl;
#endif
this->sparse = false;
this->M = vector(this->m);
addToRegisters(this->sparseList);
this->sparseList.clear();
#ifdef HLL_DEBUG
cerr << " est after: " << cardinality() << endl;
#endif
}
// add sparseList to the registers of M
template
void HyperLogLogPlusMinus::addToRegisters(const SparseListType &sparseList) {
if (sparse) {
cerr << "Cannot add to registers of a sparse HLL" << endl;
return;
}
if (sparseList.size() == 0) {
return;
}
for (auto encoded_hash_value_ptr = sparseList.begin(); encoded_hash_value_ptr != sparseList.end(); ++encoded_hash_value_ptr) {
size_t idx = getIndex(*encoded_hash_value_ptr, p);
assert(idx < M.size());
uint8_t rank_val = getEncodedRank(*encoded_hash_value_ptr, pPrime, p);
if (rank_val > this->M[idx]) {
this->M[idx] = rank_val;
}
}
}
template
uint64_t HyperLogLogPlusMinus::nObserved() const {
return n_observed;
}
template
void HyperLogLogPlusMinus::merge(HyperLogLogPlusMinus&& other) {
if (this->p != other.p) {
throw std::invalid_argument("precisions must be equal");
}
if (other.n_observed == 0)
return;
if (this->n_observed == 0) {
n_observed = other.n_observed;
sparse = other.sparse;
sparseList = std::move(other.sparseList);
M = std::move(other.M);
} else {
n_observed += other.n_observed;
if (this->sparse && other.sparse) {
// this->merge(static_cast&>(other));
// consider using addHashToSparseList(this->sparseList, val, pPrime) and checking for sizes
this->sparseList.insert(other.sparseList.begin(), other.sparseList.end());
} else if (other.sparse) {
// other is sparse, but this is not
addToRegisters(other.sparseList);
} else {
if (this->sparse) {
this->sparse = false;
M = std::move(other.M);
addToRegisters(this->sparseList);
this->sparseList.clear();
} else {
// merge registers
for (size_t i = 0; i < other.M.size(); ++i) {
if (other.M[i] > this->M[i]) {
this->M[i] = other.M[i];
}
}
}
}
}
}
// Merge other HyperLogLogPlusMinus into this one. May convert to normal representation
template
void HyperLogLogPlusMinus::merge(const HyperLogLogPlusMinus& other) {
if (this->p != other.p) {
throw std::invalid_argument("precisions must be equal");
}
if (other.n_observed == 0)
return;
if (this->n_observed == 0) {
// TODO: Make this more efficient when other is disowned
n_observed = other.n_observed;
sparse = other.sparse;
sparseList = other.sparseList;
M = other.M;
} else {
n_observed += other.n_observed;
if (this->sparse && other.sparse) {
// consider using addHashToSparseList(this->sparseList, val, pPrime) and checking for sizes
this->sparseList.insert(other.sparseList.begin(), other.sparseList.end());
} else if (other.sparse) {
// other is sparse, but this is not
addToRegisters(other.sparseList);
} else {
if (this->sparse) {
this->sparse = false;
M = other.M;
addToRegisters(this->sparseList);
this->sparseList.clear();
} else {
// merge registers
for (size_t i = 0; i < other.M.size(); ++i) {
if (other.M[i] > this->M[i]) {
this->M[i] = other.M[i];
}
}
}
}
}
}
template
HyperLogLogPlusMinus& HyperLogLogPlusMinus::operator+=(HyperLogLogPlusMinus&& other) {
merge(std::move(other));
return *this;
}
template
HyperLogLogPlusMinus& HyperLogLogPlusMinus::operator+=(const HyperLogLogPlusMinus& other) {
merge(other);
return *this;
}
template<>
uint64_t HyperLogLogPlusMinus::flajoletCardinality(bool use_sparse_precision) const {
vector M = this->M;
if (sparse) {
if (use_sparse_precision) {
return round(linearCounting(mPrime, mPrime-uint32_t(sparseList.size())));
} else{
// For testing purposes. Put sparse list into a standard register
M = vector(m, 0);
for (const auto& val : sparseList) {
size_t idx = getIndex(val, p);
assert(idx < M.size());
uint8_t rank_val = getEncodedRank(val, pPrime, p);
if (rank_val > M[idx]) {
M[idx] = rank_val;
}
}
}
}
double est = calculateRawEstimate(M);
if (est <= 2.5*m) {
uint32_t v = countZeros(M);
if (v > 0)
est = linearCounting(m, v);
} /* else if (est > 1/30 * pow(2,64) {
// No bias correction - should not run in the problem with 64-bit hashes
} */
return (use_n_observed && n_observed < est)? n_observed : round(est);
}
/**
* Improved cardinality estimator of Ertl, 2017 (arXiv, section 4)
* Based on the underlying distribution, the estimator employs correction
* factors for zero and 'over-subscribed' registers. It does not depend on
* emprically defined bias correction values or a switch between linear
* counting and loglog estimation
*
* Formula:
* alpha_inf * m^2
* --------------------------------------------------------------------------------------
* ( m * sigma(C_0/m) + sum[from k=1 to q] C_k * 2^(-k) + m * tau(1-C_(q+1)/m) * 2^(-q)
*/
template<>
uint64_t HyperLogLogPlusMinus::ertlCardinality() const {
size_t q, m;
vector C;
if (sparse) {
q = 64 - pPrime;
m = mPrime;
C = sparseRegisterHistogram(sparseList, pPrime, p, q);
} else {
q = 64 - p;
m = this->m;
C = registerHistogram(M, q);
}
D(cerr << "\n1. hist. q=" << q << "; m=" << m << endl;)
D(cerr << "2. m * tau(m*(1.0 - "<= 1; --k) {
est_denominator += C[k];
est_denominator *= 0.5;
}
D(cerr << est_denominator << endl; )
D(cerr << "4. sigma(" << C[0] << "/m) = " << sigma(double(C[0])/double(m)); )
est_denominator += m * sigma(double(C[0])/double(m));
D(cerr << endl;)
double m_sq_alpha_inf = (m / (2.0*std::log(2))) * m;
double est = m_sq_alpha_inf / est_denominator;
return (use_n_observed && n_observed < est)? n_observed : round(est);
}
template<>
uint64_t HyperLogLogPlusMinus::heuleCardinality(bool correct_bias) const {
if (p > 18) {
cerr << "Heule HLL++ estimate only works with value of p up to 18 - returning Ertl estimate." << endl;
return(ertlCardinality());
}
if (sparse) {
// if we are 'sparse', then use linear counting with increased precision pPrime
uint64_t lc_estimate = round(linearCounting(mPrime, mPrime-uint32_t(sparseList.size())));
D(cerr << "sparse representation - return linear counting estimate("<= 0);
if (lc_estimate <= double(threshold[p-4])) {
//D(cerr << "below threshold - return it " << endl;)
return lc_estimate;
}
D(cerr << "above threshold of " << threshold[p-4] << " - calculate raw estimate " << endl;)
}
// calculate raw estimate on registers
//double est = alpha(m) * harmonicMean(M, m);
double est = calculateRawEstimate(M);
D(cerr << "raw estimate: " << est << endl;)
// correct for biases if estimate is smaller than 5m
if (correct_bias && est <= double(m)*5.0) {
D(cerr << "correct bias; subtract " << getEstimateBias(est, p) << endl;)
assert(est > getEstimateBias(est, p));
est -= getEstimateBias(est, p);
}
return (use_n_observed && n_observed < est)? n_observed : round(est);
}
template<>
uint64_t HyperLogLogPlusMinus::cardinality() const {
return ertlCardinality();
}
template<>
uint64_t HyperLogLogPlusMinus::size() const {
return cardinality();
}
/////////////////////////////////////////////////////////////////////
// Hash and other functions
/**
* from Numerical Recipes, 3rd Edition, p 352
* Returns hash of u as a 64-bit integer.
*
*/
inline uint64_t ranhash (uint64_t u) {
uint64_t v = u * 3935559000370003845 + 2691343689449507681;
v ^= v >> 21; v ^= v << 37; v ^= v >> 4;
v *= 4768777513237032717;
v ^= v << 20; v ^= v >> 41; v ^= v << 5;
return v;
}
/**
* Avalanche mixer/finalizer from MurMurHash3
* https://github.com/aappleby/smhasher
* from https://github.com/aappleby/smhasher/blob/master/src/MurmurHash3.cpp
*/
uint64_t murmurhash3_finalizer (uint64_t key) {
key += 1; // murmurhash returns a hash value of 0 for the key 0 - avoid that.
key ^= key >> 33;
key *= 0xff51afd7ed558ccd;
key ^= key >> 33;
key *= 0xc4ceb9fe1a85ec53;
key ^= key >> 33;
return key;
}
/*
64-bit mixer developed by Thomas Wang
Proposed for HLL by https://github.com/dnbaker/hll
https://gist.github.com/badboy/6267743
*/
inline uint64_t wang_mixer(uint64_t key) {
key = (~key) + (key << 21); // key = (key << 21) - key - 1;
key = key ^ (key >> 24);
key = (key + (key << 3)) + (key << 8); // key * 265
key = key ^ (key >> 14);
key = (key + (key << 2)) + (key << 4); // key * 21
key = key ^ (key >> 28);
key = key + (key << 31);
return key;
}
/*
inline
void merge_lists(vector& vec1, const vector& vec2) {
auto it = std::lower_bound( vec.begin(), vec.end(), val); // find proper position in descending order
if (it == vec.end()) {
vec.insert( it, val ); // insert before iterator it
}
}
*/
//#else
//#endif
template
struct NoHash {
size_t operator() (const T &u) const {
return u;
}
};
// Always compile 64-bit HLL class
template class HyperLogLogPlusMinus;
kraken2-2.1.3/src/hyperloglogplus.h 0000664 0000000 0000000 00000010074 14437747201 0017264 0 ustar 00root root 0000000 0000000 /*
* Copyright 2017-2018, Florian Breitwieser
*
* This file was originally developed for the KrakenUniq taxonomic classification system.
*/
/*
* Implementation of 64-bit HyperLogLog algorithm by Flajolet et al.,
* with sparse mode for increased precision with low cardinalities (Stefan Heule et al.), and
* an improved estimator that does not rely on empirical bias correction data (Otmar Ertl)
*/
#ifndef HYPERLOGLOGPLUS_H_
#define HYPERLOGLOGPLUS_H_
#include "kraken2_headers.h"
using namespace std;
//#define HLL_DEBUG
//#define HLL_DEBUG2
#ifdef HLL_DEBUG
#define D(x) x
#else
#define D(x)
#endif
// 64-bit Mixer/finalizer from MurMurHash3 https://github.com/aappleby/smhasher
uint64_t murmurhash3_finalizer (uint64_t key);
// Heule et al. encode the sparse list with variable length encoding
// see section 5.3.2. This implementation just uses a sorted vector or unordered_set.
typedef unordered_set SparseListType;
// Other possible SparseList types:
// // typedef vector SparseListType;
// The sorted vector implementation is pretty inefficient currently, as the vector
// is always kept sorted.
// //typedef unordered_set > SparseListType;
// No real performance gain from using the hash value directly - and probably problems
// with the bucket assignment, since unordered_set expects size_t hashes
/**
* HyperLogLogPlusMinus class for counting the number of unique 64-bit values in stream
* Note that only HASH=uint64_t is implemented.
*/
template
class HyperLogLogPlusMinus {
private:
uint8_t p; // precision, set in constructor
size_t m = 1 << p; // number of registers
vector M; // registers, size m
uint64_t n_observed = 0;
bool sparse; // sparse representation of the data?
SparseListType sparseList;
HASH (*bit_mixer) (uint64_t);
// sparse versions of p and m
static const uint8_t pPrime = 25; // precision when using a sparse representation
// which encodes the rank + index in 32 bits:
// 25 bits for index +
// 6 bits for rank +
// 1 flag bit indicating if bits p..pPrime are 0
static const uint32_t mPrime = 1 << pPrime; // 2^pPrime
public:
bool use_n_observed = true; // return min(estimate, n_observed) instead of estimate
// Construct HLL with precision bits
HyperLogLogPlusMinus(uint8_t precision=12, bool sparse=true, HASH (*bit_mixer) (uint64_t) = murmurhash3_finalizer);
HyperLogLogPlusMinus(const HyperLogLogPlusMinus& other);
HyperLogLogPlusMinus(HyperLogLogPlusMinus&& other);
HyperLogLogPlusMinus& operator= (HyperLogLogPlusMinus&& other);
HyperLogLogPlusMinus& operator= (const HyperLogLogPlusMinus& other);
~HyperLogLogPlusMinus() {};
void reset(); // Note: sets sparse=true
// Add items or other HLL to this sketch
void insert(uint64_t item);
void insert(const vector& items);
// Merge another sketch into this one
// TODO: assumes equal bit_mixers! but does not check that
void merge(HyperLogLogPlusMinus&& other);
void merge(const HyperLogLogPlusMinus& other);
HyperLogLogPlusMinus& operator+=(HyperLogLogPlusMinus&& other);
HyperLogLogPlusMinus& operator+=(const HyperLogLogPlusMinus& other);
// Calculate cardinality estimates
uint64_t cardinality() const; // returns ertlCardinality()
uint64_t size() const; // returns ertlCardinality()
// HLL++ estimator of Heule et al., 2015. Uses empirical bias correction factors
uint64_t heuleCardinality(bool correct_bias = true) const;
// Improved estimator of Ertl, 2017. Does not rely on empirical data
uint64_t ertlCardinality() const;
// Flajolet's cardinality estimator without bias correction
uint64_t flajoletCardinality(bool use_sparse_precision = true) const;
uint64_t nObserved() const;
private:
void switchToNormalRepresentation();
void addToRegisters(const SparseListType &sparseList);
};
#endif /* HYPERLOGLOGPLUS_H_ */
kraken2-2.1.3/src/k2mask.cc 0000664 0000000 0000000 00000024355 14437747201 0015362 0 ustar 00root root 0000000 0000000 /*
* Symmetric Dustmasker mostly based on a similarly named implementation
* by Heng Li as part of the minimap project.
*/
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include "gzstream.h"
#include "seqreader.h"
#include "threadpool.h"
using namespace kraken2;
uint8_t asc2dna[] = {
/* 0 */ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* 16 */ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* 32 */ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* - */
/* 48 */ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* 64 */ 4, 0, 4, 1, 4, 4, 4, 2, 4, 4, 4, 4, 4, 4, 4, 4,
/* A B C D G H K M N */
/* 80 */ 4, 4, 4, 4, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* R S T U V W Y */
/* 96 */ 4, 0, 4, 1, 4, 4, 4, 2, 4, 4, 4, 4, 4, 4, 4, 4,
/* a b c d g h k m n */
/* 112 */ 4, 4, 4, 4, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* r s t u v w y */
/* 128 */ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* 144 */ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* 160 */ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* 176 */ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* 192 */ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* 208 */ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* 224 */ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
/* 240 */ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
};
struct PerfectInterval {
int start;
int finish;
int left;
int right;
};
struct MaskRange {
int start;
int finish;
};
struct SDust {
SDust(): rw(0), rv(0), l(0) {
memset(cv, 0, 64 * sizeof(int));
memset(cw, 0, 64 * sizeof(int));
}
void reset() {
kmers.clear();
perfectIntervals.clear();
ranges.clear();
memset(cw, 0, 64 * sizeof(int));
memset(cv, 0, 64 * sizeof(int));
l = rw = rv = 0;
}
std::deque kmers;
std::vector perfectIntervals;
std::vector ranges;
Sequence seq;
int cw[64];
int cv[64];
int rw;
int rv;
int l;
};
int windowSize = 64;
int threshold = 20;
std::function processMaskedNucleotide = tolower;
void usage(const char *prog) {
std::cerr << "usage: " << prog
<< " [-T | -level threshold] [-W | -window window size] [-i | -in input file]"
<< std::endl
<< " [-w | -width sequence width] [-o | -out output file] [-r | -replace-masked-with char]"
<< std::endl
<< " [-f | -outfmt output format] [-t | -threads threads]"
<< std::endl;
exit(1);
}
bool stricasecmp(std::string &s1, std::string &s2) {
if (s1.size() != s2.size())
return false;
for (size_t i = 0; i < s1.size(); i++) {
if (tolower(s1[i]) != tolower(s2[i]))
return false;
}
return true;
}
void shiftWindow(SDust &sd, int t) {
int s;
if ((int)sd.kmers.size() >= windowSize - 2) {
s = sd.kmers.front();
sd.kmers.pop_front();
sd.rw -= --sd.cw[s];
if (sd.l > (int)sd.kmers.size()) {
--sd.l;
sd.rv -= --sd.cv[s];
}
}
sd.kmers.push_back(t);
sd.l++;
sd.rw += sd.cw[t]++;
sd.rv += sd.cv[t]++;
if (sd.cv[t] * 10 > threshold * 2) {
do {
s = sd.kmers.at(sd.kmers.size() - sd.l);
sd.rv -= --sd.cv[s];
sd.l--;
} while (s != t);
}
}
void saveMaskedRegions(SDust &sd, int windowStart) {
bool saved = false;
if (sd.perfectIntervals.size() == 0
|| sd.perfectIntervals.back().start >= windowStart)
return;
PerfectInterval &p = sd.perfectIntervals.back();
if (!sd.ranges.empty()) {
int start = sd.ranges.back().start;
int finish = sd.ranges.back().finish;
if (p.start <= finish) {
sd.ranges.back() = MaskRange { start, std::max(p.finish, finish) };
saved = true;
}
}
if (!saved)
sd.ranges.push_back(MaskRange { p.start, p.finish});
while (!sd.perfectIntervals.empty()
&& sd.perfectIntervals.back().start < windowStart)
sd.perfectIntervals.pop_back();
}
void findPerfect(SDust &sd, int windowStart) {
int cv[64];
int maxLeft = 0;
int maxRight = 0;
int newLeft = 0;
int newRight = sd.rv;
memcpy(cv, sd.cv, 64 * sizeof(int));
for (int i = sd.kmers.size() - sd.l - 1; i >= 0; --i) {
size_t j;
int kmer = sd.kmers.at(i);
newRight += cv[kmer]++;
newLeft = sd.kmers.size() - i - 1;
if (newRight * 10 > threshold * newLeft) {
for (j = 0; j < sd.perfectIntervals.size() && sd.perfectIntervals[j].start >= i + windowStart; ++j) {
PerfectInterval &p = sd.perfectIntervals[j];
if (maxRight == 0 || p.right * maxLeft > maxRight * p.left) {
maxLeft = p.left;
maxRight = p.right;
}
}
if (maxRight == 0 || newRight * maxLeft >= maxRight * newLeft) {
maxLeft = newLeft;
maxRight = newRight;
PerfectInterval p;
p.start = i + windowStart;
p.finish = sd.kmers.size() + 2 + windowStart;
p.left = newLeft;
p.right = newRight;
auto position = sd.perfectIntervals.begin() + j;
sd.perfectIntervals.insert(position, p);
}
}
}
}
void runSymmetricDust(SDust &sd, char *seq, size_t size, int offset) {
int triplet = 0;
int windowStart = 0;
int l = 0;
for (size_t i = 0; i < size; i++) {
int base = asc2dna[(int)seq[i]];
if (base < 4) {
l++;
triplet = (triplet << 2 | base) & 63;
if (l >= 3) {
windowStart = std::max(l - (int)windowSize, 0);
saveMaskedRegions(sd, windowStart);
shiftWindow(sd, triplet);
if (sd.rw * 10 > sd.l * threshold)
findPerfect(sd, windowStart);
}
}
}
while (!sd.perfectIntervals.empty())
saveMaskedRegions(sd, windowStart++);
}
void printFasta(Sequence seq, std::ostream& out, int width) {
out.write(&seq.header[0], seq.header.size());
out << '\n';
for (size_t i = 0; i < seq.seq.size(); i += width) {
if (i + width >= seq.seq.size())
width = seq.seq.size() - i;
out.write(&seq.seq[i], width);
out << '\n';
}
}
SDust *mask(SDust *sd) {
size_t i = 0;
std::string &seq = sd->seq.seq;
for (; i < seq.size(); i++) {
if (asc2dna[(int)seq[i]] != 4) {
int start = i;
for (;;) {
seq[i] = toupper(seq[i]);
if ((i+1) == seq.size()
|| asc2dna[(int)seq[i+1]] == 4)
break;
i++;
}
char *s = &seq[0] + start;
runSymmetricDust(*sd, s, i - start + 1, start);
for (size_t j = 0; j < sd->ranges.size(); j++) {
for (int i = sd->ranges[j].start; i < sd->ranges[j].finish; i++)
s[i] = processMaskedNucleotide(s[i]);
}
sd->ranges.clear();
}
}
return sd;
}
int main(int argc, char **argv) {
int ch;
int lineWidth = 72;
int threads = 1;
std::string infile = "/dev/stdin";
std::string outfile = "/dev/stdout";
std::string buffer;
const char *prog = "k2mask";
struct option longopts[] = {
{"window", required_argument, NULL, 'W'},
{"level", required_argument, NULL, 'T'},
{"in", required_argument, NULL, 'i'},
{"out", required_argument, NULL, 'o'},
{"width", required_argument, NULL, 'w'},
{"outfmt", required_argument, NULL, 'f'},
{"threads", required_argument, NULL, 't'},
{"replace-masked-with", required_argument, NULL, 'r'},
{"help", no_argument, NULL, 'h'},
{NULL, 0, NULL, 0}
};
while ((ch = getopt_long_only(argc, argv, "W:T:hi:w:o:r:t:", longopts, NULL)) != -1) {
switch (ch) {
case 'W':
windowSize = atoi(optarg);
break;
case 'T':
threshold = atoi(optarg);
break;
case 'i':
// Input file name (default: stdin)
infile = optarg;
break;
case 'w':
// Wrap sequence every after outputting this many (72) characters
lineWidth = atoi(optarg);
break;
case 'f': {
// Output format, currently FASTA only
std::string arg(optarg);
std::string format("fasta");
if (!stricasecmp(arg, format)) {
std::cerr << prog << ": currently only supports outputting FASTA."
<< std::endl;
std::exit(1);
}
break;
}
case 'o':
// Output file name (default: stdout)
outfile = optarg;
break;
case 't':
// Number of threads. If _n_ threads are specified,
// _n - 1_ threads will be used for masking and the
// final thread used to manage I/O.
threads = atoi(optarg);
break;
case 'r': {
// Replace masked character with a specified letter. (default tolower)
if (strlen(optarg) != 1) {
std::cerr << prog << ": -r expects a single character, "
<< optarg << " given." << std::endl;
usage(prog);
}
int r = optarg[0];
processMaskedNucleotide = [=](char c) { return r; };
break;
}
case 'h':
default:
usage(prog);
}
}
argc -= optind;
argv += optind;
if (argc > 0) {
std::cerr << prog << ": reads from stdin and writes to stdout"
<< std::endl;
usage(prog);
}
gzistream is(infile.c_str());
std::ofstream out(outfile, std::ios::out | std::ios::trunc);
std::vector sds(threads);
for (size_t i = 0; i < sds.size(); i++) {
sds[i] = new SDust();
}
thread_pool pool(threads - 1);
std::queue> tasks;
for (SDust *sd = sds.back(); BatchSequenceReader::ReadNextSequence(is, sd->seq, buffer); sd = sds.back()) {
sds.pop_back();
if (threads > 1) {
tasks.push(pool.submit(mask, sd));
while (sds.empty()) {
sd = tasks.front().get();
printFasta(sd->seq, out, lineWidth);
tasks.pop();
sd->reset();
sds.push_back(sd);
}
} else {
mask(sd);
printFasta(sd->seq, out, lineWidth);
sds.push_back(sd);
}
}
while (!tasks.empty()) {
SDust *sd = tasks.front().get();
printFasta(sd->seq, out, lineWidth);
tasks.pop();
}
for (size_t i = 0; i < sds.size(); i++)
delete(sds[i]);
out.flush();
}
kraken2-2.1.3/src/kraken2_data.h 0000664 0000000 0000000 00000001757 14437747201 0016363 0 ustar 00root root 0000000 0000000 /*
* Copyright 2013-2023, Derrick Wood
*
* This file is part of the Kraken 2 taxonomic sequence classification system.
*/
#ifndef KRAKEN2_KRAKEN2_DATA_H_
#define KRAKEN2_KRAKEN2_DATA_H_
#include "kraken2_headers.h"
#include "readcounts.h"
namespace kraken2 {
struct IndexOptions {
size_t k;
size_t l;
uint64_t spaced_seed_mask;
uint64_t toggle_mask;
bool dna_db;
uint64_t minimum_acceptable_hash_value;
int revcom_version; // Fix bug from before K2.0.8
int db_version; // To allow for future database structural changes
int db_type; // To allow for future use of other data structures
};
typedef uint64_t taxid_t;
const taxid_t TAXID_MAX = (taxid_t) -1;
typedef std::unordered_map taxon_counts_t;
#ifdef EXACT_COUNTING
typedef ReadCounts> READCOUNTER;
#else
typedef ReadCounts> READCOUNTER;
#endif
typedef std::unordered_map taxon_counters_t;
}
#endif
kraken2-2.1.3/src/kraken2_headers.h 0000664 0000000 0000000 00000001275 14437747201 0017060 0 ustar 00root root 0000000 0000000 #include
#include
#include
#include
#include
#include
#include
#include
#include
#include