Ver 1.5.1 (20261004) ---------------------------------------
'--amr' works with PlasmidFinder 3.x installed with 'pip install plasmidfinder' only. DFAST runs it as 'python -m plasmidfinder' when no plasmidfinder.py command is on PATH, and aborts with an install hint when PlasmidFinder 3.x is missing. Bioconda provides only PlasmidFinder 2.x, which DFAST cannot use, so the Bioconda package of DFAST no longer depends on plasmidfinder; install PlasmidFinder 3.x from PyPI to use '--amr'.

'dfast_file_downloader.py --mefinder' now installs MobileElementFinder when it is not installed yet (it stopped with FileNotFoundError).

The configuration for the MBGD version of the web service (example/dfc_config/dfast_web_config_mbgd.py) has been removed.


Ver 1.5.0 (20261004) ---------------------------------------
DFAST stops bundling executables so that its dependencies can be installed from Bioconda or apt and maintained more easily. Together with Ver 1.6, this will let DFAST run on linux-aarch64 and osx-arm64 as well. Ver 1.5 keeps the results compatible with earlier versions as far as possible; MGA and CRT are still bundled and will be replaced in Ver 1.6 (see Roadmap below).

The following programs are no longer bundled and must be on PATH (e.g. conda install -c bioconda -c conda-forge blast rpsbproc hmmer last aragorn ghostx diamond):
  - BLAST+ 2.13 or later (blastp, blastn, makeblastdb, blastdbcmd, rpsblast). Databases are created in the BLAST database version 5 format; version 4 databases created by older DFAST can still be used.
  - rpsbproc 0.5 or later for CDDsearch.
  - HMMER 3 (hmmscan, hmmsearch, hmmpress, and nhmmer and hmmbuild for rRNA prediction).
  - LAST 1180 or later. PseudoGeneDetection passes the genetic code with 'lastal -G <transl_table>' instead of the custom lastal4 binary.
  - Aragorn for tRNA/tmRNA prediction.
  - GHOSTX, still the default aligner. DIAMOND is needed only for '--aligner diamond'. The bundled GHOSTZ has been removed.
Perl is no longer required for the default workflow. Java is still required for CRT until Ver 1.6.

rRNA prediction: Barrnap (Perl) has been reimplemented in Python, with reference to the source code of Barrnap by Torsten Seemann. DFAST runs nhmmer directly against rRNA profile HMMs and applies the Barrnap 0.8 rules; with the Barrnap models the rRNA results are identical to the bundled Barrnap 0.8. The models are no longer bundled: run 'dfast_file_downloader.py --rrna barrnap' (Barrnap 0.9 models). The inference qualifier changes to "COORDINATES:profile:BarrnapHmm:0.9". New options:
  - '--rrna_model rfam' uses models built from Rfam 15.1 seed alignments ('dfast_file_downloader.py --rrna rfam'). Planned to become the default in a future version.
  - '--rrna_kingdom bac arc' searches the bacterial and archaeal models; overlapping hits compete and the best-scoring model wins.
  - Eukaryotic rRNA models are always searched as a contamination check (check_eukaryotic in the config). Eukaryotic 18S/28S rRNAs that outscore the prokaryotic models (e.g. host contamination in MAGs) become misc_feature ("eukaryotic 18S ribosomal RNA-like sequence, possible contamination"), and hypothetical CDSs overlapping them on either strand are removed.

The COG reference data for RPS-BLAST (CDDsearch) has been updated from COG v1.0 (2020, equivalent to CDD 3.18, distributed from the DFAST server) to COG v2.0 in NCBI CDD 3.21 (5,137 PSSMs). 'dfast_file_downloader.py --cdd Cog' now downloads it from NCBI together with the annotation data for rpsbproc from the same CDD release (cdd/rpsbproc_data). The old pair of COG data and rpsbproc data did not match, so some hits were reported only as superfamilies and no COG was assigned; COG assignments increase with the new data.

HMMscan uses the NCBI HMM collection 20.0 used by PGAP instead of TIGRFAMs 15.0. Run 'dfast_file_downloader.py --hmm NCBIfam'. By default its TIGR models are searched with trusted cutoffs ('--hmm_db ncbifam' searches the whole collection), and CDSs without a database hit are named by HMMs marked for naming (product, gene, EC_number and /inference="protein motif:HMM:<accession>"). This reduces hypothetical proteins and changes such CDSs from earlier versions.

DIAMOND ('--aligner diamond') runs in the --very-sensitive mode as one process with multiple threads, which is faster and uses less memory than one process per CPU.

To update the reference data from Ver 1.4.x: dfast_file_downloader.py --rrna barrnap --cdd Cog --hmm NCBIfam

Roadmap:
  - Ver 1.6 (next): replace the last bundled programs to complete the switch. MGA will be replaced by Prodigal for CDS prediction, and CRT by Diced (a reimplementation of MinCED) for CRISPR detection with the CRT parameters, which also removes the Java dependency. DFAST will then contain no bundled executables. CDS predictions will change with Prodigal.
  - GHOSTX will run as one process with multiple threads once the Bioconda package is built with OpenMP.
  - '--use_rnammer' is deprecated and will be removed in Ver 1.6 (RNAmmer needs HMMER 2). Use the default rRNA prediction instead.


Ver 1.4.3 (20261002) ---------------------------------------
Partial rRNAs are now reported as misc_feature. Barrnap hits aligned to less than 80% of the expected length ("aligned only N percent") become misc_feature with note "putative rRNA, aligned only ..." and no product; those less than 10 bp from a contig end are extended to the end and shown as partial (< or >). Barrnap now runs with '--reject 0.5' unless --reject is set in cmd_options, since the default differs between Barrnap 0.8 and 0.9.

rRNAs overlapping an assembly gap are trimmed at the gap and shown as partial, and an rRNA spanning a gap is split into two misc_features. Truncated features get a "truncated at the contig end" or "truncated at an assembly gap" note.

Same-strand overlaps rejected by the DDBJ validator are resolved before locus_tags are assigned: rRNA overlapping a CDS or another rRNA (ANN5310) and tRNA contained in a CDS (ANN5320). A hypothetical CDS is removed; if the CDS has a functional product, the RNA is removed instead; of two overlapping rRNAs the shorter is removed. Overlap removal by feature type priority now runs after these rules as a fallback, and removed features are reported in the log.


Ver 1.4.2 (20260722) ---------------------------------------
Fixed incorrect coordinates for putative composite transposons detected by MobileElementFinder ('--mge'). MobileElementFinder reports the start of putative composite transposons as 0-based while other elements are 1-based, which shifted the feature by 1 bp and produced an invalid '<0' location on contig ends (resulting in a null location under Biopython 1.83+). The coordinate system is now detected per feature from the reported allele length, so the correct start is used and a future upstream fix will not cause a double correction.

Putative composite transposons are no longer written to the DDBJ submission file (.ann) by default, since they frequently overlap or duplicate and are hard to curate for submission. They are still included in the GenBank and GFF outputs. Set DDBJ_SUBMISSION["include_putative_composite"] to True in the config to include them in the submission file.


Ver 1.4.1 (20260705) ---------------------------------------
Added MAG/MAG-WGS submission support for DDBJ ENV division. Use '--mag' to enable MAG mode (sets project_type=mag-wgs; use with '--complete t' for project_type=mag). The '--isolate' option sets the isolate qualifier (mutually exclusive with '--strain'). ENV/MAG keywords and required source qualifiers (metagenome_source, environmental_sample) are added automatically. project_type can also be specified in a metadata file.

Fixed UnicodeDecodeError when parsing VFDB FASTA files containing non-UTF-8 bytes (latin-1 encoding is now used).

'scripts/reference_util_for_nucl.py' now automatically fetches the latest CARD version from card.mcmaster.ca when '--card_version' is omitted. The '--vfdb_update_date' option now defaults to today's date when omitted.


Ver 1.4.0 (20260611) ---------------------------------------
Python 3.10 or later is now required.

Mobile genetic element (MGE) detection has been implemented using MobileElementFinder. Enable it with the "--mge" option. Detected MGEs are annotated as mobile_element or misc_feature features in the result. MobileElementFinder can be installed and its MGEdb reference index built with "dfast_file_downloader.py --mefinder".

Extra databases have been added to the PlasmidFinder reference data, downloaded and registered automatically by "dfast_file_downloader.py --plasmidfinder". The following databases are now included in addition to the default PlasmidFinder database:
  - PseudomonasRepDB: PseudomonasRepDB database v1.0
  - WHRepDB: WHRepDB database v1.0
  - APT DB: AcinetobacterPlasmidTyping database v3


Ver 1.3.9 (20260410) ---------------------------------------
Minimum contig length has been changed to 1000bp from 200bp (in accordance with INSDC minimum requirements).
Fixed output in MSS format for transl_except on the complement strand.


Ver 1.3.6 (20250225) ---------------------------------------
Minor fix. dfast_results.json has been renamed as dfast_record, which is generated using the DR_tools module. (Python>=3.10, experimental implementation)

Ver 1.3.5 (20250203) ---------------------------------------
Minor fix. dfast_results.json will be generated in the result file, which contains information required for DDBJ submission (experimental implementation)

Ver 1.3.4 (20241220) ---------------------------------------
Minor fix. phone and fax are no longer required in DDBJ submission file, so they have been removed.

Ver 1.3.3 (20241007) ---------------------------------------
Minor fix. Inference will not be described when cyanobase_aa is used as the reference database.

Ver 1.3.2 (20240824) ---------------------------------------
Minor fix for DDBJ submission. ff_definition has been changed.


Ver 1.3.1 (20240530) ---------------------------------------
The "country" qualifier in the DDBJ submission file has been changed to "geo_loc_name" to be compliant to the INSDC requirements (https://www.ddbj.nig.ac.jp/news/en/2024-05-09-e.html)


Ver 1.3.0 (20240509) ---------------------------------------
Preliminary implementation for AMR/VFG annotation.
Use '--amr' to enable this function.



Ver 1.2.21 (20240208) ---------------------------------------
Adjusted to changes in the SeqFeature specifications of Biopython.


Ver 1.2.19 (20230515) ---------------------------------------
Accepts a gzip-compressed FASTA file as input.

20230404 ---------------------------------------

The reference protein data have been updated. The DFAST default reference database now consists of mainly from RefSeq WP proteins.
The latest version is "20230404".
You can update it with the following command.
dfast_file_downloader.py --protein dfast


Ver 1.2.0 (20190124) ---------------------------------------
 

# Import annotation from GFF

    Genes annotated in the GFF format can be imported with using --gff option.
    By default, CDS features are imported. Other features such as tRNA and rRNA can be imported by changing the config file.
    The sequence names in the genome FASTA file and those in the GFF files must be identical.
    If you use --gff option, --use_original_name option is automatically set to enabled.

    Note that this option is preliminary, so the result may not be compliant to DDBJ data submission format.

    dfast --genome example/sample.genome.fna --gff example/sample.prodigal.gff 

    default configuration is as below:
        {
            # Preliminary implementation.
            "tool_name": "GFF_import",
            "enabled": False,
            "options": {
                # "targets": ["CDS", "tRNA", "rRNA"],
                "targets": ["CDS"],  # feature types to be imported
                "gff_file_name": None,
            },
        },


# Database name and the pident (percent of identity) threshold for homology search can be specified.
    The corresponding part in the config file is as follows:
        {   
            # Sequence homology search against the default DB.
            "component_name": "DBsearch",
            "enabled": True,
            "options": {
                # "cpu": 2,  # Uncomment this to set the component-specific number of CPUs, or the global setting is used.
                "skipAnnotatedFeatures": True,
                "evalue_cutoff": 1e-6,
                "qcov_cutoff": 75,
                "scov_cutoff": 75,
                "pident_cutoff": 0,   # NEW! cutoff for identity percentage (default:0)
                "aligner": "ghostx", 
                "aligner_options": {},  # Normally, leave this empty. (Current version does not use this option.)
                "database": "@@APP_ROOT@@/db/protein/DFAST-default.ref",
                "db_name": "DFAST-default",    # NEW! Database name will appear in /note qualifier
            }
        }    
    
    Accordingly, the output format is slightly changed.
        (Before)
                     /note="Partial hit; WP_003131759.1
                     methylphosphotriester-DNA alkyltransferase (Lactococcus
                     lactis subsp. lactis Il1403) [pid:52.0%, q_cov:99.0%,
                     s_cov:54.8%, Eval:6.9e-26]"

        (After)
                     /note="DFAST-default:WP_003131759.1
                     methylphosphotriester-DNA alkyltransferase (Lactococcus
                     lactis subsp. lactis Il1403) [pid:52.0%, q_cov:99.0%,
                     s_cov:54.8%, Eval:6.9e-26, partial hit]"

                     /note="DFAST-default:WP_000531736.1 cadmium resistance
                     protein CadD (Streptococcus agalactiae 2603V/R) [pid:57.6%,
                     q_cov:100.0%, s_cov:99.5%, Eval:7.2e-64, low identity]"
    
    The default cutoff value for pident is 0(%), so no filtering is performed.


# Command line option for cutoff values for default homology search (--threshold)
    format: --threshold pident,q_cov,s_cov,e_value
    default: 0,75,75,1e-6
    example: --threshold 80,,,1e-10
             (Default values are used if not specified)
    

# Command line option for cutoff values for additional homology search (--database)
    In the previous versions, --database option simply accepts the path to the reference database.
        e.g. --database db/protein/DFAST-ECOLI.ref
    Now, database name and cutoff values can be specified.
        Format: db_path[,db_name[,pident,q_cov,s_cov,e_value]]
        e.g. --database your_database.ref,your_DBNAME
        e.g. --database your_database.ref,your_DBNAME,90,80,80,1e-10
    
    If you have a reference database from very closely-related species, using more stringent thresholds is recommendable.
    (Predicted protein sequences are queried against the additional database first, then against the default database.)      
        e.g. dfast --genome your.genome.fna --threshold 0,75,75,1e-6 --database db/protein/DFAST-ECOLI.ref,ECOLI,90,85,85,1e-10
    
    
# New commandline options:
    --no_cds    Disable CDS prediction
    
    --no_rrna    Disable rRNA prediction
    
    --no_trna    Disable tRNA prediction
    
    --no_crispr    Disable CRISPR prediction