DFAST — Implementation Report / 2026-06-24

MAG / MAG-WGS Support
Implementation Report

New features added relative to the main branch — DDBJ ENV division submission support

5Files Modified
2Files Added
46New Tests
+220Lines Added

Overview

In the main branch, project_type was parsed from metadata but used only to set complete=True for gnm. DDBJ submission requirements specific to MAG (Metagenome-Assembled Genome) and MAG-WGS — ENV division, MAG keywords, isolate qualifier, metagenome_source, environmental_sample — were not supported.

This implementation adds the following new features relative to the main branch.

+
MAG / MAG-WGS project_type recognition and complete flag
mag now triggers complete=True automatically. project_type is stored in GENOME_CONFIG and propagated throughout the pipeline.
+
Automatic ENV division and MAG keywords
For MAG/MAG-WGS, DIVISION: ENV and keywords ENV, Metagenome Assembled Genome, MAG are emitted automatically.
+
isolate field support
isolate is now propagated end-to-end: metadata file → GENOME_SOURCE_INFORMATION → Genome object → source feature qualifiers → .ann file.
+
TSV-driven conditional field control
required_for / excluded_for columns added to the TSV. Excluding strain and requiring isolate/metagenome_source/environmental_sample for MAG is now controlled purely via TSV — no code changes needed for future project types.
✎
ff_definition: @@[strain]@@ → @@[isolate]@@ for MAG
The DEFINITION line template in the DDBJ flat file is switched automatically based on project_type.
✔
Boolean field render() fix
environmental_sample (boolean type) now correctly emits a valueless qualifier row even when the value is empty, enabling its use as a MAG placeholder.

project_type Recognition and complete Flag

config_util.py / genome.py

config_util.py changes

dfc/utils/config_util.py
- keys = ["organism", "strain", "seq_names", "seq_types", "seq_topologies", "additional_modifiers"] + keys = ["organism", "strain", "isolate", "seq_names", "seq_types", "seq_topologies", "additional_modifiers"] - if project_type == "gnm": - complete = True + if project_type in ("gnm", "mag"): + complete = True + config.GENOME_CONFIG["project_type"] = project_type

genome.py changes

dfc/genome.py
self.complete = config.GENOME_CONFIG.get("complete", False) +self.project_type = config.GENOME_CONFIG.get("project_type", "") +self.is_mag = self.project_type in ("mag", "mag-wgs") self.organism = config.GENOME_SOURCE_INFORMATION.get("organism", "") self.strain = config.GENOME_SOURCE_INFORMATION.get("strain", "") +self.isolate = config.GENOME_SOURCE_INFORMATION.get("isolate", "")
project_typecompleteis_mag
mag✔ True✔ True
mag-wgs— False✔ True
gnm✔ True— False
wgs— False— False

ENV Division and MAG Keywords

metadata_util.py / genome.py

MAG/MAG-WGS submissions go to DDBJ's ENV division. The COMMON entry in the .ann file must include a dedicated block of division and keyword rows.

WGS (unchanged)

COMMON entry
DATATYPEtype: WGS
KEYWORDkeyword: WGS
keyword: STANDARD_DRAFT

MAG-WGS (new)

COMMON entry
DIVISIONdivision: ENV
KEYWORDkeyword: ENV
keyword: Metagenome Assembled Genome
keyword: MAG
DATATYPEtype: WGS
KEYWORDkeyword: WGS
keyword: STANDARD_DRAFT

GNM / complete (unchanged)

COMMON entry
(no DATATYPE / KEYWORD rows)

MAG / complete (new)

COMMON entry
DIVISIONdivision: ENV
KEYWORDkeyword: ENV
keyword: Metagenome Assembled Genome
keyword: MAG

The internal data_file_division field (used in GenBank/EMBL output) also switches from BCT to ENV via genome.py add_source_information().

isolate Field Added

New

The main branch had no isolate support anywhere. Since MAG submissions use isolate instead of strain as the specimen identifier, it was added throughout the pipeline.

Propagation path

metadata file → config_util → Genome.isolate → record.annotations → source feature → .ann file

SOURCE line in source feature

dfc/genome.py — add_source_information()
-source = self.organism + " strain " + self.strain if self.strain else self.organism +if self.is_mag: + source = self.organism + " isolate " + self.isolate if self.isolate else self.organism +else: + source = self.organism + " strain " + self.strain if self.strain else self.organism
ℹ
isolate also works for non-MAG project types: although isolate is declared required_for=mag,mag-wgs, it is rendered for WGS/GNM too when a value is present — the should_render() rule "value present → always render" applies.

TSV-Driven Conditional Field Control

New Architecture

In the main branch, MetadataField had no concept of project_type, and EX_SOURCE rendering used the simple condition field.value or field.mss_required. This implementation replaces that with two new TSV columns and four new methods.

New and updated rows in metadata_definition.tsv

Fieldtypefeaturemss_requiredrequired_for (new)excluded_for (new)
strain (existing)stringsource TRUE—mag,mag-wgs
isolate ★ newstringsource FALSEmag,mag-wgs—
metagenome_source ★ newstringEX_SOURCE FALSEmag,mag-wgs—
environmental_sample ★ newbooleanEX_SOURCE FALSEmag,mag-wgs—

New methods on MetadataField

def is_required_for(self, project_type: str) -> bool:
    return project_type in self.required_for

def is_excluded_for(self, project_type: str) -> bool:
    return project_type in self.excluded_for

def effective_required(self, project_type: str) -> bool:
    if project_type in self.excluded_for: return False
    if project_type in self.required_for:  return True
    return self.mss_required  # project_type-independent default

def should_render(self, project_type: str) -> bool:
    if project_type in self.excluded_for: return False  # excluded even if value is set
    if self.value:                          return True   # value present → always render
    return self.effective_required(project_type)         # render placeholder if required

render_ex_source() changes

dfc/utils/metadata_util.py
-def render_ex_source(self): +def render_ex_source(self, project_type: str = ""): RET = [] for field in self.fields.values(): if field.feature == "source" and field.entry == "EX_SOURCE": - if field.value or field.mss_required: - RET += field.render() + if field.should_render(project_type): + rows = field.render() + if rows: + RET += rows + else: # boolean with empty value: emit valueless placeholder row + RET.append(["", "", "", field.qualifier, ""])

get_field_definition() — new module-level function

Used by genome.py to look up TSV field definitions when building source feature qualifiers. Parses the TSV once and caches the result.

# module level
_FIELD_DEF_CACHE: "dict | None" = None

def get_field_definition(name: str):
    global _FIELD_DEF_CACHE
    if _FIELD_DEF_CACHE is None:
        _FIELD_DEF_CACHE = {}
        # parse TSV once, populate cache
        ...
    return _FIELD_DEF_CACHE.get(name)

ff_definition Template Switch

ddbj_submission.py

In the main branch, ff_definition always used @@[strain]@@. For MAG/MAG-WGS, where strain is excluded, it now switches to @@[isolate]@@.

dfc/utils/ddbj_submission.py — create_ff_definiton()
-def create_ff_definiton(seq_rank, plasmid): +def create_ff_definiton(seq_rank, plasmid, project_type=""): assert seq_rank in ["complete", "scaffold", "contig"] + identifier = "@@[isolate]@@" if project_type in ("mag", "mag-wgs") else "@@[strain]@@" if seq_rank == "complete": if plasmid: - ff_definition = "@@[organism]@@ @@[strain]@@ plasmid @@[plasmid]@@ DNA, complete sequence" + ff_definition = f"@@[organism]@@ {identifier} plasmid @@[plasmid]@@ DNA, complete sequence" - ff_definition = "@@[organism]@@ @@[strain]@@ DNA, complete genome" + ff_definition = f"@@[organism]@@ {identifier} DNA, complete genome" - ff_definition = "@@[organism]@@ @@[strain]@@ DNA, @@[submitter_seqid]@@" + ff_definition = f"@@[organism]@@ {identifier} DNA, @@[submitter_seqid]@@"
project_typeff_definition template
mag@@[organism]@@ @@[isolate]@@ DNA, complete genome
mag-wgs@@[organism]@@ @@[isolate]@@ DNA, @@[submitter_seqid]@@
gnm@@[organism]@@ @@[strain]@@ DNA, complete genome
wgs@@[organism]@@ @@[strain]@@ DNA, @@[submitter_seqid]@@

Output File Naming

ddbj_submission.py

The main branch always used the biosample_strain pattern for output file names. Since MAG does not use strain, the identifier is now chosen based on is_mag.

dfc/utils/ddbj_submission.py — get_file_prefix()
- strain = self.genome.strain - if bio_sample_id and strain: - return bio_sample_id + "_" + strain + if getattr(self.genome, "is_mag", False): + identifier = self.genome.isolate + else: + identifier = self.genome.strain + if bio_sample_id and identifier: + return bio_sample_id + "_" + identifier
ConditionOutput filename
MAG + biosample set + isolate setSAMDxxxxxxxx_MAG-001.ann
MAG + biosample onlySAMDxxxxxxxx.ann
WGS + biosample set + strain setSAMDxxxxxxxx_strain.ann
None setmss.ann

Boolean Field render() Fix

metadata_util.py

In the main branch, the boolean render path emitted a valueless qualifier only when the value was explicitly set (anything other than "NO"). With an empty value it returned [], making it impossible to emit environmental_sample as a placeholder for MAG submissions.

dfc/utils/metadata_util.py — MetadataField.render()
+ if self.type == "boolean": + if str(self.value).lower() in ("yes", "true"): + RET.append(["", "", "", self.qualifier, ""]) + return RET # returns [] when empty; render_ex_source handles placeholder if self.value == "" or self.value == []: RET.append(["", "", "", self.qualifier, ""]) - elif self.type == "boolean" and self.value: - if self.value != "NO": - RET.append(["", "", "", self.qualifier, ""])
⚠
When render() returns [], render_ex_source() adds a fallback valueless placeholder row if should_render() is True. This ensures environmental_sample always appears in MAG output, even when the user has not set a value.

Tests

46 new tests
46
Total tests (all pass)
0
Failures / errors

Test categories

CategoryWhat it coversCount
MetadataField unit should_render, effective_required, is_excluded_for, etc. ~12
TSV parsing required_for/excluded_for columns, backward compatibility with 9-column rows ~6
render_ex_source Per-field output for MAG/WGS, boolean valueless placeholder ~8
Genome source features strain excluded, isolate placeholder, isolate rendered for WGS when set ~8
render_common_entry DIVISION/KEYWORD block output for MAG and MAG-WGS ~4
DDBJ submission output create_ff_definiton identifier switch, file prefix generation ~8

Changed Files

Modified dfc/utils/config_util.py Added isolate to GENOME_SOURCE_INFORMATION keys. Added mag to complete=True trigger. Stored project_type in GENOME_CONFIG.
Modified dfc/utils/metadata_util.py Added required_for/excluded_for to MetadataField with 4 new methods. Extended TSV parser. Added render_common_entry(project_type) ENV/MAG keyword output. TSV-driven render_ex_source(project_type). New get_field_definition(). Fixed boolean render.
Modified dfc/utils/metadata_definition.tsv Added columns 10/11 (required_for/excluded_for). Added 3 new rows: isolate, metagenome_source, environmental_sample. Set excluded_for=mag,mag-wgs on strain.
Modified dfc/genome.py Added self.project_type, self.is_mag, self.isolate. Updated add_source_information() for ENV division and isolate. Changed add_source_features() to TSV-driven strain/isolate logic.
Modified dfc/utils/ddbj_submission.py Removed is_mag parameter; replaced with project_type. Switched @@[strain]@@ → @@[isolate]@@ in create_ff_definiton. Updated get_file_prefix() to use isolate for MAG. Added ENV division to JSON conversion.
Added tests/test_mag.py New test file dedicated to MAG/MAG-WGS support. 46 tests, all passing.
Added example/sample.metadata.mag*.txt Sample metadata files for MAG and MAG-WGS (full / nofill / minimal variants).