Validate Annotations and Reconstructed Morphology

Good to see you again. You now have a frozen cohort manifest: a versioned list of root IDs, a stated selection predicate, a raw annotation export, and a fixed materialization context. That establishes how neurons entered the cohort. This lesson asks the next, more demanding question: should each candidate remain in it?

Validation is not an attempt to prove that a reconstruction or cell-type label is flawless. It is a bounded, auditable check that the annotation is being interpreted correctly and that the morphology contains no observed issue serious enough to compromise the capstone’s connectivity claims. By the end, you will have a review table that separates included neurons from documented exclusions and unresolved cases—before you compute cohort connectivity.

---

Validation has two independent targets

A cohort member can be problematic in two different ways.

1. Annotation validity: the metadata does not actually support its inclusion under your written rule. For example, the exported row may use a different type field than you intended, have a conflicting side label, or contain a community tag that has been mistaken for a consolidated annotation.

2. Morphological validity: the root ID is formally valid and its annotation may be plausible, but its segmentation has a suspected false merge, false split, missing branch, or other reconstruction concern that could affect connectivity.

Keep these questions separate. A neuron can have a sound annotation but an uncertain reconstruction; conversely, a clean-looking reconstruction does not independently establish a cell-type label.

For your running example, the selection rule is:

cell_type == DA1_lPN && side == left

The validation claim should therefore be modest and testable:

At the recorded annotation release and materialization version, this neuron satisfies the stated annotation rule, and the bounded morphology review found no issue requiring exclusion from the primary cohort.

That wording does not claim that a neuron is biologically perfect, fully proofread, or permanently assigned to a type.

The three possible outcomes

Use a small, predeclared decision vocabulary. It prevents quietly dropping inconvenient candidates once connectivity results are visible.

Decision	Meaning	Role in the primary connectivity analysis
included	Annotation supports the rule; morphology review found no material concern within the defined inspection scope.	Included
excluded	A documented annotation contradiction or morphology problem makes the candidate unsuitable for this cohort.	Excluded
unresolved	Evidence is insufficient or contradictory; the issue cannot responsibly be resolved in the allotted review.	Excluded from primary analysis, retained for sensitivity analysis

An unresolved case is a legitimate result, not a failure. In dense EM regions, image defects or ambiguous continuations may make a confident decision impossible without deeper proofreading.

---

First: verify what the annotations actually say

The manifest records the annotation fields used for selection. Now inspect those fields for every candidate and distinguish them from supplementary evidence.

FlyWire-related annotations commonly fall into two categories:

• Systematic or hierarchical annotations often provide one structured row per neuron, with fields such as cell_class, cell_sub_class, cell_type, hemibrain_type, side, and neurotransmitter-related fields.
• Community annotations are user-contributed tags associated with a neuron or location. They can be extremely useful evidence, but multiple tags, duplicated tags, outdated labels, and disagreements are all possible.

A community tag is not automatically a vote that overrides the selection field. Treat it as an observation with provenance: who applied it, when, at what location, and at what materialization context.

Pin the annotation release as well

Your prior manifest already fixed a CAVE materialization version. The fafbseg documentation makes an additional point: systematic annotation releases are technically separate from the segmentation, even though they are often updated to work with a particular materialization.

Therefore, add these fields to your project provenance:

systematic_annotation_source: fafbseg / flywire_annotations
systematic_annotation_version: <exact release or commit>
systematic_annotation_materialization_compatibility: <documented pairing, if known>
community_annotation_source: CAVE community annotation table
community_annotation_query_time_utc: <timestamp>

The documentation’s example states that annotation release v2.0.0 was based on materialization 783 at the time it was written. Do not infer that this pairing is appropriate for your own current materialization. Use the version information applicable to the release you selected.

A small annotation-audit cell

The following is deliberately an evidence collection cell, not an automated classifier. A script can retrieve fields and flag discrepancies, but it cannot decide whether two taxonomies mean the same biological class.

import pandas as pd
from fafbseg import flywire

# Set this to the exact annotation release you have chosen and recorded.
ANNOTATION_VERSION = "<your pinned annotation version>"
flywire.set_default_annotation_version(ANNOTATION_VERSION)

def first_value(frame, column):
    """Return the first non-null value from a result DataFrame."""
    if frame.empty or column not in frame.columns:
        return pd.NA

    values = frame[column].dropna()
    return values.iloc[0] if not values.empty else pd.NA

annotation_audit = []

for root_id_text in cohort["root_id"]:
    root_id = int(root_id_text)

    systematic = flywire.search_annotations(root_id)
    community = flywire.search_community_annotations(root_id)

    annotation_audit.append(
        {
            "root_id": str(root_id),
            "systematic_rows": len(systematic),
            "cell_type": first_value(systematic, "cell_type"),
            "hemibrain_type": first_value(systematic, "hemibrain_type"),
            "side": first_value(systematic, "side"),
            "morphology_group": first_value(systematic, "morphology_group"),
            "systematic_status": first_value(systematic, "status"),
            "community_annotation_rows": len(community),
        }
    )

annotation_audit = pd.DataFrame(annotation_audit)
display(annotation_audit)

For each root ID, compare the returned fields against:

• the original raw Codex export;
• the exact selection predicate in cohort_manifest.json;
• the materialization version printed or reported by the annotation query;
• any annotation-release compatibility notes you have recorded.

If the systematic query returns zero rows, multiple unexpected rows, a contradictory side, or an unfamiliar type field, do not improvise a correction. Mark the candidate as unresolved until you can explain the discrepancy from documented provenance.

A useful distinction is:

• consistent: the relevant structured fields directly meet the inclusion rule;
• incomplete but not contradictory: a selected field is missing, but other recorded provenance supports the original inclusion;
• contradictory: the returned metadata conflicts with the stated rule;
• ambiguous taxonomy: labels exist but require an interpretation you have not predeclared.

Only the first category should normally move straight to morphology review without a note.

---

Then: inspect morphology as evidence, not as decoration

For this capstone, do not attempt a complete proofread of every neuron. Instead, perform a bounded morphology validation pass on every included candidate. The purpose is to find errors large enough to make the neuron a questionable member of the cohort or to substantially distort its connectivity.

Use both views:

• 3D morphology is best for identifying global problems: a detached arbor, an implausible excursion, an abrupt bulge, an unexpected second tract, or a conspicuous gap in an otherwise coherent branching field.
• 2D serial EM imagery is the evidence for a local claim. It tells you whether the membrane boundary and continuation actually support the apparent 3D connection or break.

The 3D view generates hypotheses; the 2D imagery tests them.

A repeatable inspection pass

For a small cohort, inspect every neuron using the same sequence. The consistency matters more than the total number of minutes spent on any one neuron.

1. Open the candidate root ID in the fixed materialization context. Confirm that the displayed object corresponds to the reviewed ID and record the viewer location or root ID in your validation table.

2. Perform a coarse 3D scan. Rotate and zoom out. Look for disconnected-looking fragments, unusually long branches entering a different territory, missing-looking sectors of a broad arbor, or abrupt changes in caliber and branching pattern.

3. Choose a small set of review targets. These should include any visually suspicious location, a main tract or branch point, and—if relevant to your type definition—the soma or soma tract region. Do not select targets based on synapse counts; connectivity has not yet been analyzed.

4. Inspect each target through consecutive EM sections. Follow the same neurite before, at, and after the suspected boundary. Check whether the membrane contour and local context support continuity.

5. Return to 3D. Ask whether the local interpretation produces a coherent whole-cell morphology. If local and global evidence disagree, retain the case as unresolved rather than forcing a conclusion.

6. Place annotation points at meaningful evidence locations. Use a concise label such as candidate false merge, possible split, misalignment boundary, or reviewed intact tract. Export the points or record their coordinates with their coordinate system.

What warrants a closer look?

Common segmentation errors have recognizable visual signatures, but signatures are prompts for inspection, not conclusions.

Use the following interpretation guide during review:

Observation	Plausible concern	Evidence required before an exclusion
Two branches form an apparent X or cross	False merge	Serial sections show distinct neurites with a segmentation bridge that lacks membrane-supported continuity.
Two parallel neurites appear joined in the middle	H-shaped or parallel false merge	Inspect beyond the apparent junction; the true bridge may be displaced from its 3D appearance.
A thin terminal fragment extends into a dense neighboring arbor	Twig-to-twig merge	2D boundary evidence and an assessment that the fragment does not fit the target morphology.
A broad dendritic field has an unusual empty sector	Possible false split or missing branch	A physically plausible nearby continuation across several sections, not merely a shape that “looks incomplete.”
A neurite suddenly changes direction or joins a similar bundle after a poor image region	Possible path swap or image artifact	Continuity evidence across the image region; otherwise classify it as unresolved.
A soma tract is absent or detached	Possible missing thin extension	Search for a parallel thin process and inspect the EM, rather than assuming the closest fragment belongs to the neuron.

Misalignment is an uncertainty signal

Image shifts, blackout regions, and poor-quality sections can make both AI segmentation and manual interpretation unreliable. When this happens, record the problematic interval rather than treating the visible discontinuity as definite evidence of a split or merge.

A useful review note might read:

Morphology status: unresolved.
Concern: possible false split near the soma tract.
Evidence: target process terminates immediately before a two-section image misalignment; a plausible continuation exists afterward but cannot be linked confidently through the affected interval.
Next step: defer to targeted proofreading review; omit from primary cohort analysis.

That note is much more informative than either “bad neuron” or “looks wrong.”

---

Turn visual inspection into an auditable dataset

A screenshot alone is not enough evidence; neither is a bare coordinate. Preserve enough context that you—or another reviewer—can reconstruct why a decision was made.

Create two files:

data/derived/
├── cohort_validation.csv
└── morphology_observations.csv

Use one row per candidate in cohort_validation.csv. Use one row per reviewed location in morphology_observations.csv, because a single neuron can have several targets.

Candidate-level validation table

A practical schema is:

Field	Purpose
root_id	Candidate identifier, stored as text
annotation_version	Pinned systematic annotation release or commit
annotation_status	consistent, incomplete, contradictory, or ambiguous
annotation_evidence	Relevant raw fields and their values
morphology_status	reviewed_no_material_concern, concern_found, or uncertain
review_scope	What was checked, such as 3D scan plus three 2D targets
decision	included, excluded, or unresolved
analysis_role	primary, excluded, or sensitivity_only
decision_rationale	Concise explanation tied to evidence
reviewer and reviewed_at_utc	Audit provenance

Location-level observations

For every suspicious or affirmatively reviewed location, record:

Field	Example meaning
root_id	Parent candidate
observation_id	Stable local identifier, such as DA1L_03_obs_02
location_x, location_y, location_z	Coordinates exactly as displayed/exported
coordinate_space	The stated FlyWire coordinate convention or viewer context
issue_class	possible false merge, possible false split, image misalignment, or reviewed intact
evidence_2d	Serial sections inspected and the observed membrane/continuity evidence
evidence_3d	Whole-morphology feature that prompted or resolved the check
annotation_point_label	Label of any placed FlyWire annotation point
evidence_reference	Screenshot filename, exported annotation CSV row, or saved viewer-state reference
outcome	resolved intact, excluded concern, or unresolved

The coordinate-space field is essential. A triple of numbers without a declared coordinate convention is like a frontend bug report that says only “the button is over there.”

Here is a safe starting point for the candidate-level table:

from datetime import datetime, timezone
from pathlib import Path

validation = cohort[["root_id", "cell_type", "side"]].copy()

validation["annotation_version"] = ANNOTATION_VERSION
validation["annotation_status"] = "not_reviewed"
validation["annotation_evidence"] = pd.NA
validation["morphology_status"] = "not_reviewed"
validation["review_scope"] = pd.NA
validation["decision"] = "not_reviewed"
validation["analysis_role"] = "not_assigned"
validation["decision_rationale"] = pd.NA
validation["reviewer"] = "<your handle>"
validation["reviewed_at_utc"] = pd.NA

validation_path = Path("data/derived/cohort_validation.csv")
validation_path.parent.mkdir(parents=True, exist_ok=True)
validation.to_csv(validation_path, index=False)

display(validation)

After manual review, your finished table should contain no not_reviewed rows. For a candidate with no material concern, a concise completed record could be:

annotation_status: consistent
morphology_status: reviewed_no_material_concern
review_scope: 3D whole-cell scan; soma tract; main tract; two distal arbor targets
decision: included
analysis_role: primary
decision_rationale: Exported cell_type and side match predicate; no merge, split, or major missing-arbor concern observed in reviewed regions.

For an uncertain candidate:

annotation_status: consistent
morphology_status: uncertain
review_scope: 3D scan; suspect distal branch reviewed over 12 sections
decision: unresolved
analysis_role: sensitivity_only
decision_rationale: Putative continuation crosses an image-artifact interval; available EM evidence does not distinguish a false split from a true termination.

Do not delete excluded rows from cohort_validation.csv. The point of the file is to preserve the complete path from the original candidate list to the final analytical cohort.

---

Define the boundary of your claim

Before considering the validation pass complete, write the scope in the manifest or notebook:

Morphology review scope:
- Every manifest candidate received a whole-cell 3D scan.
- Each candidate received 2D serial-EM inspection at its main tract and at least two
  additional morphology targets, plus all visually suspicious sites.
- No edits were performed during cohort validation.
- Unresolved reconstruction concerns were excluded from the primary analysis and
  retained in a sensitivity-only list.

This boundary is scientifically useful because it says exactly what “validated” means in your project. It also prevents a later reader from assuming you performed exhaustive proofreading.

A final checklist for this lesson:

• Every manifest root ID has an annotation review result.
• The systematic annotation version is pinned and recorded.
• Community tags, if used, are recorded as supplementary evidence rather than silently merged into the selection rule.
• Every candidate has received the same bounded morphology-review pass.
• Each meaningful morphology concern has a coordinate, issue class, and 2D/3D evidence note.
• Exclusions remain visible in the validation table.
• Unresolved cases are retained with a stated role in the later analysis.
• The primary cohort is defined before any connectivity ranking is inspected.

---

Wrap-up

You have moved from a cohort that is merely queryable to one that is defensible. The key discipline is to keep annotation evidence, morphology evidence, and analytical decisions distinct:

• structured annotations justify membership under a stated rule;
• serial EM and 3D morphology test whether the reconstruction is suitable for the intended claim;
• exclusions and unresolved cases remain part of the project record rather than disappearing from it.

Next, you will use the validated primary cohort to build a directed connectivity table, calculate in-degree and out-degree summaries, and preserve the versioned context that makes those results reproducible.