Assessing Findings Affected by Data and Cohort Uncertainty

Welcome to the final capstone lesson. Your notebook now contains a selected and validated cohort, versioned connectivity queries, summary tables, and figures of its strongest directed connections. The remaining task is epistemic rather than graphical: state which findings are stable under reasonable changes in the data and analysis, and which are conditional.

In this lesson, you will turn the project from “a set of observed edges” into a reproducible result with explicit limitations. You will assess four sources of change: reconstruction uncertainty, annotation ambiguity, cohort-selection choices, and FlyWire materialization versions. Plan for about 40 minutes, including two focused readings and a sensitivity-analysis addition to your notebook.

---

Treat every finding as a conditional claim

A connectomics result is never simply “neuron A connects strongly to neuron B.” For this capstone, the defensible form is closer to:

In the selected FlyWire materialization, under the recorded synapse filters and cohort-selection rules, this source-target pair has one of the largest observed cohort-internal synapse counts.

That wording is not empty caution. Each clause points to something that could change:

Source of possible change	What can change	Example impact on your capstone
Reconstruction uncertainty	Which neurites, branches, or synapses belong to a neuron	A false split can divide one neuron's synapses across two roots
Annotation ambiguity	The label or type assigned to a root	“Strong input to type X” may become “strong input to a composite X/Y type”
Cohort selection	Which neurons count as members	A top-$K$ edge can disappear when a borderline member is excluded
Data-version change	Root IDs, synapse-to-root mappings, annotations, and tables tied to a snapshot	Re-running at a later materialization can change counts and rankings

The important distinction is between the observation and the conclusion drawn from it.

• An edge table reports counts under one defined query.
• A network figure ranks and displays some of those counts.
• A biological interpretation adds claims about neuron identity, type, circuit organization, or generality.

Uncertainty should be evaluated at the level of the claim. If an uncertain branch lies far from all relevant synapses, it may matter little for a claim about the cohort’s strongest edge. Conversely, one false merge at a highly connected endpoint can alter several rankings at once.

---

Calibrate what “stable” means

The following reading provides useful empirical context. It compares FlyWire and hemibrain connectivity and distinguishes technical factors from biological variability. Its numerical heuristics come from large-scale, cell-type-level comparisons, so do not convert them into universal pass-fail thresholds for your small, individual-neuron cohort. Instead, use them to avoid over-interpreting small differences.

The comparison workflow below illustrates why type labels and reconstructions are not fixed facts attached permanently to an ID. Matching begins with morphology, is manually reviewed, and can be revised when connectivity evidence is considered.

For your project, use three practical categories of finding:

1. Robust within the tested scope: the result survives your conservative sensitivity scenario, such as excluding all unresolved cohort members.
2. Conditional: the result changes meaningfully under a documented, plausible alternative. It can still be reported, but must name the condition.
3. Unresolved: the result depends on an unreviewed reconstruction, ambiguous annotation, unavailable comparison version, or another issue you cannot yet test.

“Robust” does not mean biologically universal or error-free. It means the result survived the particular alternatives you actually documented and tested.

---

Four ways your conclusion can change

1. Reconstruction uncertainty changes the identity behind a root

A segmentation gives an algorithmic partition of EM imagery; proofreading improves that partition but does not make every remaining branch equally certain.

A false split assigns parts of one biological neuron to separate roots. In the connectivity table, its synapses may be divided across those roots, making each apparent partner relationship smaller than it should be. A false merge combines material from separate neurons under one root, potentially creating inflated or spurious connectivity. A missing branch can remove synapses; an uncertain small terminal may have little effect unless it contains relevant synapses.

This makes an evidence record from Module 2 valuable now. For every cohort member, ask:

• Is there an unresolved false split, false merge, or missing branch?
• Does that uncertainty involve a branch containing relevant input or output synapses?
• Was the neuron included because its morphology was sufficient for the cohort rule, or because it was fully proofread?
• Would excluding this neuron be a reasonable conservative alternative?

Do not invent a corrected synapse count for an unresolved reconstruction. Use one of two honest approaches:

• Strict scenario: exclude affected neurons and test whether the headline result survives.
• Rerun scenario: after a supported edit or a later materialization, rerun the original data-extraction pipeline and compare the rebuilt result.

The strict scenario measures sensitivity to membership uncertainty. It does not simulate the exact biological reconstruction that a future correction will produce.

2. Annotation ambiguity changes naming and grouping claims

Connectivity counts are attached to roots and synapse locations, whereas a label such as type, cell_class, or a community annotation is metadata. These are related, but they are not the same kind of evidence.

If your cohort consists of explicitly named individual roots, a label revision may leave the edge table completely unchanged while altering a sentence such as “type X preferentially targets type Y.” If the cohort was selected because of a label, annotation ambiguity can also alter membership and therefore edge counts.

Keep these cases separate in your report:

Situation	What remains unchanged	What may change
Root IDs are fixed; labels are revised	Individual source-target synapse counts	Type names, grouping, and type-level interpretation
Cohort selected by a type annotation	Counts among retained roots	Membership, totals, ranked edges, and type-level conclusions
A label becomes a composite type	Root-level graph	Whether the result supports a specific subtype claim
A neuron is relabelled or unresolved	Other validated roots	Claims that generalize from that neuron to a named class

A good capstone phrasing is: “Among the roots annotated as type X in the recorded annotation source and materialization…” This is much more accurate than treating the label as timeless ground truth.

3. Cohort selection changes the graph you decided to analyze

Cohort selection is an analysis decision, not merely a preprocessing detail. Small cohorts are particularly sensitive because each member can contribute a large fraction of the possible source-target pairs.

For example, if the strongest displayed edge touches one borderline neuron, excluding that root may remove the edge entirely. If your conclusion instead concerns a repeated pattern among several well-validated members, it may survive unchanged.

Use a small set of purposeful scenarios rather than an uncontrolled collection of alternatives:

Scenario	Membership rule	What it tests
Baseline	Your recorded validated primary cohort	The reported capstone result
Strict	Baseline minus roots with unresolved reconstruction or identity concerns	Dependence on uncertain members
Inclusive	Baseline plus explicitly documented plausible members	Dependence on a narrow inclusion boundary
Label alternative	Rebuild membership under a credible alternate annotation interpretation	Dependence on type-label decisions

Only run an inclusive scenario when you can define it before viewing its desired outcome. If the added neurons were not included in the original edge table, rerun the original synapse query for the expanded cohort rather than treating absent rows as zero connectivity.

4. Materialization versions change the data snapshot

FlyWire root IDs are not permanent biological names. They identify a collection of supervoxels at a particular segmentation state. A merge or split creates new roots and invalidates the prior root or roots. This is why version control is central to connectome analysis rather than administrative bookkeeping.

A materialization is a coherent snapshot in which synapse coordinates and annotations have been mapped to roots. Mixing roots that never coexisted at one materialization can produce invalid metadata queries or misleading comparisons.

If a later materialization is available, a proper version comparison has four stages:

1. Resolve the selected coordinates or old roots against the new snapshot.
2. Reapply the original cohort-selection criteria, recording any identity or label changes.
3. Rerun the synapse query and aggregation with the same filters.
4. Compare the resulting edge tables, rankings, and conclusions.

Do not compare a new root ID to an old edge count as though only the identifier changed. A segmentation edit can change which synapses map to the reconstructed neuron.

---

Add an uncertainty ledger to the capstone notebook

Create one compact table that connects proofreading evidence and annotation decisions to the final analysis. This is more useful than a vague “limitations” paragraph because it records what would trigger a rerun.

from pathlib import Path
import pandas as pd

output_dir = Path("data/derived")

uncertainty_ledger = pd.DataFrame(
    columns=[
        "issue_id",
        "category",
        "affected_root_ids",
        "status",
        "evidence_reference",
        "plausible_alternative",
        "expected_effect_on_findings",
        "decision_or_next_step",
    ]
)

# Add one row for each real unresolved or conditional case.
# Example categories:
# "reconstruction", "annotation", "cohort_selection", "version"
#
# Keep affected_root_ids as semicolon-separated root IDs so that
# the table remains easy to inspect and parse later.

uncertainty_ledger.to_csv(
    output_dir / "cohort_uncertainty_ledger.csv",
    index=False,
)

uncertainty_ledger

A useful record is specific enough that another analyst can act on it. For example:

Field	Appropriate content
category	reconstruction
affected_root_ids	The root or roots actually affected
status	unresolved, reviewed, or version-dependent
evidence_reference	Your proofreading case note, annotation source, or query provenance
plausible_alternative	Exclude root; use composite label; rerun at a later materialization
expected_effect_on_findings	“May remove strongest edge rank 2” rather than “could affect results”
decision_or_next_step	“Include only in baseline; exclude in strict scenario”

The ledger should include ordinary decisions as well as problems. For instance, the choice to exclude autapses from a visualization but retain them in the canonical edge table is a documented scope choice, even if it does not indicate a data error.

---

Run a conservative cohort sensitivity analysis

The following code compares your baseline cohort with a strict scenario that excludes roots you have recorded as unresolved. It works with the capstone files created in earlier lessons and only uses the already queried canonical edge table.

Before running it, fill UNCERTAIN_ROOTS using the IDs in your real uncertainty ledger. An empty set is legitimate if your validation found no unresolved cohort members, but do not use it merely to avoid testing a difficult result.

import json
import pandas as pd

edge_table = pd.read_csv(
    output_dir / "cohort_connectivity_edges.csv",
    dtype={
        "source_root_id": "string",
        "target_root_id": "string",
    },
)

validation = pd.read_csv(
    output_dir / "cohort_validation.csv",
    dtype={"root_id": "string"},
)

provenance = json.loads(
    (output_dir / "cohort_connectivity_provenance.json").read_text()
)

primary = validation.loc[
    (validation["decision"] == "included")
    & (validation["analysis_role"] == "primary")
].copy()

baseline_members = set(primary["root_id"].dropna().astype(str))

# Copy unresolved reconstruction or identity cases from your ledger.
# Leave the set empty only if there are genuinely no such cases.
UNCERTAIN_ROOTS = {
    # "720575940000000000",
}

if not UNCERTAIN_ROOTS.issubset(baseline_members):
    unknown = UNCERTAIN_ROOTS - baseline_members
    raise ValueError(
        "Every strict-scenario exclusion must be a baseline cohort member: "
        f"{sorted(unknown)}"
    )

strict_members = baseline_members - UNCERTAIN_ROOTS

print(f"Dataset: {provenance['dataset']}")
print(f"Materialization: {provenance['materialization_version']}")
print(f"Baseline members: {len(baseline_members)}")
print(f"Strict members: {len(strict_members)}")
print(f"Excluded unresolved members: {len(UNCERTAIN_ROOTS)}")

Next, calculate the strongest displayed edges under each membership rule. This code keeps the previous lesson’s convention: exclude autapses from the strongest-edge display, retain all ties at the cutoff, and preserve the canonical table as the full record.

TOP_K = 15

def select_top_edges_with_ties(edges, members, top_k=15):
    """Return displayed non-autapse edges internal to one scenario."""
    scoped = edges.loc[
        edges["source_root_id"].isin(members)
        & edges["target_root_id"].isin(members)
    ].copy()

    non_autapse = scoped.loc[
        scoped["source_root_id"] != scoped["target_root_id"]
    ].copy()

    non_autapse = non_autapse.sort_values(
        ["synapse_count", "source_root_id", "target_root_id"],
        ascending=[False, True, True],
        kind="stable",
    ).reset_index(drop=True)

    if non_autapse.empty:
        return non_autapse.assign(display_rank=pd.Series(dtype="int64"))

    requested = min(top_k, len(non_autapse))
    cutoff = int(non_autapse.iloc[requested - 1]["synapse_count"])

    shown = non_autapse.loc[
        non_autapse["synapse_count"] >= cutoff
    ].copy()

    shown.insert(0, "display_rank", range(1, len(shown) + 1))
    return shown


def summarize_scenario(edges, members, top_k=15):
    scoped = edges.loc[
        edges["source_root_id"].isin(members)
        & edges["target_root_id"].isin(members)
    ].copy()

    shown = select_top_edges_with_ties(scoped, members, top_k=top_k)

    outgoing = (
        scoped.groupby("source_root_id", as_index=False)["synapse_count"]
        .sum()
        .rename(
            columns={
                "source_root_id": "root_id",
                "synapse_count": "outgoing_synapses",
            }
        )
        .sort_values("outgoing_synapses", ascending=False)
    )

    incoming = (
        scoped.groupby("target_root_id", as_index=False)["synapse_count"]
        .sum()
        .rename(
            columns={
                "target_root_id": "root_id",
                "synapse_count": "incoming_synapses",
            }
        )
        .sort_values("incoming_synapses", ascending=False)
    )

    return {
        "member_count": len(members),
        "edge_count": len(scoped),
        "total_internal_synapses": int(scoped["synapse_count"].sum()),
        "shown_edges": shown,
        "top_outgoing": outgoing.head(5),
        "top_incoming": incoming.head(5),
    }


baseline_result = summarize_scenario(
    edge_table,
    baseline_members,
    top_k=TOP_K,
)

strict_result = summarize_scenario(
    edge_table,
    strict_members,
    top_k=TOP_K,
)

scenario_overview = pd.DataFrame(
    [
        {
            "scenario": "baseline",
            "member_count": baseline_result["member_count"],
            "directed_edge_count": baseline_result["edge_count"],
            "total_internal_synapses": (
                baseline_result["total_internal_synapses"]
            ),
            "displayed_strong_edges": len(baseline_result["shown_edges"]),
        },
        {
            "scenario": "strict",
            "member_count": strict_result["member_count"],
            "directed_edge_count": strict_result["edge_count"],
            "total_internal_synapses": (
                strict_result["total_internal_synapses"]
            ),
            "displayed_strong_edges": len(strict_result["shown_edges"]),
        },
    ]
)

scenario_overview

Compare the identities of displayed strong edges, not just the total number of synapses. A total may remain similar while a headline pair changes.

def edge_keys(frame):
    return {
        (str(row.source_root_id), str(row.target_root_id))
        for row in frame.itertuples(index=False)
    }

baseline_keys = edge_keys(baseline_result["shown_edges"])
strict_keys = edge_keys(strict_result["shown_edges"])

retained = baseline_keys & strict_keys
baseline_only = baseline_keys - strict_keys
strict_only = strict_keys - baseline_keys

union = baseline_keys | strict_keys
jaccard = len(retained) / len(union) if union else 1.0

stability_report = {
    "baseline_displayed_edges": len(baseline_keys),
    "strict_displayed_edges": len(strict_keys),
    "retained_displayed_edges": len(retained),
    "baseline_only_edges": len(baseline_only),
    "strict_only_edges": len(strict_only),
    "displayed_edge_jaccard_similarity": round(jaccard, 3),
}

print(stability_report)

if baseline_only:
    print("\nEdges shown only in the baseline scenario:")
    print(sorted(baseline_only))

if strict_only:
    print("\nEdges shown only in the strict scenario:")
    print(sorted(strict_only))

print("\nBaseline top outgoing roots:")
display(baseline_result["top_outgoing"])

print("\nStrict top outgoing roots:")
display(strict_result["top_outgoing"])

Interpret this output according to the kind of change you observe:

• If the main strong edges and leading input/output roots remain visible in the strict scenario, report them as robust within the tested cohort uncertainty.
• If a finding disappears only because an excluded root was one endpoint, report the finding as conditional on that root’s inclusion and reconstruction.
• If the ordering shifts among similarly weighted edges, avoid strong rank language such as “the dominant connection.” Prefer “among the strongest observed connections.”
• If an issue changes the overall qualitative pattern, such as the leading source or target, elevate it from a footnote to a central limitation.

Save the scenario outputs beside your original result:

baseline_result["shown_edges"].to_csv(
    output_dir / "scenario_baseline_strong_edges.csv",
    index=False,
)

strict_result["shown_edges"].to_csv(
    output_dir / "scenario_strict_strong_edges.csv",
    index=False,
)

scenario_overview.to_csv(
    output_dir / "cohort_sensitivity_overview.csv",
    index=False,
)

pd.DataFrame(
    [
        {
            "source_root_id": source,
            "target_root_id": target,
            "status": "retained",
        }
        for source, target in sorted(retained)
    ]
    + [
        {
            "source_root_id": source,
            "target_root_id": target,
            "status": "baseline_only",
        }
        for source, target in sorted(baseline_only)
    ]
    + [
        {
            "source_root_id": source,
            "target_root_id": target,
            "status": "strict_only",
        }
        for source, target in sorted(strict_only)
    ]
).to_csv(
    output_dir / "cohort_sensitivity_edge_comparison.csv",
    index=False,
)

---

Audit version dependence without mixing snapshots

If fafbseg is available in your notebook environment, use the following audit to record whether your baseline IDs are current and to save their current lineage mapping. Ensure that the FlyWire dataset has already been configured exactly as in your earlier query notebook.

from fafbseg import flywire

materialization = provenance["materialization_version"]
root_ids = sorted(baseline_members)

available_materializations = flywire.get_materialization_versions()

root_status = pd.DataFrame(
    {
        "root_id": root_ids,
        "is_current_at_pinned_materialization": flywire.is_latest_root(
            root_ids,
            timestamp=f"mat_{materialization}",
        ),
        "is_current_now": flywire.is_latest_root(root_ids),
    }
)

current_id_mapping = flywire.update_ids(root_ids)

root_status.to_csv(
    output_dir / "cohort_root_status_audit.csv",
    index=False,
)

current_id_mapping.to_csv(
    output_dir / "cohort_root_id_current_mapping.csv",
    index=False,
)

print("Available materializations:")
display(available_materializations)

print("Root status relative to the pinned materialization and current state:")
display(root_status)

print("Current ID mapping:")
display(current_id_mapping)

This audit supports two different conclusions:

• Your pinned materialization remains the correct reference for the published capstone output, because all original tables and figures were queried consistently against it.
• Some roots may have changed since that snapshot, which means a future update analysis requires a full re-query, not an ID substitution.

If no alternative public materialization is available, say so explicitly:

Version sensitivity was audited through root-status and lineage checks. A direct materialization comparison was not performed because no suitable second public materialization was available for the selected analysis scope.

That is an accurate limitation, not a failed analysis.

---

Write the limitations and stability statement

Add a final Markdown section to your notebook. It should distinguish what you tested from what remains possible but untested.

A concise template is:

Scope and stability. Connectivity was calculated for the validated primary cohort using FlyWire dataset $D$, materialization $M$, and the recorded synapse filters. The strict sensitivity analysis excluded $N$ roots with unresolved reconstruction or identity concerns. The principal finding [was retained / changed] under this scenario: [state the concrete result]. Annotation-based interpretations are conditional on the recorded annotation source and label definitions. Root-status checks showed [state result]; therefore, results should not be assumed to transfer unchanged to a later materialization without rerunning cohort resolution and synapse queries.

Then add an issue-specific list:

• Reconstruction: identify which roots were uncertain, which candidate edges they affected, and whether strict exclusion changed the finding.
• Annotation: state whether labels affected only interpretation or also membership.
• Cohort selection: name the baseline and alternative membership rules.
• Version: state the exact materialization, root-status audit result, and whether another materialization was compared.

Avoid vague statements such as “results may change with future data.” Every result may change. Your ledger and scenario outputs let you say how it could change, which conclusions are exposed, and what rerun would resolve the issue.

---

Wrap-up

You have completed the capstone’s final scientific safeguard: evaluating the stability of findings rather than presenting a single graph as definitive.

Key takeaways:

• A FlyWire connectivity result is conditional on a reconstruction state, annotations, cohort definition, query filters, and materialization version.
• Reconstruction uncertainty can alter root membership and synapse assignment; annotation ambiguity can alter labels, groupings, and sometimes cohort membership.
• A strict exclusion scenario is a practical way to test whether headline findings depend on unresolved members.
• Root-ID updates provide lineage information, but version comparisons require rebuilding the analysis from a coherent materialized snapshot.
• Your final notebook should include an uncertainty ledger, scenario outputs, version audit, and a precise stability statement.

With the provenance files, validation records, connectivity tables, figures, sensitivity analysis, and limitations now packaged together, your project is a reproducible FlyWire cohort-connectomics analysis rather than just a one-time visualization.