Documenting a Neuron Cohort and Its Provenance

Hello again. In the previous lesson, you defined a bounded capstone question: a source population, a direction of connectivity, a synapse-based metric, and a reporting limit. This lesson turns that design into something a notebook can actually analyze: a frozen cohort manifest containing a small set of neuron root IDs, the version context in which those IDs are meaningful, the exact selection rule, and the source of every annotation used to include them.

For the worked example, continue with the question about the strongest postsynaptic partners of left DA1_lPN neurons. You can substitute another narrowly defined type later, but do not change criteria after inspecting connectivity results without recording the change.

---

A cohort is a versioned data object

A biological category such as “left DA1 lateral projection neurons” is not yet a computational cohort. To query synapses, you need an explicit set of root identifiers:

$$S=\{r_1,r_2,\dots ,r_n\}$$

Each $r_i$ identifies one reconstructed neuron at a particular state of the segmentation. Proofreading can split or merge objects, so a root ID should not be treated as a permanent biological identifier. It is closer to a versioned primary key in a database: useful and precise only when accompanied by its dataset and time context.

The key distinction is:

Term	What it fixes	Why you record it
Dataset	The underlying connectomics project, such as FlyWire FAFB	Prevents accidental mixing with BANC, MANC, or another dataset
Codex snapshot / release	A static published release used for search and annotation discovery	Makes the search result reproducible
Materialization version	A timestamped, internally consistent snapshot of queryable tables	Keeps IDs and synapse queries consistent
Root ID	One segmentation object within that versioned context	Defines the actual cohort membership
Annotation provenance	Where the inclusion labels came from and what they mean	Lets a reader assess how the cohort was constructed

The supplied figure illustrates why this discipline matters.

Your goal is not to identify an eternally immutable list of neurons. Your goal is to make a list that another person can recreate or audit for one stated release.

---

Pin the analysis to a materialization version

By default, query clients often select the most recent materialization. That is convenient for exploration but unsafe for a capstone: a later proofreading edit may alter root IDs or connectivity between two runs of the same notebook.

In your notebook, begin by explicitly choosing a version and recording its timestamp. The following is a compact setup cell; replace the datastack name only if your existing FlyWire setup uses a different one.

from caveclient import CAVEclient

DATASTACK = "flywire_fafb_public"

client = CAVEclient(DATASTACK)

available_versions = client.materialize.get_versions()
print(available_versions)

# Choose and record one available version for the whole capstone.
MAT_VERSION = 783  # Replace only after checking available_versions.
assert MAT_VERSION in available_versions

client.materialize.version = MAT_VERSION

materialization_timestamp = client.materialize.get_timestamp()
materialization_metadata = client.materialize.get_version_metadata(MAT_VERSION)

print("Version:", MAT_VERSION)
print("Timestamp:", materialization_timestamp)

Do not blindly use 783 just because it is shown here. It is a worked placeholder consistent with the example release naming, not a promise that it is currently available to your account or appropriate for your selected dataset.

Two important rules follow:

1. Set client.materialize.version once near the top of the notebook. All cohort validation and all later synapse queries must use that same version.
2. Do not substitute a live query using the current time for the final capstone analysis. Live queries have a legitimate role when inspecting recent proofreading, but they do not provide a stable common state for a multi-step analysis.

A Codex release and a CAVE materialization version may use similar version labels, but do not assume they are equivalent merely because their numbers look alike. Record both. If your chosen Codex snapshot is documented as corresponding to a CAVE materialization, note that relationship explicitly in the manifest.

---

Make the selection rule executable

Your previous question card used a biological description. Now express that description as a selection predicate with a defined annotation layer.

For the running example:

dataset: FlyWire FAFB
selection rule: cell_type == DA1_lPN && side == left
membership rule: include every returned row with a valid root_id

This is preferable to a hand-picked list because it states what would happen if someone reran the selection at the same release. It also makes exclusions visible rather than implicit.

Codex is suitable for constructing and exporting a small candidate cohort. It exposes both static releases and searchable annotation fields.

A practical Codex selection pass

In Codex:

1. Set the dataset to FlyWire FAFB and deliberately select the snapshot you intend to record.

2. Run the structured query:

   cell_type == DA1_lPN && side == left
   

3. Inspect the result count and a few rows. Confirm that cell_type and side contain the values your rule claims to use.

4. Download the CSV summary of the result table, not only a bare list of cell IDs. The CSV preserves useful annotation columns as evidence.

5. Save the unmodified export in your project under a name that embeds the snapshot, for example:

   data/raw/codex_DA1_lPN_left_snapshot_v783.csv
   

If the expected cell type does not produce a useful result, do not switch casually between cell_type, resolved_type, free-text labels, or fuzzy search. Each represents a different rule. Decide which field supports your biological definition, update the question card, and record that decision.

Annotation provenance: the minimum useful record

For a systematic annotation-based cohort, provenance should state:

• Source system: Codex / FlyWire annotation index.
• Dataset and static release: for example, FlyWire FAFB and the selected Codex snapshot.
• Annotation layer and fields: for example, cell_type and side.
• Predicate: the exact query text, including operators and spelling.
• Retrieval time: in UTC.
• Raw export: filename and ideally a checksum.
• Interpretation: whether the type is a consolidated/resolved classification, a direct field value, or a community-provided label.

If you use a community label, record that it is a community label and preserve the raw label evidence or attribution visible in the cell details. Do not quietly rewrite it as a systematic type.

---

Build the cohort artifact, not just a notebook variable

Use three small files. This separates raw source evidence, the analysis-ready membership table, and the explanation of how membership was derived.

capstone/
├── notebook.ipynb
├── data/
│   ├── raw/
│   │   └── codex_DA1_lPN_left_snapshot_vNNN.csv
│   └── derived/
│       └── cohort_DA1_lPN_left_vNNN.csv
├── cohort_manifest.json
└── decisions.csv

Keep root IDs as strings when reading and writing tables. They are large integer identifiers; spreadsheet software and JavaScript-based tools can silently round sufficiently large integers.

from pathlib import Path
import pandas as pd

raw_path = Path("data/raw/codex_DA1_lPN_left_snapshot_vNNN.csv")

raw = pd.read_csv(raw_path, dtype={"root_id": "string"})

# Inspect actual column names before selecting them.
print(raw.columns.tolist())

required = ["root_id", "cell_type", "side"]
missing = set(required) - set(raw.columns)
assert not missing, f"Missing expected columns: {missing}"

cohort = raw.loc[:, required].copy()
cohort["root_id"] = cohort["root_id"].str.strip()

assert cohort["root_id"].notna().all()
assert cohort["root_id"].str.fullmatch(r"\d+").all()
assert cohort["root_id"].is_unique, "Unexpected duplicate root IDs"

cohort["annotation_source"] = "Codex consolidated annotation index"
cohort["selection_status"] = "included"

derived_path = Path("data/derived/cohort_DA1_lPN_left_vNNN.csv")
derived_path.parent.mkdir(parents=True, exist_ok=True)
cohort.to_csv(derived_path, index=False)

cohort

Adapt the column mapping only if the downloaded CSV uses different names. If you do, record the mapping in the manifest. Do not rename a column such as resolved_type to cell_type without preserving the original field name.

Next, test that each selected root ID is valid at the materialization timestamp chosen for the analysis.

root_ids = [int(root_id) for root_id in cohort["root_id"]]

valid_at_materialization = client.chunkedgraph.is_latest_roots(
    root_ids,
    timestamp=materialization_timestamp
)

cohort["valid_at_materialization"] = valid_at_materialization

invalid = cohort.loc[~cohort["valid_at_materialization"]]
print(f"Valid roots: {cohort['valid_at_materialization'].sum()} / {len(cohort)}")

if not invalid.empty:
    display(invalid)
    raise ValueError(
        "At least one exported root ID is not valid at the chosen "
        "materialization timestamp. Reconcile the release/version context "
        "before querying connectivity."
    )

A failed validation is useful information, not an inconvenience to bypass. It often means the selected Codex snapshot and CAVE materialization have not been aligned, or that proofreading changed an object after the relevant snapshot. Record the issue in decisions.csv; do not replace IDs with newer ones without documenting why.

---

Write the manifest before analyzing synapses

The manifest is the cohort’s data contract. It should make the selection reproducible without forcing a reader to infer it from notebook code.

import hashlib
import json
from datetime import datetime, timezone

raw_sha256 = hashlib.sha256(raw_path.read_bytes()).hexdigest()

manifest = {
    "project": "Strongest postsynaptic partners of left DA1_lPN neurons",
    "dataset": "FlyWire FAFB",
    "codex_snapshot": "vNNN",
    "materialization_version": MAT_VERSION,
    "materialization_timestamp_utc": str(materialization_timestamp),
    "selection": {
        "source_system": "Codex",
        "annotation_layer": "consolidated annotation index",
        "query": "cell_type == DA1_lPN && side == left",
        "membership_rule": "All rows returned by the recorded query",
        "root_id_field": "root_id",
        "included_count": len(cohort),
    },
    "annotation_provenance": {
        "fields_used": ["cell_type", "side"],
        "raw_export": str(raw_path),
        "raw_export_sha256": raw_sha256,
        "retrieved_at_utc": datetime.now(timezone.utc).isoformat(),
    },
    "derived_cohort_file": str(derived_path),
    "root_ids_valid_at_materialization": True,
    "notes": [
        "Codex snapshot and CAVE materialization are recorded separately.",
        "No manual additions or removals were made after export."
    ],
}

with open("cohort_manifest.json", "w", encoding="utf-8") as f:
    json.dump(manifest, f, indent=2)

print(json.dumps(manifest, indent=2))

If you exclude a returned neuron because of a duplicated row, an invalid root ID, or a later morphology concern, do not erase it from history. Put the decision in decisions.csv:

root_id	decision	reason	evidence	reviewer
...	excluded	Invalid at chosen materialization	Validation cell output	your name or handle

At this stage, exclusions should be technical and explicit. The next lesson will address biological and reconstruction-based validation of the cohort, where morphology and annotation plausibility may justify further documented exclusions.

Checkpoint: what you should now possess

Before moving on, confirm that you have:

• one small cohort_*.csv with one row per included neuron;
• root IDs preserved as text;
• a selected CAVE materialization version and timestamp;
• a recorded Codex dataset and snapshot;
• an exact query predicate and named annotation fields;
• a raw result export that supports the cohort table;
• a manifest that connects all of those pieces;
• no unrecorded manual membership changes.

---

Wrap-up

A cohort is not merely “neurons of type X.” For a reproducible FlyWire analysis, it is a versioned list of root IDs plus a documented selection rule and annotation trail. Codex gives you a static, searchable release for cohort discovery, while a fixed CAVE materialization version provides the consistent query state required for subsequent synapse analysis.

Next, you will validate this cohort: inspect whether its annotations and reconstructed morphologies support inclusion, and document exclusions or unresolved cases before treating the group as a biological population.