Creating a Comprehensive Dataset Provenance Manifest

Good to see you again. In the previous lesson, you inspected what your files say about themselves: CRS, bounds, grids, resolution, declared NoData, and validity masks. That inspection is evidence. A provenance manifest is where you preserve that evidence alongside information a file usually cannot supply on its own, such as its authoritative source, license, and product version.

This closes the first module’s data-intake workflow. By the end of the lesson, you will have a version-controlled YAML manifest and a small script that records observed spatial metadata for every raw input to an ecological mapping project. That includes field observations, boundaries, terrain rasters, and eventually satellite-image assets.

---

Provenance: an auditable record, not a file inventory

A file path such as data/raw/canopy_cover.tif tells a collaborator where a file happened to be on your machine. It does not tell them:

• who produced it;
• whether it was a particular release or a silently revised download;
• what rights govern reuse;
• whether “30” means 30 m or 30 degrees;
• whether a value of 0, 255, or -9999 represents data or missingness.

A provenance manifest answers those questions in one reviewable project artifact. Its purpose is not bureaucratic completeness. It lets you trace a model result back to the exact inputs and assumptions that produced it.

For a predictor or response dataset, preserve three kinds of information:

Kind of information	Examples	How it is obtained
Source claims	Provider, product identifier, version, acquisition date, license	Product landing page, metadata record, field-data documentation
Observed file facts	CRS, bounds, dimensions, raster transform, declared NoData, mask flags	GeoPandas, Rasterio, or format-specific metadata
Project interpretation	Intended role, relevant bands or fields, NoData treatment	Your documented modeling decisions

Keeping these categories separate matters. Rasterio can report that a GeoTIFF declares -9999 as NoData, but it cannot establish whether the provider’s documentation says that cloudy pixels were already removed. Conversely, a product web page can describe a 30 m product, but only inspection confirms the grid and CRS of the exact file you downloaded.

A completed manifest should be specific enough to reproduce the input set, yet concise enough that someone can actually audit it.

---

Decide what counts as one input

The right unit of provenance is the logical input you could replace independently.

For example:

• A GeoPackage of amphibian survey observations is one input.
• A DEM GeoTIFF is one input.
• A three-band multispectral scene may be one logical input with several listed assets, or three inputs if the bands are downloaded and managed independently.
• A Shapefile is a multi-file input: its .shp, .shx, .dbf, and .prj files form one dataset and must remain together.

For this course, use one record per raw vector layer, raster file, imagery scene, or other independently sourced data product. Derived layers, such as slope calculated from a DEM, should not be presented as if they were raw source data. Later, you will document those as derived products with a link back to their source DEM and the code that created them.

A useful distinction is:

1. Raw input: obtained from an external producer or an original field-data workflow.
2. Intermediate data: created inside this project, such as aligned predictors or extracted observation values.
3. Output: a trained model, validation predictions, or a wall-to-wall prediction raster.

This lesson creates the manifest for category 1. Version control and later workflow records will provide the lineage for categories 2 and 3.

---

Required fields and their ecological meaning

Every raw-input record should include the following.

Field	What to record	Important distinction
id	Stable project identifier, such as elevation_dem_2023	Use a meaningful ID rather than a generic filename.
role	Response observations, prediction boundary, continuous predictor, categorical predictor, imagery	Makes the model’s input logic reviewable.
source	Provider, landing page or catalog record, source asset identifier	A local path is not a source.
version_or_acquisition	Product version, release date, or observation/acquisition date or range	A download date is not the same as acquisition date.
license	License name or SPDX identifier, terms URL, verification status	“Publicly downloadable” does not establish permission to reuse or redistribute.
crs	Native CRS reported by the file, ideally an EPSG code plus any necessary WKT	Record the native CRS, not the future analysis CRS.
resolution_or_support	Raster x/y pixel size and native units; point, polygon, or line support for vectors	A point layer has no raster pixel resolution. Its coordinate accuracy or sampling support is different information.
nodata_convention	Declared sentinel values, validity-mask behavior, and documented interpretation	Do not equate NoData with zero without evidence.
sha256	Cryptographic fingerprint of the downloaded local file	Identifies the exact bytes used even if a provider later changes a file.

Dates are not interchangeable

Spatial ecological data commonly have several dates. Record their meanings explicitly:

• Acquisition date: when a satellite observed the Earth.
• Observation period: when field observations were collected.
• Product-generation or release date: when a provider processed or issued a product.
• Download date: when your project retrieved a copy.
• Inspection date: when your script read the local metadata.

For example, a Landsat scene acquired in June might have been processed in July and downloaded by your project in August. A habitat model can be sensitive to that distinction: an observation from 2024 paired with a canopy raster derived from 2016 imagery may represent a real temporal mismatch, not a minor metadata detail.

---

NoData needs both a machine-readable and a human-readable record

Your previous inspection script showed that NoData can be represented by declared band values, masks, NaN, or some combination. A strong manifest records both:

1. Observed convention: what Rasterio reports in the local file.
2. Documented meaning: what the product documentation says those invalid values represent.
3. Planned treatment: what this project will do with them.

For a continuous canopy-cover predictor, the manifest might eventually state:

nodata_convention:
  observed:
    declared_by_band:
      band_1: -9999.0
    validity_mask: "MaskFlags.nodata"
  documented_meaning: "Pixels outside the mapped terrestrial domain."
  planned_treatment: "Treat as missing; exclude affected prediction cells and never replace with zero."

For a categorical land-cover layer, 0 or 255 might be an actual class code, a background code, or a missing-value sentinel. The manifest must state which one it is based on the class legend or product documentation. This prevents a later preprocessing step from turning an omitted category into an apparently meaningful land-cover class.

For a vector observation dataset, NoData is different. Record how missing geometry, missing target values, censored counts, and unknown survey effort are encoded in its relevant fields. The next module will address how those conditions affect model training; for now, ensure they are not invisible.

---

A practical YAML manifest

Create data/provenance.yml. YAML is a good fit because it is readable in a code review, supports nested metadata, and works cleanly with Python. JSON would also work; the important property is that the file is committed to version control alongside the configuration and code.

Start with this template. Values written as REPLACE_... or UNKNOWN are deliberate prompts: replace them with verified information before treating the input as ready for a defensible analysis.

manifest_schema_version: 1
project: "ecological_mapping"

datasets:
  - id: "amphibian_surveys"
    kind: "vector"
    role: "response observations"
    local_path: "data/raw/amphibian_surveys.gpkg"

    source:
      provider: "REPLACE_WITH_DATA_CUSTODIAN"
      landing_page: "REPLACE_WITH_CITATION_OR_LANDING_PAGE"
      source_identifier: "REPLACE_WITH_DATASET_OR_EXPORT_IDENTIFIER"

    version_or_acquisition:
      version: "UNKNOWN"
      acquisition_start: "REPLACE_WITH_FIRST_SURVEY_DATE"
      acquisition_end: "REPLACE_WITH_LAST_SURVEY_DATE"
      downloaded_at: "REPLACE_WITH_ISO_8601_TIMESTAMP"

    license:
      name_or_spdx: "UNKNOWN"
      terms_url: "REPLACE_WITH_LICENSE_URL"
      status: "unverified"

    spatial_support:
      geometry_type: "point"
      resolution: "not applicable"
      positional_accuracy: "UNKNOWN"

    nodata_convention:
      documented_meaning: "Missing geometry or missing response values require review."
      planned_treatment: "Do not use records with missing geometry or target values for model fitting."

    tracked_fields:
      response: "REPLACE_WITH_RESPONSE_FIELD"
      observation_id: "REPLACE_WITH_STABLE_ID_FIELD"

  - id: "elevation_dem"
    kind: "raster"
    role: "continuous predictor"
    local_path: "data/raw/predictors/elevation.tif"

    source:
      provider: "REPLACE_WITH_PROVIDER"
      landing_page: "REPLACE_WITH_PRODUCT_PAGE_OR_CITATION"
      source_identifier: "REPLACE_WITH_TILE_OR_PRODUCT_ID"

    version_or_acquisition:
      version: "REPLACE_WITH_PRODUCT_VERSION_OR_UNKNOWN"
      acquisition_start: "UNKNOWN"
      acquisition_end: "UNKNOWN"
      downloaded_at: "REPLACE_WITH_ISO_8601_TIMESTAMP"

    license:
      name_or_spdx: "REPLACE_WITH_LICENSE"
      terms_url: "REPLACE_WITH_LICENSE_URL"
      status: "unverified"

    nodata_convention:
      documented_meaning: "REPLACE_AFTER_CHECKING_PRODUCT_DOCUMENTATION"
      planned_treatment: "Treat invalid pixels as missing; do not substitute zero."

  - id: "canopy_cover"
    kind: "raster"
    role: "continuous predictor"
    local_path: "data/raw/predictors/canopy_cover.tif"

    source:
      provider: "REPLACE_WITH_PROVIDER"
      landing_page: "REPLACE_WITH_PRODUCT_PAGE_OR_CITATION"
      source_identifier: "REPLACE_WITH_TILE_OR_PRODUCT_ID"

    version_or_acquisition:
      version: "REPLACE_WITH_PRODUCT_VERSION_OR_UNKNOWN"
      acquisition_start: "REPLACE_WITH_IMAGE_OR_COMPOSITE_DATE_RANGE"
      acquisition_end: "REPLACE_WITH_IMAGE_OR_COMPOSITE_DATE_RANGE"
      downloaded_at: "REPLACE_WITH_ISO_8601_TIMESTAMP"

    license:
      name_or_spdx: "REPLACE_WITH_LICENSE"
      terms_url: "REPLACE_WITH_LICENSE_URL"
      status: "unverified"

    nodata_convention:
      documented_meaning: "REPLACE_AFTER_CHECKING_PRODUCT_DOCUMENTATION"
      planned_treatment: "Treat invalid pixels as missing; do not substitute zero."

A temporary UNKNOWN is preferable to a confident fabrication. But it is not a final answer. Before a dataset supports a published result or a portfolio project, resolve its source, license, and version or acquisition information, or explicitly justify why the data cannot be used.

Notice that the manifest does not yet state a CRS, raster resolution, checksum, or mask flags. Those values will be inserted from the local files rather than typed by hand.

---

Automate the facts that the file can prove

Create src/update_provenance.py. The script below reads the hand-authored source information, inspects each local file, calculates a SHA-256 digest, and writes observed properties back into the same manifest.

from datetime import datetime, timezone
from hashlib import sha256
from pathlib import Path

import geopandas as gpd
import rasterio
import yaml


PROJECT_ROOT = Path(__file__).resolve().parents[1]
MANIFEST_PATH = PROJECT_ROOT / "data" / "provenance.yml"


def utc_now():
    return datetime.now(timezone.utc).replace(microsecond=0).isoformat()


def sha256_file(path, chunk_size=1024 * 1024):
    """Return a SHA-256 digest without loading the whole file into memory."""
    digest = sha256()

    with path.open("rb") as file:
        while chunk := file.read(chunk_size):
            digest.update(chunk)

    return digest.hexdigest()


def raster_units(crs):
    if crs is None:
        return None

    if crs.is_geographic:
        return "degrees"

    return "native projected CRS units"


def invalid_fraction(src, band):
    """Calculate invalid-pixel fraction block by block."""
    invalid_pixels = 0
    total_pixels = 0

    for _, window in src.block_windows(band):
        mask = src.read_masks(band, window=window)
        invalid_pixels += int((mask == 0).sum())
        total_pixels += mask.size

    if total_pixels == 0:
        return None

    return invalid_pixels / total_pixels


def inspect_raster(path):
    with rasterio.open(path) as src:
        return {
            "format": src.driver,
            "crs": src.crs.to_string() if src.crs else None,
            "bounds": [float(value) for value in src.bounds],
            "width": src.width,
            "height": src.height,
            "band_count": src.count,
            "data_types": list(src.dtypes),
            "resolution": {
                "x": float(src.res[0]),
                "y": float(src.res[1]),
                "unit": raster_units(src.crs),
            },
            "transform": [float(value) for value in src.transform],
            "nodata_observed": {
                "declared_by_band": list(src.nodatavals),
                "mask_flags_by_band": [
                    [str(flag) for flag in flags]
                    for flags in src.mask_flag_enums
                ],
                "invalid_fraction_by_band": {
                    f"band_{band}": invalid_fraction(src, band)
                    for band in src.indexes
                },
            },
        }


def inspect_vector(path):
    gdf = gpd.read_file(path)

    return {
        "crs": gdf.crs.to_string() if gdf.crs else None,
        "feature_count": int(len(gdf)),
        "geometry_types": sorted(
            gdf.geom_type.dropna().astype(str).unique().tolist()
        ),
        "bounds": [float(value) for value in gdf.total_bounds],
        "missing_geometry_count": int(gdf.geometry.isna().sum()),
        "empty_geometry_count": int(gdf.geometry.is_empty.sum()),
        "resolution_or_support": (
            "Vector geometry; inspect the declared spatial support "
            "and positional accuracy in the hand-authored record."
        ),
    }


def inspect_dataset(entry):
    local_path = PROJECT_ROOT / entry["local_path"]

    if not local_path.exists():
        raise FileNotFoundError(
            f"Manifest path does not exist: {local_path}"
        )

    if entry["kind"] == "raster":
        observed = inspect_raster(local_path)
    elif entry["kind"] == "vector":
        observed = inspect_vector(local_path)
    else:
        raise ValueError(
            f"Unsupported kind for {entry['id']}: {entry['kind']}"
        )

    observed["file_bytes"] = local_path.stat().st_size
    observed["sha256"] = sha256_file(local_path)
    observed["inspected_at"] = utc_now()

    return observed


def main():
    with MANIFEST_PATH.open("r", encoding="utf-8") as file:
        manifest = yaml.safe_load(file)

    dataset_ids = [entry["id"] for entry in manifest["datasets"]]

    if len(dataset_ids) != len(set(dataset_ids)):
        raise ValueError("Every dataset id in the manifest must be unique.")

    for entry in manifest["datasets"]:
        entry["observed"] = inspect_dataset(entry)
        print(f"Recorded observed metadata: {entry['id']}")

    with MANIFEST_PATH.open("w", encoding="utf-8") as file:
        yaml.safe_dump(
            manifest,
            file,
            sort_keys=False,
            allow_unicode=False,
        )


if __name__ == "__main__":
    main()

Run it from the project root:

python src/update_provenance.py

After running, inspect the resulting YAML rather than assuming the script has made the records correct. For a raster in a geographic CRS, a resolution such as 0.0027 will properly be labelled as degrees. That is the truth about the raw file, even if its product documentation describes an approximate metre-scale ground sampling distance.

The script also preserves a key discipline:

• Python populates observed metadata.
• You verify provenance claims and interpretation from authoritative documentation.
• The manifest records both.

Do not edit an input file merely to make its observed metadata look consistent. If the manifest reveals a missing CRS, unexpected NoData fraction, or uncertain license, retain that discrepancy and investigate the data source.

For multi-file formats, adapt the checksum logic. A GeoPackage is one file, so its SHA-256 digest is straightforward. For a Shapefile, calculate and record a digest for every required component, or archive the complete input deterministically before hashing it.

---

STAC as a provenance pattern for satellite data

You will work directly with STAC catalogs in the satellite-imagery module. For now, recognize why STAC is useful for provenance: it formalizes the relationship among catalogs, collections, individual acquisitions, and downloadable assets.

If a satellite source provides STAC metadata, preserve at least:

• the collection ID;
• the item ID for the acquisition;
• the acquisition datetime;
• the asset name or band identifier;
• the source catalog or item URL;
• the native CRS, ground sampling distance, and relevant masking conventions.

An expiring signed asset URL is useful for downloading, but it is not a durable citation. The catalog, collection, and item identifiers are the durable provenance anchors.

---

Implementation checkpoint

Complete this small intake routine for the files currently listed in config/project.yml:

1. Add one manifest entry for every raw vector and raster input.
2. Fill source, version or acquisition information, and license from the authoritative dataset record.
3. State the documented meaning and intended treatment of missing data.
4. Run src/update_provenance.py.
5. Review the generated CRS, resolution, bounds, NoData facts, and SHA-256 fingerprints.
6. Commit data/provenance.yml and the inspection script together.

Treat a new download, a changed file digest, or a new product version as a meaningful project change. Update the manifest in the same commit that introduces the changed input.

---

Key takeaways and next step

A provenance manifest connects your ecological model to defensible evidence about its inputs.

• It records source, version or acquisition timing, license, CRS, resolution or spatial support, and NoData conventions for every raw input.
• It separates provider documentation from facts observed in the downloaded file.
• Raster resolution must include its native units; point and polygon data require a statement of spatial support rather than a pixel size.
• NoData documentation must cover declared values, validity masks, their meaning, and their planned treatment.
• A SHA-256 digest identifies the exact file used, even when a provider revises a product without changing its familiar filename.
• STAC’s Collection, Item, and Asset structure is a useful model for recording satellite-image provenance.

Next, you will begin assembling ecological training data by auditing observations for spatial clustering, sampling bias, and mismatch with the area where you intend to make predictions.