Retrieving Financial Data with Python APIs

Hello. In the previous lesson, you translated an investment mandate into a concrete research specification: a point-in-time universe, benchmark, rebalancing convention, objective, and constraints. That specification tells us which instruments and fields we need; this lesson begins the operational work of obtaining them.

For a portfolio project, “download prices” is not a sufficient data process. You need a reproducible retrieval layer that distinguishes market prices from cash distributions, share-structure events, and security-reference data. By the end of this lesson, you will have a Python pattern for retrieving daily prices, dividends, stock splits, and as-of security metadata from a documented REST API, while preserving enough provenance to audit the data later.

---

Treat market-data retrieval as a data contract

A ticker is an input to a data request, not a permanent definition of an asset. Tickers can be reused, renamed, delisted, or represent different share classes. Likewise, a closing price alone does not reveal whether a large one-day move reflects market performance, a dividend, or a stock split.

For the US equity strategy drafted in the previous lesson, the retrieval layer should create four separate datasets.

Dataset	One row represents	Why the optimizer or backtest will need it
Price bars	One ticker on one trading session	Returns, volatility, liquidity proxies, execution-price assumptions
Distributions	One cash dividend event	Total-return construction and dividend validation
Corporate actions	One split or other share-structure event	Interpretation and normalization of price history
Security metadata	A security’s reference attributes as of a date	Universe filters, identifier continuity, exchange and instrument-type checks

The separation is deliberate. A dividend is economically different from a split:

• A cash dividend transfers cash to shareholders. On the ex-dividend date, the share price commonly falls by roughly the cash amount, all else equal.
• In a two-for-one split, the shareholder has twice as many shares, each worth roughly half as much. A move from $100$ to $50$ is not a 50% investment loss.
• Metadata such as active status, primary exchange, ticker, CIK, FIGI, market capitalization, and industry classification define what the security is. These fields may also change over time.

Later, you will combine these sources carefully to construct or validate total-return series. Today’s priority is narrower and more foundational: retrieve and preserve the inputs without silently blending their meanings.

Select one canonical source for a research run

The Massive Stocks REST API documentation provides a useful example of a documented vendor interface. It exposes:

Need	Documented endpoint family
Security discovery and details	/v3/reference/tickers and /v3/reference/tickers/{ticker}
Historical daily OHLCV bars	/v2/aggs/ticker/{stocksTicker}/range/...
Dividends	/stocks/v1/dividends
Splits	/stocks/v1/splits

The source you choose need not be Massive in every professional setting. What matters is that a project has a named source, documented semantics, known coverage limits, and a stable retrieval process. Do not build a price table from one provider, add dividends from another, and assume their adjustment conventions are compatible.

The dividends endpoint deserves separate attention because it contains fields that are easy to confuse. The ex-dividend date is the relevant date for a standard price-return to total-return adjustment; the payment may arrive later, on the pay date. Keep declaration, ex-dividend, record, and pay dates rather than reducing an event immediately to a single number.

One practical warning from the documentation: vendor plans can have different historical coverage. If a free tier supplies only two years of history, it cannot support a five-year backtest, no matter how correct your Python is. Record your data-plan limitation in the project’s assumptions.

---

A retrieval design that remains auditable

A job-ready data pull should preserve three kinds of information:

1. The response content
   The raw bars, dividends, split events, and metadata returned by the vendor.

2. The request context
   Vendor, endpoint, ticker, query parameters, retrieval time, and adjustment setting.

3. The interpretation choices
   For example, whether price bars were requested adjusted or unadjusted, how timestamps were converted, and which metadata date was used.

A clean directory layout might be:

project/
  data/
    raw/
      MSFT_bars_2022-01-01_2024-12-31.json
      MSFT_dividends_2022-01-01_2024-12-31.json
      MSFT_splits_2022-01-01_2024-12-31.json
      MSFT_metadata_2024-12-31.json
    processed/
      prices_daily.parquet
      dividends.parquet
      splits.parquet
      security_metadata.parquet

The raw files are evidence of what the source returned. The processed files are tabular versions used downstream. In a mature system, you would also store a run identifier, code version, data-vendor version where available, and a hash of each raw file. Module 10 will formalize that reproducibility layer; begin the habit now.

Decide explicitly how prices are adjusted

Most vendors support some form of historical adjustment. That convenience can create a serious error:

Never construct a dividend-based total-return series by adding dividends to a price series that is already dividend-adjusted.

For the next lesson, you need either:

• Unadjusted prices plus separate dividend and split records, allowing you to build and test the adjustment logic yourself; or
• Vendor-adjusted prices plus separate corporate-action records, allowing you to validate the vendor’s treatment without reapplying it.

For this course project, request unadjusted daily bars where the endpoint supports that option, store the actions separately, and label the choice in the saved request metadata. We will not calculate returns in this lesson.

---

Build a small, reusable Python client

The code below uses requests rather than hiding calls behind a notebook cell or a vendor SDK. This makes the endpoint and query parameters visible during review. An SDK can be entirely appropriate in production, but the same principles still apply: log request context, handle pagination, validate responses, and preserve raw payloads.

Set your API key as an environment variable rather than embedding it in a notebook or committing it to Git:

export MASSIVE_API_KEY="your_key_here"

Install the minimal dependencies:

pip install pandas requests pyarrow

Now create src/data_client.py or an equivalent module.

from __future__ import annotations

import json
import os
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import parse_qsl, urlencode, urljoin, urlparse, urlunparse

import pandas as pd
import requests


BASE_URL = "https://api.massive.com/"
API_KEY = os.environ["MASSIVE_API_KEY"]
RAW_DIR = Path("data/raw")
RAW_DIR.mkdir(parents=True, exist_ok=True)


def with_api_key(url: str) -> str:
    """Add the API key without overwriting existing pagination parameters."""
    parsed = urlparse(url)
    query = dict(parse_qsl(parsed.query, keep_blank_values=True))
    query["apiKey"] = API_KEY
    return urlunparse(parsed._replace(query=urlencode(query)))


def make_url(path: str, params: dict | None = None) -> str:
    """Construct a vendor URL from an endpoint path and query parameters."""
    endpoint = urljoin(BASE_URL, path.lstrip("/"))
    if params:
        endpoint = f"{endpoint}?{urlencode(params)}"
    return with_api_key(endpoint)


def get_json(url: str) -> dict:
    """Request one page and fail loudly on HTTP or malformed JSON responses."""
    response = requests.get(with_api_key(url), timeout=30)
    response.raise_for_status()

    payload = response.json()
    if not isinstance(payload, dict):
        raise ValueError(f"Expected an object response, received {type(payload)}")

    return payload


def get_paginated_records(path: str, params: dict | None = None) -> list[dict]:
    """Retrieve every page from an endpoint that returns results and next_url."""
    url = make_url(path, params)
    records: list[dict] = []

    while url:
        payload = get_json(url)
        page_records = payload.get("results", [])

        if not isinstance(page_records, list):
            raise ValueError("Expected a list in the 'results' field.")

        records.extend(page_records)

        next_url = payload.get("next_url")
        if not next_url:
            break

        candidate = urljoin(BASE_URL, next_url)
        if urlparse(candidate).netloc != urlparse(BASE_URL).netloc:
            raise ValueError("Unexpected pagination host.")

        url = candidate

    return records


def save_raw(name: str, payload: dict) -> Path:
    """Save source payload plus retrieval provenance as JSON."""
    output_path = RAW_DIR / f"{name}.json"

    with output_path.open("w", encoding="utf-8") as file:
        json.dump(payload, file, indent=2, default=str)

    return output_path

A few details in this client are worth noticing.

• raise_for_status() prevents an authentication error or rate-limit response from being mistaken for valid empty data.
• Pagination is mandatory whenever the API provides next_url; retrieving only page one is a subtle form of data loss.
• The pagination-host check prevents your code from blindly following an unexpected URL.
• The raw JSON saving function stores data and context. A CSV of closing prices alone cannot tell a reviewer whether the source was adjusted, which endpoint was used, or when the data was retrieved.

Retrieve a complete four-part snapshot

Add the following functions beneath the client code.

def fetch_security_snapshot(
    ticker: str,
    start_date: str,
    end_date: str,
    metadata_as_of: str,
) -> dict:
    """
    Retrieve unadjusted daily bars, dividends, splits, and metadata.

    Dates use ISO format: YYYY-MM-DD.
    metadata_as_of is the date on which the reference record is requested.
    """
    ticker = ticker.upper()
    retrieved_at = datetime.now(timezone.utc).isoformat()

    bars_path = (
        f"/v2/aggs/ticker/{ticker}/range/1/day/{start_date}/{end_date}"
    )
    bars = get_paginated_records(
        bars_path,
        params={
            "adjusted": "false",
            "sort": "asc",
            "limit": 50000,
        },
    )

    dividends = get_paginated_records(
        "/stocks/v1/dividends",
        params={
            "ticker": ticker,
            "limit": 5000,
            "sort": "ex_dividend_date.asc",
        },
    )

    splits = get_paginated_records(
        "/stocks/v1/splits",
        params={
            "ticker": ticker,
            "limit": 5000,
        },
    )

    metadata_response = get_json(
        make_url(
            f"/v3/reference/tickers/{ticker}",
            params={"date": metadata_as_of},
        )
    )
    metadata = metadata_response.get("results", {})

    snapshot = {
        "source": "Massive Stocks REST API",
        "retrieved_at_utc": retrieved_at,
        "ticker_requested": ticker,
        "start_date": start_date,
        "end_date": end_date,
        "metadata_as_of": metadata_as_of,
        "price_adjustment_requested": "unadjusted",
        "bars": bars,
        "dividends": dividends,
        "splits": splits,
        "metadata": metadata,
    }

    return snapshot

Run a small retrieval first. A single liquid equity is enough to test the pipeline.

snapshot = fetch_security_snapshot(
    ticker="MSFT",
    start_date="2023-01-01",
    end_date="2024-12-31",
    metadata_as_of="2024-12-31",
)

save_raw(
    name="MSFT_snapshot_2023-01-01_2024-12-31",
    payload=snapshot,
)

print(f"Bars: {len(snapshot['bars'])}")
print(f"Dividend records: {len(snapshot['dividends'])}")
print(f"Split records: {len(snapshot['splits'])}")
print(f"Metadata keys: {sorted(snapshot['metadata'].keys())[:10]}")

Do not assume that an empty split table is a failure. Many stocks simply had no split during the period. By contrast, an empty price table for an actively traded, well-known stock should trigger an investigation of the ticker syntax, API key, date range, entitlement, or request construction.

---

Convert responses to analysis-ready tables, without discarding meaning

A vendor’s JSON field names need not become your project’s final schema. Convert them into consistent, explicit columns while retaining the raw JSON separately.

Daily aggregate responses commonly use compact field names, such as o, h, l, c, v, and t. The following function turns those into readable names. It also keeps both a UTC timestamp and a US market-session date.

def normalise_daily_bars(bar_records: list[dict], ticker: str) -> pd.DataFrame:
    bars = pd.DataFrame(bar_records).rename(
        columns={
            "t": "timestamp_epoch",
            "o": "open",
            "h": "high",
            "l": "low",
            "c": "close",
            "v": "volume",
            "vw": "vwap",
            "n": "transaction_count",
        }
    )

    if bars.empty:
        return pd.DataFrame(
            columns=[
                "ticker",
                "timestamp_utc",
                "session_date",
                "open",
                "high",
                "low",
                "close",
                "volume",
                "vwap",
                "transaction_count",
            ]
        )

    required = {"timestamp_epoch", "open", "high", "low", "close", "volume"}
    missing = required.difference(bars.columns)
    if missing:
        raise ValueError(f"Daily-bar response is missing fields: {sorted(missing)}")

    epoch_values = pd.to_numeric(bars["timestamp_epoch"], errors="raise")

    # The endpoint schema is authoritative. This defensive check handles
    # seconds versus milliseconds safely when normalizing stored responses.
    epoch_unit = "ms" if epoch_values.abs().median() > 1e11 else "s"

    bars["timestamp_utc"] = pd.to_datetime(
        epoch_values,
        unit=epoch_unit,
        utc=True,
    )

    bars["session_date"] = (
        bars["timestamp_utc"]
        .dt.tz_convert("America/New_York")
        .dt.normalize()
        .dt.tz_localize(None)
    )

    bars["ticker"] = ticker.upper()

    return (
        bars[
            [
                "ticker",
                "timestamp_utc",
                "session_date",
                "open",
                "high",
                "low",
                "close",
                "volume",
                "vwap",
                "transaction_count",
            ]
        ]
        .sort_values("session_date")
        .reset_index(drop=True)
    )

Normalize dividends with the fields documented by the provider. Crucially, retain distribution_type. A special dividend is not necessarily comparable to the recurring quarterly distribution that preceded it.

def normalise_dividends(
    dividend_records: list[dict],
    ticker: str,
    retrieved_at_utc: str,
) -> pd.DataFrame:
    columns = [
        "ticker",
        "id",
        "cash_amount",
        "split_adjusted_cash_amount",
        "currency",
        "distribution_type",
        "frequency",
        "declaration_date",
        "ex_dividend_date",
        "record_date",
        "pay_date",
        "historical_adjustment_factor",
        "vendor_retrieved_at_utc",
    ]

    dividends = pd.DataFrame(dividend_records)

    if dividends.empty:
        return pd.DataFrame(columns=columns)

    dividends["ticker"] = ticker.upper()
    dividends["vendor_retrieved_at_utc"] = retrieved_at_utc

    for date_column in [
        "declaration_date",
        "ex_dividend_date",
        "record_date",
        "pay_date",
    ]:
        if date_column in dividends.columns:
            dividends[date_column] = pd.to_datetime(
                dividends[date_column],
                errors="coerce",
            ).dt.date

    return dividends.reindex(columns=columns).sort_values(
        "ex_dividend_date"
    ).reset_index(drop=True)

For splits, preserve all source fields initially. Split schemas often include an execution date and ratio-related fields, but the exact representation is source-specific. Do not invent a split ratio by parsing a display string or assume every action has the same event-date convention.

def normalise_splits(
    split_records: list[dict],
    ticker: str,
    retrieved_at_utc: str,
) -> pd.DataFrame:
    splits = pd.DataFrame(split_records)

    if splits.empty:
        return pd.DataFrame(
            columns=["ticker", "vendor_retrieved_at_utc"]
        )

    splits["ticker"] = ticker.upper()
    splits["vendor_retrieved_at_utc"] = retrieved_at_utc

    for possible_date in ["execution_date", "date"]:
        if possible_date in splits.columns:
            splits[possible_date] = pd.to_datetime(
                splits[possible_date],
                errors="coerce",
            ).dt.date

    return splits.sort_index(axis=1)

Finally, convert the metadata dictionary into a one-row, dated reference table:

def normalise_metadata(
    metadata: dict,
    ticker_requested: str,
    metadata_as_of: str,
    retrieved_at_utc: str,
) -> pd.DataFrame:
    row = dict(metadata)
    row["ticker_requested"] = ticker_requested.upper()
    row["metadata_as_of"] = pd.Timestamp(metadata_as_of).date()
    row["vendor_retrieved_at_utc"] = retrieved_at_utc

    return pd.DataFrame([row])

Putting the pieces together:

prices = normalise_daily_bars(snapshot["bars"], snapshot["ticker_requested"])

dividends = normalise_dividends(
    snapshot["dividends"],
    snapshot["ticker_requested"],
    snapshot["retrieved_at_utc"],
)

splits = normalise_splits(
    snapshot["splits"],
    snapshot["ticker_requested"],
    snapshot["retrieved_at_utc"],
)

metadata = normalise_metadata(
    snapshot["metadata"],
    snapshot["ticker_requested"],
    snapshot["metadata_as_of"],
    snapshot["retrieved_at_utc"],
)

processed_dir = Path("data/processed")
processed_dir.mkdir(parents=True, exist_ok=True)

prices.to_parquet(processed_dir / "prices_daily.parquet", index=False)
dividends.to_parquet(processed_dir / "dividends.parquet", index=False)
splits.to_parquet(processed_dir / "splits.parquet", index=False)
metadata.to_parquet(processed_dir / "security_metadata.parquet", index=False)

At this point you have not calculated a return. That is a feature, not an omission. You have created a clean boundary:

• retrieval obtains and records vendor facts;
• normalization creates usable tables;
• return construction will make explicit economic and adjustment assumptions.

---

Inspect the retrieved data before scaling up

Before requesting hundreds of tickers, perform a concise inspection on the one-security sample.

print(prices.head(3))
print(prices.tail(3))

print(
    dividends[
        [
            "ex_dividend_date",
            "pay_date",
            "cash_amount",
            "currency",
            "distribution_type",
        ]
    ].tail()
)

print(splits.tail())
print(metadata.T.head(20))

Use this inspection to verify the following.

Price-bar checks

• session_date should be strictly increasing for one ticker.
• OHLC fields should be positive for ordinary equity observations.
• Daily high should not be less than daily low.
• Volume should not be interpreted as liquidity without considering units, market session, and corporate events.
• Missing calendar dates are normal on weekends and exchange holidays. Missing expected trading dates require later investigation.

Dividend checks

• Confirm that the ticker, currency, ex-dividend date, and cash amount are populated where expected.
• Preserve every reported distribution type. Filtering special dividends is a modeling choice to make later, not a retrieval decision.
• Do not substitute pay_date for ex_dividend_date when relating an event to price history.

Split checks

• Examine the exact vendor fields in your response before using them in arithmetic.
• If a price discontinuity coincides with a split execution date, flag it for later adjustment validation.
• A split event should affect historical-price interpretation and shares held; it does not create investment profit by itself.

Metadata checks

For the long-only US common-equity universe from the previous lesson, inspect at least:

• returned ticker and security name;
• active status;
• market and primary exchange;
• locale and currency;
• instrument type;
• standardized identifiers such as CIK, composite FIGI, and share-class FIGI when available;
• reference date and vendor retrieval timestamp.

This is where you would reject an ETF, preferred share, index, or non-US listing when the mandate permits common US equities only. Do not use a current metadata snapshot to claim historical eligibility. A metadata record requested “as of” a date supports historical reference, but later lessons will distinguish an event’s economic date from the date information became available to the strategy.

---

A convenient prototype option: yfinance

For exploratory work, yfinance is convenient because a ticker object can expose price history, dividends, splits, and broad metadata. The important warning is adjustment semantics: the video notes that auto_adjust defaults to True in current versions, meaning downloaded prices may already incorporate split and dividend adjustments.

For a prototype, an explicit retrieval might look like this:

import yfinance as yf

instrument = yf.Ticker("MSFT")

history = instrument.history(
    start="2023-01-01",
    end="2024-12-31",
    auto_adjust=False,
    actions=True,
)

dividends = instrument.dividends
splits = instrument.splits
metadata = instrument.info

This is useful for a quick experiment, but retain the same discipline:

• pin and record the package version;
• record the retrieval timestamp and request parameters;
• preserve whether prices were auto-adjusted;
• do not assume the metadata fields or their historical semantics are complete;
• choose one canonical source for a single backtest.

For the capstone-quality version of the project, a documented API retrieval layer with raw-response storage is easier to defend in a code review than a sequence of manually rerun notebook downloads.

---

Key takeaways

A credible portfolio data pipeline collects more than closing prices.

• Keep daily bars, cash distributions, split events, and security metadata as separate datasets with explicit schemas.
• Record the source, endpoint, parameters, retrieval time, and price-adjustment choice alongside raw data.
• Follow vendor pagination; incomplete retrieval can otherwise look like valid data.
• Convert timestamps from UTC deliberately and derive US trading-session dates in Eastern Time.
• Request and label unadjusted prices if you intend to construct total returns from raw prices and actions later.
• Use metadata and stable identifiers to support universe checks; do not treat a ticker as a timeless security identity.
• Validate one liquid ticker end to end before scaling the pipeline to a full universe.

Next, you will move from Python retrieval to relational analysis: querying normalized price, corporate-action, and security-reference tables with SQL joins and window functions.