Loading, Cleaning, and Serializing Text Data with Python

Good to see you again. In the previous lesson, you used NumPy to transform in-memory numerical arrays efficiently while tracking shapes and axes. Before text reaches a tokenizer, though, it has to become a trustworthy dataset: correctly decoded, structurally parsed, cleaned according to an explicit policy, and saved in a reusable form.

This lesson builds a small ingestion pipeline using only Python’s standard library. You will load a CSV file of text snippets, validate its expected columns, apply conservative text cleaning, reject unusable records, and serialize the cleaned result as UTF-8 JSON. Plan for roughly 40–45 minutes: short documentation and video segments, followed by a practical implementation.

---

1. Treat a dataset as an interface, not just a file

A dataset is an interface between stages of a system. Its schema specifies the fields each record must contain; its cleaning policy specifies what transformations are permitted; and its serialized form lets the next stage consume the same artifact reproducibly.

For this lesson, assume a raw CSV dataset with these fields:

Field	Meaning	Example
id	Stable identifier, kept as text	"001"
source	Origin or category of the sample	"docs"
text	The training-relevant text	"Build model cards."

Keeping IDs as strings avoids accidental loss of meaningful leading zeroes. CSV readers also return field values as strings by default, which is appropriate here.

Create this project layout:

project/
├── data/
│   ├── raw_documents.csv
│   └── cleaned_documents.json
└── prepare_dataset.py

Put the following content in data/raw_documents.csv using a UTF-8-capable editor:

id,source,text
001,docs,"  Build   model   cards. "
002,forum,"Café owners discussed tokenization."
003,docs,"    "
004,docs,"Use UTF-8, not a platform default."
005,forum,"Model ""cards"" record limitations."

Two details matter:

• The text column contains a comma in record 004, so the value is quoted.
• Record 005 contains literal double quotes. In CSV, a quote inside a quoted field is represented by two quotes.

This is why parsing CSV with line.split(",") is unsafe. The csv module understands quoted delimiters, embedded quotes, and the other format rules that a manual split ignores.

Read the relevant parts of the official csv documentation before implementing the loader.

---

2. Text, bytes, encodings, and safe file access

On disk, a text file is bytes. In Python, useful text manipulation happens with str values. An encoding defines how Python converts between those two representations.

Always specify encoding="utf-8" for text datasets unless you have verified that the source requires another encoding. Relying on a platform default is risky: a file that appears correct on one developer’s Windows machine can fail or silently display corrupted characters elsewhere.

The basic pattern is:

from pathlib import Path

path = Path("data/raw_documents.csv")

with path.open("r", encoding="utf-8", newline="") as file:
    contents = file.read()

with is Python’s context manager. It guarantees that the file is closed even if parsing raises an exception. Path also avoids manual string construction with operating-system-specific path separators.

The newline="" argument is specifically recommended when a file is passed to csv.reader or csv.DictReader. It lets the CSV module handle line-ending details itself.

For a small dataset, reading all valid records into a list is reasonable. For the large corpora involved in pretraining, you will usually retain the same cleaning logic but yield one record at a time, applying the generator pattern from the earlier lesson.

---

3. Define a conservative cleaning policy

“Cleaning” should not mean “delete anything unusual.” Text that looks odd may be meaningful: casing, punctuation, newlines, code snippets, Markdown, URLs, or non-English characters can all matter for language modeling.

For our short standalone snippets, use this deliberately modest policy:

1. Normalize Unicode to NFC, so equivalent Unicode representations become consistent.
2. Strip leading and trailing whitespace.
3. Collapse internal whitespace runs to one ordinary space.
4. Reject records whose required fields are blank after cleaning.
5. Reject duplicate IDs.

We will not lowercase text, remove punctuation, discard accents, or rewrite words. Those are task-specific transformations that can destroy useful information.

Here is the complete loader. Put it in prepare_dataset.py:

import csv
import unicodedata
from pathlib import Path

REQUIRED_FIELDS = ("id", "source", "text")


def clean_snippet(value: str) -> str:
    """Normalize a short text snippet without changing its wording."""
    normalized = unicodedata.normalize("NFC", value)
    return " ".join(normalized.split())


def load_and_clean_csv(path: Path) -> tuple[list[dict[str, str]], list[str]]:
    """Load required CSV fields and return valid records plus rejection notes."""
    records: list[dict[str, str]] = []
    rejected: list[str] = []
    seen_ids: set[str] = set()

    with path.open("r", encoding="utf-8", newline="") as file:
        reader = csv.DictReader(file)

        if reader.fieldnames is None:
            raise ValueError("CSV file has no header row.")

        missing_fields = set(REQUIRED_FIELDS) - set(reader.fieldnames)
        if missing_fields:
            missing = ", ".join(sorted(missing_fields))
            raise ValueError(f"CSV is missing required columns: {missing}")

        for line_number, row in enumerate(reader, start=2):
            record = {
                field: clean_snippet(row.get(field) or "")
                for field in REQUIRED_FIELDS
            }

            blank_fields = [
                field for field, value in record.items()
                if not value
            ]

            if blank_fields:
                fields = ", ".join(blank_fields)
                rejected.append(
                    f"line {line_number}: blank required field(s): {fields}"
                )
                continue

            if record["id"] in seen_ids:
                rejected.append(
                    f"line {line_number}: duplicate id: {record['id']}"
                )
                continue

            seen_ids.add(record["id"])
            records.append(record)

    return records, rejected

A few design decisions are worth noticing.

Validate schema before processing rows

This check:

missing_fields = set(REQUIRED_FIELDS) - set(reader.fieldnames)

fails early if the dataset uses an unexpected schema, such as document instead of text. Failing explicitly is safer than quietly generating records whose text is always empty.

Clean missing CSV values safely

DictReader can produce None for a missing value. This expression makes such values safe to clean:

row.get(field) or ""

It retrieves the field if present; otherwise it uses an empty string. The subsequent blank-field check then rejects the incomplete row with a useful message.

Keep rejections visible

The function does not merely discard bad rows. It returns a list of reasons such as:

line 4: blank required field(s): text

For a production data pipeline, you might write these rejection records to a separate audit artifact. Even at this small scale, the important habit is to distinguish “accepted data” from “data that disappeared without explanation.”

Make cleaning scope explicit

" ".join(value.split()) collapses spaces, tabs, and newlines. That is sensible for short snippets such as our examples. It would be a poor default for long-form documents, where paragraph boundaries or code formatting may be meaningful. A cleaning function should encode a specific policy, not pretend to be universally correct.

---

4. Serialize the cleaned dataset as JSON

CSV is convenient for flat tabular input. JSON is often a better cleaned-data artifact because it represents Python lists and dictionaries naturally, supports nested fields when needed, and is easy to inspect, exchange, and version in Git.

Serialization means converting an in-memory Python object into a representation that can be stored or transmitted. It is separate from character encoding:

• Serialization turns list and dict structures into JSON syntax.
• UTF-8 encoding turns that JSON text into bytes for storage.

Watch this concise portion of Corey Schafer’s JSON tutorial to see the json.load and json.dump distinction in file-based use.

Add this serialization function beneath the loader:

import json


def write_json(records: list[dict[str, str]], path: Path) -> None:
    """Serialize cleaned records as readable UTF-8 JSON."""
    path.parent.mkdir(parents=True, exist_ok=True)

    with path.open("w", encoding="utf-8", newline="\n") as file:
        json.dump(
            records,
            file,
            ensure_ascii=False,
            indent=2,
            sort_keys=True,
        )
        file.write("\n")

Each option has a purpose:

Argument	Why it is used
encoding="utf-8"	Reads and writes the standard Unicode encoding explicitly.
newline="\n"	Produces consistent line endings across operating systems.
ensure_ascii=False	Keeps "Café" readable instead of escaping it as an ASCII sequence.
indent=2	Makes the artifact practical to inspect and review.
sort_keys=True	Gives dictionary keys a stable order in the output.

Do not write the result back over the raw input file. Raw data and derived data serve different purposes: keeping the raw input lets you improve and rerun the cleaning policy later.

---

5. Run the pipeline and verify the saved artifact

Finish the script with a small orchestration and verification section:

RAW_PATH = Path("data/raw_documents.csv")
CLEANED_PATH = Path("data/cleaned_documents.json")

records, rejected = load_and_clean_csv(RAW_PATH)
write_json(records, CLEANED_PATH)

print(f"Accepted records: {len(records)}")
print(f"Rejected records: {len(rejected)}")

for reason in rejected:
    print(f"  - {reason}")

with CLEANED_PATH.open("r", encoding="utf-8") as file:
    reloaded_records = json.load(file)

if reloaded_records != records:
    raise RuntimeError("JSON round-trip verification failed.")

print(f"Verified JSON round trip: {CLEANED_PATH}")

Run it from the project directory:

python prepare_dataset.py

Expected output:

Accepted records: 4
Rejected records: 1
  - line 4: blank required field(s): text
Verified JSON round trip: data/cleaned_documents.json

Your generated cleaned_documents.json will contain a JSON list of four normalized record dictionaries. The first record’s text, for example, will be:

{
  "id": "001",
  "source": "docs",
  "text": "Build model cards."
}

The round-trip comparison is a compact integrity check:

1. json.dump writes the Python records to disk.
2. json.load reconstructs Python values from the saved file.
3. Comparing the reconstructed values with the original records checks that the basic serialization contract held.

JSON is well suited here because our dataset contains only strings, dictionaries, and a list. It is not designed to preserve every Python-specific type, such as sets, tuples, arbitrary class instances, or NumPy arrays, without additional conventions.

---

6. Practical data-handling habits to retain

This small pipeline contains several habits that carry directly into LLM data work:

• Declare UTF-8 explicitly. Encoding errors are data bugs, not presentation issues.
• Use format-aware parsers. csv.DictReader handles CSV syntax and exposes meaningful field names.
• Validate expected columns early. A schema mismatch should stop processing rather than produce silently bad data.
• Use conservative, documented cleaning. Preserve semantics unless there is a task-specific reason not to.
• Track rejected records. Data quality decisions should be inspectable.
• Keep raw and processed artifacts separate.
• Reload serialized output. A round-trip check verifies that the saved artifact is usable.

---

You can now take a small textual CSV dataset through a complete standard-library workflow: load it safely, clean it according to an explicit policy, preserve Unicode correctly, serialize it as readable JSON, and verify that it reloads correctly.

Next, you will add automated tests with pytest around data-processing and numerical code. This loader is a strong candidate for testing: normalization behavior, blank-row rejection, duplicate-ID handling, schema validation, and JSON round trips can all be checked without manual inspection.