Memory-Efficient Data Processing with Iterators and Generators

Hello again. In the previous lesson, you organized Python code into importable modules, used classes to preserve valid state, and separated domain-level exceptions from application-level recovery. That structure becomes especially useful when your data is too large to fit comfortably in memory.

This lesson introduces iterables, iterators, and generators: Python’s mechanism for reading and transforming data incrementally. You will learn to recognize one-pass data sources, write generator functions with yield, compose streaming stages, and avoid an easy mistake: making a transformation lazy while still eagerly loading the entire input file.

Plan for roughly 45 minutes: a short video, focused reading, and an implementation walkthrough.

---

1. An iterable is not necessarily an iterator

You have already used iteration extensively:

for sample in samples:
    print(sample.preview())

The for loop works with an iterable: an object Python can traverse. Lists, tuples, strings, dictionaries, range objects, files, and generators are all common iterables.

An iterator is the stateful object that actually produces the next value. It knows where it currently is in a traversal.

Object	Iterable?	Iterator?	Practical implication
list[str]	Yes	No	Can create a fresh traversal each time
range(10)	Yes	No	Can be traversed again
A file opened for reading	Yes	Yes	Advances through the file line by line
A generator object	Yes	Yes	Usually single-use and stateful
A generator function	No	No	Calling it creates a generator object

At the protocol level:

• An iterable provides __iter__(), which supplies an iterator.
• An iterator provides __next__(), which returns the next item or raises StopIteration when no items remain.

This is what Python does conceptually for a for loop:

iterator = iter(["train", "validation", "test"])

while True:
    try:
        split = next(iterator)
    except StopIteration:
        break

    print(split)

Normally, you should write the clear for loop rather than this expansion. The value of seeing the mechanics is that it explains two important behaviors:

1. Calling next() advances an iterator.
2. Once an iterator is exhausted, it does not reset itself.

Corey Schafer’s video gives a compact, concrete explanation of this distinction before moving into generators.

Try the state change yourself in a Python REPL:

splits = ["train", "validation", "test"]

split_iterator = iter(splits)

print(next(split_iterator))  # train
print(next(split_iterator))  # validation

for split in split_iterator:
    print(split)             # test

for split in split_iterator:
    print(split)             # No output: it is exhausted

The original splits list still exists and can be traversed again. The particular iterator obtained from it has already reached its end:

for split in splits:
    print(split)  # train, validation, test

This distinction matters in data work. A dataset stored in a list is convenient for repeated passes and random access, but requires memory proportional to the dataset. A stream can be processed with a bounded working set, but you must design around its single-pass nature.

---

2. Generators: concise, lazy iterators

A generator function contains yield rather than an ordinary return for its produced values.

from collections.abc import Iterable, Iterator


def clean_nonempty_lines(lines: Iterable[str]) -> Iterator[str]:
    """Yield whitespace-normalized, nonempty lines."""
    for raw_line in lines:
        cleaned_line = " ".join(raw_line.split())

        if cleaned_line:
            yield cleaned_line

Calling this function does not immediately run the loop and build a result list:

records = clean_nonempty_lines(
    ["  Attention uses queries and keys.  ", "", "  Tokenizers split text. "]
)

print(records)
# <generator object clean_nonempty_lines at ...>

The function starts when a consumer asks for an item:

print(next(records))
# Attention uses queries and keys.

print(next(records))
# Tokenizers split text.

At each yield, Python:

1. Produces the current value for the caller.
2. Suspends the function, retaining its local state.
3. Resumes immediately after that yield when another item is requested.

Once the input is exhausted, the generator ends and Python raises StopIteration internally. A for loop handles that signal for you.

The return type annotation is worth noticing:

def clean_nonempty_lines(lines: Iterable[str]) -> Iterator[str]:

The function accepts any source that can be iterated over, not just a list. It can therefore receive:

• a list while testing;
• a tuple or another generator;
• a file object;
• eventually, a stream supplied by a dataset library.

It returns an Iterator[str], signaling a one-pass result rather than a fully materialized collection.

yield versus return

A normal function returns once:

def first_nonempty_line(lines: Iterable[str]) -> str | None:
    for line in lines:
        cleaned = " ".join(line.split())
        if cleaned:
            return cleaned

    return None

That is appropriate when the program needs exactly one answer.

A generator can produce many answers over time:

def nonempty_lines(lines: Iterable[str]) -> Iterator[str]:
    for line in lines:
        cleaned = " ".join(line.split())
        if cleaned:
            yield cleaned

Both tools are useful. Use return for a final result; use yield when the caller should consume a sequence incrementally.

Generator expressions

A generator expression is the compact counterpart of a generator function. Compare these two forms:

# Eager: build every transformed value now.
lengths_list = [len(text) for text in texts]

# Lazy: compute each length when requested.
lengths_iterator = (len(text) for text in texts)

Square brackets create a list. Parentheses create a generator expression.

A generator expression is particularly effective for a simple local transformation. Once validation, error reporting, resource management, or multiple lines of logic enter the picture, a named generator function is usually clearer and easier to test.

---

3. Avoiding the real memory bottleneck: stream the source too

Suppose corpus.txt is much larger than a typical source-code file. This approach is eager at several points:

from pathlib import Path

path = Path("corpus.txt")

raw_lines = path.read_text(encoding="utf-8").splitlines()

cleaned_lines = [
    " ".join(line.split())
    for line in raw_lines
    if line.strip()
]

transformer_lines = [
    line
    for line in cleaned_lines
    if "transformer" in line.casefold()
]

print(len(transformer_lines))

This code may be perfectly reasonable for a small file. But it can retain all of the following in memory:

• the complete file contents as one string;
• a list of all raw lines;
• a list of all cleaned lines;
• a list of all matching lines.

Replacing only the final list comprehension with parentheses would not solve the central issue if path.read_text() has already loaded the entire file.

Instead, open the file and iterate over it directly. Files provide lines incrementally, using internal I/O buffering rather than loading the complete file at once.

from collections.abc import Iterable, Iterator
from pathlib import Path


def clean_nonempty_lines(
    lines: Iterable[str],
) -> Iterator[tuple[int, str]]:
    """Yield line number and normalized text for each nonblank line."""
    for line_number, raw_line in enumerate(lines, start=1):
        cleaned_line = " ".join(raw_line.split())

        if cleaned_line:
            yield line_number, cleaned_line


def lines_containing(
    records: Iterable[tuple[int, str]],
    phrase: str,
) -> Iterator[tuple[int, str]]:
    """Yield records whose normalized text contains phrase, case-insensitively."""
    normalized_phrase = phrase.casefold()

    for line_number, text in records:
        if normalized_phrase in text.casefold():
            yield line_number, text


def count_phrase_matches(path: Path, phrase: str) -> int:
    """Count matching nonblank lines without materializing the corpus."""
    with path.open("r", encoding="utf-8") as source:
        cleaned_records = clean_nonempty_lines(source)
        matching_records = lines_containing(cleaned_records, phrase)

        return sum(1 for _ in matching_records)

Use it as follows:

match_count = count_phrase_matches(
    Path("corpus.txt"),
    "transformer",
)

print(f"Matching lines: {match_count}")

There are three separate responsibilities here:

Component	Responsibility
source	Supplies raw lines from disk
clean_nonempty_lines()	Normalizes and removes blank records
lines_containing()	Applies a domain-specific filter
sum()	Consumes the stream and produces one final number

This style resembles a staged processing architecture. Each stage exposes a small contract: accept an iterable, yield a transformed iterable. Because the stages do not care whether their input is a list, file, or generator, they are independently reusable and testable.

The actual processing is deferred until sum() requests values. At that point, Python obtains a line from source, normalizes it, tests the phrase, and either contributes it to the count or moves on. It does not need a collection of every cleaned or matching line.

A useful nuance: “memory efficient” does not mean “zero memory.” The program still holds the current line, transformed strings, generator state, and file buffers. A single unusually large line can still be a memory concern. But memory use no longer grows with the number of ordinary records in the corpus.

Resource ownership stays at the boundary

Notice that the with block is in count_phrase_matches():

with path.open("r", encoding="utf-8") as source:
    ...

The function that opens the file also controls when it is closed. This is a sound boundary: clean_nonempty_lines() merely transforms any iterable and does not need to know whether its source is a file, a list, or a network-backed stream.

This also extends the exception-handling principle from the previous lesson. A generator’s body runs during consumption, not merely when the generator is created. Therefore, if a later input record is invalid, its exception emerges while the caller is iterating. Put handling policy around the consuming loop or terminal operation, where the application can decide whether to report, skip, or stop.

---

4. Practical constraints: laziness changes the contract

Generators trade convenience for bounded memory and composability. Use them deliberately.

A generator is usually one-pass

This is a common bug:

records = clean_nonempty_lines(["one", "", "two"])

print(list(records))
# [(1, 'one'), (3, 'two')]

print(list(records))
# []

The first list(records) consumes every remaining item and stores it in a list. The second call sees an exhausted generator.

If you require two passes over a source, choose one of these designs:

• Re-create the generator from a repeatable source.
• Reopen the file and build a fresh pipeline for each pass.
• Materialize the data into a list only if it safely fits in memory and repeated access is genuinely needed.
• Compute multiple summaries in one pass when possible.

For example, when processing a very large corpus, it is often better to count records and retain only three representative examples during one traversal than to make two complete passes.

Some operations force materialization

The following operations consume an iterator and create a full collection:

all_records = list(records)
ordered_records = sorted(records)

That may be entirely appropriate for a small dataset, but it removes the streaming memory advantage.

By contrast, these can consume one item at a time and retain only a small amount of state:

record_count = sum(1 for _ in records)
has_records = any(True for _ in records)

max() and min() can also stream their input, retaining only the best item seen so far.

Generators are not automatically faster

Laziness principally addresses peak memory use and enables processing of sources of unknown or unbounded size. It can add per-item overhead. A list can be the better tool when:

• the collection is small;
• you need indexing or random access;
• you need repeated passes;
• profiling shows that eager evaluation is preferable and memory is not a constraint.

For example, a list of a few hundred model names for a UI dropdown should simply be a list. A multi-gigabyte text corpus or a continuing event stream should be processed incrementally.

Prefer generator functions over custom iterator classes

You can implement a custom iterator with __iter__() and __next__(). That is useful when you are modeling a stateful object with a richer public API.

For ordinary data transformation, however, this:

def normalized_texts(lines: Iterable[str]) -> Iterator[str]:
    for line in lines:
        yield " ".join(line.split())

is clearer and less error-prone than a class that manually tracks an index and explicitly raises StopIteration. Python handles the iterator protocol on your behalf.

---

5. A small implementation checkpoint

Place the two generator functions and count_phrase_matches() in a new module in the package from the previous lesson, for example text_ingest/streaming.py.

Keep the entry point separate:

# text_ingest/app.py
from pathlib import Path

from .streaming import count_phrase_matches


def main() -> None:
    count = count_phrase_matches(
        Path("corpus.txt"),
        "attention",
    )
    print(f"Lines mentioning attention: {count}")


if __name__ == "__main__":
    main()

Run it from the directory containing the text_ingest package:

python -m text_ingest.app

For a quick behavioral check:

• Run it on a short text file containing blank lines and mixed capitalization.
• Confirm that blank lines do not count.
• Confirm that "Attention" and "attention" both match.
• Temporarily add a print() immediately before yield in clean_nonempty_lines() and observe that it runs only while sum() consumes the pipeline.
• Call the counting function a second time. It should work because the function opens the file and constructs a new pipeline on each call.

---

You now have the core pattern for streaming data in Python:

• An iterable can provide an iterator.
• An iterator advances statefully through values and is commonly one-pass.
• A generator function uses yield to create a lazy iterator without implementing protocol methods manually.
• A memory-efficient pipeline must stream both the source and its transformations.
• with should own the lifetime of files and other external resources.
• Laziness is valuable for large datasets, but lists remain appropriate for small, repeatable, random-access collections.

Next, you will move from Python objects to NumPy arrays, learning indexing, vectorization, and broadcasting. Generators handle a dataset incrementally; NumPy will let you process the numerical data within each batch efficiently.