Asynchronous Generators with `async def` and `yield`

Hello! Welcome back to our course on asyncio.

In our last two lessons, we explored the asynchronous iteration protocol from both sides. First, we built a custom asynchronous iterator class, NetworkDataStream, by implementing the __aiter__ and __anext__ methods. Then, we learned how to consume such an iterator using the async for statement, appreciating it as syntactic sugar that simplifies the process.

Today, we'll connect these ideas and introduce a much more direct and Pythonic way to create asynchronous iterators. Our learning outcome is to define an asynchronous generator using async def and yield. This pattern is the most common and readable way to produce asynchronous streams of data, and it builds directly on concepts you are likely already familiar with from standard Python generators.

1. The Journey from Generators to Coroutines

Before we define an asynchronous generator, it's valuable to understand its conceptual origins. Standard Python generators (functions with yield) are not just for creating simple iterators; their ability to pause and resume execution makes them the direct ancestors of asyncio coroutines.

The evolution from a simple generator to a full-fledged coroutine involved enhancing generators with the ability to receive data and exceptions from the outside world. This bidirectional communication turned them from simple data producers into cooperative units of execution.

For a deeper insight into this evolution and how the internal mechanics of generators (specifically, their frame objects) laid the groundwork for asyncio, the following segment from a talk by Python core developer Łukasz Langa is excellent.

This context is key: an asynchronous generator isn't a completely new invention but rather the logical culmination of this journey, combining the iteration-producing nature of a generator with the await-able nature of a coroutine.

2. Defining an Asynchronous Generator

With that background, the definition of an asynchronous generator becomes quite intuitive.

An asynchronous generator is a function defined using async def that contains one or more yield expressions.

That's it. This simple combination of keywords creates a powerful construct with the following properties:

• It looks like a coroutine but behaves like an iterator.
• It can contain await expressions, allowing it to perform non-blocking I/O operations between yields.
• It returns an asynchronous generator iterator, which can be consumed by an async for loop.

The following resource provides a very clear definition and comparison between classical and asynchronous generators.

Let's look at a canonical example that puts this definition into practice. The function simulates fetching a stream of data where each chunk requires an asynchronous wait.

import asyncio
import time

# Define an asynchronous generator
async def async_data_stream(num_chunks: int):
    """
    An asynchronous generator that simulates streaming data chunks
    with a delay between each chunk.
    """
    print("--- Stream starting ---")
    for i in range(num_chunks):
        # Simulate a non-blocking I/O operation (e.g., a network call)
        await asyncio.sleep(1)
        
        # Yield the data chunk
        yield f"Chunk {i+1}/{num_chunks} at {time.strftime('%X')}"
    print("--- Stream finished ---")

async def main():
    # Consume the asynchronous generator with an async for loop
    print(f"Main coroutine started at {time.strftime('%X')}")
    async for chunk in async_data_stream(5):
        print(f"  -> Received: {chunk}")
    print(f"Main coroutine finished at {time.strftime('%X')}")

# Run the main coroutine
asyncio.run(main())

Trace the execution:

1. asyncio.run(main()) starts the event loop.
2. main() begins. The async for loop calls async_data_stream(5). This does not run the function body yet; it returns an asynchronous generator object.
3. The async for loop calls __anext__() on this object for the first time.
4. Execution enters async_data_stream. It prints "--- Stream starting ---" and hits await asyncio.sleep(1).
5. async_data_stream is suspended, and control returns to the event loop.
6. After 1 second, the event loop resumes async_data_stream. It proceeds to the yield statement, yielding the first chunk.
7. The async for loop in main() receives the value, and the loop body executes, printing the "Received" message.
8. The loop continues to the next iteration, implicitly calling __anext__() again, and the cycle repeats until the generator function completes.

3. The Power and Elegance of Async Generators

Now that you know how to define an async generator, let's focus on why it's the preferred approach for creating asynchronous iterators in most scenarios.

The primary advantage is the dramatic improvement in readability and conciseness compared to a class-based implementation. State is managed via local variables within the function's scope, not through instance attributes like self._current_chunk and self._total_chunks as in our NetworkDataStream class. The logic flows linearly, just like a regular function.

The following video segments articulate these benefits clearly.

The key takeaway is that asynchronous generators allow you to write code that produces a stream of data as a simple, sequential function, while asyncio handles all the complex state management of pausing and resuming the function around the await and yield points.

Conclusion

In this lesson, we've formally defined and explored asynchronous generators.

• Key Takeaway: An asynchronous generator is a function created with async def that includes at least one yield statement. It seamlessly blends the non-blocking capabilities of a coroutine with the iterative nature of a generator.
• Core Concept: It serves as a concise and highly readable factory for creating asynchronous iterators. The state of the iteration is implicitly managed by the generator's local scope, avoiding the boilerplate of a custom iterator class.
• Practicality: This is the idiomatic and most common way to implement producers of I/O-bound data streams in modern asynchronous Python.

You have now seen both the manual, class-based way to create an async iterator and the far more elegant generator-based approach. In our next lesson, we will make this comparison concrete by applying what you've just learned to refactor our NetworkDataStream class into a simple and clean asynchronous generator function.