API Pagination Strategies

Hello! Welcome to your next lesson in mastering n8n.

In our recent lessons, we've focused on connecting to APIs with the HTTP Request node and parsing various response formats like JSON and XML. These are essential skills, but they've dealt with fetching a single "chunk" of data at a time. However, most real-world APIs don't send you their entire database in one go. To manage server load and ensure fast responses, they break large datasets into smaller pieces, or "pages."

This brings us to our current learning outcome: Implement pagination patterns to retrieve large datasets from APIs.

Today, you'll learn how to instruct n8n to automatically fetch page after page until you have all the data you need. We'll explore two primary methods for this: using n8n's powerful built-in pagination feature and constructing a custom loop for more complex scenarios. Your background as a software developer will be a great asset here, as these patterns are analogous to common programming constructs for handling large collections of data.

1. Understanding API Pagination

When you request a list of items from an API (like contacts, orders, or posts), and the total number of items is large, the API will typically return only the first subset (e.g., the first 100 contacts) along with information on how to get the next subset. This mechanism is called pagination.

There are two dominant pagination strategies you'll encounter:

• Offset/Page-based Pagination: This is the simplest form. The API exposes parameters like page and limit. To get the second page of 100 items, you would make a new request with page=2&limit=100.
• Cursor-based Pagination: This method is more robust. In its response, the API provides a "cursor"—a unique identifier (often a string or token) that points to the next item or page. Your next request must include this cursor to get the subsequent set of data. This avoids issues with data changing while you are fetching it.

Failing to handle pagination means you're only working with a fraction of the available data. Furthermore, sending too many requests too quickly can get you temporarily blocked by the API, a measure known as rate-limiting. Proper pagination helps you manage this by fetching data methodically.

2. Method 1: The Built-in HTTP Request Pagination Feature

The most direct way to handle pagination in n8n is by using the dedicated feature within the HTTP Request node. It's designed to handle most common pagination patterns automatically.

A fantastic, practical guide for this is an article by Tugui Dragos-Constantin on the Medium platform. It uses the GoHighLevel API as an example, which employs a cursor-based pagination strategy.

Key Steps for Built-in Pagination

As you've seen in the article, the process boils down to these key configurations within the HTTP Request node:

1. Set up the initial request: Configure the URL, authentication, and any static parameters (like limit=100) just as you would for a single request.
2. Add the Pagination Option: Click Add Option and select Pagination.
3. Choose Pagination Mode: Set this to Update a Parameter in Each Request. This mode is versatile and can handle both cursor- and page-based patterns.
4. Define the Dynamic Parameter(s): This is the core of the logic.
   • For Cursor-based (like the article): You add a Query parameter for the cursor (e.g., startAfterId). The Value is set to an Expression that extracts the cursor from the previous response: {{ $response['body']['meta']['startAfterId'] }}. The $response variable here specifically refers to the output from the previous paginated call made by this same node.
   • For Page-based: The logic is even simpler. You can use the special $pageCount variable that n8n provides, which tracks the number of requests made.

5. Set the Stopping Condition: How does n8n know when to stop? You must configure this. Common choices for Pagination Complete When are:
   • Response Is Empty: The loop stops when the API returns an empty list of items.
   • Last Response Has No Field: The loop stops when a specific field (e.g., a next_page_cursor) is missing from the API's response.
6. Respect Rate Limits: Always set an Interval Between Requests (e.g., 1000 ms) to avoid overwhelming the API.

{
  "data": [ ... items ... ],
  "pagination": {
    "next_url": "https://api.example.com/items?page=2"
  }
}

3. Method 2: The Custom Loop Pattern

While the built-in feature is convenient, sometimes you need more control. As a developer, you'll appreciate this pattern, which is essentially a do-while loop constructed from n8n nodes. It's more verbose but also more flexible and robust, especially for non-standard APIs or web scraping.

This approach is well-documented in an article on dev.to for a web scraping scenario, but the principles apply to any paginated data source.

The Custom Loop Architecture

This pattern consists of a few key components that form a loop:

1. Page Manager (Code Node): This node acts as your loop counter. On the first run, it outputs page: 1. On subsequent runs, it takes the page number from the previous iteration, increments it, and outputs the new number.
2. HTTP Request Node: This fetches data for the page number provided by the Page Manager.
3. Data Aggregation (Code Node): This is a crucial step. After fetching data, this node takes the items and appends them to a temporary, workflow-wide storage area using const workflowStaticData = $getWorkflowStaticData('global');. This ensures data isn't lost between loop iterations.
4. IF Node: This is your loop's exit condition. It checks, for example, if the last HTTP request returned zero items.
   • If true (no items found), the loop stops, and the workflow proceeds along the true path.
   • If false (items were found), the workflow is routed back to the Page Manager node to start the next iteration.
5. Final Collector (Code Node): Placed on the true path of the IF node, this final node retrieves all the aggregated data from workflowStaticData and outputs it as a single list for the rest of your workflow to use.

This pattern gives you the flexibility to add other logic inside the loop, such as transforming data on a per-page basis or handling complex error conditions before deciding whether to continue.

Conclusion

You now have a robust toolkit for retrieving large datasets from any API that uses pagination. This is a massive step up from fetching single responses and is critical for building production-ready automations.

Key Takeaways:

• APIs use pagination (offset- or cursor-based) to serve large datasets in manageable chunks.
• n8n's built-in pagination feature in the HTTP Request node is the quickest way to handle most standard APIs. The key is to correctly configure the dynamic parameters and the stopping condition.
• For maximum flexibility or for non-standard APIs, you can build a custom pagination loop using Code nodes for state management, an IF node for the condition, and workflowStaticData for aggregation.

In our next lesson, we'll address a common follow-up challenge. Now that you can fetch thousands of items, how do you process them without overwhelming your workflow or downstream services? We'll learn to use the Split In Batches node to process items in groups, a perfect companion to the pagination skills you've learned today.