Partner API pagination: how to collect every record safely

One page is not the whole answer

A partner's list endpoint may return a successful response while showing only a slice of the available records. GitHub documents a Link header with a next relation for its REST API; Microsoft Graph instead places an @odata.nextLink in a response body. Those are examples of different provider contracts, not interchangeable syntax. Before an SME promises that an import is complete, it must identify the actual endpoint's continuation signal and its end condition. A status of 200, a visually full page, or a plausible row count alone does not establish completion. Write the provider-specific rule into the integration specification and keep a sample response for review.

The decision is about the business action after collection, not merely the loop that fetches pages. An inventory feed may drive availability, a customer directory may drive support work, and a booking export may drive reminders. If the process quietly stops halfway, staff could act on incomplete information while the dashboard still says 'success'. Define a completion result with the endpoint, filters, extraction window, pages followed, unique records accepted, and any unresolved error. Present an incomplete run as incomplete, and prevent downstream jobs from treating it as an authoritative full snapshot. This boundary is especially important when an empty page could mean either a genuine endpoint result or an incorrectly constructed request.

Follow the provider's continuation contract

Treat a next URL or token as an opaque instruction supplied by the provider. Microsoft Graph says to request the URL returned in @odata.nextLink until that property no longer appears. GitHub says to follow the next URL in its Link header where present. Do not assume page numbers are consecutive, construct the next token by arithmetic, or copy a Graph rule into a GitHub client. An API may cap page size or change the shape of its continuation later. The consumer should also keep the same authorization and endpoint policy for each follow-up request, and should not send a provider token to an unrelated host.

Agree what a stable record identity means before importing. For example, a partner's immutable product ID can be the local upsert key, while its display name cannot because names change and may repeat. If the provider exposes an update timestamp, preserve it as source data rather than pretending it proves a globally consistent snapshot. Store source ID, fetch time, source update time when available, and the run ID separately. That separation lets the team distinguish an item seen twice across pages from two different items with similar names. It also supports a later reconciliation without rewriting history or erasing which run supplied a value.

Make a failed run restartable

Network interruptions should not force an operator to guess which rows reached the database. Persist progress only after a page's records have passed validation and have been committed. The checkpoint should identify the last completed continuation, the query and filter settings, and the run that owns the checkpoint. If the provider offers a next URL, store it securely as operational state because it can contain sensitive tokens. On restart, retry the same page or resume from a durable checkpoint according to that provider's documented behavior. Never advance the checkpoint before the associated records are durable; doing so creates a gap that ordinary retries cannot repair.

Microsoft Graph's paging guidance includes a specific retry warning: for certain large directory result sets, a token obtained from a retry response may not be a valid token for later pages. Its documented advice is to use the continuation from the last successful non-retry response when requesting the subsequent page. This is a Graph-specific condition rather than a universal API rule. The broader lesson for a partner integration is to test retry semantics with the actual provider and preserve enough state to repeat the request it expects. Write that edge case into an operational runbook only after the provider's contract and your own test confirm it applies.

Plan for records changing during the walk

A paginated list is not automatically a frozen picture of the provider's system. While the consumer reads page two, a record may be inserted, deleted, or moved by the provider's sort order. Offset-style pagination can then repeat or skip a record; a continuation token may reduce some risks but does not by itself promise snapshot isolation. Ask whether the endpoint documents a snapshot, change feed, stable sort key, or update-since filter. If it does not, describe the import as a best-effort traversal and plan a reconciliation pass. Do not assert that a completed traversal equals an exact point-in-time copy unless the provider says so.

A workable example is a supplier product feed that changes all day. First complete the ordinary page traversal into a staging table keyed by supplier product ID. Then compare the staged IDs and update markers with the previous accepted run, and inspect suspicious gaps or sudden drops before publishing availability to staff. If the provider offers an incremental endpoint, use its documented cursor or time window and keep overlap and deduplication rules explicit. An overlap is an editorially proposed safeguard, not a measured requirement of every API. If the provider has no count or reliable update marker, sample a few known records with the partner and record the remaining uncertainty.

A worked import decision

Suppose a local distributor exposes product updates to a Hong Kong shop. The shop's test request returns one page of records plus a next URL, and the documented endpoint says to follow that URL until it disappears. The shop records a run ID and original filter, validates each product ID and price field, then commits each page and checkpoint together. During a simulated interruption after page three, it restarts at the documented continuation and upserts by product ID. A repeated product ID updates one local record instead of creating a duplicate. Those page numbers are merely an illustrative test sequence, not a claim about a particular provider's page count or performance.

The shop then checks the run summary before making the new catalogue available to staff. If the next URL was missing because the final page was reached, the batch may be complete under the documented contract. If the request failed, a response could not be parsed, or a validation rule rejected material records, the batch remains incomplete. A mismatch with the supplier's documented total, where such a total exists, needs investigation before acceptance. The operator can compare source IDs in the old and new runs and ask whether a large deletion reflects real business changes. The key decision is when data becomes trusted enough for a downstream action, not whether a script exited without an exception.

Test the failure states and handover

Use a small test fixture whose expected identities are known. It should include more than one page, a final page, a repeated identity, a changed record, and a failure between fetching and committing a page. Verify that restarting does not lose a record or create an extra local row. Test a malformed continuation and an authorization failure separately; neither should be mistaken for end of list. If the provider has a test environment, exercise its real header or body format there. A local mock helps with edge cases but cannot prove that the live provider will use the same ordering, expiry, or retry rules.

Assign one person to own the provider contract and another operational role to review incomplete runs; these can be responsibilities within the same small team. Document where the checkpoint lives, how to rerun safely, how to compare accepted batches, and whom to contact when the provider changes its API. Keep the business-facing summary honest: last complete import time, data scope, and any unresolved exceptions. If a source cannot support a complete run, publish a clear stale-data notice internally or pause the dependent decision. That makes pagination a controlled data process instead of a hidden implementation detail that only becomes visible after a customer problem.

A final acceptance rule should also separate disappearance from deletion. If a product ID is absent from one traversal, do not immediately erase it from the shop unless the provider guarantees that the run is a complete authoritative snapshot. Check whether the endpoint filters inactive items, whether the query changed, and whether the supplier provides explicit deletion markers. Keep the previous accepted state available while staff investigate a suspicious drop. If the feed is known to be partial, update only the records actually observed and use a separate agreed process for removals. This distinction prevents a temporary pagination gap from becoming an irreversible business action. It also gives operations a precise question to put to the partner: does absence in this endpoint mean the item no longer exists, or merely that it was outside this result set?