Why a successful request can still be incomplete
An import job may report success because its first HTTP request returned a normal response, yet the business report contains only the first page of records. Many APIs divide a collection into pages. Microsoft Graph, for example, returns an @odata.nextLink when another page exists; its documentation says a client must follow successive links until one is absent to read all results. A requested page size is not a promise that the entire collection fits in one response. The owner should define success as a complete, reconciled run, not as one green network status. This distinction matters before stock, customer or invoice data is trusted in a dashboard or another system.
First decide exactly which records the run intends to capture: a full collection, one date range, or changes since a prior checkpoint. Define which system is authoritative and what happens when a source record changes while pages are being read. A live list is not necessarily a frozen snapshot. An importer should state the consistency guarantee the source API actually provides; if none is documented, reconciliation and a later catch-up pass may be necessary. Keep the business unit of work visible: “all orders updated since the last completed checkpoint” is more useful than “the request finished.” A partial result must never silently replace the previous complete dataset.
Follow the server’s cursor
Treat the next-page URL or cursor returned by the provider as opaque. Do not infer that page two can be constructed by adding a number to the first URL unless the specific API says so. Microsoft Graph notes that support for $top, $skip and skip tokens varies by endpoint; its /users example supports $top but not $skip. Preserve the query that established the run, follow the returned link and process each page once it has passed validation. The final page is the one for which the documented continuation value is absent. A short page is not, by itself, evidence of completion when the server supplies an explicit continuation link.
A useful importer records the run identifier, source endpoint, start boundary, continuation state, pages fetched, accepted record IDs and the last completed checkpoint. Protect continuation URLs as operational secrets if they contain tokens or sensitive parameters; do not paste them into public error logs. Before loading a page, verify that its response has the expected shape and that mandatory identifiers are present. Separate raw retrieval from publishing the finished dataset. This allows staff to investigate one bad page without presenting a mixture of old and new data as a clean report. It also lets the business distinguish a missing source record from a record omitted by its own importer.
Recover from throttling and interrupted pages
A 429 response is not an empty page and must not advance the checkpoint. Microsoft Graph tells clients to wait for the Retry-After interval on throttling, then retry; if that header is absent, its guidance recommends exponential backoff. Repeated immediate calls can prolong the throttle. Other providers may use different status codes or limits, so implement their documented policy rather than copying Graph rules blindly. Distinguish temporary service failure from a permanent permission or query error. An alert should say which run and page failed, how many attempts were made, and whether the last complete dataset remains available to users.
Retry the same logical page before committing progress. Microsoft Graph documents a subtle paging issue: tokens returned during a retry can lead to DirectoryPageTokenNotFoundException if used as the basis for subsequent requests; follow its page-token guidance for that service. For a generic integration, design an idempotent page load: identify source records by stable IDs and use upsert or a staging table so a retry does not create duplicate business rows. Do not assume a page is safely stored merely because the response arrived. The checkpoint belongs after durable validation and write, with the exact continuation state and run context retained for resumption.
Work through a partial run
Imagine a wholesaler importing supplier product records overnight. The first page contains familiar products, the second includes recently added items, and a third page fails temporarily. If the job publishes pages one and two as a complete catalogue, staff may conclude those newer items were deleted. The correct run state is incomplete. Keep the prior verified catalogue available, retry the failed page under the provider’s rules, and publish the new set only after the continuation chain finishes and validation passes. This is a hypothetical illustration, not an observed incident or claim about a particular supplier. Its purpose is to make the consequence of a silent partial import concrete.
The acceptance test should inject several conditions: a multi-page result, an empty final page where permitted, a 429 with Retry-After, a network timeout after receiving data, and a malformed record. Check that none marks the run complete too early. Repeat a page and confirm the destination still has one row per source ID. Interrupt the process and resume from its saved state. Then compare a known source sample and record counts or other provider-supported reconciliation evidence. Counts alone cannot prove every item is correct, but a sudden drop is a useful warning. Define which discrepancies stop publication and which are routed for human review.
Choose a sustainable update pattern
A complete initial import and a recurring update are different jobs. A nightly full scan can be simple, but large collections may consume time and hit limits. Microsoft Graph’s throttling guidance warns that regular full collection scans and continuous polling are more likely to be throttled. If the specific source supports delta or change queries, evaluate those instead, including how deletions, expired tokens and catch-up work are represented. Never switch to an incremental method merely because its name sounds efficient; test that it yields the business changes the destination needs. Periodic reconciliation may still be necessary to catch missed or misunderstood events.
Give the business a clear operational dashboard: last complete run time, source coverage, pages read, records accepted, records rejected, pending retries and age of the published dataset. Avoid displaying “up to date” when a run is still staging. Set an alert threshold from the business’s actual tolerance for stale information, not an invented universal interval. The next action is to trace one existing import from first request to final publication and ask where it proves that all pages arrived. If there is no proof, add completion and reconciliation checks before using the data for consequential decisions. Astera can help design this integration contract and its monitoring around the source API’s real capabilities.
Preserve permissions and ownership
Completeness does not justify asking the API for unlimited access. Give the importer only the permissions and fields required for its defined business use, and document who can rotate its credential and view its logs. When a query changes, restart or explicitly version the run rather than applying an old continuation link to a new scope. A field added by the provider may be absent from older records; a field removed from a response may indicate a permission change. Treat such schema changes as review events. They can make a technically complete page chain produce an incomplete business dataset if required information is silently discarded.
Assign one person to own the meaning of “complete” for each imported dataset. An engineer can prove the code followed all continuation links, but a warehouse manager may notice that discontinued items, variants or returns were excluded by the query. Agree on sample records that exercise those cases before going live. Keep an audit trail from each published row to the source identifier and import run so staff can trace a disputed figure. If the API offers no reliable count or snapshot, be candid about that limit and use targeted checks against the source interface. This is stronger evidence than a generic success message, while still acknowledging what cannot be proven automatically.
Before trusting the import
- Follow each documented continuation value until the source says the collection is finished.
- Treat throttling and timeouts as incomplete runs and resume without duplicate rows.
- Reconcile and publish only a validated complete dataset.
