← Back to Insights
Software Architecture

API Design Principles for Data Platforms

Joshua Garza

Key Takeaways

  • A shared database connection string is not an API strategy — every internal schema detail becomes an implicit contract that breaks consumers when changed.
  • Choosing the right protocol (REST, GraphQL, or gRPC) depends on your consumer segments; mature platforms often expose multiple protocols for different workloads.
  • Cursor-based pagination provides constant-time performance at any depth, making it the correct default for data platforms serving large result sets.
  • Contract-first development with OpenAPI turns the API design into a reviewable artifact before any implementation code is written, enabling generated docs, SDKs, and mock servers.
  • Authentication and rate limiting are non-negotiable at the API layer — without them, audit trails and access revocation are impossible.

Every team that builds a data warehouse eventually faces the same question: how should other systems access the data? The answer matters more than most teams realize at the time they make it.

The default answer is often the wrong one. Too often, the default is a shared database connection string — quick to provision, immediately functional, and quietly catastrophic. Every table name, column type, and internal join relationship becomes an implicit contract. Rename a column, and someone's dashboard breaks. Migrate to a new storage engine, and a downstream pipeline fails at 2 AM.

This post covers the concrete design decisions that prevent that outcome: protocol selection, versioning, pagination, authentication, and contract-first specification. These are not abstract principles — they are the specific choices that determine whether a data platform can evolve independently or becomes a coordination bottleneck for every team that touches it. Skip any one of them, and the platform drifts toward implicit contracts — where a column rename becomes a cross-team incident and nobody can answer who accessed what or why.

Why Data Platforms Need Real API Boundaries

A database connection is not a contract. It exposes every table, column, and index — turning internal implementation into an implicit promise. Rename a column, and every consumer breaks. Schema evolution becomes an organizational coordination event rather than an engineering decision.

An API boundary separates what the platform promises from how it stores. Behind that boundary, you can migrate storage engines, restructure schemas, or add caching — without notifying anyone.

Operational concerns live at the API layer. Rate limiting, authentication, audit logging, access control — none are enforceable when consumers hold direct database credentials.

REST, GraphQL, and gRPC: Protocol Trade-offs for Data Workloads

With an API boundary established, the first concrete decision is protocol selection. Each option brings distinct trade-offs for data workloads:

  • REST is the default for good reasons: resource-oriented design maps naturally to datasets, and stateless requests make network-layer caching straightforward for slowly-changing aggregates. The trade-off is over-fetching from wide records.
  • GraphQL lets consumers request exactly the fields they need, but introduces unpredictable query complexity and N+1 resolver problems that are difficult to govern at scale.
  • gRPC pairs Protocol Buffers with HTTP/2 for high-throughput internal pipeline communication. Binary serialization is fast but harder to debug, and browser support requires proxying.

Mature platforms often expose multiple protocols for different consumer segments.

Backward Compatibility and Versioning

Regardless of protocol, the API will need to change over time. Versioning discipline determines whether those changes are routine or disruptive.

URI versioning (/v1/datasets, /v2/datasets) is the most explicit approach — visible in logs, easy to route. The discipline that makes it sustainable: additive changes (new fields, new optional parameters, new endpoints) never require a version bump. Breaking changes (removed fields, altered types, changed semantics) always do.

When retiring a version, communicate timelines through Sunset and Deprecation response headers. Removing a version without notice destroys platform trust faster than any bug.

Pagination and Filtering for Large Datasets

Data platforms serve large result sets, making pagination strategy a critical design choice rather than an afterthought.

Offset pagination (?page=3&limit=100) is familiar but deceptive. The database still scans and discards rows to reach the offset, degrading at depth. Concurrent writes shift results between pages.

Cursor-based pagination solves both problems. The API returns an opaque cursor; the next request anchors from it. Performance is constant regardless of depth. The Google API Design Guide¹ codifies this as the page_token pattern.

For filtering, accept constrained, field-specific parameters (?status=active&created_after=2024-01-01) — not arbitrary query passthrough. Constrained filters are indexable, cacheable, and auditable.

Authentication and Rate Limiting

Authentication is non-negotiable, even for internal APIs. Without it, you cannot audit who accessed what, and you cannot revoke access when a consumer misbehaves. OAuth 2.0 with short-lived bearer tokens is the standard approach. API keys work for machine-to-machine scenarios if you enforce automated rotation.

Rate limiting protects platform stability. Return X-RateLimit-Remaining and X-RateLimit-Reset headers on every response so consumers can self-regulate, and HTTP 429 with a Retry-After header when limits are exceeded. Differentiate limits by operation — a batch export scanning millions of rows deserves a stricter ceiling than a metadata lookup returning kilobytes.

Contract-First Development with OpenAPI

Each of the decisions above needs to be captured in a single, authoritative artifact. That is where contract-first development comes in.

Contract-first development means writing the API specification before implementation code. The OpenAPI Specification² provides a machine-readable format for endpoints, parameters, response schemas, authentication requirements, and error codes.

The benefits compound:

  • Generated documentation stays accurate
  • Client SDKs can be auto-generated
  • Validation middleware catches response regressions
  • Mock servers let consumer teams integrate before implementation is complete

There is a cultural benefit too. The API design becomes a reviewable artifact — product managers, security reviewers, and consumer leads evaluate the contract before any code exists.

Conclusion

The decisions covered here are not implementation details to revisit later — they are the primary interface through which a data platform delivers value:

  • Protocol selection
  • Versioning discipline
  • Pagination strategy
  • Authentication
  • Rate limiting
  • Contract-first specification

Every integration a consumer builds will depend on these choices holding steady.

Make them before the first consumer connects. Encode them in an OpenAPI specification. Treat the contract as a first-class product artifact. The platform earns trust one reliable API response at a time.

References

  1. Google API Design Guide — https://cloud.google.com/apis/design
  2. OpenAPI Specification — https://www.openapis.org