Skip to content
Kiran
← Work

A published internal package · adopted by other teams

Multi-Provider LLM Batch Runtime

Two ideas make this worth more than its size. First, rate limits are treated as an unknown to be discovered: where a provider publishes a quota, a token bucket enforces it; where it does not, an additive-increase multiplicative-decrease controller finds the edge and backs off from it: TCP congestion control, applied to an undocumented API quota. Second, the safe path is the easy path: construction from the environment resolves credentials from the platform secret manager, so no consuming project handles a key and nobody has a reason to create a local secrets file.

2026platformeconomics

What I owned

Sole author. Design, implementation, packaging, and a substantial README.

Providers supported
4codeProviders supported
Other projects that adopted it
4+codeOther projects that adopted it
Initial adaptive delay
6.0scodeInitial adaptive delay
Built onPython package with an internal indexPydanticSecret manager integration
01Problem02My role03Decisions04What broke05Impact

The problem

What was actually wrong

Several projects needed to run large batches of model calls across different providers, and each was solving the same problems independently: rate limiting, retry on throttling, structured-output validation, and credential handling.

The credential part was the one that mattered most. Every project rolling its own meant every project inventing a way to hold a key, and the easiest way to hold a key is a file on a laptop.

Constraints

The conditions the design had to hold under

Constraints are the interesting part of an architecture. Without them any diagram looks reasonable.

Some provider quotas are undocumented
Internal gateways in particular publish no rate limit. You find it by hitting it, which means the runtime has to be able to hit it safely.
The consumers are other people's projects
Anything requiring setup will be set up wrong somewhere. The default path has to be the correct one.

My role

Sole author. Design, implementation, packaging, and a substantial README.

Designed

  • The two-strategy rate-limiting interface: token bucket where the quota is known, adaptive control where it is not
  • Zero-configuration construction that resolves credentials from the platform secret manager
  • A provider base class such that adding a provider is one file

Built

  • The adaptive controller, throttle re-queuing, structured-output validation, callback hooks and tracing integration
  • The package itself, versioned and published to an internal index

Decisions

The calls I would defend

Each one with the alternatives I rejected, what the choice cost, and how it turned out.

Decision

Two limiter strategies behind one interface: a token bucket when the requests-per-minute figure is published, and an adaptive controller when it is not.

Context

Internal gateways frequently publish no quota, and the real limit changes without notice.

Alternatives, and why not

  • Require every consumer to configure a rateThey will guess. Guessing low wastes hours on a batch; guessing high produces a throttling storm and a support conversation.
  • Retry with exponential backoff and no pacingIt converges on the limit by repeatedly exceeding it, which is the behaviour that gets a service throttled at the account level.

Rationale

Start conservative at six seconds between requests, reduce the delay by ten percent after three consecutive successes, multiply it by 1.8 with a ten-second cooldown on a throttle, and clamp to a configured range. Additive increase, multiplicative decrease: the same control loop that governs TCP, applied to an API quota.

What it cost

The first minute of a batch is slower than a correctly configured constant rate would be. The trade buys never needing that configuration to be correct.

Outcome

Batches self-tune to whatever the provider is currently allowing, including when that changes mid-run.

What broke

Failures, and what they changed

Every one of these is a thing that went wrong in a system I own. They are here because the architecture is largely a record of them.

Throttled requests were being lost rather than retried in order

What happened
Early batches dropped throttled items instead of returning them to the queue, so a large batch would silently complete with gaps.
Root cause
Throttling was handled as an error at the call site rather than as a queue state.
What I did
Throttled requests are re-queued and the controller's delay is increased, so the batch slows down rather than shrinking.
What changed in the architecture
Rate limiting and queue management became one concern rather than two.
What it taught me
A partial batch that reports success is worse than a slow one. Make the degradation latency, never completeness.

Impact

What changed, and how it is known

Every figure carries its basis. Nothing here is rounded up, and nothing modelled is presented as a result.

4code
Providers supportedVertex, an internal gateway, a direct provider API, and any LangChain-compatible model.
4+code
Other projects that adopted itVendored or depended upon, including one where I authored only a small fraction of the commits.
6.0scode
Initial adaptive delayReduced 10% after three consecutive successes; multiplied 1.8× with a 10s cooldown on a throttle.
Business
The clearest evidence of reuse in my work: a library I wrote alone was picked up and vendored by other engineers into projects I barely contributed to.
Engineering
Adding a provider is one file. Rate limiting, throttle re-queuing, structured-output validation and tracing come for free to every consumer.

Leadership and hindsight

What I influenced, and what I would change

Technical leadership

  • Wrote a genuinely long README, because a library nobody can adopt without asking me is not a library. The adoption evidence follows from that more than from the code.

What I would do differently

  • The adaptive controller has no memory across runs. It rediscovers the same limit every time, which costs the first minute of every batch. Persisting the learned delay per provider would be a small change with an obvious payoff.
  • The two limiter strategies share an interface but not their observability. The token bucket reports utilisation; the adaptive controller reports its current delay. A consumer switching between them loses continuity in their dashboards.

Stack

What it is built on, and why that

A technology list without reasons is a list of things I have heard of.

Python package with an internal index
Versioned distribution across teams.Distribution is what turns a useful module into something other teams can adopt without copying it.
Pydantic
Structured-output validation on model responses.A batch that returns unvalidated JSON pushes the parsing failure into the consumer's loop, where it costs the whole batch.
Secret manager integration
Automatic credential resolution.It removes the reason anyone would create a local secrets file.

3 more sections are written and hidden: constraints, leadership, stack.

Next