WardrikGateway

Rate limits, circuit breakers and model failover for Wardrik agents' model calls.

def deps do
  [
    {:wardrik_gateway, "~> 0.1.0-rc.1"}
  ]
end

Usage

children = [
  {WardrikGateway,
   name: MyApp.Gateway,
   fallbacks: %{"anthropic:claude-sonnet-4-5" => ["openai:gpt-5", "zai:glm-5.1"]},
   limits: %{"anthropic" => [requests_per_minute: 4000, tokens_per_minute: 400_000]}},
  {MyApp.AgentRuntime, middleware: [{WardrikGateway, MyApp.Gateway}]}
]

With WardrikCore alone, pass the same entry in an agent's :middleware.

• A model with an open breaker is skipped. The call goes to the next model in fallbacks.
• A call to a provider at its limit is delayed until the provider has capacity.
• A 429 delays calls to that provider for its Retry-After, or one second.
• A 5xx, a 529 or a timeout counts towards the model's breaker. The call goes to the next model.
• After a fallback, the rest of the turn uses the fallback.
• A call is delayed at most :wait, 10 minutes by default. Then WardrikCore.Retry applies.

configure/2 replaces the settings at runtime:

WardrikGateway.configure(MyApp.Gateway, limits: %{"anthropic" => [requests_per_minute: 2000]})

Scale

Gateway overhead per model call, from mix run bench/gateway.exs: AMD Ryzen AI Max+ 395 (32 threads), Elixir 1.20, OTP 28, range of five runs.

Limits

• Limits are counted per node. On several nodes, divide each provider's quota between them.
• Fallbacks use the deployment's API key for their provider. Agent definitions can't set them.

Related

• Running Wardrik: :middleware, :model_options and the connection :pool.
• WardrikCore.Middleware: the behaviour this package implements.
• Model gateway: the same settings in the daemon's config.yaml.

License

Apache-2.0