small-router
Provider-Agnostic Inference Layer for Coding Agents
Diagramas
Request Path
Project Overview
small-router is the execution plane of AI Factory: the layer that decides which model — and which provider — runs each call. It exposes an OpenAI-compatible endpoint and, behind it, routes by policy or by a direct model ID, with a fallback chain, retry with exponential backoff and jitter, and a per-provider circuit breaker. The same endpoint serves both models from a self-hosted fleet and serverless providers in the same catalogue — the client asks for a capability, not a machine name. It is also the security boundary: keys are isolated in the router's environment, prompts are sanitised before leaving, and prompts, completions, tool arguments, and API keys are deliberately excluded from logs and metrics. On top of it, an embedded dashboard shows real-time KPIs — cost per session, latency, active sessions, loop detection, and a daily spend cap.
Architecture & Implementation
The path of a call is short and explicit: the agent makes POST /v1/chat/completions, the router authenticates, sanitises the prompt, and applies the routing policy; from there it picks the provider, with retry and circuit breaker along the way. Non-retryable errors (400/401/403) halt the fallback chain; retryable errors (429/500/503/timeout) trigger retry and fallback. The provider never accesses the codebase directly — the router is what receives and cleans up before proxying. Configuration is declarative (router.yaml): model registry, routing policies, aliases, and fallback chains, without touching code. Observability is deliberately blind to content: structured JSON logs, per-request cost estimation, and Prometheus metrics record what happened, never the text. Tests run 100% offline, against mocks and fake providers, without touching the network.