Software Architect & Builder In Production (Private)

small-router

Provider-Agnostic Inference Layer for Coding Agents

GoOpenAI-compatible APIPrometheusSSEYAML config

Diagramas

Request Path

REQUEST PATH — SMALL-ROUTERCoding AgentPOST /v1/chat/completionssmall-routerauth · sanitise promptrouting policyResilienceretry · backoff + jitterper-provider circuit breakerfallback chain429/500/503 → retry + fallback400/401/403 → haltProvidersin-house fleet + serverlesssame catalogueSecurity boundarykeys isolated in the router environmentprompt · completion · tools · keys — excluded from logsObservability — blind to contentcost per session · latency · Prometheus metricsembedded dashboard · loop guard · daily spend capone endpoint · policy or direct model ID · declarative config in router.yamlthe provider never accesses the codebase · tests run 100% offline

Project Overview

small-router is the execution plane of AI Factory: the layer that decides which model — and which provider — runs each call. It exposes an OpenAI-compatible endpoint and, behind it, routes by policy or by a direct model ID, with a fallback chain, retry with exponential backoff and jitter, and a per-provider circuit breaker. The same endpoint serves both models from a self-hosted fleet and serverless providers in the same catalogue — the client asks for a capability, not a machine name. It is also the security boundary: keys are isolated in the router's environment, prompts are sanitised before leaving, and prompts, completions, tool arguments, and API keys are deliberately excluded from logs and metrics. On top of it, an embedded dashboard shows real-time KPIs — cost per session, latency, active sessions, loop detection, and a daily spend cap.

Architecture & Implementation

The path of a call is short and explicit: the agent makes POST /v1/chat/completions, the router authenticates, sanitises the prompt, and applies the routing policy; from there it picks the provider, with retry and circuit breaker along the way. Non-retryable errors (400/401/403) halt the fallback chain; retryable errors (429/500/503/timeout) trigger retry and fallback. The provider never accesses the codebase directly — the router is what receives and cleans up before proxying. Configuration is declarative (router.yaml): model registry, routing policies, aliases, and fallback chains, without touching code. Observability is deliberately blind to content: structured JSON logs, per-request cost estimation, and Prometheus metrics record what happened, never the text. Tests run 100% offline, against mocks and fake providers, without touching the network.