Rate Limiting

Architecture

Ensemble uses a local-first rate management design optimized for zero hot-path latency:

Request path (microseconds):
  Local atomic counter check → Allow/Deny

Background (every 1 second):
  Local counters → Redis → Global view update

No Redis queries on the request path. Local counters use atomic.Int64 for lock-free operation.

Rate Limit Tracking

Per-Endpoint Limits

Each endpoint has RPM (requests per minute) and TPM (tokens per minute) limits:

endpoints:
  - id: anthropic-primary
    rpm_limit: 1000
    tpm_limit: 100000

Local Counter Structure

Counters are packed into a single atomic.Uint64 for cache-line efficiency:

// Upper 32 bits: TPM count
// Lower 32 bits: RPM count
packed := atomic.Uint64{}

Window rollover uses CompareAndSwap — no mutex, no contention.

Global View

Background sync publishes local counters to Redis and reads global aggregates:

Redis key: {namespace}:ratelimit:{endpoint_id}:{model_id}

The global view is used for routing decisions (avoid sending traffic to endpoints that other instances have already saturated) but is never on the request path.

Mock Endpoint Detection

Endpoints with TPM limits above a configurable threshold (MockEndpointTPMThreshold) are treated as unlimited locally — useful for testing and development.

RateDecision

Every rate limit check produces a RateDecision:

type RateDecision struct {
    Allowed     bool
    CurrentTPM  int64
    LimitTPM    int64
    Utilization float64        // 0.0-1.0
    WindowStart time.Time
    RetryAfter  *time.Duration // Set when rate limited
}

Per-Key Rate Limits

API keys can have their own rate limits (in addition to endpoint limits):

{
  "rate_limit_rpm": 100,
  "rate_limit_tpm": 50000
}

Key-level limits are checked before endpoint-level limits.

Redis Namespace Isolation

Rate limit keys are namespaced by environment:

Priority 1: REDIS_NAMESPACE env var
Priority 2: ENSEMBLE_ENVIRONMENT env var (dev/staging/production)
Priority 3: Test mode detection (unique per test run)
Default: "default"

This prevents development instances from interfering with production rate limit state.