HAOL 0.5.0: The Router That Watches Itself

When I started rebuilding HAOL (Heterogeneous Agent Orchestration Layer) in March, the routing layer was a regex engine pointed at four tiers of agents. In May, it became a system that captures every routing decision, surfaces the patterns to me on demand, gates regressions in CI, and uses real production data to tune itself. This post is about that journey — the bits I’m proud of, the bugs I tripped over, and what it took to go from “the router works” to “the router watches itself work.”

The starting point: rules and tiers#

The classifier scored prompts into four complexity tiers (T1 cheap and fast, T4 expensive and capable) and selected an agent with a weighted formula: capability × 0.5 + cost × 0.3 + latency × 0.2. That worked for a hello-world demo, but it had no way of knowing whether its decisions were any good. A T3 prompt routed to a T1 agent that returned garbage looked exactly the same in the logs as a T1 prompt routed to a T1 agent that returned a perfect answer. There was no signal coming back.

So the first thing I built was the signal.

March: capturing outcomes, then closing the loop#

The first change added a four-tier outcome taxonomy (success, partial, failure, rejected) and an endpoint to record downstream signals against every task. From day one, every routing decision got annotated with how it actually went. Even if I didn’t use the data yet, I was now collecting it.

The hard part was making sure the data was clean. A parade of tiny correctness bugs surfaced once real outcome data started flowing — a missing size limit on a JSON field, miscounted pending records, non-deterministic sort order, swallowed fallback errors, connection pool state bleeding between tests. None of them are interesting individually, but collectively they’re the difference between an outcome dataset I trust and an outcome dataset I don’t.

Two weeks later, I built the routing tuner — a closed-loop learning system that aggregates agent performance per (agent × tier) combination from accumulated outcome signals, watches for repeated high-confidence LLM escalations and crystallizes them into cheap deterministic rules (“the LLM keeps deciding 'kubernetes' is T3 — let’s make that a regex, save the call”), and promotes successful fallback prompts into reference utterances for the semantic similarity layer. The tuner runs as a single command (haol tune), is fully reversible via Dolt’s commit history, and has a --dry-run flag to preview what it’d do. It only fires when sample sizes are above a threshold and confidence is high.

This was the first inkling of HAOL as a self-modifying system. The router had started to learn from its own decisions.

March–April: production guardrails and a face#

By mid-March, the system was working well enough that I needed to stop people from breaking it. Bearer-token API key authentication shipped (with timing-safe comparison via SHA-256 hashing), along with per-IP rate limiting with proper Retry-After and X-RateLimit-* headers, prompt size caps, the database indexes I’d been meaning to add, and a fix for Dolt connection safety where the withConnection / withBranchConnection split protects branch-mutating operations from racing.

Then came the cascade trace: every routing attempt across all four layers (deterministic rules → semantic similarity → LLM escalation → fallback) is now captured in a structured CascadeTrace object — which agent was selected at each layer, why subsequent layers were skipped, latency per layer. The full journey of every decision is preserved.

A small but satisfying side quest followed: a static demo UI that visualizes classification in real time. You paste a prompt, you see the cascade light up layer by layer with similarity scores and final tier assignment. It’s the kind of thing that’s a nightmare to build and a delight to use.

By April 4, all of this was tagged as v0.4.0. The system was production-shaped (if you squint).

May: making it measurable#

This is where the second half of the story starts. By May, HAOL had been making routing decisions for a while and accumulating cascade trace data — but that data lived per-task, with no aggregate view. “Is the routing brain degrading?” was answerable only by reading individual traces or grepping logs.

Three new features fixed it.

The first was a load test as a CI gate. The harness submits 23 scenarios spanning T1–T4 and edge cases against a running HAOL server, computes p50/p95/p99 percentiles, per-tier breakdowns, and routing-assertion mismatches — each scenario carries an expectedTier and the load test reports when actual ≠ expected. Threshold flags for max p95 latency, max cost, and max failure rate cause non-zero exit. A GitHub Actions workflow provisions Dolt, seeds, starts the server, runs the load test against real LLM providers, and posts the report to the job summary. It’s manual-trigger-only by default — calling real APIs costs real money.

The second was contract tests for the security middleware. Four middleware modules (api-key-auth, rate-limit, error-handler, request-id) had zero direct test coverage — exactly the kind of code where a silent regression would be a high-severity production incident. Forty new tests filled the gap, and two real bugs surfaced just from writing them. The rate limiter’s no-socket fallback used key = "global" (its initial value), the same key as global-mode buckets — within a single instance the closure-scoped Map made it harmless, but the comment lied and any future refactor would silently break. And request-id passed the raw header through unsanitized, making anyone logging request IDs (i.e., anyone) a CRLF-injection or arbitrary-control-char log injection target. It now sanitizes and falls back to UUID if the cleaned value is empty.

The third was observability for the routing brain itself: a new GET /observability/cascade endpoint that aggregates routing_log into a snapshot of per-layer hit-rate, tier distribution, latency percentiles (overall and per-layer), confidence and similarity distributions, and the top 20 near-miss decisions sorted by similarity_score DESC. A companion /cascade/timeseries returns bucketed escalation rate over time. A careful line-level review caught a subtle issue here: the four underlying queries fire concurrently via Promise.all on independent connections, so under heavy concurrent writes the counts and percentile distributions can briefly disagree. Strong consistency for monitoring data is overkill, but the inconsistency must be visible, so I added snapshot_at (ISO timestamp) and consistency: "best_effort" to the response.

The payoff: fixing T3 over-escalation#

I now had three things I didn’t have a week earlier: a load test that flags routing mismatches, an endpoint that aggregates real routing data, and test coverage on the safety-critical middleware.

Running the load test for the first time, the routing-assertion section showed 13 of 23 prompts went to the wrong tier. The observability endpoint confirmed it: 61% of all decisions were hitting T3 (the most expensive tier). Something was very wrong.

The classifier flattens instructions and data into one string and pattern-matches against the whole thing. So a simple T1 task like “Extract dates from this contract” would match a T3 keyword like “function” if the contract data happened to mention functions.

Three issues were conspiring against accurate tier matching. The regex patterns matched anywhere in the prompt — \b(implement|function|debug|refactor)\b would catch “extract from this function spec” or “the function returned an error” with no concept of intent vs. mention. Then max(tier) clobbered priority: when multiple rules matched, runDeterministicRules picked the highest tier, so “Summarize this analysis report” hit both rule-summarize (T1) and rule-reasoning (T3) and resolved to T3. And the priority column was effectively dead — the matcher iterated all rules and picked max tier, so the column influenced iteration order, but iteration order doesn’t matter for max(). The field that looked like it should make T1 rules short-circuit T3 rules had no effect on the outcome.

The fix did three things at once. Priority became a real short-circuit: the first matched rule wins for tier, while capabilities still aggregate across all matches. The regex patterns tightened around intent — strong code verbs (implement/debug/refactor/optimize) match alone, generic verbs (write/build/create) require a code-noun within ~40 characters, reasoning rules match verb forms only (which drops “analysis” / “comparison” / “evaluation” as descriptive nouns), and tool-use requires a phrase match or action verb. And a defensive sort in the matcher itself ensures the priority-order contract isn’t only enforced by loadRules()’s ORDER BY clause.

Did the fix work?#

Re-running the load test against the new rules:

Three specific scenarios I’d targeted moved from T3 to T2 exactly as predicted: “JSON structured output,” where Analyze no longer triggers rule-reasoning; “Data table generation,” where comparing no longer triggers rule-reasoning and rule-structured (priority 15) wins via short-circuit; and “Complex data analysis,” where analysis is a noun form and no longer matches.

The raw count didn’t move — 13 of 23 still flagged — but the composition flipped entirely. The remaining mismatches are cases where the load test’s expected tier is debatable (T2 vs T3 for “Refactor this middleware” is a judgment call, not an error). The over-escalation that wasn’t debatable is gone. The router stopped over-escalating; the load test’s expectations are the next thing to calibrate.

What I’d take away#

Three things stand out from the spring.

Outcome data is the prerequisite for everything. I built the outcome capture in week one and didn’t use it for anything for two weeks. Then once I had a routing tuner, an observability endpoint, and a load test, every one of them depended on having clean outcome and trace data already in place. Build the dataset before you need it.

Observability and synthetic regression tests are mutually reinforcing. The load test catches things in CI that would otherwise need real traffic. The observability endpoint catches things in real traffic that the load test doesn’t cover. Either alone is half a system; together they form a tight feedback loop. The T3 over-escalation fix was diagnosed in days because both surfaces were measuring it.

Human code review catches what tests don’t. A thorough mental walkthrough of the code found real bugs the tests missed: the rate-limit fallback-key naming, the X-Request-ID injection vector, and the snapshot’s read-skew. Tests verified the code did what I wrote; review verified that what I wrote was what I meant. Both are necessary.

The system isn’t done — there’s still no real-traffic dashboard, the migration runner has fragile parsing, and a handful of routing edge cases fall through to T3 by default. But it’s measurable now. And once a system is measurable, every problem with it becomes a tractable one.