Safe auto-answers — when your bot should ask instead of guess

The expensive failure for a support bot or an auto-resolver is not “I don’t know.” It’s a confident wrong answer — auto-replying when retrieval didn’t actually match, or auto-picking one option when the request was ambiguous. The fix is a gate: auto-answer only when you’re confident, and otherwise ask a clarifying question or hand off to a human.

RedHop does not ship a router, a threshold, or a confidence score baked into the answer. By design (it’s a context runtime, not an agent framework) it gives you the signals to gate on and a deterministic eval to prove the gate works. You own the “if confident, answer; else ask” logic. This guide shows the pattern end to end. The runnable version is examples/*/15_safe_auto_answer.

The signals RedHop gives you

Every doc.context(query) returns a Decision Report, and redhop.evaluate(query, ctx) adds self-eval metrics with no ground truth required. The ones to gate on:

Signal	Where	What it tells you
`low_confidence_retrieval`	`ctx.report`	The primary gate. `true` when every selected chunk is at or below the grounding floor — nothing relevant matched. If this fires, ask.
`mean_grounding`	`evaluate(query, ctx)`	A no-gold confidence strength in `[0,1]`: how query-relevant the assembled context is. A clear match scores high, a vague query (“can you help me”) scores ~0.
`diagnosis.score_spread`	`ctx.report`	A margin in `[0,1]` over the retrieved set: `(top − k-th) / top`. A clear winner among several candidates has high spread, a tight race has low spread. It’s `None` when a single chunk dominates (no competition), so use it for the ambiguous-between-candidates case, not as the only gate.
`set_coverage`	`evaluate(query, ctx, gold_families=…)`	For set-valued queries (all sizes of a product, both clauses of an override): `< 1.0` means a family is half-retrieved, so you can’t offer the choice. Ask, don’t auto-pick. See Choosing a configuration → catalogs.

The gate

Route each turn on the signals. A minimal, honest gate: auto-answer only when retrieval is not low-confidence and the grounding strength clears a threshold tau; otherwise clarify.

ctx = doc.context(user_query)
r = redhop.evaluate(user_query, ctx)        # no gold needed for mean_grounding

confident = (not ctx.report.low_confidence_retrieval) and r.mean_grounding >= TAU
if confident:
    answer = call_your_llm(ctx.text(), user_query)   # auto-answer
else:
    answer = "Can you tell me a bit more about what you're looking for?"  # ask

const ctx = doc.context(userQuery);
const r = evaluate(userQuery, ctx);          // no gold needed for meanGrounding

const confident = !ctx.report.lowConfidenceRetrieval && r.meanGrounding >= TAU;
const answer = confident
  ? callYourLlm(ctx.text, userQuery)         // auto-answer
  : "Can you tell me a bit more about what you're looking for?"; // ask

let ctx = doc.context(user_query)?;
let r = evaluate(&Query::new(user_query), &ctx, None, EvalGold::None, None, EvalConfig::default());

let confident = !ctx.report.low_confidence_retrieval && r.mean_grounding >= TAU;
// confident => auto-answer with ctx.text(); else ask a clarifying question.

Add the other two signals as your domain needs them: score_spread < tau_margin when two candidates are genuinely tied (ambiguous → offer both), and set_coverage < 1.0 when the answer is a set and you couldn’t fetch all of it.

Derive the threshold, don’t pick it

tau is not a magic number you guess. Sweep it on a labeled dev set and pick the smallest value that hits your precision target (say 99% auto-precision) — that maximizes how often you can auto-answer at the safety you require.

# Dev set: (query, gold_chunk_ids_or_None). None = "should clarify".
samples = [(q, gold), ...]

rows = []
for q, gold in samples:
    ctx = doc.context(q)
    r = redhop.evaluate(q, ctx, gold_chunks=gold) if gold else redhop.evaluate(q, ctx)
    confident_signal = 0.0 if ctx.report.low_confidence_retrieval else r.mean_grounding
    correct = bool(gold) and (r.context_recall or 0.0) >= 1.0
    rows.append((confident_signal, correct, gold is not None))

# Sweep tau over the observed signal values; pick the smallest tau whose
# auto-precision >= target. Report the whole curve so the choice is evidence.
for tau in sorted({s for s, _, _ in rows}):
    autos   = [(c, ans) for (s, c, ans) in rows if s >= tau]
    correct = sum(1 for c, _ in autos if c)
    unsafe  = sum(1 for _, answerable in autos if not answerable)  # auto'd a "should-clarify"
    precision = correct / len(autos) if autos else 1.0
    print(f"tau={tau:.2f}  auto_rate={len(autos)/len(rows):.2f}  "
          f"precision={precision:.3f}  unsafe_auto={unsafe}")

Pick the smallest tau with precision >= 0.99 and unsafe_auto == 0. Re-derive it per corpus and re-check it as the corpus grows — the right threshold moves when retrieval gets harder (see the scale story below).

Measure the gate: auto-precision and unsafe-auto

Two numbers tell you whether the gate is safe, both computable with evaluate on a labeled set (no LLM judge):

auto-precision = correct ÷ auto-answered. Your safety gate. Aim ≥ 99%.
unsafe-auto = how many should-have-clarified queries you auto-answered anyway. The scary one. Target 0. A single confidently-wrong answer to an ambiguous request is worse than a hundred clarifying questions.
auto-rate = fraction you answered without asking. The throughput you’re trading against safety.

redhop.summarize(reports) rolls these over a whole test set, and summary.low_confidence_rate tells you what fraction of traffic your corpus can’t confidently answer yet (a content backlog, see Evaluation in production).

The headline: gets cautious, not wrong

A correctly-derived gate has a property worth designing for: when retrieval degrades (a bigger, noisier corpus; a query the KB doesn’t cover), the system routes more queries to clarify rather than producing bad auto-answers. In a catalog-scale stress test, when the corpus grew large enough that retrieval started missing, the auto-rate fell but auto-precision stayed ~1.0 and unsafe-auto stayed 0 — the system got cautious, not wrong. That is the safety property the whole gate exists to buy. (Illustrative, from one workload — measure it on yours.)

When the answer is a set, ask instead of picking

If a query maps to several valid options (every size of a product, both regional overrides of a clause), auto-picking one is an unsafe-auto even if it’s a plausible pick. Score set_coverage with gold_families: when it’s < 1.0 you couldn’t even retrieve the whole option set, so clarify and offer what you have.

r = redhop.evaluate(query, ctx, gold_families=[["sku_a1", "sku_a2"], ["sku_b1", "sku_b2"]])
if (r.set_coverage or 0.0) < 1.0:
    ask_user_to_disambiguate(ctx)   # don't auto-pick one variant

Localizing a failure (replay with gold)

When an auto-answer was wrong, replay the query and check context_recall. If the gold chunk wasn’t retrieved (context_recall low), the failure is retrieval — fix the corpus, the analyzer (language="char_ngram" for noisy tokens), or field weights. If it was retrieved but the answer was still wrong, the failure is generation — fix the prompt or model, not the retriever. The Evaluation page has the full signal-to-stage table.

What RedHop does, and what you own

RedHop gives you	You build
`low_confidence_retrieval`, `mean_grounding`, `score_spread`, `set_coverage`	the routing rule (auto / clarify / human)
a deterministic, no-LLM `evaluate` + `summarize` to score the gate	the threshold `tau`, derived on your dev set
`context_recall` for replay-based attribution	the clarifying-question and human-handoff UX

This split is deliberate. The signals and the measurement are the hard, get-them-wrong-and-you-ship-a-confidently-wrong-bot part, and they’re the same primitives RedHop uses for its own Decision Report, so they can’t drift from what the runtime actually did. The routing policy is yours because it’s domain-specific — your precision target, your “ambiguous,” your handoff.