| métrica | naive | reranked (llm) | Δ |
|---|---|---|---|
| hit@8 higher better | +0.167 | ||
| unique_ratio higher better | +0.609 | ||
| abstention higher better | +0.000 | ||
| MRR (known-item) ⚠ dedup-biased | -0.092 | ||
| NDCG@8 ⚠ dedup-biased | -0.050 |
| percentil | naive | reranked | Δ |
|---|---|---|---|
| avg | 3070 ms | 4946 ms | +1876 ms |
| p50 | 3083 ms | 4781 ms | +1698 ms |
| p95 | 3158 ms | 6363 ms | +3205 ms |
mb_search não dedupa: 61% duplicata16/16 casos. G02 retornou o mesmo doc ×8 (unique_ratio 0.12). O consumidor de copy recebe ~3 docs úteis num top-8. Fix: portar dedup (kind,slug) que o sibling reranked já tem.
Default MB_RERANK_MODE=off → reranked devolve só dedup, sem cross-encoder. E adversariais off-domain (score 0.18–0.26) recebem 8 chunks de copy mesmo com sinal limpo para abster. Fix: mode=llm default + confidence-floor 0.32 + web-fallback.
A03 caiu em error_fallback / parse_failed e degradou para ordem baseline sem lançar exceção. Contrato "never raise" confirmado. Qualidade, não bug.
"níveis de consciência" → naive trouxe wesley-cleam ×8 (Schwartz ausente); reranked pôs eugene-schwartz #1. É por isso que hit@8 sobe apesar do MRR estrito cair.
| id | tipo | naive | reranked | vencedor |
|---|---|---|---|---|
| A01 | abstention | abstain ✅ (0.18) | abstain ✅ (0.18) | = |
| A02 | abstention | abstain ✅ (0.24) | abstain ✅ (0.22) | = |
| A03 | abstention | abstain ✅ (0.25) | abstain ✅ (0.25) | = |
| A04 | abstention | abstain ✅ (0.26) | abstain ✅ (0.25) | = |
| G01 | normal | hit ✅ · mrr 0.33 · uniq 0.38 | hit ✅ · mrr 0.20 · uniq 1.00 | rerank dedup |
| G02 | normal | hit ✅ · mrr 1.00 · uniq 0.12 | hit ✅ · mrr 0.20 · uniq 1.00 | rerank dedup |
| G03 | normal | hit ✅ · mrr 1.00 · uniq 0.38 | hit ✅ · mrr 0.17 · uniq 1.00 | rerank dedup |
| G04 | normal | hit ✅ · mrr 1.00 · uniq 0.12 | hit ✅ · mrr 0.17 · uniq 1.00 | rerank dedup |
| G05 | normal | hit ❌ · mrr 0.00 · uniq 0.38 | hit ❌ · mrr 0.00 · uniq 1.00 | rerank dedup |
| G06 | normal | hit ✅ · mrr 1.00 · uniq 0.25 | hit ✅ · mrr 1.00 · uniq 1.00 | rerank dedup |
| G07 | normal | hit ✅ · mrr 0.12 · uniq 0.38 | hit ✅ · mrr 0.12 · uniq 1.00 | rerank dedup |
| G08 | normal | hit ❌ · mrr 0.00 · uniq 0.62 | hit ❌ · mrr 0.00 · uniq 1.00 | rerank dedup |
| G09 | disambiguation | hit ❌ · mrr 0.00 · uniq 0.12 | hit ✅ · mrr 1.00 · uniq 1.00 | rerank↑ |
| G10 | disambiguation | hit ❌ · mrr 0.00 · uniq 0.38 | hit ❌ · mrr 0.00 · uniq 1.00 | rerank dedup |
| G11 | disambiguation | hit ❌ · mrr 0.00 · uniq 0.12 | hit ✅ · mrr 0.50 · uniq 1.00 | rerank↑ |
| G12 | disambiguation | hit ✅ · mrr 1.00 · uniq 0.38 | hit ✅ · mrr 1.00 · uniq 1.00 | rerank dedup |