feat(review): add token-aware context control with tokenlens

Replace hardcoded char-count context limits with token-based budgets using
tokenlens (data from models.dev). TokenCounter provides 3-tier context window
lookup: dynamic catalog (refreshed every 24h) → static tokenlens → 128k default.

- specialist-agent: token budget from model context window instead of MAX_CONTEXT_CHARS=100k
- critic-agent/reflexion-agent: tokenCounter.clip() instead of diff.slice(0, 3000/2000)
- diff-extractor: raw diff clipping at 30k tokens
- engine.ts: refreshCatalog() at startup, stopRefresh() at shutdown

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
This commit is contained in:
jeffusion
2026-03-05 22:03:00 +08:00
committed by 路遥知码力
parent 86480dec07
commit ec2029a942
9 changed files with 447 additions and 17 deletions

View File

@@ -15,6 +15,7 @@
"hono": "^4.11.9",
"lodash-es": "^4.17.21",
"openai": "^4.87.3",
"tokenlens": "^1.3.1",
"zod": "^3.25.1",
"zod-to-json-schema": "^3.25.1"
},