One Wikipedia page costs your AI agent 68,000 tokens
Comments
bugalati
ohadkr
Token reduction is useful, but knowing whether the agent actually accessed the real page is even more important.
chonghaoju
stripping to markdown with Jina Reader or Trafilatura before passing to the agent cuts that 68k down to ~3-5k for most Wikipedia pages, and handles the JS-rendered case too.
armanluthra_
+1. i think the the additional value prop is the bypassing blockers. ideally op should just use jina on top of whatever they are building.
arthurcolle
one of the early jina.ai products was/is reader api, and they trained ReaderLM for this purpose. definitely a good idea to check out existing implementations
m463
> on js-rendered and some anti-bot pages it returns nothing
yeah, and my (human piloted) browser gets blocked by so many websites now. I routinely get "Sorry." when trying to log into hn.
sigh.
dlcarrier
Hacker News is the one page that hasn't assumed I'm a bot, yet.
At this rate, we'll need to use bots to browse the web for us, because they're the only way to get through the anti-bot filters.
mkbkn
And how much water?
gonna try this out this week, one ask: when a wall does beat it, return "blocked, here's why" instead of empty, avoiding silent fetch failures.