Incidentally, according to https://gpt-tokenizer.dev, in gpt-5, "literature" is ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

valleyer 30 days ago | parent | context | favorite | on: Why does Opus 5 feel worse to work with?

Incidentally, according to https://gpt-tokenizer.dev, in gpt-5, "literature" is two tokens ("liter" + "ature"), whereas "文学" is one.

argee 30 days ago [–]

My bad, the English word I had in mind was "technology" (技術) but I misremembered due to feeling like "tech" should be its own token.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact