-
is openai imposing a token tax on non-english languages
The token count on international languages are significantly higher than English. The post discusses the issue of OpenAI's tokenizer assigning more tokens to non-English languages compared to English, effectively imposing a "token tax" on intern...The token count on international languages are significantly higher than English. The post discusses the issue of OpenAI's tokenizer assigning more tokens to non-English languages compared to English, effectively imposing a "token tax" on intern...
Page 1 of 1
Powered by Optimal Access