Token

Meaning

A token is the smallest unit a model uses to read and write text. Sometimes one word is one token; long or rare words split into several; a single Korean syllable can be one or two tokens. Each model has its own tokenizer, so exact counts differ, but a working rule is roughly 1.3 tokens per English word and 1 to 2 tokens per Korean character. Context window limits, a tool's usage accounting and model providers' pricing are all in tokens. Input tokens (the prompt) and output tokens (the answer) are counted separately, and output is usually more expensive.

Why it matters

Tokens matter to writers because Korean is expensive. A scene that costs 500 tokens in English can run around 1,000 in Korean. With the same context window, a Korean manuscript fits only half as much, and the same credits generate about half the text. Korean writing tools therefore have to manage bible excerpts and the length of preceding text more tightly than English tools do. Understanding tokens also makes a tool's usage display legible: "this request costs N credits" is a conversion of input tokens plus estimated output tokens. Longer prompts and longer requested output cost more. A 3,000-character style sample versus an 800-character one makes a small difference in output and a large difference in cost.

Example

Token counts for the same sentences (varies by model; illustrative)

Sentence	Chars / words	Approx. tokens
She opened the drawer.	4 words	5
그녀는 서랍을 열었다.	11 chars incl. spaces	9–12
The bankbook had her name on it.	7 words	8
통장에는 그녀의 이름이 적혀 있었다.	19 chars	15–20

Korean runs around twice English. One 5,000-character episode is roughly 4,000 to 8,000 tokens, and inserting all of it as continuation input consumes that much room. This is where the practice of including only the last 1,500 characters of the previous chapter comes from.

Common mistakes

• Equating characters with tokens. Korean can have more tokens than characters. Do not compute limits in characters.
• Not economizing on input. A long prompt does not improve output; it only raises cost. Include what is needed.
• Forgetting output tokens. Requesting a whole 5,000-character episode at once is thousands of output tokens. Requesting scene by scene usually gives better results too.