Well a token can be thought of as a "context".It's not how it exactly works, but a very simplified version would be (27)"ing", denoting past tense, and "run"(47) being the verb, and .(294) denoting the end of a sentence. so it takes 3 "tokens" [27, 47, 294] to signify the word "running."
Assuming the pre-prompts are in normal english, hard checking against tokens would match against everything.
Assuming the pre-prompts are in normal english, hard checking against tokens would match against everything.