Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Well a token can be thought of as a "context".It's not how it exactly works, but a very simplified version would be (27)"ing", denoting past tense, and "run"(47) being the verb, and .(294) denoting the end of a sentence. so it takes 3 "tokens" [27, 47, 294] to signify the word "running."

Assuming the pre-prompts are in normal english, hard checking against tokens would match against everything.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: