Associations to the word «Tokenize» - Word Associations Network

Noun

Basic
Byte
Identifier
Token
Code
Output
File
Text
User
Memory
Program
Data
Source
Word
Example
Language
Form
Line
Use

Adjective

Lexical

Verb

Store

Wiktionary

TOKENIZE, verb. (transitive) (computing) To reduce to a set of tokens by parsing.

TOKENIZE, verb. (transitive) To treat as a token minority.

Wise words

Don't use words too big for the subject. Don't say "infinitely" when you mean "very"; otherwise you'll have no word left when you want to talk about something really infinite.

C. S. Lewis