With one-hot encoding we prevent a model from learning categories as an ordinal (order/ranking). We basically put in each category not as a number denoting the category, but as a column with a binary digit representing whether the row is part of this column.
The ‘hot’ is the 1. So each column that has a ‘1’ for the particular row we are evaluating.
(Small caveat: a transformer doesn’t actually take one-hot columns as input. It looks up a learned embedding for every token ID, which is basically the same as multiplying a one-hot vector by a matrix, just a lot cheaper. And the embedding is where the meaning lives: similar tokens end up close together. One-hot does show up as the training target, the ‘correct’ next token.)
Read about this in Book - Natural Language Processing With Transformers.