[Notes] A Hackers Guide to LLMs

from blog callmephilip, | ↗ original
Notes for A Hackers’ Guide to Language Models video by Jeremy Howard companion notebook language models predict next tokens based on the input token can either be a whole word or part of a word/puctuation/number example of using OpenAI’s tokenizer from tiktoken import encoding_for_model enc = encoding_for_model("text-davinci-003") toks =...