Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.
moe
mixture-of-experts
kv-cache
llm
rmsnorm
swiglu
pytorch-llm
byte-pair-tokenizer
rotational-positional-embedding
-
Updated
Nov 18, 2024 - Python