Lab 2: Build a Small Language Model

Build a small language model from scratch to understand how LLMs work.

Overview

Go deeper into Code 3.0 by building a small language model from scratch. Following the Karpathy approach, you’ll implement tokenization, embeddings, and a transformer-style architecture to understand how large language models work from first principles.

Learning Objectives

• Implement tokenization and text encoding
• Build embeddings and understand their role in language models
• Construct and train a small transformer-based model
• Generate text from your trained model

Tasks

To be determined.

Deliverables

• Working language model that generates text
• Written analysis of model behavior and limitations