{ "cells": [ { "cell_type": "markdown", "id": "c000", "metadata": {}, "source": [ "# L3 In-Class Activity: NumPy, Broadcasting, and Backpropagation\n", "\n", "### E190BO — Fall 2026\n", "\n", "Three parts, and they build on each other:\n", "\n", "1. **NumPy basics** — arrays vs. Python lists, shapes, `*` vs. `@`, reshaping.\n", "2. **Broadcasting** — the rule behind $Z^{(\\ell)} = A^{(\\ell-1)} W^{(\\ell)} \\oplus \\bar{b}^{(\\ell)}$ from the L3 slides.\n", "3. **Backpropagation** — the 4-step procedure, run by hand and then in code on a single neuron.\n", "\n", "Work with a partner. Do the parts in order — Part 3 assumes you have the shape\n", "intuition from Parts 1 and 2. Everything you build here is machinery you will need\n", "for Lab 2, where you write a 2-layer network from scratch." ] }, { "cell_type": "code", "execution_count": null, "id": "c001", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- run this cell first. You do not need to modify it.\n", "import numpy as np\n", "import time\n", "\n", "np.set_printoptions(precision=3, suppress=True)\n", "print(\"numpy version:\", np.__version__)" ] }, { "cell_type": "markdown", "id": "c002", "metadata": {}, "source": [ "# Part 1 — NumPy arrays vs. Python lists\n", "\n", "A Python `list` is a general-purpose container. A NumPy `ndarray` is a grid of numbers\n", "with a fixed shape, and **arithmetic on it is elementwise**. That one difference is the\n", "source of most first-week NumPy confusion, so start here." ] }, { "cell_type": "code", "execution_count": null, "id": "c003", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- read the output carefully. You do not need to modify this.\n", "py_list = [1, 2, 3]\n", "np_arr = np.array([1, 2, 3])\n", "\n", "print(\"list * 2 ->\", py_list * 2) # repeats the list\n", "print(\"array * 2 ->\", np_arr * 2) # multiplies every element\n", "\n", "print(\"list + list ->\", py_list + [4, 5, 6]) # concatenates\n", "print(\"array + array ->\", np_arr + np.array([4, 5, 6])) # adds elementwise" ] }, { "cell_type": "markdown", "id": "c004", "metadata": {}, "source": [ "## Shapes\n", "\n", "Every array has a `.shape` — a tuple giving its size along each axis — and a `.ndim`,\n", "the number of axes. Three things that all \"hold three numbers\" are **not** the same\n", "object to NumPy:\n", "\n", "| object | shape | `ndim` |\n", "|---|---|---|\n", "| 1-D array | `(3,)` | 1 |\n", "| row vector | `(1, 3)` | 2 |\n", "| column vector | `(3, 1)` | 2 |\n", "\n", "You will spend real time in Lab 2 chasing shape bugs. Get in the habit of printing\n", "`.shape` whenever something behaves oddly." ] }, { "cell_type": "code", "execution_count": null, "id": "c005", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- you do not need to modify this.\n", "v = np.array([1, 2, 3]) # 1-D\n", "row = np.array([[1, 2, 3]]) # 2-D, one row\n", "col = np.array([[1], [2], [3]]) # 2-D, one column\n", "M = np.array([[1, 2, 3], [4, 5, 6]]) # 2-D, 2 rows x 3 cols\n", "\n", "for name, arr in [(\"v\", v), (\"row\", row), (\"col\", col), (\"M\", M)]:\n", " print(f\"{name:4s} shape={str(arr.shape):8s} ndim={arr.ndim}\")" ] }, { "cell_type": "markdown", "id": "c006", "metadata": {}, "source": [ "## Matrix multiplication: `@` vs. `*`\n", "\n", "- `A * B` is **elementwise** — `A` and `B` must have the same shape (or be broadcastable, see Part 2).\n", "- `A @ B` is **matrix multiplication** — the *inner* dimensions must match, and the *outer* dimensions survive:\n", "\n", "$$(n \\times k) \\; @ \\; (k \\times m) \\;\\longrightarrow\\; (n \\times m)$$\n", "\n", "This is exactly the shape rule behind $\\bar{a}^{(\\ell-1)} W^{(\\ell)}$: a\n", "$1 \\times s_{\\ell-1}$ activation times an $s_{\\ell-1} \\times s_{\\ell}$ weight matrix\n", "gives a $1 \\times s_{\\ell}$ result." ] }, { "cell_type": "markdown", "id": "c007", "metadata": {}, "source": [ "### TODO 1\n", "\n", "`A` is $2 \\times 3$ and `B` is $3 \\times 2$. **Before you run the cell**, predict with your\n", "partner what shape `A @ B` and `B @ A` will have. Then compute them." ] }, { "cell_type": "code", "execution_count": null, "id": "c008", "metadata": {}, "outputs": [], "source": [ "A = np.array([[1, 2, 3],\n", " [4, 5, 6]])\n", "B = np.array([[1, 0],\n", " [0, 1],\n", " [1, 1]])\n", "\n", "# ============ TODO 1: the two matrix products ============\n", "AB = ...\n", "BA = ...\n", "# =========================================================\n", "\n", "print(\"A @ B =\\n\", AB, \"\\nshape:\", AB.shape)\n", "print(\"B @ A =\\n\", BA, \"\\nshape:\", BA.shape)" ] }, { "cell_type": "markdown", "id": "c009", "metadata": {}, "source": [ "Note that `A @ B` and `B @ A` are not even the same *shape*, let alone the same\n", "matrix. Matrix multiplication does not commute.\n", "\n", "### TODO 2\n", "\n", "`A` and `C` below are both $2 \\times 3$. Compute the elementwise product `A * C` and the\n", "matrix product `A @ C.T` (`.T` is the transpose)." ] }, { "cell_type": "code", "execution_count": null, "id": "c010", "metadata": {}, "outputs": [], "source": [ "C = np.array([[1, 1, 1],\n", " [2, 2, 2]])\n", "\n", "# ============ TODO 2: elementwise vs. matrix product ============\n", "elementwise = ...\n", "matmul = ...\n", "# ================================================================\n", "\n", "print(\"A * C =\\n\", elementwise, \"\\nshape:\", elementwise.shape)\n", "print(\"A @ C.T =\\n\", matmul, \"\\nshape:\", matmul.shape)" ] }, { "cell_type": "markdown", "id": "c011", "metadata": {}, "source": [ "**Question.** In one sentence: why is `A * C` a $2 \\times 3$ matrix while `A @ C.T` is\n", "$2 \\times 2$?\n", "\n", "*Your answer here.*" ] }, { "cell_type": "markdown", "id": "c012", "metadata": {}, "source": [ "## Reshaping\n", "\n", "`arr.reshape(new_shape)` returns the same data laid out with a different shape. NumPy\n", "fills the new array in **row-major order** — it walks across each row before moving to\n", "the next. A `-1` in one position means \"figure this dimension out from the total number\n", "of elements,\" which makes `reshape(-1, 1)` (column) and `reshape(1, -1)` (row) the\n", "idiomatic way to turn a 1-D array into a 2-D one.\n", "\n", "### TODO 3" ] }, { "cell_type": "code", "execution_count": null, "id": "c013", "metadata": {}, "outputs": [], "source": [ "x = np.arange(6) # array([0, 1, 2, 3, 4, 5])\n", "\n", "# ============ TODO 3: four reshapes of the same six numbers ============\n", "as_2x3 = ... # 2 rows, 3 columns\n", "as_3x2 = ... # 3 rows, 2 columns\n", "as_col = ... # a column vector, shape (6, 1) -- use -1\n", "as_row = ... # a row vector, shape (1, 6) -- use -1\n", "# =======================================================================\n", "\n", "for name, arr in [(\"as_2x3\", as_2x3), (\"as_3x2\", as_3x2),\n", " (\"as_col\", as_col), (\"as_row\", as_row)]:\n", " print(f\"{name} shape={arr.shape}\\n{arr}\\n\")" ] }, { "cell_type": "markdown", "id": "c014", "metadata": {}, "source": [ "### Reference: the handful of NumPy calls you actually need\n", "\n", "Run this cell if you want to see what each does. Nothing to fill in." ] }, { "cell_type": "code", "execution_count": null, "id": "c015", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- reference only. You do not need to modify this.\n", "print(\"np.zeros((2, 3)) ->\\n\", np.zeros((2, 3)))\n", "print(\"np.ones(3) ->\", np.ones(3))\n", "print(\"np.arange(2, 8, 2) ->\", np.arange(2, 8, 2))\n", "print(\"np.linspace(0, 1, 5) ->\", np.linspace(0, 1, 5))\n", "print(\"M.T ->\\n\", M.T)\n", "print(\"np.sum(M) ->\", np.sum(M)) # everything\n", "print(\"np.sum(M, axis=0) ->\", np.sum(M, axis=0)) # collapse rows -> one value per column\n", "print(\"np.sum(M, axis=1) ->\", np.sum(M, axis=1)) # collapse cols -> one value per row" ] }, { "cell_type": "markdown", "id": "c016", "metadata": {}, "source": [ "# Part 2 — Broadcasting\n", "\n", "**Broadcasting** is how NumPy handles arithmetic between arrays of *different* shapes.\n", "The rule: line the shapes up from the **right**, and at each position the two sizes must\n", "either be equal, or one of them must be 1 — in which case that array is stretched\n", "(conceptually, not in memory) to match.\n", "\n", "Work up from the case you already believe." ] }, { "cell_type": "markdown", "id": "c017", "metadata": {}, "source": [ "### 1 & 2. A scalar plus anything\n", "\n", "Adding a scalar to a vector or a matrix adds it to every element. You already expect\n", "this to work, and it works for exactly the broadcasting reason: a scalar is stretched\n", "to whatever shape it meets." ] }, { "cell_type": "code", "execution_count": null, "id": "c018", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- you do not need to modify this.\n", "vec = np.array([10, 20, 30])\n", "mat = np.array([[1, 2, 3, 4],\n", " [5, 6, 7, 8],\n", " [9, 10, 11, 12]])\n", "\n", "print(\"vec + 100 ->\", vec + 100)\n", "print(\"mat + 100 ->\\n\", mat + 100)" ] }, { "cell_type": "markdown", "id": "c019", "metadata": {}, "source": [ "### 3. A column plus a matrix\n", "\n", "`col` has shape $(3, 1)$ and `mat` has shape $(3, 4)$. Line them up from the right:\n", "\n", "```\n", "mat (3, 4)\n", "col (3, 1)\n", " ^ 4 vs 1 -> stretch the column across all 4 columns\n", "```\n", "\n", "So the same column is added to **every column** of the matrix — equivalently, one value\n", "is added per *row*." ] }, { "cell_type": "code", "execution_count": null, "id": "c020", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- you do not need to modify this.\n", "col = np.array([[0], [100], [200]]) # shape (3, 1)\n", "print(\"col shape:\", col.shape, \" mat shape:\", mat.shape)\n", "print(\"mat + col ->\\n\", mat + col)" ] }, { "cell_type": "markdown", "id": "c021", "metadata": {}, "source": [ "### 4. A row plus a matrix\n", "\n", "`row` has shape $(1, 4)$ and `mat` has shape $(3, 4)$:\n", "\n", "```\n", "mat (3, 4)\n", "row (1, 4)\n", " ^ 3 vs 1 -> stretch the row down all 3 rows\n", "```\n", "\n", "The same row is added to **every row**. This is precisely the batched linear layer from\n", "the L3 slides:\n", "\n", "$$Z^{(\\ell)} = A^{(\\ell-1)} W^{(\\ell)} \\oplus \\bar{b}^{(\\ell)}, \\qquad\n", "A^{(\\ell-1)}W^{(\\ell)} \\in \\mathbb{R}^{N \\times s_\\ell}, \\quad\n", "\\bar{b}^{(\\ell)} \\in \\mathbb{R}^{1 \\times s_\\ell}$$\n", "\n", "The $\\oplus$ *is* broadcasting: one bias vector, added identically to all $N$ rows of\n", "the batch. That is why the bias has shape $1 \\times s_\\ell$ and not $N \\times s_\\ell$." ] }, { "cell_type": "code", "execution_count": null, "id": "c022", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- you do not need to modify this.\n", "row = np.array([[0, 100, 200, 300]]) # shape (1, 4)\n", "print(\"row shape:\", row.shape, \" mat shape:\", mat.shape)\n", "print(\"mat + row ->\\n\", mat + row)" ] }, { "cell_type": "markdown", "id": "c023", "metadata": {}, "source": [ "### 5. When broadcasting fails\n", "\n", "Shapes $(3, 1)$ and $(4, 1)$ cannot be broadcast: reading from the right, $1$ vs $1$ is\n", "fine, but then $3$ vs $4$ — not equal, and neither is 1. NumPy raises a `ValueError`.\n", "Read that error message now, because you will see it again in Lab 2." ] }, { "cell_type": "code", "execution_count": null, "id": "c024", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- this cell is SUPPOSED to print an error. Do not modify it.\n", "try:\n", " np.array([[1], [2], [3]]) + np.array([[1], [2], [3], [4]])\n", "except ValueError as err:\n", " print(\"ValueError:\", err)" ] }, { "cell_type": "markdown", "id": "c025", "metadata": {}, "source": [ "## Exercise: build the same matrix four ways\n", "\n", "Build the $4 \\times 5$ matrix `M` whose entries are the sum of the row and column index:\n", "\n", "$$M_{ij} = i + j, \\qquad i = 0 \\dots 3, \\; j = 0 \\dots 4$$\n", "\n", "$$M = \\begin{bmatrix}\n", "0 & 1 & 2 & 3 & 4\\\\\n", "1 & 2 & 3 & 4 & 5\\\\\n", "2 & 3 & 4 & 5 & 6\\\\\n", "3 & 4 & 5 & 6 & 7\n", "\\end{bmatrix}$$\n", "\n", "You will build it four times, each with less machinery than the last. The point is not\n", "that one is \"clever\" — it is that all four compute the same thing, and by the fourth you\n", "should be able to *read* broadcasting as a description of what the matrix is." ] }, { "cell_type": "code", "execution_count": null, "id": "c026", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- shared setup. You do not need to modify this.\n", "n_rows, n_cols = 4, 5" ] }, { "cell_type": "markdown", "id": "c027", "metadata": {}, "source": [ "### TODO 4 — double `for` loop\n", "\n", "Preallocate with `np.zeros` and fill one element at a time." ] }, { "cell_type": "code", "execution_count": null, "id": "c028", "metadata": {}, "outputs": [], "source": [ "# ============ TODO 4: double loop, one element at a time ============\n", "M1 = np.zeros((n_rows, n_cols))\n", "for i in range(n_rows):\n", " for j in range(n_cols):\n", " M1[i, j] = ...\n", "# ====================================================================\n", "\n", "print(M1)" ] }, { "cell_type": "markdown", "id": "c029", "metadata": {}, "source": [ "### TODO 5 — single `for` loop, one row at a time\n", "\n", "Row $i$ is the vector $[\\,i+0,\\; i+1,\\; \\dots,\\; i+4\\,]$. Build that whole row at once\n", "with `np.arange` and a scalar add — no inner loop." ] }, { "cell_type": "code", "execution_count": null, "id": "c030", "metadata": {}, "outputs": [], "source": [ "# ============ TODO 5: one loop, a whole row per iteration ============\n", "M2 = np.zeros((n_rows, n_cols))\n", "for i in range(n_rows):\n", " M2[i, :] = ...\n", "# =====================================================================\n", "\n", "np.testing.assert_array_equal(M2, M1) # provided check -- silent means correct\n", "print(M2)" ] }, { "cell_type": "markdown", "id": "c031", "metadata": {}, "source": [ "### TODO 6 — no loops, using broadcasting\n", "\n", "Make a **column** of row indices, shape $(4, 1)$, and a **row** of column indices,\n", "shape $(1, 5)$. Add them. Line the shapes up from the right and convince yourself the\n", "result must be $4 \\times 5$ before you run it." ] }, { "cell_type": "code", "execution_count": null, "id": "c032", "metadata": {}, "outputs": [], "source": [ "# ============ TODO 6: broadcasting, named intermediates ============\n", "i_col = ... # shape (n_rows, 1) -- row indices as a column\n", "j_row = ... # shape (1, n_cols) -- column indices as a row\n", "M3 = ...\n", "# ===================================================================\n", "\n", "print(\"i_col shape:\", i_col.shape, \" j_row shape:\", j_row.shape, \" M3 shape:\", M3.shape)\n", "np.testing.assert_array_equal(M3, M1)\n", "print(M3)" ] }, { "cell_type": "markdown", "id": "c033", "metadata": {}, "source": [ "### TODO 7 — the whole thing in one line" ] }, { "cell_type": "code", "execution_count": null, "id": "c034", "metadata": {}, "outputs": [], "source": [ "# ============ TODO 7: one expression, no intermediates ============\n", "M4 = ...\n", "# ==================================================================\n", "\n", "np.testing.assert_array_equal(M4, M1)\n", "print(M4)" ] }, { "cell_type": "markdown", "id": "c035", "metadata": {}, "source": [ "### Does it matter?\n", "\n", "The provided cell below times the double loop against the broadcast version at\n", "$n = 500$ (a 500 × 500 matrix — 250,000 entries)." ] }, { "cell_type": "code", "execution_count": null, "id": "c036", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- timing comparison. You do not need to modify this.\n", "n = 500\n", "\n", "t0 = time.perf_counter()\n", "Mloop = np.zeros((n, n))\n", "for i in range(n):\n", " for j in range(n):\n", " Mloop[i, j] = i + j\n", "t_loop = time.perf_counter() - t0\n", "\n", "t0 = time.perf_counter()\n", "Mbcast = np.arange(n).reshape(-1, 1) + np.arange(n).reshape(1, -1)\n", "t_bcast = time.perf_counter() - t0\n", "\n", "np.testing.assert_array_equal(Mloop, Mbcast)\n", "print(f\"double loop : {t_loop*1000:8.2f} ms\")\n", "print(f\"broadcasting: {t_bcast*1000:8.2f} ms\")\n", "print(f\"speedup : {t_loop/t_bcast:8.1f}x\")" ] }, { "cell_type": "markdown", "id": "c037", "metadata": {}, "source": [ "**Discussion.** Two questions, one minute each:\n", "\n", "1. `i_col` is $(4,1)$ and `j_row` is $(1,5)$. Why is the result $(4,5)$ — what does the\n", " broadcasting rule do at each of the two positions?\n", "2. Which of the four versions would you rather debug at 11pm, and why?\n", "\n", "*Your answers here.*" ] }, { "cell_type": "markdown", "id": "c038", "metadata": {}, "source": [ "# Part 3 — Backpropagation on a single neuron\n", "\n", "Recall the 4-step procedure from L3:\n", "\n", "1. decompose into a composition of simple functions\n", "2. derive the derivative of each simple function\n", "3. compute the intermediate outputs and derivatives (**forward pass**)\n", "4. multiply the derivatives via the chain rule (**backward pass**)\n", "\n", "The L3 slide example, $y = [\\ln(x^2+1)]^2$, is a single chain. Here is a graph that\n", "actually looks like a piece of a network — one neuron with two inputs, a ReLU, and a\n", "squared-error loss against a target $t$:\n", "\n", "```\n", " x1 ---(* w1)---\\\n", " (+ b) ---> z ---[ReLU]---> a ---[(a - t)^2]---> J\n", " x2 ---(* w2)---/\n", "```\n", "\n", "$$z = w_1 x_1 + w_2 x_2 + b, \\qquad a = \\mathrm{ReLU}(z), \\qquad J = (a - t)^2$$\n", "\n", "The inputs $x_1, x_2$ and the target $t$ are **data** — fixed. The things we\n", "differentiate with respect to are the **parameters** $w_1, w_2, b$, because those are\n", "what gradient descent will update.\n", "\n", "Work with these numbers throughout:\n", "\n", "$$x_1 = 2.0,\\quad x_2 = -1.0,\\quad t = 1.0,\\qquad w_1 = 0.5,\\quad w_2 = 1.0,\\quad b = 0.25$$" ] }, { "cell_type": "markdown", "id": "c039", "metadata": {}, "source": [ "## (a) Do it on paper first — ~10 minutes\n", "\n", "Do all of part (a) on paper with your partner *before* touching the code. Fill in your\n", "answers in the tables below as you go.\n", "\n", "### (a-i) Forward pass\n", "\n", "Compute the intermediate values by hand.\n", "\n", "| quantity | value |\n", "|---|---|\n", "| $z$ |   |\n", "| $a$ |   |\n", "| $J$ |   |\n" ] }, { "cell_type": "markdown", "id": "c040", "metadata": {}, "source": [ "### (a-ii) Local derivatives\n", "\n", "One row per edge of the graph. Write each derivative **symbolically** first, then\n", "evaluate it at the numbers from (a-i).\n", "\n", "| local derivative | symbolic | value here |\n", "|---|---|---|\n", "| $\\dfrac{\\partial J}{\\partial a}$ |   |   |\n", "| $\\dfrac{d a}{d z}$ |   |   |\n", "| $\\dfrac{\\partial z}{\\partial w_1}$ |   |   |\n", "| $\\dfrac{\\partial z}{\\partial w_2}$ |   |   |\n", "| $\\dfrac{\\partial z}{\\partial b}$ |   |   |\n", "\n", "*Note: $\\mathrm{ReLU}$ is not differentiable at $z = 0$. The universal convention is to\n", "define the derivative there as 0, so $\\frac{da}{dz} = 1$ if $z > 0$ and $0$ otherwise.*\n" ] }, { "cell_type": "markdown", "id": "c041", "metadata": {}, "source": [ "### (a-iii) Backward pass\n", "\n", "Multiply the local derivatives along the path from $J$ back to each parameter. It is\n", "worth computing $\\dfrac{\\partial J}{\\partial z}$ once and reusing it three times.\n", "\n", "$$\\frac{\\partial J}{\\partial z} = \\qquad\\qquad\n", "\\frac{\\partial J}{\\partial w_1} = \\qquad\\qquad\n", "\\frac{\\partial J}{\\partial w_2} = \\qquad\\qquad\n", "\\frac{\\partial J}{\\partial b} =$$\n", "\n", "*Your answer here.*" ] }, { "cell_type": "code", "execution_count": null, "id": "114992d7", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "c042", "metadata": {}, "source": [ "### (a-iv) A different setting (skip if low on time)\n", "\n", "Suppose $b = -3.0$ instead, with everything else unchanged. Recompute $z$ and $a$, then\n", "work out all three gradients. What happened, and what does it mean for gradient descent?\n", "\n", "*Your answer here.*" ] }, { "cell_type": "markdown", "id": "c043", "metadata": {}, "source": [ "## (b) Now write it in code — ~15 minutes\n", "\n", "Two functions. `forward` computes and **returns the intermediate values**, because\n", "`backward` needs them — that is the whole reason frameworks \"cache activations\" during\n", "the forward pass.\n", "\n", "### TODO 8 — the forward pass" ] }, { "cell_type": "code", "execution_count": null, "id": "c044", "metadata": {}, "outputs": [], "source": [ "def forward(w1, w2, b, x1, x2, t):\n", " \"\"\"Forward pass through the single-neuron graph.\n", "\n", " Returns\n", " -------\n", " z : float, the pre-activation w1*x1 + w2*x2 + b\n", " a : float, the activation ReLU(z)\n", " J : float, the squared-error loss (a - t)**2\n", " \"\"\"\n", " # ============ TODO 8: three lines, following the graph left to right ============\n", " z = ...\n", " a = ...\n", " J = ...\n", " # ===============================================================================\n", " return z, a, J\n", "\n", "\n", "# --- PROVIDED --- check against your (a-i) answers.\n", "x1, x2, t = 2.0, -1.0, 1.0\n", "w1, w2, b = 0.5, 1.0, 0.25\n", "\n", "z, a, J = forward(w1, w2, b, x1, x2, t)\n", "print(f\"z = {z} a = {a} J = {J}\")" ] }, { "cell_type": "markdown", "id": "c045", "metadata": {}, "source": [ "### TODO 9 — the backward pass\n", "\n", "One line per row of your (a-ii) table, then the three products from (a-iii). Note that\n", "`backward` takes `z` and `a` as arguments — the cached forward values — rather than\n", "recomputing them." ] }, { "cell_type": "code", "execution_count": null, "id": "c046", "metadata": {}, "outputs": [], "source": [ "def backward(z, a, t, x1, x2):\n", " \"\"\"Backward pass. Takes the cached forward values and returns the parameter gradients.\n", "\n", " Returns\n", " -------\n", " dJ_dw1, dJ_dw2, dJ_db : floats\n", " \"\"\"\n", " # ============ TODO 9: local derivatives, then the chain rule ============\n", " dJ_da = ... # derivative of (a - t)^2 with respect to a\n", " da_dz = ... # derivative of ReLU at z (1 if z > 0 else 0)\n", " dJ_dz = ... # chain rule: combine the two above\n", "\n", " dJ_dw1 = ... # dJ_dz times dz/dw1\n", " dJ_dw2 = ... # dJ_dz times dz/dw2\n", " dJ_db = ... # dJ_dz times dz/db\n", " # ========================================================================\n", " return dJ_dw1, dJ_dw2, dJ_db\n", "\n", "\n", "# --- PROVIDED --- these should match your (a-iii) answers exactly.\n", "grads = backward(z, a, t, x1, x2)\n", "print(f\"dJ/dw1 = {grads[0]} dJ/dw2 = {grads[1]} dJ/db = {grads[2]}\")" ] }, { "cell_type": "markdown", "id": "c047", "metadata": {}, "source": [ "### Gradient check\n", "\n", "Same idea as in Lab 1, and the same idea as the L3 slides: nudge one parameter by\n", "$\\pm\\epsilon$, re-run only the *forward* pass, and estimate\n", "\n", "$$\\frac{\\partial J}{\\partial w_i} \\approx \\frac{J^+ - J^-}{2\\epsilon}.$$\n", "\n", "If your analytic gradient disagrees with this, the analytic gradient is almost always\n", "the one that is wrong. Run the provided cell." ] }, { "cell_type": "code", "execution_count": null, "id": "c048", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- gradient check. You do not need to modify this.\n", "def grad_check(w1, w2, b, x1, x2, t, eps=1e-5, tol=1e-7):\n", " z, a, _ = forward(w1, w2, b, x1, x2, t)\n", " analytic = backward(z, a, t, x1, x2)\n", "\n", " params = [w1, w2, b]\n", " numeric = []\n", " for i in range(3):\n", " up, dn = list(params), list(params)\n", " up[i] += eps\n", " dn[i] -= eps\n", " Jp = forward(up[0], up[1], up[2], x1, x2, t)[2]\n", " Jm = forward(dn[0], dn[1], dn[2], x1, x2, t)[2]\n", " numeric.append((Jp - Jm) / (2 * eps))\n", "\n", " print(f\"{'param':6s} {'analytic':>12s} {'numeric':>12s} {'rel err':>12s} result\")\n", " for name, ana, num in zip([\"w1\", \"w2\", \"b\"], analytic, numeric):\n", " denom = max(abs(ana), abs(num), 1e-12)\n", " rel = abs(ana - num) / denom\n", " print(f\"{name:6s} {ana:12.6f} {num:12.6f} {rel:12.2e} {'PASS' if rel < tol else 'FAIL'}\")\n", "\n", "grad_check(w1, w2, b, x1, x2, t)" ] }, { "cell_type": "markdown", "id": "c049", "metadata": {}, "source": [ "### TODO 10 — the dead neuron\n", "\n", "Close the loop on (a-iv): rerun the forward and backward pass with $b = -3.0$ and check\n", "that all three gradients are exactly zero." ] }, { "cell_type": "code", "execution_count": null, "id": "c050", "metadata": {}, "outputs": [], "source": [ "# ============ TODO 10: same neuron, b = -3.0 ============\n", "b_dead = ...\n", "z_d, a_d, J_d = ...\n", "grads_dead = ...\n", "# ========================================================\n", "\n", "print(f\"z = {z_d} a = {a_d} J = {J_d}\")\n", "print(\"gradients:\", grads_dead)\n", "assert all(g == 0.0 for g in grads_dead), \"expected every gradient to be zero\"\n", "print(\"All gradients are zero -- this neuron cannot learn.\")" ] }, { "cell_type": "markdown", "id": "c051", "metadata": {}, "source": [ "## (c) Two data points: gradients accumulate — ~5 minutes\n", "\n", "So far, one training point. With **two** points sharing the same $w_1, w_2, b$, the loss\n", "is the sum of the two individual losses:\n", "\n", "$$J = J^{(1)} + J^{(2)}$$\n", "\n", "Now $w_1$ feeds **two** paths through the graph — one per data point. The chain rule for\n", "a variable that fans out says: **sum the contributions from every path**.\n", "\n", "$$\\frac{\\partial J}{\\partial w_1}\n", "= \\frac{\\partial J^{(1)}}{\\partial w_1} + \\frac{\\partial J^{(2)}}{\\partial w_1}$$\n", "\n", "This is not a new rule to memorize — it is the reason Lab 1's `X.T @ residuals` is a sum\n", "over samples, and it is what the batched backward pass in Lab 2 is doing under the hood.\n", "\n", "### TODO 11\n", "\n", "Loop over the two points, calling your existing `forward` and `backward` once per point,\n", "and accumulate." ] }, { "cell_type": "code", "execution_count": null, "id": "c052", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- two training points, same parameters.\n", "data = [(2.0, -1.0, 1.0), # (x1, x2, t) -- the point from part (a)\n", " (1.0, 3.0, 0.0)]\n", "\n", "total_grads = np.zeros(3)\n", "total_loss = 0.0\n", "\n", "for (xi1, xi2, ti) in data:\n", " # ============ TODO 11: forward, backward, accumulate ============\n", " zi, ai, Ji = ...\n", " gi = ...\n", " total_loss += ...\n", " total_grads += ...\n", " # ===============================================================\n", "\n", "print(f\"total loss = {total_loss}\")\n", "print(f\"dJ/dw1 = {total_grads[0]} dJ/dw2 = {total_grads[1]} dJ/db = {total_grads[2]}\")" ] }, { "cell_type": "markdown", "id": "c053", "metadata": {}, "source": [ "## What to carry into Lab 2\n", "\n", "- **`*` is elementwise, `@` is matrix multiplication.** When something breaks, print\n", " `.shape` before you print values.\n", "- **Broadcasting is the bias add.** $A W \\oplus \\bar b$ works because a $1 \\times s_\\ell$\n", " bias stretches down all $N$ rows of the batch. Shapes align from the right.\n", "- **The backward pass is cached forward values times local derivatives.** Decompose,\n", " differentiate each simple piece, then multiply along the path. Compute shared\n", " quantities like $\\partial J/\\partial z$ once.\n", "- **A zero anywhere in the chain zeroes everything behind it** — that is a dead ReLU.\n", "- **When the gradient is suspect, gradient-check it.** You will use the exact same\n", " central-difference trick in Lab 2." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.13" } }, "nbformat": 4, "nbformat_minor": 5 }