{ "cells": [ { "cell_type": "markdown", "id": "c000", "metadata": {}, "source": [ "# L3 In-Class Activity: NumPy, Broadcasting, and Backpropagation\n", "\n", "### E190BO — Fall 2026\n", "\n", "Three parts, and they build on each other:\n", "\n", "1. **NumPy basics** — arrays vs. Python lists, shapes, `*` vs. `@`, reshaping.\n", "2. **Broadcasting** — the rule behind $Z^{(\\ell)} = A^{(\\ell-1)} W^{(\\ell)} \\oplus \\bar{b}^{(\\ell)}$ from the L3 slides.\n", "3. **Backpropagation** — the 4-step procedure, run by hand and then in code on a single neuron.\n", "\n", "Work with a partner. Do the parts in order — Part 3 assumes you have the shape\n", "intuition from Parts 1 and 2. Everything you build here is machinery you will need\n", "for Lab 2, where you write a 2-layer network from scratch." ] }, { "cell_type": "code", "execution_count": null, "id": "c001", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- run this cell first. You do not need to modify it.\n", "import numpy as np\n", "import time\n", "\n", "np.set_printoptions(precision=3, suppress=True)\n", "print(\"numpy version:\", np.__version__)" ] }, { "cell_type": "markdown", "id": "c002", "metadata": {}, "source": [ "# Part 1 — NumPy arrays vs. Python lists\n", "\n", "A Python `list` is a general-purpose container. A NumPy `ndarray` is a grid of numbers\n", "with a fixed shape, and **arithmetic on it is elementwise**. That one difference is the\n", "source of most first-week NumPy confusion, so start here." ] }, { "cell_type": "code", "execution_count": null, "id": "c003", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- read the output carefully. You do not need to modify this.\n", "py_list = [1, 2, 3]\n", "np_arr = np.array([1, 2, 3])\n", "\n", "print(\"list * 2 ->\", py_list * 2) # repeats the list\n", "print(\"array * 2 ->\", np_arr * 2) # multiplies every element\n", "\n", "print(\"list + list ->\", py_list + [4, 5, 6]) # concatenates\n", "print(\"array + array ->\", np_arr + np.array([4, 5, 6])) # adds elementwise" ] }, { "cell_type": "markdown", "id": "c004", "metadata": {}, "source": [ "## Shapes\n", "\n", "Every array has a `.shape` — a tuple giving its size along each axis — and a `.ndim`,\n", "the number of axes. Three things that all \"hold three numbers\" are **not** the same\n", "object to NumPy:\n", "\n", "| object | shape | `ndim` |\n", "|---|---|---|\n", "| 1-D array | `(3,)` | 1 |\n", "| row vector | `(1, 3)` | 2 |\n", "| column vector | `(3, 1)` | 2 |\n", "\n", "You will spend real time in Lab 2 chasing shape bugs. Get in the habit of printing\n", "`.shape` whenever something behaves oddly." ] }, { "cell_type": "code", "execution_count": null, "id": "c005", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- you do not need to modify this.\n", "v = np.array([1, 2, 3]) # 1-D\n", "row = np.array([[1, 2, 3]]) # 2-D, one row\n", "col = np.array([[1], [2], [3]]) # 2-D, one column\n", "M = np.array([[1, 2, 3], [4, 5, 6]]) # 2-D, 2 rows x 3 cols\n", "\n", "for name, arr in [(\"v\", v), (\"row\", row), (\"col\", col), (\"M\", M)]:\n", " print(f\"{name:4s} shape={str(arr.shape):8s} ndim={arr.ndim}\")" ] }, { "cell_type": "markdown", "id": "c006", "metadata": {}, "source": [ "## Matrix multiplication: `@` vs. `*`\n", "\n", "- `A * B` is **elementwise** — `A` and `B` must have the same shape (or be broadcastable, see Part 2).\n", "- `A @ B` is **matrix multiplication** — the *inner* dimensions must match, and the *outer* dimensions survive:\n", "\n", "$$(n \\times k) \\; @ \\; (k \\times m) \\;\\longrightarrow\\; (n \\times m)$$\n", "\n", "This is exactly the shape rule behind $\\bar{a}^{(\\ell-1)} W^{(\\ell)}$: a\n", "$1 \\times s_{\\ell-1}$ activation times an $s_{\\ell-1} \\times s_{\\ell}$ weight matrix\n", "gives a $1 \\times s_{\\ell}$ result." ] }, { "cell_type": "markdown", "id": "c007", "metadata": {}, "source": [ "### TODO 1\n", "\n", "`A` is $2 \\times 3$ and `B` is $3 \\times 2$. **Before you run the cell**, predict with your\n", "partner what shape `A @ B` and `B @ A` will have. Then compute them." ] }, { "cell_type": "code", "execution_count": null, "id": "c008", "metadata": {}, "outputs": [], "source": [ "A = np.array([[1, 2, 3],\n", " [4, 5, 6]])\n", "B = np.array([[1, 0],\n", " [0, 1],\n", " [1, 1]])\n", "\n", "# ============ TODO 1: the two matrix products ============\n", "AB = A @ B\n", "BA = B @ A\n", "# =========================================================\n", "\n", "print(\"A @ B =\\n\", AB, \"\\nshape:\", AB.shape)\n", "print(\"B @ A =\\n\", BA, \"\\nshape:\", BA.shape)" ] }, { "cell_type": "markdown", "id": "c009", "metadata": {}, "source": [ "Note that `A @ B` and `B @ A` are not even the same *shape*, let alone the same\n", "matrix. Matrix multiplication does not commute.\n", "\n", "### TODO 2\n", "\n", "`A` and `C` below are both $2 \\times 3$. Compute the elementwise product `A * C` and the\n", "matrix product `A @ C.T` (`.T` is the transpose)." ] }, { "cell_type": "code", "execution_count": null, "id": "c010", "metadata": {}, "outputs": [], "source": [ "C = np.array([[1, 1, 1],\n", " [2, 2, 2]])\n", "\n", "# ============ TODO 2: elementwise vs. matrix product ============\n", "elementwise = A * C\n", "matmul = A @ C.T\n", "# ================================================================\n", "\n", "print(\"A * C =\\n\", elementwise, \"\\nshape:\", elementwise.shape)\n", "print(\"A @ C.T =\\n\", matmul, \"\\nshape:\", matmul.shape)" ] }, { "cell_type": "markdown", "id": "c011", "metadata": {}, "source": [ "**Question.** In one sentence: why is `A * C` a $2 \\times 3$ matrix while `A @ C.T` is\n", "$2 \\times 2$?\n", "\n", "`A * C` pairs up entries in the same position and multiplies them, so the result has the\n", "same shape as its inputs; `A @ C.T` is $(2\\times3)@(3\\times2)$, where the inner dimension\n", "3 is summed away and the two outer dimensions, 2 and 2, are what survive." ] }, { "cell_type": "markdown", "id": "c012", "metadata": {}, "source": [ "## Reshaping\n", "\n", "`arr.reshape(new_shape)` returns the same data laid out with a different shape. NumPy\n", "fills the new array in **row-major order** — it walks across each row before moving to\n", "the next. A `-1` in one position means \"figure this dimension out from the total number\n", "of elements,\" which makes `reshape(-1, 1)` (column) and `reshape(1, -1)` (row) the\n", "idiomatic way to turn a 1-D array into a 2-D one.\n", "\n", "### TODO 3" ] }, { "cell_type": "code", "execution_count": null, "id": "c013", "metadata": {}, "outputs": [], "source": [ "x = np.arange(6) # array([0, 1, 2, 3, 4, 5])\n", "\n", "# ============ TODO 3: four reshapes of the same six numbers ============\n", "as_2x3 = x.reshape(2, 3) # 2 rows, 3 columns\n", "as_3x2 = x.reshape(3, 2) # 3 rows, 2 columns\n", "as_col = x.reshape(-1, 1) # a column vector, shape (6, 1) -- use -1\n", "as_row = x.reshape(1, -1) # a row vector, shape (1, 6) -- use -1\n", "# =======================================================================\n", "\n", "for name, arr in [(\"as_2x3\", as_2x3), (\"as_3x2\", as_3x2),\n", " (\"as_col\", as_col), (\"as_row\", as_row)]:\n", " print(f\"{name} shape={arr.shape}\\n{arr}\\n\")" ] }, { "cell_type": "markdown", "id": "c014", "metadata": {}, "source": [ "### Reference: the handful of NumPy calls you actually need\n", "\n", "Run this cell if you want to see what each does. Nothing to fill in." ] }, { "cell_type": "code", "execution_count": null, "id": "c015", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- reference only. You do not need to modify this.\n", "print(\"np.zeros((2, 3)) ->\\n\", np.zeros((2, 3)))\n", "print(\"np.ones(3) ->\", np.ones(3))\n", "print(\"np.arange(2, 8, 2) ->\", np.arange(2, 8, 2))\n", "print(\"np.linspace(0, 1, 5) ->\", np.linspace(0, 1, 5))\n", "print(\"M.T ->\\n\", M.T)\n", "print(\"np.sum(M) ->\", np.sum(M)) # everything\n", "print(\"np.sum(M, axis=0) ->\", np.sum(M, axis=0)) # collapse rows -> one value per column\n", "print(\"np.sum(M, axis=1) ->\", np.sum(M, axis=1)) # collapse cols -> one value per row" ] }, { "cell_type": "markdown", "id": "c016", "metadata": {}, "source": [ "# Part 2 — Broadcasting\n", "\n", "**Broadcasting** is how NumPy handles arithmetic between arrays of *different* shapes.\n", "The rule: line the shapes up from the **right**, and at each position the two sizes must\n", "either be equal, or one of them must be 1 — in which case that array is stretched\n", "(conceptually, not in memory) to match.\n", "\n", "Work up from the case you already believe." ] }, { "cell_type": "markdown", "id": "c017", "metadata": {}, "source": [ "### 1 & 2. A scalar plus anything\n", "\n", "Adding a scalar to a vector or a matrix adds it to every element. You already expect\n", "this to work, and it works for exactly the broadcasting reason: a scalar is stretched\n", "to whatever shape it meets." ] }, { "cell_type": "code", "execution_count": null, "id": "c018", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- you do not need to modify this.\n", "vec = np.array([10, 20, 30])\n", "mat = np.array([[1, 2, 3, 4],\n", " [5, 6, 7, 8],\n", " [9, 10, 11, 12]])\n", "\n", "print(\"vec + 100 ->\", vec + 100)\n", "print(\"mat + 100 ->\\n\", mat + 100)" ] }, { "cell_type": "markdown", "id": "c019", "metadata": {}, "source": [ "### 3. A column plus a matrix\n", "\n", "`col` has shape $(3, 1)$ and `mat` has shape $(3, 4)$. Line them up from the right:\n", "\n", "```\n", "mat (3, 4)\n", "col (3, 1)\n", " ^ 4 vs 1 -> stretch the column across all 4 columns\n", "```\n", "\n", "So the same column is added to **every column** of the matrix — equivalently, one value\n", "is added per *row*." ] }, { "cell_type": "code", "execution_count": null, "id": "c020", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- you do not need to modify this.\n", "col = np.array([[0], [100], [200]]) # shape (3, 1)\n", "print(\"col shape:\", col.shape, \" mat shape:\", mat.shape)\n", "print(\"mat + col ->\\n\", mat + col)" ] }, { "cell_type": "markdown", "id": "c021", "metadata": {}, "source": [ "### 4. A row plus a matrix\n", "\n", "`row` has shape $(1, 4)$ and `mat` has shape $(3, 4)$:\n", "\n", "```\n", "mat (3, 4)\n", "row (1, 4)\n", " ^ 3 vs 1 -> stretch the row down all 3 rows\n", "```\n", "\n", "The same row is added to **every row**. This is precisely the batched linear layer from\n", "the L3 slides:\n", "\n", "$$Z^{(\\ell)} = A^{(\\ell-1)} W^{(\\ell)} \\oplus \\bar{b}^{(\\ell)}, \\qquad\n", "A^{(\\ell-1)}W^{(\\ell)} \\in \\mathbb{R}^{N \\times s_\\ell}, \\quad\n", "\\bar{b}^{(\\ell)} \\in \\mathbb{R}^{1 \\times s_\\ell}$$\n", "\n", "The $\\oplus$ *is* broadcasting: one bias vector, added identically to all $N$ rows of\n", "the batch. That is why the bias has shape $1 \\times s_\\ell$ and not $N \\times s_\\ell$." ] }, { "cell_type": "code", "execution_count": null, "id": "c022", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- you do not need to modify this.\n", "row = np.array([[0, 100, 200, 300]]) # shape (1, 4)\n", "print(\"row shape:\", row.shape, \" mat shape:\", mat.shape)\n", "print(\"mat + row ->\\n\", mat + row)" ] }, { "cell_type": "markdown", "id": "c023", "metadata": {}, "source": [ "### 5. When broadcasting fails\n", "\n", "Shapes $(3, 1)$ and $(4, 1)$ cannot be broadcast: reading from the right, $1$ vs $1$ is\n", "fine, but then $3$ vs $4$ — not equal, and neither is 1. NumPy raises a `ValueError`.\n", "Read that error message now, because you will see it again in Lab 2." ] }, { "cell_type": "code", "execution_count": null, "id": "c024", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- this cell is SUPPOSED to print an error. Do not modify it.\n", "try:\n", " np.array([[1], [2], [3]]) + np.array([[1], [2], [3], [4]])\n", "except ValueError as err:\n", " print(\"ValueError:\", err)" ] }, { "cell_type": "markdown", "id": "c025", "metadata": {}, "source": [ "## Exercise: build the same matrix four ways\n", "\n", "Build the $4 \\times 5$ matrix `M` whose entries are the sum of the row and column index:\n", "\n", "$$M_{ij} = i + j, \\qquad i = 0 \\dots 3, \\; j = 0 \\dots 4$$\n", "\n", "$$M = \\begin{bmatrix}\n", "0 & 1 & 2 & 3 & 4\\\\\n", "1 & 2 & 3 & 4 & 5\\\\\n", "2 & 3 & 4 & 5 & 6\\\\\n", "3 & 4 & 5 & 6 & 7\n", "\\end{bmatrix}$$\n", "\n", "You will build it four times, each with less machinery than the last. The point is not\n", "that one is \"clever\" — it is that all four compute the same thing, and by the fourth you\n", "should be able to *read* broadcasting as a description of what the matrix is." ] }, { "cell_type": "code", "execution_count": null, "id": "c026", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- shared setup. You do not need to modify this.\n", "n_rows, n_cols = 4, 5" ] }, { "cell_type": "markdown", "id": "c027", "metadata": {}, "source": [ "### TODO 4 — double `for` loop\n", "\n", "Preallocate with `np.zeros` and fill one element at a time." ] }, { "cell_type": "code", "execution_count": null, "id": "c028", "metadata": {}, "outputs": [], "source": [ "# ============ TODO 4: double loop, one element at a time ============\n", "M1 = np.zeros((n_rows, n_cols))\n", "for i in range(n_rows):\n", " for j in range(n_cols):\n", " M1[i, j] = i + j\n", "# ====================================================================\n", "\n", "print(M1)" ] }, { "cell_type": "markdown", "id": "c029", "metadata": {}, "source": [ "### TODO 5 — single `for` loop, one row at a time\n", "\n", "Row $i$ is the vector $[\\,i+0,\\; i+1,\\; \\dots,\\; i+4\\,]$. Build that whole row at once\n", "with `np.arange` and a scalar add — no inner loop." ] }, { "cell_type": "code", "execution_count": null, "id": "c030", "metadata": {}, "outputs": [], "source": [ "# ============ TODO 5: one loop, a whole row per iteration ============\n", "M2 = np.zeros((n_rows, n_cols))\n", "for i in range(n_rows):\n", " M2[i, :] = i + np.arange(n_cols)\n", "# =====================================================================\n", "\n", "np.testing.assert_array_equal(M2, M1) # provided check -- silent means correct\n", "print(M2)" ] }, { "cell_type": "markdown", "id": "c031", "metadata": {}, "source": [ "### TODO 6 — no loops, using broadcasting\n", "\n", "Make a **column** of row indices, shape $(4, 1)$, and a **row** of column indices,\n", "shape $(1, 5)$. Add them. Line the shapes up from the right and convince yourself the\n", "result must be $4 \\times 5$ before you run it." ] }, { "cell_type": "code", "execution_count": null, "id": "c032", "metadata": {}, "outputs": [], "source": [ "# ============ TODO 6: broadcasting, named intermediates ============\n", "i_col = np.arange(n_rows).reshape(-1, 1) # shape (n_rows, 1) -- row indices as a column\n", "j_row = np.arange(n_cols).reshape(1, -1) # shape (1, n_cols) -- column indices as a row\n", "M3 = i_col + j_row\n", "# ===================================================================\n", "\n", "print(\"i_col shape:\", i_col.shape, \" j_row shape:\", j_row.shape, \" M3 shape:\", M3.shape)\n", "np.testing.assert_array_equal(M3, M1)\n", "print(M3)" ] }, { "cell_type": "markdown", "id": "c033", "metadata": {}, "source": [ "### TODO 7 — the whole thing in one line" ] }, { "cell_type": "code", "execution_count": null, "id": "c034", "metadata": {}, "outputs": [], "source": [ "# ============ TODO 7: one expression, no intermediates ============\n", "M4 = np.arange(n_rows).reshape(-1, 1) + np.arange(n_cols).reshape(1, -1)\n", "# ==================================================================\n", "\n", "np.testing.assert_array_equal(M4, M1)\n", "print(M4)" ] }, { "cell_type": "markdown", "id": "c035", "metadata": {}, "source": [ "### Does it matter?\n", "\n", "The provided cell below times the double loop against the broadcast version at\n", "$n = 500$ (a 500 × 500 matrix — 250,000 entries)." ] }, { "cell_type": "code", "execution_count": null, "id": "c036", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- timing comparison. You do not need to modify this.\n", "n = 500\n", "\n", "t0 = time.perf_counter()\n", "Mloop = np.zeros((n, n))\n", "for i in range(n):\n", " for j in range(n):\n", " Mloop[i, j] = i + j\n", "t_loop = time.perf_counter() - t0\n", "\n", "t0 = time.perf_counter()\n", "Mbcast = np.arange(n).reshape(-1, 1) + np.arange(n).reshape(1, -1)\n", "t_bcast = time.perf_counter() - t0\n", "\n", "np.testing.assert_array_equal(Mloop, Mbcast)\n", "print(f\"double loop : {t_loop*1000:8.2f} ms\")\n", "print(f\"broadcasting: {t_bcast*1000:8.2f} ms\")\n", "print(f\"speedup : {t_loop/t_bcast:8.1f}x\")" ] }, { "cell_type": "markdown", "id": "c037", "metadata": {}, "source": [ "**Discussion.** Two questions, one minute each:\n", "\n", "1. `i_col` is $(4,1)$ and `j_row` is $(1,5)$. Why is the result $(4,5)$ — what does the\n", " broadcasting rule do at each of the two positions?\n", "2. Which of the four versions would you rather debug at 11pm, and why?\n", "\n", "**1.** Line the shapes up from the right. Rightmost position: $1$ vs $5$ — one of them is\n", "1, so `i_col` is stretched across 5 columns. Next position: $4$ vs $1$ — again one is 1,\n", "so `j_row` is stretched down 4 rows. Both stretch, and the result takes the larger size at\n", "each position: $(4, 5)$.\n", "\n", "**2.** Reasonable people differ, but the broadcast version has one line and no index\n", "arithmetic, so there is nowhere for an off-by-one to hide — and if the shapes are wrong,\n", "NumPy raises immediately rather than silently filling the wrong entries. The cost is that\n", "you have to *read* it, which is what this exercise is for. The loop versions are easier to\n", "step through but far slower, and in Lab 2 a loop over samples inside a training loop is the\n", "difference between seconds and minutes." ] }, { "cell_type": "markdown", "id": "c038", "metadata": {}, "source": [ "# Part 3 — Backpropagation on a single neuron\n", "\n", "Recall the 4-step procedure from L3:\n", "\n", "1. decompose into a composition of simple functions\n", "2. derive the derivative of each simple function\n", "3. compute the intermediate outputs and derivatives (**forward pass**)\n", "4. multiply the derivatives via the chain rule (**backward pass**)\n", "\n", "The L3 slide example, $y = [\\ln(x^2+1)]^2$, is a single chain. Here is a graph that\n", "actually looks like a piece of a network — one neuron with two inputs, a ReLU, and a\n", "squared-error loss against a target $t$:\n", "\n", "```\n", " x1 ---(* w1)---\\\n", " (+ b) ---> z ---[ReLU]---> a ---[(a - t)^2]---> J\n", " x2 ---(* w2)---/\n", "```\n", "\n", "$$z = w_1 x_1 + w_2 x_2 + b, \\qquad a = \\mathrm{ReLU}(z), \\qquad J = (a - t)^2$$\n", "\n", "The inputs $x_1, x_2$ and the target $t$ are **data** — fixed. The things we\n", "differentiate with respect to are the **parameters** $w_1, w_2, b$, because those are\n", "what gradient descent will update.\n", "\n", "Work with these numbers throughout:\n", "\n", "$$x_1 = 2.0,\\quad x_2 = -1.0,\\quad t = 1.0,\\qquad w_1 = 0.5,\\quad w_2 = 1.0,\\quad b = 0.25$$" ] }, { "cell_type": "markdown", "id": "c039", "metadata": {}, "source": [ "## (a) Do it on paper first — ~10 minutes\n", "\n", "Do all of part (a) on paper with your partner *before* touching the code. Fill in your\n", "answers in the tables below as you go.\n", "\n", "### (a-i) Forward pass\n", "\n", "Compute the intermediate values by hand.\n", "\n", "| quantity | value |\n", "|---|---|\n", "| $z$ | $0.5(2.0) + 1.0(-1.0) + 0.25 = 0.25$ |\n", "| $a$ | $\\mathrm{ReLU}(0.25) = 0.25$ |\n", "| $J$ | $(0.25 - 1.0)^2 = 0.5625$ |" ] }, { "cell_type": "markdown", "id": "c040", "metadata": {}, "source": [ "### (a-ii) Local derivatives\n", "\n", "One row per edge of the graph. Write each derivative **symbolically** first, then\n", "evaluate it at the numbers from (a-i).\n", "\n", "| local derivative | symbolic | value here |\n", "|---|---|---|\n", "| $\\dfrac{\\partial J}{\\partial a}$ | $2(a - t)$ | $2(0.25 - 1.0) = -1.5$ |\n", "| $\\dfrac{d a}{d z}$ | $1$ if $z > 0$, else $0$ | $z = 0.25 > 0 \\Rightarrow 1$ |\n", "| $\\dfrac{\\partial z}{\\partial w_1}$ | $x_1$ | $2.0$ |\n", "| $\\dfrac{\\partial z}{\\partial w_2}$ | $x_2$ | $-1.0$ |\n", "| $\\dfrac{\\partial z}{\\partial b}$ | $1$ | $1$ |\n", "\n", "*Note: $\\mathrm{ReLU}$ is not differentiable at $z = 0$. The universal convention is to\n", "define the derivative there as 0, so $\\frac{da}{dz} = 1$ if $z > 0$ and $0$ otherwise.*\n", "\n", "Every one of these is a derivative of a *simple* function — that is the whole point of\n", "step 1. None of them required the chain rule." ] }, { "cell_type": "markdown", "id": "c041", "metadata": {}, "source": [ "### (a-iii) Backward pass\n", "\n", "Multiply the local derivatives along the path from $J$ back to each parameter. It is\n", "worth computing $\\dfrac{\\partial J}{\\partial z}$ once and reusing it three times.\n", "\n", "$$\\frac{\\partial J}{\\partial z} = \\frac{\\partial J}{\\partial a}\\cdot\\frac{da}{dz}\n", " = (-1.5)(1) = -1.5$$\n", "\n", "$$\\frac{\\partial J}{\\partial w_1} = \\frac{\\partial J}{\\partial z}\\cdot\\frac{\\partial z}{\\partial w_1}\n", " = (-1.5)(2.0) = -3.0$$\n", "\n", "$$\\frac{\\partial J}{\\partial w_2} = \\frac{\\partial J}{\\partial z}\\cdot\\frac{\\partial z}{\\partial w_2}\n", " = (-1.5)(-1.0) = 1.5$$\n", "\n", "$$\\frac{\\partial J}{\\partial b} = \\frac{\\partial J}{\\partial z}\\cdot\\frac{\\partial z}{\\partial b}\n", " = (-1.5)(1) = -1.5$$\n", "\n", "Reusing $\\partial J/\\partial z$ is not just tidiness — it is the reason backpropagation is\n", "efficient. Everything upstream of $z$ shares that one number, so it gets computed once no\n", "matter how many parameters feed into $z$.\n", "\n", "Sanity check on the sign: $\\partial J/\\partial w_1 = -3.0$ is negative, so *increasing*\n", "$w_1$ decreases the loss. That makes sense — $a = 0.25$ is below the target $t = 1.0$, and\n", "$x_1$ is positive, so a larger $w_1$ pushes the prediction up toward the target." ] }, { "cell_type": "markdown", "id": "c042", "metadata": {}, "source": [ "### (a-iv) A different setting (skip if low on time)\n", "\n", "Suppose $b = -3.0$ instead, with everything else unchanged. Recompute $z$ and $a$, then\n", "work out all three gradients. What happened, and what does it mean for gradient descent?\n", "\n", "Now $z = 0.5(2.0) + 1.0(-1.0) - 3.0 = -3.0$, so $a = \\mathrm{ReLU}(-3.0) = 0$ and\n", "$J = (0 - 1)^2 = 1$. The loss is *worse*. But:\n", "\n", "$$\\frac{da}{dz} = 0 \\;\\Longrightarrow\\; \\frac{\\partial J}{\\partial z} = (-2)(0) = 0\n", "\\;\\Longrightarrow\\; \\frac{\\partial J}{\\partial w_1} = \\frac{\\partial J}{\\partial w_2}\n", "= \\frac{\\partial J}{\\partial b} = 0$$\n", "\n", "Every gradient is exactly zero, so the gradient descent update\n", "$w \\leftarrow w - \\gamma \\cdot 0$ changes nothing. The neuron is **dead**: it is wrong, it\n", "knows it is wrong (the loss is 1), and it has no signal telling it which way to move. One\n", "zero factor anywhere in the chain kills the entire path behind it.\n", "\n", "This is the well-known \"dying ReLU\" problem, and it is why initialization, learning rate,\n", "and alternatives like Leaky ReLU matter in practice. Keep it in mind in Lab 2: if a chunk\n", "of your network's gradients are identically zero, this is the first thing to check." ] }, { "cell_type": "markdown", "id": "c043", "metadata": {}, "source": [ "## (b) Now write it in code — ~15 minutes\n", "\n", "Two functions. `forward` computes and **returns the intermediate values**, because\n", "`backward` needs them — that is the whole reason frameworks \"cache activations\" during\n", "the forward pass.\n", "\n", "### TODO 8 — the forward pass" ] }, { "cell_type": "code", "execution_count": null, "id": "c044", "metadata": {}, "outputs": [], "source": [ "def forward(w1, w2, b, x1, x2, t):\n", " \"\"\"Forward pass through the single-neuron graph.\n", "\n", " Returns\n", " -------\n", " z : float, the pre-activation w1*x1 + w2*x2 + b\n", " a : float, the activation ReLU(z)\n", " J : float, the squared-error loss (a - t)**2\n", " \"\"\"\n", " # ============ TODO 8: three lines, following the graph left to right ============\n", " z = w1 * x1 + w2 * x2 + b\n", " a = max(z, 0.0)\n", " J = (a - t) ** 2\n", " # ===============================================================================\n", " return z, a, J\n", "\n", "\n", "# --- PROVIDED --- check against your (a-i) answers.\n", "x1, x2, t = 2.0, -1.0, 1.0\n", "w1, w2, b = 0.5, 1.0, 0.25\n", "\n", "z, a, J = forward(w1, w2, b, x1, x2, t)\n", "print(f\"z = {z} a = {a} J = {J}\")" ] }, { "cell_type": "markdown", "id": "c045", "metadata": {}, "source": [ "### TODO 9 — the backward pass\n", "\n", "One line per row of your (a-ii) table, then the three products from (a-iii). Note that\n", "`backward` takes `z` and `a` as arguments — the cached forward values — rather than\n", "recomputing them." ] }, { "cell_type": "code", "execution_count": null, "id": "c046", "metadata": {}, "outputs": [], "source": [ "def backward(z, a, t, x1, x2):\n", " \"\"\"Backward pass. Takes the cached forward values and returns the parameter gradients.\n", "\n", " Returns\n", " -------\n", " dJ_dw1, dJ_dw2, dJ_db : floats\n", " \"\"\"\n", " # ============ TODO 9: local derivatives, then the chain rule ============\n", " dJ_da = 2.0 * (a - t) # derivative of (a - t)^2 with respect to a\n", " da_dz = 1.0 if z > 0 else 0.0 # derivative of ReLU at z (1 if z > 0 else 0)\n", " dJ_dz = dJ_da * da_dz # chain rule: combine the two above\n", "\n", " dJ_dw1 = dJ_dz * x1 # dJ_dz times dz/dw1\n", " dJ_dw2 = dJ_dz * x2 # dJ_dz times dz/dw2\n", " dJ_db = dJ_dz * 1.0 # dJ_dz times dz/db\n", " # ========================================================================\n", " return dJ_dw1, dJ_dw2, dJ_db\n", "\n", "\n", "# --- PROVIDED --- these should match your (a-iii) answers exactly.\n", "grads = backward(z, a, t, x1, x2)\n", "print(f\"dJ/dw1 = {grads[0]} dJ/dw2 = {grads[1]} dJ/db = {grads[2]}\")" ] }, { "cell_type": "markdown", "id": "c047", "metadata": {}, "source": [ "### Gradient check\n", "\n", "Same idea as in Lab 1, and the same idea as the L3 slides: nudge one parameter by\n", "$\\pm\\epsilon$, re-run only the *forward* pass, and estimate\n", "\n", "$$\\frac{\\partial J}{\\partial w_i} \\approx \\frac{J^+ - J^-}{2\\epsilon}.$$\n", "\n", "If your analytic gradient disagrees with this, the analytic gradient is almost always\n", "the one that is wrong. Run the provided cell." ] }, { "cell_type": "code", "execution_count": null, "id": "c048", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- gradient check. You do not need to modify this.\n", "def grad_check(w1, w2, b, x1, x2, t, eps=1e-5, tol=1e-7):\n", " z, a, _ = forward(w1, w2, b, x1, x2, t)\n", " analytic = backward(z, a, t, x1, x2)\n", "\n", " params = [w1, w2, b]\n", " numeric = []\n", " for i in range(3):\n", " up, dn = list(params), list(params)\n", " up[i] += eps\n", " dn[i] -= eps\n", " Jp = forward(up[0], up[1], up[2], x1, x2, t)[2]\n", " Jm = forward(dn[0], dn[1], dn[2], x1, x2, t)[2]\n", " numeric.append((Jp - Jm) / (2 * eps))\n", "\n", " print(f\"{'param':6s} {'analytic':>12s} {'numeric':>12s} {'rel err':>12s} result\")\n", " for name, ana, num in zip([\"w1\", \"w2\", \"b\"], analytic, numeric):\n", " denom = max(abs(ana), abs(num), 1e-12)\n", " rel = abs(ana - num) / denom\n", " print(f\"{name:6s} {ana:12.6f} {num:12.6f} {rel:12.2e} {'PASS' if rel < tol else 'FAIL'}\")\n", "\n", "grad_check(w1, w2, b, x1, x2, t)" ] }, { "cell_type": "markdown", "id": "c049", "metadata": {}, "source": [ "### TODO 10 — the dead neuron\n", "\n", "Close the loop on (a-iv): rerun the forward and backward pass with $b = -3.0$ and check\n", "that all three gradients are exactly zero." ] }, { "cell_type": "code", "execution_count": null, "id": "c050", "metadata": {}, "outputs": [], "source": [ "# ============ TODO 10: same neuron, b = -3.0 ============\n", "b_dead = -3.0\n", "z_d, a_d, J_d = forward(w1, w2, b_dead, x1, x2, t)\n", "grads_dead = backward(z_d, a_d, t, x1, x2)\n", "# ========================================================\n", "\n", "print(f\"z = {z_d} a = {a_d} J = {J_d}\")\n", "print(\"gradients:\", grads_dead)\n", "assert all(g == 0.0 for g in grads_dead), \"expected every gradient to be zero\"\n", "print(\"All gradients are zero -- this neuron cannot learn.\")" ] }, { "cell_type": "markdown", "id": "c051", "metadata": {}, "source": [ "## (c) Two data points: gradients accumulate — ~5 minutes\n", "\n", "So far, one training point. With **two** points sharing the same $w_1, w_2, b$, the loss\n", "is the sum of the two individual losses:\n", "\n", "$$J = J^{(1)} + J^{(2)}$$\n", "\n", "Now $w_1$ feeds **two** paths through the graph — one per data point. The chain rule for\n", "a variable that fans out says: **sum the contributions from every path**.\n", "\n", "$$\\frac{\\partial J}{\\partial w_1}\n", "= \\frac{\\partial J^{(1)}}{\\partial w_1} + \\frac{\\partial J^{(2)}}{\\partial w_1}$$\n", "\n", "This is not a new rule to memorize — it is the reason Lab 1's `X.T @ residuals` is a sum\n", "over samples, and it is what the batched backward pass in Lab 2 is doing under the hood.\n", "\n", "### TODO 11\n", "\n", "Loop over the two points, calling your existing `forward` and `backward` once per point,\n", "and accumulate." ] }, { "cell_type": "code", "execution_count": null, "id": "c052", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- two training points, same parameters.\n", "data = [(2.0, -1.0, 1.0), # (x1, x2, t) -- the point from part (a)\n", " (1.0, 3.0, 0.0)]\n", "\n", "total_grads = np.zeros(3)\n", "total_loss = 0.0\n", "\n", "for (xi1, xi2, ti) in data:\n", " # ============ TODO 11: forward, backward, accumulate ============\n", " zi, ai, Ji = forward(w1, w2, b, xi1, xi2, ti)\n", " gi = backward(zi, ai, ti, xi1, xi2)\n", " total_loss += Ji\n", " total_grads += np.array(gi)\n", " # ===============================================================\n", "\n", "print(f\"total loss = {total_loss}\")\n", "print(f\"dJ/dw1 = {total_grads[0]} dJ/dw2 = {total_grads[1]} dJ/db = {total_grads[2]}\")" ] }, { "cell_type": "markdown", "id": "c053", "metadata": {}, "source": [ "## What to carry into Lab 2\n", "\n", "- **`*` is elementwise, `@` is matrix multiplication.** When something breaks, print\n", " `.shape` before you print values.\n", "- **Broadcasting is the bias add.** $A W \\oplus \\bar b$ works because a $1 \\times s_\\ell$\n", " bias stretches down all $N$ rows of the batch. Shapes align from the right.\n", "- **The backward pass is cached forward values times local derivatives.** Decompose,\n", " differentiate each simple piece, then multiply along the path. Compute shared\n", " quantities like $\\partial J/\\partial z$ once.\n", "- **A zero anywhere in the chain zeroes everything behind it** — that is a dead ReLU.\n", "- **When the gradient is suspect, gradient-check it.** You will use the exact same\n", " central-difference trick in Lab 2." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.13" } }, "nbformat": 4, "nbformat_minor": 5 }