{ "cells": [ { "cell_type": "markdown", "id": "b9a9fff5", "metadata": {}, "source": [ "# L5 Optional Extension: Attacking Your Own Classifier\n", "\n", "### E190BO — Fall 2026\n", "\n", "*This notebook is completely optional — it is for people who have finished Lab 2 and want\n", "to keep going. Nothing in this notebook will be on a quiz or needed in future assignments.*\n", "\n", "In Lab 2 you computed $\\frac{dJ}{dW}$: how the loss changes when you nudge a **weight**.\n", "That is what you need to train a network.\n", "\n", "Today you take the same chain rule **one step further** and compute $\\frac{dJ}{dX}$: how\n", "the loss changes when you nudge a **pixel**. The weights stay frozen; the image becomes\n", "the thing you differentiate with respect to.\n", "\n", "That one extra step is worth an hour because of what it lets you do:\n", "\n", "1. **Checkpoint 1 — Saliency.** Ask your network which pixels it is actually looking at.\n", "2. **Checkpoint 2 — Adversarial examples.** Change an image by an amount you can barely\n", " see, and make the network confidently wrong. Then make it wrong in a *direction you\n", " choose*.\n", "3. **Checkpoint 3 — What it learned.** Look inside $W_1$, and at which digits it confuses.\n", "\n", "Each checkpoint is about 20 minutes. Work with whoever is nearby.\n", "\n", "**A warning about Checkpoint 3:** it is the one most likely to disappoint you, and the\n", "disappointment is the point. Read the text there before you decide your code is broken." ] }, { "cell_type": "markdown", "id": "68e8edd3", "metadata": {}, "source": [ "## Setup\n", "\n", "Three provided cells: load your Lab 2 functions, load MNIST, train a network. Run them\n", "in order. Nothing below works until all three run clean." ] }, { "cell_type": "code", "execution_count": null, "id": "470d9016", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- load the functions YOU wrote in Lab 2. Run this cell first.\n", "#\n", "# This reads your completed Lab 2 notebook and executes every cell that defines a\n", "# function, so that `forward_propagation_using_vectorization` and friends are available\n", "# here. Nothing is copied -- it is your code, running.\n", "import glob, json, warnings\n", "from pathlib import Path\n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "\n", "np.set_printoptions(precision=4, suppress=True)\n", "\n", "# numpy 2.1.3 on Apple Silicon reports spurious \"... encountered in matmul\" warnings on\n", "# perfectly ordinary matrix multiplies. The results are correct and finite -- it is a bug\n", "# in that numpy/BLAS combination, and you may have seen it in Lab 2 already. The line below\n", "# silences that one message and nothing else: a genuine overflow or divide-by-zero in your\n", "# own code will still be reported.\n", "warnings.filterwarnings(\"ignore\", message=\".*encountered in matmul\", category=RuntimeWarning)\n", "\n", "# If the search below fails, set this to the full path of your Lab 2 notebook, e.g.\n", "# LAB2 = \"/Users/you/e190bo/hw2.ipynb\"\n", "LAB2 = None\n", "\n", "NEEDED = [\"relu\", \"softmax\", \"cross_entropy_loss\",\n", " \"initialize_weights\", \"initialize_bias\",\n", " \"forward_propagation_using_vectorization\",\n", " \"back_propagation_using_vectorization\",\n", " \"training_using_vectorization\", \"predict_using_vectorization\"]\n", "\n", "def load_lab2(path):\n", " \"\"\"Execute every `def ...` cell of a completed Lab 2 notebook in this namespace.\"\"\"\n", " for cell in json.load(open(path))[\"cells\"]:\n", " if cell[\"cell_type\"] == \"code\":\n", " source = \"\".join(cell[\"source\"])\n", " if source.lstrip().startswith(\"def \"):\n", " exec(compile(source, str(path), \"exec\"), globals())\n", "\n", "if LAB2 is None:\n", " found = []\n", " for pattern in (\"hw2.ipynb\", \"*hw2*.ipynb\", \"*Lab2*.ipynb\", \"*Lab 2*.ipynb\"):\n", " for hit in glob.glob(pattern) + glob.glob(\"../\" + pattern) + glob.glob(\"../../\" + pattern):\n", " if \"soln\" not in hit.lower() and hit not in found:\n", " found.append(hit)\n", " LAB2 = found[0] if found else None\n", "\n", "if LAB2 is None:\n", " print(\"Could not find your Lab 2 notebook.\")\n", " print(\"Set LAB2 at the top of this cell to its full path, then re-run.\")\n", "else:\n", " load_lab2(LAB2)\n", " missing = [name for name in NEEDED if name not in globals()]\n", " print(f\"Loaded from: {LAB2}\")\n", " if missing:\n", " print(\"MISSING -- this notebook will not work until these are defined:\")\n", " for name in missing:\n", " print(f\" - {name}\")\n", " else:\n", " print(\"All nine Lab 2 functions loaded.\")" ] }, { "cell_type": "code", "execution_count": null, "id": "a8cf652a", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- MNIST, exactly as in Lab 2 Part 3.\n", "MNIST = \"mnist.npz\"\n", "if Path(MNIST).exists():\n", " data = np.load(MNIST)\n", "else:\n", " import io, urllib.request\n", " url = \"https://course.aiprototypes.org/resources/files/eng190bo-fa26-lab-2-mnist.npz\"\n", " data = np.load(io.BytesIO(urllib.request.urlopen(url).read()))\n", "\n", "# One difference from Lab 2: we cast the pixels to float64. Checkpoint 1 compares your\n", "# gradient against a finite difference taken with a step of 1e-6, and float32 carries\n", "# only about 7 decimal digits -- not enough to make that comparison mean anything.\n", "X_train = data[\"X_train\"].astype(np.float64)\n", "X_test = data[\"X_test\"].astype(np.float64)\n", "y_train, y_test = data[\"y_train\"], data[\"y_test\"]\n", "Y_train, Y_test = np.eye(10)[y_train], np.eye(10)[y_test]\n", "\n", "print(f\"train {X_train.shape} test {X_test.shape} pixel range [{X_train.min()}, {X_train.max()}]\")" ] }, { "cell_type": "code", "execution_count": null, "id": "498a31ad", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- train the network you will spend the rest of the hour attacking.\n", "# 64 hidden units, learning rate 0.5, 600 epochs. A few seconds on a laptop.\n", "W1, W2, b1, b2 = training_using_vectorization(X_train, Y_train, 64, 0.5, 600)\n", "\n", "train_pred, _ = predict_using_vectorization(W1, W2, b1, b2, X_train)\n", "test_pred, _ = predict_using_vectorization(W1, W2, b1, b2, X_test)\n", "train_acc = np.mean(train_pred == y_train)\n", "test_acc = np.mean(test_pred == y_test)\n", "print(f\"\\ntrain accuracy {train_acc:.3f} test accuracy {test_acc:.3f}\")\n", "\n", "# Expect roughly 1.00 train and 0.85-0.92 test. If test accuracy is below 0.8, the\n", "# problem is in your Lab 2 code, not here -- fix that first or nothing below will mean\n", "# anything. (Train 1.00 against test ~0.89 is overfitting. Hold that thought.)" ] }, { "cell_type": "markdown", "id": "3dd5ef51", "metadata": {}, "source": [ "---\n", "\n", "## Checkpoint 1 — What is the network looking at?\n", "\n", "### The one new idea\n", "\n", "Every gradient you wrote in Lab 2 answered the same question: *if I nudge this **weight**,\n", "how does the loss change?* Training needs that, because weights are what you get to change.\n", "\n", "But the loss depends on the input pixels too, and nothing stops you from asking:\n", "\n", "$$\\frac{dJ}{dX}$$\n", "\n", "*If I nudge this **pixel**, how does the loss change?*\n", "\n", "This has the same shape as $X$: one number per pixel per image, $(m, 784)$. A large positive\n", "entry means *brightening this pixel makes the network more wrong*. A large negative entry\n", "means *brightening this pixel makes the network more right*. Near zero means the network\n", "does not care about that pixel at all.\n", "\n", "Displayed as a $28 \\times 28$ picture, that is a **saliency map**.\n", "\n", "### Getting there\n", "\n", "You already have almost all of it. Your `back_propagation_using_vectorization` walks the\n", "chain from $J$ backward and stops when it has what it needs for the weights. You need the\n", "same walk, one step further.\n", "\n", "The last step is the only new one, and you have already done its twin. The input enters\n", "the network through\n", "\n", "$$Z_1 = X W_1 + b_1$$\n", "\n", "In Lab 2 you differentiated that expression with respect to $W_1$. Now differentiate it\n", "with respect to $X$. It is the same product rule on the same line, with the roles of the\n", "two matrices swapped — and, as always, **the shape settles the transpose**: the answer\n", "must come out $(m, 784)$.\n", "\n", "*Two hints:*\n", "\n", "- `forward_propagation_using_vectorization` hands back `A1`, not `Z1`, so you cannot write\n", " `Z1 > 0` directly. You do not need to: think about what ReLU does, and convince yourself\n", " that `A1 > 0` is the identical mask. (It is worth thirty seconds — it is the kind of\n", " thing that saves you a recomputation in real code.)\n", "- Keep the $\\frac{1}{m}$ exactly where Lab 2 put it. The check below will catch you if\n", " you drop it." ] }, { "cell_type": "markdown", "id": "7f6a6a22", "metadata": {}, "source": [ "### TODO 1 — the gradient with respect to the input" ] }, { "cell_type": "code", "execution_count": null, "id": "d6e7fa1d", "metadata": {}, "outputs": [], "source": [ "def input_gradient(W1, W2, b1, b2, X, Y):\n", " \"\"\"dJ/dX -- the gradient of the loss with respect to the INPUT PIXELS.\n", "\n", " Parameters\n", " ----------\n", " W1, W2, b1, b2 : the trained network. These are held FIXED here.\n", " X : (m, n0) images\n", " Y : (m, n2) one-hot labels\n", "\n", " Returns\n", " -------\n", " dX : (m, n0) array, same shape as X. dX[i, j] is the partial derivative of the\n", " loss with respect to pixel j of image i.\n", " \"\"\"\n", " m = X.shape[0]\n", " A1, A2 = forward_propagation_using_vectorization(W1, W2, b1, b2, X)\n", "\n", " # ============ TODO 1 ============\n", " dZ2 = ...\n", " dA1 = ...\n", " dZ1 = ...\n", " dX = ...\n", " # ================================\n", "\n", " return dX" ] }, { "cell_type": "markdown", "id": "6f9a1bdd", "metadata": {}, "source": [ "### Check your work\n", "\n", "Same idea as the gradient checking you wrote in Lab 2 — compare against a finite\n", "difference — except that now you perturb a **pixel** instead of a weight.\n", "\n", "This runs on a batch of 3 images, on purpose: with a single image the $\\frac{1}{m}$ factor\n", "is invisible, so a batch is the only way to catch a missing one." ] }, { "cell_type": "code", "execution_count": null, "id": "0077b3d3", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- run this to check TODO 1. You do not need to modify it.\n", "def check_input_gradient(m=3, num_pixels=40, epsilon=1e-6):\n", " X, Y = X_test[:m].copy(), Y_test[:m]\n", " try:\n", " G = input_gradient(W1, W2, b1, b2, X, Y)\n", " except TypeError:\n", " print(\"Finish TODO 1 first.\")\n", " return\n", " if G is Ellipsis or G is None:\n", " print(\"Finish TODO 1 first.\")\n", " return\n", " if G.shape != X.shape:\n", " print(f\"dX came out {G.shape}, but it must match X at {X.shape}.\")\n", " print(\"The shape is what tells you where the transpose goes -- fix that first.\")\n", " return\n", "\n", " rng = np.random.default_rng(0)\n", " worst = 0.0\n", " for _ in range(num_pixels):\n", " i = int(rng.integers(m)) # which image\n", " j = int(rng.integers(X.shape[1])) # which pixel\n", " X_plus, X_minus = X.copy(), X.copy()\n", " X_plus[i, j] += epsilon\n", " X_minus[i, j] -= epsilon\n", " J_plus = cross_entropy_loss(forward_propagation_using_vectorization(W1, W2, b1, b2, X_plus)[1], Y)\n", " J_minus = cross_entropy_loss(forward_propagation_using_vectorization(W1, W2, b1, b2, X_minus)[1], Y)\n", " worst = max(worst, abs((J_plus - J_minus) / (2 * epsilon) - G[i, j]))\n", "\n", " print(f\"largest gap between your gradient and the finite difference: {worst:.3e}\")\n", " if worst < 1e-6:\n", " print(\"PASS\")\n", " else:\n", " print(\"FAIL -- recheck TODO 1. A missing 1/m or a missing ReLU mask both land here.\")\n", "\n", "check_input_gradient()" ] }, { "cell_type": "markdown", "id": "0b4a5756", "metadata": {}, "source": [ "### Look at it\n", "\n", "Two pictures per digit, and the difference between them is the interesting part." ] }, { "cell_type": "code", "execution_count": null, "id": "42143039", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- saliency maps for the first five test images.\n", "def show_saliency(indices=range(5)):\n", " indices = list(indices)\n", " G = input_gradient(W1, W2, b1, b2, X_test[indices], Y_test[indices])\n", " if G is Ellipsis:\n", " print(\"Finish TODO 1 first.\")\n", " return\n", "\n", " fig, axes = plt.subplots(3, len(indices), figsize=(2.2 * len(indices), 6.6))\n", " for col, (k, idx) in enumerate(zip(indices, range(len(indices)))):\n", " image = X_test[k]\n", " axes[0, col].imshow(image.reshape(28, 28), cmap=\"gray\")\n", " axes[0, col].set_title(f\"digit {y_test[k]}\", fontsize=9)\n", " axes[1, col].imshow(np.abs(G[idx]).reshape(28, 28), cmap=\"hot\")\n", " axes[1, col].set_title(\"|dJ/dx|\", fontsize=9)\n", " axes[2, col].imshow(np.abs(G[idx] * image).reshape(28, 28), cmap=\"hot\")\n", " axes[2, col].set_title(\"|x * dJ/dx|\", fontsize=9)\n", " for row in range(3):\n", " axes[row, col].axis(\"off\")\n", " plt.tight_layout()\n", " plt.show()\n", "\n", "show_saliency()" ] }, { "cell_type": "markdown", "id": "157b807b", "metadata": {}, "source": [ "### Think about it\n", "\n", "Middle row: the raw gradient. Bottom row: the gradient multiplied elementwise by the image\n", "itself. The bottom row traces the stroke. The middle row is a mess.\n", "\n", "**Q1.** The middle row puts a lot of weight on pixels that are *black* in that image. Is\n", "that a bug in your gradient, or a true statement about the network? (Run the cell below\n", "before you answer.)\n", "\n", "**Q2.** Given your answer to Q1, what does multiplying by `x` accomplish? Which of the two\n", "pictures is the honest answer to \"what is the network looking at\"?" ] }, { "cell_type": "code", "execution_count": null, "id": "d77dd536", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- how much of the gradient sits on pixels that are currently black?\n", "indices = list(range(5))\n", "G = input_gradient(W1, W2, b1, b2, X_test[indices], Y_test[indices])\n", "ink = X_test[indices] > 0\n", "print(f\"{ink.mean():.0%} of pixels have any ink in them\")\n", "print(f\"{np.abs(G * ink).sum() / np.abs(G).sum():.0%} of the total |gradient| sits on those pixels\")" ] }, { "cell_type": "markdown", "id": "1d90d0da", "metadata": {}, "source": [ "---\n", "\n", "## Checkpoint 2 — Breaking it on purpose\n", "\n", "### The idea\n", "\n", "Training does gradient **descent** on the weights: step downhill, loss goes down.\n", "\n", "Freeze the weights, and do gradient **ascent** on the pixels instead: step *uphill* in $X$,\n", "and the loss goes up — the network gets more wrong. Nothing about the network changed. Only\n", "the image did.\n", "\n", "The version you will implement is the **Fast Gradient Sign Method** (Goodfellow et al.,\n", "2014), which is one line:\n", "\n", "$$X_{\\text{adv}} = \\text{clip}\\left(X + \\epsilon \\cdot \\text{sign}\\left(\\frac{dJ}{dX}\\right),\\ 0,\\ 1\\right)$$\n", "\n", "Two details worth pausing on:\n", "\n", "- **Why `sign` and not the gradient itself?** Because it puts a hard cap on how visible the\n", " attack is. With `sign`, *every* pixel moves by exactly $\\epsilon$ — never more — so\n", " $\\epsilon$ is a direct budget on the largest change any single pixel is allowed. Using the\n", " raw gradient would move a few pixels a lot and most pixels not at all, which is easier to\n", " see and a weaker attack for the same visible damage.\n", "- **Why `clip`?** Pixels live in $[0, 1]$. An \"image\" with a pixel value of $1.1$ is not an\n", " image, and an attack that needs one is cheating.\n", "\n", "$\\epsilon$ is in the units of the pixels: $\\epsilon = 0.1$ means every pixel shifts by 10%\n", "of the full black-to-white range." ] }, { "cell_type": "markdown", "id": "c67f3bed", "metadata": {}, "source": [ "### TODO 2 — FGSM" ] }, { "cell_type": "code", "execution_count": null, "id": "e18de857", "metadata": {}, "outputs": [], "source": [ "def fgsm(W1, W2, b1, b2, X, Y, epsilon):\n", " \"\"\"One step of gradient ASCENT on the pixels. Returns perturbed images.\n", "\n", " Parameters\n", " ----------\n", " X, Y : (m, n0) images and (m, n2) one-hot TRUE labels\n", " epsilon : how far each pixel may move, in pixel-intensity units\n", "\n", " Returns\n", " -------\n", " X_adv : (m, n0), same shape as X, every value still in [0, 1]\n", " \"\"\"\n", " # ============ TODO 2 ============\n", " X_adv = ...\n", " # ================================\n", "\n", " return X_adv" ] }, { "cell_type": "code", "execution_count": null, "id": "9ee8ebc9", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- run this to check TODO 2. You do not need to modify it.\n", "def check_fgsm(epsilon=0.1):\n", " X, Y = X_test[:8], Y_test[:8]\n", " try:\n", " X_adv = fgsm(W1, W2, b1, b2, X, Y, epsilon)\n", " except TypeError:\n", " print(\"Finish TODO 2 first.\")\n", " return\n", " if X_adv is Ellipsis or X_adv is None:\n", " print(\"Finish TODO 2 first.\")\n", " return\n", "\n", " ok = True\n", " if X_adv.shape != X.shape:\n", " print(f\"shape is {X_adv.shape}, expected {X.shape}\"); ok = False\n", " if X_adv.min() < 0 or X_adv.max() > 1:\n", " print(f\"values run [{X_adv.min():.2f}, {X_adv.max():.2f}] -- did you clip?\"); ok = False\n", " moved = np.abs(X_adv - X)\n", " if moved.max() > epsilon + 1e-9:\n", " print(f\"some pixel moved by {moved.max():.3f} > epsilon -- did you take the sign?\"); ok = False\n", " if np.allclose(X_adv, X):\n", " print(\"nothing moved at all\"); ok = False\n", "\n", " base, _ = predict_using_vectorization(W1, W2, b1, b2, X)\n", " adv, _ = predict_using_vectorization(W1, W2, b1, b2, X_adv)\n", " flipped = int(np.sum(base != adv))\n", " print(f\"at epsilon={epsilon}, {flipped} of 8 predictions changed\")\n", " if ok and flipped >= 3:\n", " print(\"PASS\")\n", " elif ok:\n", " print(\"Shapes and clipping are fine, but almost nothing flipped -- check the SIGN of\")\n", " print(\"your step. Descending makes the network more confident, not less.\")\n", "\n", "check_fgsm()" ] }, { "cell_type": "markdown", "id": "0d1f6e7f", "metadata": {}, "source": [ "### Look at it\n", "\n", "Red title = the network is now wrong. Look at the images, not just the labels: the question\n", "that matters is whether *you* would still read the digit correctly." ] }, { "cell_type": "code", "execution_count": null, "id": "66d7d193", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- an epsilon sweep on three test images.\n", "def show_attack(indices=(0, 1, 4), epsilons=(0.0, 0.02, 0.05, 0.10, 0.15, 0.20)):\n", " fig, axes = plt.subplots(len(indices), len(epsilons),\n", " figsize=(2.0 * len(epsilons), 2.2 * len(indices)))\n", " for row, i in enumerate(indices):\n", " X, Y = X_test[i:i+1], Y_test[i:i+1]\n", " for col, epsilon in enumerate(epsilons):\n", " X_adv = fgsm(W1, W2, b1, b2, X, Y, epsilon)\n", " pred, probs = predict_using_vectorization(W1, W2, b1, b2, X_adv)\n", " correct = pred[0] == y_test[i]\n", " ax = axes[row, col]\n", " ax.imshow(X_adv.reshape(28, 28), cmap=\"gray\", vmin=0, vmax=1)\n", " ax.set_title(f\"e={epsilon:.2f}: says {pred[0]} ({probs[0, pred[0]]:.2f})\",\n", " fontsize=9, color=\"black\" if correct else \"red\")\n", " ax.axis(\"off\")\n", " plt.tight_layout()\n", " plt.show()\n", "\n", "show_attack()" ] }, { "cell_type": "markdown", "id": "c84677f7", "metadata": {}, "source": [ "### The control\n", "\n", "Here is the objection you should already be raising: *of course the network breaks — you\n", "added noise to the image.*\n", "\n", "So test it. The cell below compares your attack against random noise of **exactly the same\n", "size**: every pixel moves by exactly $\\epsilon$, same budget, same visible damage, but in a\n", "random direction instead of the gradient direction." ] }, { "cell_type": "code", "execution_count": null, "id": "09c3d706", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- your attack vs. random noise of identical magnitude, on all 500 test images.\n", "def attack_curve(epsilons=(0.0, 0.02, 0.05, 0.10, 0.15, 0.20, 0.30)):\n", " rng = np.random.default_rng(1)\n", " fgsm_acc, noise_acc = [], []\n", " for epsilon in epsilons:\n", " X_adv = fgsm(W1, W2, b1, b2, X_test, Y_test, epsilon)\n", " pred_adv, _ = predict_using_vectorization(W1, W2, b1, b2, X_adv)\n", " fgsm_acc.append(np.mean(pred_adv == y_test))\n", "\n", " X_noise = np.clip(X_test + epsilon * rng.choice([-1.0, 1.0], size=X_test.shape), 0, 1)\n", " pred_noise, _ = predict_using_vectorization(W1, W2, b1, b2, X_noise)\n", " noise_acc.append(np.mean(pred_noise == y_test))\n", "\n", " print(f\" epsilon={epsilon:.2f} your attack {fgsm_acc[-1]:.3f} random noise {noise_acc[-1]:.3f}\")\n", "\n", " plt.figure(figsize=(5.5, 4))\n", " plt.plot(epsilons, fgsm_acc, \"o-\", label=\"FGSM (gradient direction)\")\n", " plt.plot(epsilons, noise_acc, \"s-\", label=\"random noise, same size\")\n", " plt.xlabel(\"epsilon (per-pixel budget)\"); plt.ylabel(\"test accuracy\")\n", " plt.ylim(0, 1); plt.legend(); plt.grid(alpha=0.3)\n", " plt.tight_layout(); plt.show()\n", "\n", "attack_curve()" ] }, { "cell_type": "markdown", "id": "61a77489", "metadata": {}, "source": [ "**Q3.** At $\\epsilon = 0.1$ the two curves are far apart. Both perturbations moved every\n", "pixel by the same amount. Say in one sentence what the gradient bought you." ] }, { "cell_type": "markdown", "id": "6bf673fc", "metadata": {}, "source": [ "### TODO 3 — choose the answer\n", "\n", "Making the network wrong is one thing. Making it wrong in a *direction you choose* is the\n", "part with real consequences: not \"this digit is unreadable\" but \"this digit reads as an 8\".\n", "\n", "You need exactly one change. Descending the loss makes the network *more* confident in\n", "whatever label you hand it — so hand it the label you want, and descend." ] }, { "cell_type": "code", "execution_count": null, "id": "8920e7cb", "metadata": {}, "outputs": [], "source": [ "def fgsm_targeted(W1, W2, b1, b2, X, target, epsilon):\n", " \"\"\"Perturb X so the network predicts `target`, an integer class 0-9.\n", "\n", " Hint: build the one-hot labels for `target` the same way Lab 2 built Y_train,\n", " then take a step that makes the loss for THAT label smaller.\n", " \"\"\"\n", " Y_target = np.eye(10)[np.full(X.shape[0], target)]\n", "\n", " # ============ TODO 3 ============\n", " X_adv = ...\n", " # ================================\n", "\n", " return X_adv" ] }, { "cell_type": "code", "execution_count": null, "id": "8a85fca6", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- turn the first eight test digits into the digit of your choice.\n", "TARGET = 8\n", "\n", "X_adv = fgsm_targeted(W1, W2, b1, b2, X_test[:8], TARGET, 0.15)\n", "pred, probs = predict_using_vectorization(W1, W2, b1, b2, X_adv)\n", "\n", "fig, axes = plt.subplots(1, 8, figsize=(14, 2.3))\n", "for k in range(8):\n", " axes[k].imshow(X_adv[k].reshape(28, 28), cmap=\"gray\", vmin=0, vmax=1)\n", " hit = pred[k] == TARGET\n", " axes[k].set_title(f\"{y_test[k]} -> {pred[k]} ({probs[k, pred[k]]:.2f})\",\n", " fontsize=9, color=\"red\" if hit else \"black\")\n", " axes[k].axis(\"off\")\n", "plt.tight_layout(); plt.show()\n", "print(f\"{int(np.sum(pred == TARGET))} of 8 became a {TARGET}\")" ] }, { "cell_type": "markdown", "id": "9bdd8405", "metadata": {}, "source": [ "**Q4.** Try a few values of `TARGET`. Some are easy to reach from any starting digit and\n", "some are hard. Before you look: which digits would *you* guess are the easy targets, and\n", "why?\n", "\n", "**Q5.** Everything you just did needed `W1, W2, b1, b2` — you attacked a network whose\n", "weights you could read. Real systems do not hand you their weights. Does that make this\n", "attack impossible, or only inconvenient? You do not have the tools to answer this properly\n", "yet; give your intuition and hold onto it." ] }, { "cell_type": "markdown", "id": "6dace360", "metadata": {}, "source": [ "---\n", "\n", "## Checkpoint 3 — What did it actually learn?\n", "\n", "**Read this before you run anything.** You are about to look at the first layer of a trained\n", "network, and the honest expectation is that it will be underwhelming. People expect to find\n", "crisp stroke detectors and edge detectors. You will mostly find blobs.\n", "\n", "That is not a bug in your Lab 2 code and it is not a bug here. One hidden layer, 64 units,\n", "2000 training images — this network has no reason to organize itself into features a human\n", "would name, and it does not. Individual units being uninterpretable is the *typical* case\n", "for neural networks, not the exception, and knowing that first-hand is better than believing\n", "the tidy diagrams.\n", "\n", "There are two genuinely readable things in here, and they are below." ] }, { "cell_type": "markdown", "id": "d622f2f3", "metadata": {}, "source": [ "### TODO 4 — a hidden unit as a picture\n", "\n", "`W1` has shape $(784, 64)$. One of those axes indexes pixels and the other indexes hidden\n", "units, and getting a picture out means picking the right one.\n", "\n", "Hidden unit $j$ computes $\\sum_{i=1}^{784} x_i (W_1)_{ij}$: one weight per pixel. So the 784\n", "numbers belonging to unit $j$ are its \"template\" for what it responds to — and 784 numbers\n", "reshape into $28 \\times 28$." ] }, { "cell_type": "code", "execution_count": null, "id": "e2ac0057", "metadata": {}, "outputs": [], "source": [ "def hidden_unit_image(W1, j):\n", " \"\"\"Return the weights of hidden unit j as a 28x28 array, ready to imshow.\"\"\"\n", " # ============ TODO 4 ============\n", " image = ...\n", " # ================================\n", "\n", " return image\n", "\n", "\n", "# quick shape check\n", "_image = hidden_unit_image(W1, 0)\n", "assert _image is not Ellipsis, \"Finish TODO 4 first.\"\n", "assert _image.shape == (28, 28), f\"got {_image.shape}, expected (28, 28) -- wrong axis of W1?\"\n", "print(\"shape OK\")" ] }, { "cell_type": "code", "execution_count": null, "id": "d71c20c1", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- the first 24 hidden units. Red = positive weight, blue = negative.\n", "fig, axes = plt.subplots(3, 8, figsize=(13, 5.2))\n", "for j, ax in enumerate(axes.ravel()):\n", " image = hidden_unit_image(W1, j)\n", " limit = np.abs(image).max()\n", " ax.imshow(image, cmap=\"RdBu_r\", vmin=-limit, vmax=limit)\n", " ax.set_title(f\"unit {j}\", fontsize=8)\n", " ax.axis(\"off\")\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "bb343b88", "metadata": {}, "source": [ "### The readable thing: that pale border\n", "\n", "Look at the edges of those 24 pictures. Every one of them has a washed-out frame where the\n", "weights are near zero and, more tellingly, where all 24 units look *alike* — and units that\n", "were trained to do different jobs should not agree.\n", "\n", "The cell below explains why, and the explanation is a one-line consequence of a formula you\n", "wrote in Lab 2." ] }, { "cell_type": "code", "execution_count": null, "id": "f2e20bb7", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- why the border is pale.\n", "never_inked = (X_train > 0).sum(axis=0) == 0\n", "print(f\"{never_inked.sum()} of 784 pixels are ZERO in all 2000 training images.\")\n", "\n", "W1_init, _ = initialize_weights(784, 64, 10) # re-runs the same np.random.seed(0)\n", "print(\"W1 rows for those pixels, after 600 epochs of training, are bit-for-bit identical\")\n", "print(\"to their random initial values:\", np.array_equal(W1[never_inked], W1_init[never_inked]))\n", "print(\"...and for the other pixels: \", np.array_equal(W1[~never_inked], W1_init[~never_inked]))" ] }, { "cell_type": "markdown", "id": "8aea8300", "metadata": {}, "source": [ "**Q6.** Your Lab 2 gradient was $\\frac{dJ}{dW_1} = \\frac{1}{m} X^{\\top} \\frac{dJ}{dZ_1}$.\n", "Read the `True` above off that formula in one step. Why is it exactly equal, rather than\n", "just nearly equal?\n", "\n", "**Q7.** Those 145 weights per unit are untrained noise, and they sit in the network doing\n", "nothing — until Checkpoint 2, where the attack was free to write ink into any pixel it\n", "liked. Connect those two facts." ] }, { "cell_type": "markdown", "id": "ffbfc435", "metadata": {}, "source": [ "### The other readable thing: which digits it confuses\n", "\n", "Accuracy is one number and it hides everything. A confusion matrix `C` has `C[i, j]` = the\n", "number of test images whose true label is `i` and which the network called `j`. The diagonal\n", "is what it got right; everything off-diagonal is a specific, nameable mistake." ] }, { "cell_type": "code", "execution_count": null, "id": "285c3808", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- the confusion matrix.\n", "def confusion_matrix(y_true, y_pred, num_classes=10):\n", " \"\"\"C[i, j] = how many images with true label i were predicted as j.\"\"\"\n", " C = np.zeros((num_classes, num_classes), dtype=int)\n", " for i in range(num_classes):\n", " for j in range(num_classes):\n", " # count the images whose true label is i AND whose prediction was j\n", " C[i, j] = np.sum((y_true == i) & (y_pred == j))\n", " return C\n", "\n", "\n", "C = confusion_matrix(y_test, test_pred)\n", "print(f\"{np.trace(C)} correct out of {C.sum()}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "19dd570e", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- the matrix, and the mistakes themselves.\n", "fig, ax = plt.subplots(figsize=(5.6, 5))\n", "ax.imshow(C, cmap=\"Blues\")\n", "for i in range(10):\n", " for j in range(10):\n", " if C[i, j]:\n", " ax.text(j, i, C[i, j], ha=\"center\", va=\"center\", fontsize=7,\n", " color=\"white\" if C[i, j] > C.max() / 2 else \"black\")\n", "ax.set_xlabel(\"predicted\"); ax.set_ylabel(\"true\")\n", "ax.set_xticks(range(10)); ax.set_yticks(range(10))\n", "plt.tight_layout(); plt.show()\n", "\n", "off_diagonal = C.copy()\n", "np.fill_diagonal(off_diagonal, 0)\n", "order = np.argsort(off_diagonal.ravel())[::-1][:5]\n", "print(\"most common mistakes:\")\n", "for flat in order:\n", " i, j = np.unravel_index(flat, C.shape)\n", " print(f\" true {i} called {j}: {off_diagonal[i, j]} times\")" ] }, { "cell_type": "code", "execution_count": null, "id": "88d86204", "metadata": {}, "outputs": [], "source": [ "# --- PROVIDED --- the first ten misclassified test images. Look at them.\n", "wrong = np.where(test_pred != y_test)[0][:10]\n", "_, probs_test = predict_using_vectorization(W1, W2, b1, b2, X_test)\n", "\n", "fig, axes = plt.subplots(2, 5, figsize=(11, 4.8))\n", "for k, i in enumerate(wrong):\n", " ax = axes[k // 5, k % 5]\n", " ax.imshow(X_test[i].reshape(28, 28), cmap=\"gray\")\n", " ax.set_title(f\"true {y_test[i]} -> said {test_pred[i]} ({probs_test[i, test_pred[i]]:.2f})\",\n", " fontsize=8)\n", " ax.axis(\"off\")\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "99afba80", "metadata": {}, "source": [ "**Q8.** Of those ten mistakes, how many would *you* have gotten wrong? Sort them into\n", "\"ambiguous handwriting\" and \"the network is being stupid\", and count each pile.\n", "\n", "**Q9.** Compare the confidences on these mistakes with the confidences on the adversarial\n", "images from Checkpoint 2. Both are wrong answers. Is the network equally confident in both\n", "kinds of wrongness, and should it be?" ] }, { "cell_type": "markdown", "id": "e79e2c58", "metadata": {}, "source": [ "---\n", "\n", "## Carry this forward\n", "\n", "- **A gradient is not only for training.** $\\frac{dJ}{dW}$ updates a model;\n", " $\\frac{dJ}{dX}$ interrogates or attacks it. Same chain rule, different variable held\n", " fixed. Every technique in the second category — saliency, adversarial examples, the\n", " input-optimization tricks you will meet later — is that one substitution.\n", "- **A model is only constrained where the data pushed on it.** 145 of your input pixels\n", " never received a single unit of gradient in 600 epochs. What the network does there was\n", " decided by `np.random.seed(0)`, and an attacker chooses the input.\n", "- **Confidence is not reliability.** Your network's most confident answers today were its\n", " adversarial ones.\n", "- **High dimensions are not intuitive.** A change too small to see in any single pixel,\n", " pointed in the right direction, took accuracy from 89% to 12%. Random noise of identical\n", " size did nothing.\n", "\n", "Starting Wednesday the course moves to models you did not build and cannot see inside, and\n", "you will spend the rest of the semester putting them into products. Everything above is\n", "still true of those models. It is mostly *more* true — they are larger, their training data\n", "is less knowable, and their confident wrong answers are much harder to spot than a 7 that\n", "says it is a 3." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.13" } }, "nbformat": 4, "nbformat_minor": 5 }