{ "cells": [ { "cell_type": "markdown", "id": "l9-00", "metadata": {}, "source": [ "# L9 In-Class Activity: Take Apart Two Apps from the Command Line\n", "\n", "### E190BO — Fall 2026\n", "\n", "In the video, you watched Hyperschedule get taken apart one layer at a time. Today you will do it yourself using this notebook.\n", "\n", "- **Toolbox (about 5 min).** Short demos of every tool you'll need: `curl`, pipes, headers, status codes, and `jq`. Run them and read what they print.\n", "- **Part 1 — Hyperschedule (about 22 min).** The same API from the video. **You write every command**: each question's code cell starts empty. Check your answers against the **checkpoint key**.\n", "- **Part 2 — Wordle (about 25 min).** A new app, and this time no one is giving you the URLs. Find them with your browser's devtools, then take the app apart with the same tools. It ends with the same diagram as the video, filled in for Wordle.\n", "\n", "**Work in teams of 2–3.** Everyone runs this notebook in their own codespace and types every command. Don't move on until everyone on your team has the output. Your team shares one paper worksheet.\n", "\n", "Aim to finish Part 1 by **10:05**. If you're not done by then, move on to Part 2 anyway; that's where the sleuthing is.\n", "\n", "> **Solution notebook.** Answers verified 9/25/2026; Toolbox and 1.4 outputs from 9/28/2026. Seat counts and section totals drift; the Wordle answers are for past dates only." ] }, { "cell_type": "markdown", "id": "l9-01", "metadata": {}, "source": [ "## How this notebook runs shell commands\n", "\n", "Every code cell here starts with **`%%bash`**. That first line tells Jupyter to run the rest of the cell in **bash**, the same shell as your Codespace terminal. Anything that works in these cells works in the terminal too (**Terminal → New Terminal**, then paste everything except the `%%bash` line).\n", "\n", "The Toolbox cells below are filled in. After that, each code cell holds only `%%bash` and a comment: replace the comment with your command. (Lines starting with `#` are comments; bash ignores them.) **Keep the `%%bash` line.** Without it, Jupyter reads your command as Python and reports a `SyntaxError`.\n", "\n", "Two differences from a terminal:\n", "\n", "- **Each cell starts a fresh shell.** A variable you set in one cell (`X=5`) is gone by the next one. The exception is `%env`, which sets a variable for every cell. We use it once, just below.\n", "- You may see `!curl …` in other notebooks. **Don't use it here.** Jupyter rewrites `{…}` and `$name` in `!` lines before bash ever sees them, and that breaks most `jq` commands." ] }, { "cell_type": "markdown", "id": "l9-02", "metadata": {}, "source": [ "## Setup — run this once\n", "\n", "`$B` is shorthand for Hyperschedule's API, just like in the video. `%env` makes it available in every cell below." ] }, { "cell_type": "code", "execution_count": 1, "id": "l9-03", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "env: B=https://banana.hyperschedule.io\n" ] } ], "source": [ "%env B=https://banana.hyperschedule.io" ] }, { "cell_type": "markdown", "id": "l9-t00", "metadata": {}, "source": [ "---\n", "# Toolbox\n", "\n", "Run each demo and read what it prints. The demos use the **course website** and a small **made-up file**, so none of them answers a question in Part 1 or Part 2.\n", "\n", "Use this section as a reference whenever you need a pattern." ] }, { "cell_type": "markdown", "id": "l9-t01", "metadata": {}, "source": [ "### `curl` fetches a URL\n", "\n", "`curl` prints whatever the server sends back: here, the course home page, which is HTML. `-s` (silent) hides the progress meter.\n", "\n", "The `|` (a **pipe**) sends one command's output into the next command instead of to the screen. `head -n 5` keeps only the first five lines." ] }, { "cell_type": "code", "execution_count": 2, "id": "l9-t02", "metadata": {}, "outputs": [ { "ename": "", "evalue": "", "output_type": "error", "traceback": [ "\u001b[1;31mRunning cells with 'Python 3.9.25' requires the ipykernel package.\n", "\u001b[1;31mCreate a Python Environment with the required packages.\n", "\u001b[1;31mOr install 'ipykernel' using the command: '/bin/python3 -m pip install ipykernel -U --user --force-reinstall'" ] } ], "source": [ "%%bash\n", "curl -s https://course.aiprototypes.org/ | head -n 5" ] }, { "cell_type": "markdown", "id": "l9-t03", "metadata": {}, "source": [ "### Save it and measure it\n", "\n", "`>` saves the output to a file instead of printing it, and `wc -c` counts the file's bytes.\n", "\n", "The last line gets the same number straight from `curl`: `-o /dev/null` throws the body away, and `-w` writes out a fact about the transfer, here its size." ] }, { "cell_type": "code", "execution_count": 3, "id": "l9-t04", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "28140 home.html\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "28140 bytes\n" ] } ], "source": [ "%%bash\n", "curl -s https://course.aiprototypes.org/ > home.html\n", "wc -c home.html\n", "curl -s -o /dev/null -w '%{size_download} bytes\\n' https://course.aiprototypes.org/" ] }, { "cell_type": "markdown", "id": "l9-t05", "metadata": {}, "source": [ "### Headers\n", "\n", "Before the body, a server sends **headers**: a status line (`HTTP/2 200`), then `name: value` lines about the response. `-D -` prints the headers, and `-o /dev/null` throws the body away.\n", "\n", "These are the headers for the notebook you downloaded at the start of class. Find its type (`content-type`), its size in bytes (`content-length`), how long it may be cached (`cache-control`), and who served it (`server`)." ] }, { "cell_type": "code", "execution_count": 4, "id": "l9-t06", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "HTTP/2 200 \r\n", "date: Mon, 28 Sep 2026 17:22:19 GMT\r\n", "content-type: application/octet-stream\r\n", "content-le" ] }, { "name": "stdout", "output_type": "stream", "text": [ "ngth: 23484\r\n", "x-content-type-options: nosniff\r\n", "report-to: {\"group\":\"cf-nel\",\"max_age\":604800,\"endpoin" ] }, { "name": "stdout", "output_type": "stream", "text": [ "ts\":[{\"url\":\"https://a.nel.cloudflare.com/report/v4?s=hzHkzlRVnccaTnOaz8pQoWwOavJ0s61tag5wJRHZWELSQZ" ] }, { "name": "stdout", "output_type": "stream", "text": [ "p9tCArXj3bqYNUxzoXia7XaTzkz7wwKspm1hFAWKPedAS3qzrGPonyVXxMOw91qVE8koht9YC6wig%2BNqBNJOS5k6h1S%2Bic7g" ] }, { "name": "stdout", "output_type": "stream", "text": [ "%3D%3D\"}]}\r\n", "nel: {\"report_to\":\"cf-nel\",\"success_fraction\":0.0,\"max_age\":604800}\r\n", "access-control-allo" ] }, { "name": "stdout", "output_type": "stream", "text": [ "w-origin: *\r\n", "cache-control: public, max-age=0, must-revalidate\r\n", "etag: \"21455a79c13bd9d712f7eb0840672" ] }, { "name": "stdout", "output_type": "stream", "text": [ "884\"\r\n", "referrer-policy: strict-origin-when-cross-origin\r\n", "server: cloudflare\r\n", "cf-cache-status: DYNAMIC" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r\n", "cf-ray: a4246df6d8bb16af-IAD\r\n", "alt-svc: h3=\":443\"; ma=86400\r\n", "\r\n" ] } ], "source": [ "%%bash\n", "curl -sS -D - -o /dev/null https://course.aiprototypes.org/resources/files/eng190bo-fa26-lecture-9-activity-skeleton.ipynb" ] }, { "cell_type": "markdown", "id": "l9-t07", "metadata": {}, "source": [ "`grep` keeps only the lines that match. `-i` ignores upper and lower case, and `^` means \"at the start of a line,\" so `'^server'` finds the `server:` header and nothing else." ] }, { "cell_type": "code", "execution_count": 5, "id": "l9-t08", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "server: cloudflare\r\n" ] } ], "source": [ "%%bash\n", "curl -sS -D - -o /dev/null https://course.aiprototypes.org/ | grep -i '^server'" ] }, { "cell_type": "markdown", "id": "l9-t09", "metadata": {}, "source": [ "### Status codes\n", "\n", "`-w '%{http_code}'` prints just the status code, as in the first line.\n", "\n", "To see the body **and** the code, leave out `-o /dev/null` and use `-w '\\n[%{http_code}]\\n'`, which puts the code on its own line after the body. The home page is long, so the second line keeps only its last two lines: the end of the HTML, then the code." ] }, { "cell_type": "code", "execution_count": 6, "id": "l9-t10", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "200\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "[200]\n" ] } ], "source": [ "%%bash\n", "curl -s -o /dev/null -w '%{http_code}\\n' https://course.aiprototypes.org/\n", "curl -s -w '\\n[%{http_code}]\\n' https://course.aiprototypes.org/ | tail -n 2" ] }, { "cell_type": "markdown", "id": "l9-t11", "metadata": {}, "source": [ "### `jq` reads JSON\n", "\n", "Most APIs answer in **JSON**, and `jq` is a small language for pulling pieces out of it. To practice, this cell writes a made-up library catalog to `books.json` (everything between `<<'EOF'` and `EOF` goes into the file). Then `jq .` pretty-prints it; `.` means \"the whole thing.\"" ] }, { "cell_type": "code", "execution_count": 7, "id": "l9-t12", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[\n", " {\n", " \"title\": \"A Wizard of Earthsea\",\n", " \"author\": {\n", " \"first\": \"Ursula\",\n", " \"last\": \"L" ] }, { "name": "stdout", "output_type": "stream", "text": [ "e Guin\"\n", " },\n", " \"pages\": 183,\n", " \"copies\": 3,\n", " \"checkedOut\": 3\n", " },\n", " {\n", " \"title\": \"The Dis" ] }, { "name": "stdout", "output_type": "stream", "text": [ "possessed\",\n", " \"author\": {\n", " \"first\": \"Ursula\",\n", " \"last\": \"Le Guin\"\n", " },\n", " \"pages\": 387" ] }, { "name": "stdout", "output_type": "stream", "text": [ ",\n", " \"copies\": 2,\n", " \"checkedOut\": 1\n", " },\n", " {\n", " \"title\": \"Kindred\",\n", " \"author\": {\n", " \"first" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\": \"Octavia\",\n", " \"last\": \"Butler\"\n", " },\n", " \"pages\": 264,\n", " \"copies\": 4,\n", " \"checkedOut\": 4\n", " " ] }, { "name": "stdout", "output_type": "stream", "text": [ " },\n", " {\n", " \"title\": \"Exhalation\",\n", " \"author\": {\n", " \"first\": \"Ted\",\n", " \"last\": \"Chiang\"\n", " " ] }, { "name": "stdout", "output_type": "stream", "text": [ "},\n", " \"pages\": 350,\n", " \"copies\": 2,\n", " \"checkedOut\": 0\n", " }\n", "]\n" ] } ], "source": [ "%%bash\n", "cat > books.json <<'EOF'\n", "[\n", " {\"title\": \"A Wizard of Earthsea\", \"author\": {\"first\": \"Ursula\", \"last\": \"Le Guin\"}, \"pages\": 183, \"copies\": 3, \"checkedOut\": 3},\n", " {\"title\": \"The Dispossessed\", \"author\": {\"first\": \"Ursula\", \"last\": \"Le Guin\"}, \"pages\": 387, \"copies\": 2, \"checkedOut\": 1},\n", " {\"title\": \"Kindred\", \"author\": {\"first\": \"Octavia\", \"last\": \"Butler\"}, \"pages\": 264, \"copies\": 4, \"checkedOut\": 4},\n", " {\"title\": \"Exhalation\", \"author\": {\"first\": \"Ted\", \"last\": \"Chiang\"}, \"pages\": 350, \"copies\": 2, \"checkedOut\": 0}\n", "]\n", "EOF\n", "jq . books.json" ] }, { "cell_type": "markdown", "id": "l9-t13", "metadata": {}, "source": [ "`length` counts the items in a list. `.[0]` is the first item (counting starts at 0). A dot path walks into it: `.[0].author.last` is the first book's author's last name. Separate paths with commas to print several at once." ] }, { "cell_type": "code", "execution_count": 8, "id": "l9-t14", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "4\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"title\": \"A Wizard of Earthsea\",\n", " \"author\": {\n", " \"first\": \"Ursula\",\n", " \"last\": \"Le Guin\"\n", " }," ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", " \"pages\": 183,\n", " \"copies\": 3,\n", " \"checkedOut\": 3\n", "}\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\"A Wizard of Earthsea\"\n", "\"Le Guin\"\n" ] } ], "source": [ "%%bash\n", "jq length books.json\n", "jq '.[0]' books.json\n", "jq '.[0].title, .[0].author.last' books.json" ] }, { "cell_type": "markdown", "id": "l9-t23", "metadata": {}, "source": [ "### Cheat sheet\n", "\n", "| Command | Does |\n", "|---|---|\n", "| `curl -s URL` | print the response body |\n", "| `curl -s URL \\| jq .` | …as pretty-printed JSON |\n", "| `curl -s URL > file.json` | save the body to a file instead of printing it |\n", "| `jq length file.json` | how many items are in the list? |\n", "| `jq '.[0].title' file.json` | one field of the first item |\n", "| `jq '.[] \\| select(.a == \"x\") \\| .title' file.json` | a field of every item that passes a test |\n", "| `jq '[.[] \\| select(TEST)] \\| length' file.json` | how many items pass a test |\n", "| `jq -c '.[] \\| {title, half: (.pages / 2)}' file.json` | pick and compute fields, one line per item |\n", "| `curl -sS -D - -o /dev/null URL` | headers only (the status is on the first line) |\n", "| `curl -s -o /dev/null -w '%{http_code}\\n' URL` | just the status code |\n", "| `curl -s -w '\\n[%{http_code}]\\n' URL` | the body, then the status code |\n", "| `curl -s -o /dev/null -w '%{size_download}\\n' URL` | how many bytes came back |\n", "| `… \\| grep -i '^word'` | keep only the lines that start with *word* |\n", "| `… \\| head -n 5` | keep only the first five lines |\n", "| `date -d '2026-09-30 - 7 days' +%F` | date arithmetic |\n", "| `for x in A B C; do …; done` | run a command once for each value |" ] }, { "cell_type": "markdown", "id": "l9-04", "metadata": {}, "source": [ "---\n", "# Part 1 — Hyperschedule\n", "\n", "Same API, same diagram as the video. Every question matches a layer you saw." ] }, { "cell_type": "markdown", "id": "l9-05", "metadata": {}, "source": [ "## The API server: talk to it\n", "\n", "### 1.1 What term is it?\n", "\n", "Ask the API for the current term. The route is `/v4/term/current`, so the URL is `$B/v4/term/current`. This is the first request from the video." ] }, { "cell_type": "code", "execution_count": 9, "id": "l9-06", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"year\": 2026,\n", " \"term\": \"FA\"\n", "}" ] } ], "source": [ "%%bash\n", "curl -s $B/v4/term/current" ] }, { "cell_type": "markdown", "id": "l9-07", "metadata": {}, "source": [ "Which term and year does Hyperschedule think it is? What format is the response in?\n", "\n", "**Answer:** `{\"year\": 2026, \"term\": \"FA\"}`, which is Fall 2026. It's JSON: an object with two fields." ] }, { "cell_type": "markdown", "id": "l9-08", "metadata": {}, "source": [ "### 1.2 How many sections?\n", "\n", "`$B/v4/sections` returns every section this term, as one list. Count them.\n", "\n", "Then **save** the whole list to a file named `sections.json`. It's about 2.5 MB, so from here on read the file with `jq` rather than downloading it over and over. That's faster for you and politer to the server." ] }, { "cell_type": "code", "execution_count": 10, "id": "l9-09", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2188\n" ] } ], "source": [ "%%bash\n", "curl -s $B/v4/sections | jq length\n", "curl -s $B/v4/sections > sections.json" ] }, { "cell_type": "markdown", "id": "l9-10", "metadata": {}, "source": [ "How many sections are offered this term?\n", "\n", "**Answer:** 2,188 on 9/25. The number drifts a little as the registrar adds and cancels sections." ] }, { "cell_type": "markdown", "id": "l9-11", "metadata": {}, "source": [ "### 1.3 One section up close\n", "\n", "Print the **first** section in `sections.json`, so you can see every field." ] }, { "cell_type": "code", "execution_count": 11, "id": "l9-12", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"courseAreas\": [\n", " \"1A3\",\n", " \"2FYA\",\n", " \"2RES\",\n", " \"4HSS\",\n", " \"5HUM\",\n", " \"5IUG\",\n", " \"5IUL\",\n", " \"AFRI\"\n", " ],\n", " \"credits\": 1,\n", " \"permCount\": 0,\n", " \"seatsTotal\": 25,\n", " \"seatsFilled\": 23,\n", " \"status\": \"R\",\n", " \"startDate\": {\n", " \"year\": 2026,\n", " \"month\": 8,\n", " \"day\": 31\n", " },\n", " \"endDate\": {\n", " \"year\": 2026,\n", " \"month\": 12,\n", " \"day\": 18\n", " },\n", " \"instructors\": [\n", " {\n", " \"name\": \"Maryan Soliman\"\n", " }\n", " ],\n", " \"course\": {\n", " \"code\": {\n", " \"department\": \"AFRI\",\n", " \"courseNumber\": 10,\n", " \"suffix\": \"A\",\n", " \"affiliation\": \"AF\"\n", " },\n", " \"title\": \"Intro to Africana Studies\",\n", " \"description\": \"This class will serve as a general introduction to Africana Studies. Africana studies, while still relatively young, has a vibrant history that traces the lives and scholarship of people from African descent. Its complex and latent development in academia follows from the socio-political marginalization of people within the African diaspora. Nevertheless, resilience and perseverance will be repeated themes as we study how, through different techniques and modes of understanding, people of the African diaspora have continually challenged the western hegemony of academic study.\\n\",\n", " \"primaryAssociation\": \"SC\",\n", " \"potentialError\": false\n", " },\n", " \"schedules\": [\n", " {\n", " \"startTime\": 47700,\n", " \"endTime\": 52200,\n", " \"days\": [\n", " \"T\",\n", " \"R\"\n", " ],\n", " \"locations\": [\n", " \"Humanities Building 105\"\n", " ]\n", " }\n", " ],\n", " \"potentialError\": false,\n", " \"identifier\": {\n", " \"department\": \"AFRI\",\n", " \"courseNumber\": 10,\n", " \"suffix\": \"A\",\n", " \"affiliation\": \"AF\",\n", " \"sectionNumber\": 1,\n", " \"year\": 2026,\n", " \"term\": \"FA\",\n", " \"half\": null\n", " }\n", "}\n" ] } ], "source": [ "%%bash\n", "jq '.[0]' sections.json" ] }, { "cell_type": "markdown", "id": "l9-13", "metadata": {}, "source": [ "Now print **only** three things about that section: its course title, how many seats are filled, and how many there are in total. Find the field names in the output above. The title is inside `course`." ] }, { "cell_type": "code", "execution_count": 12, "id": "l9-14", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\"Intro to Africana Studies\"\n", "23\n", "25\n" ] } ], "source": [ "%%bash\n", "jq '.[0].course.title, .[0].seatsFilled, .[0].seatsTotal' sections.json" ] }, { "cell_type": "markdown", "id": "l9-15", "metadata": {}, "source": [ "Title, seats filled, seats total:\n", "\n", "**Answer:** *Intro to Africana Studies*, 23 of 25 seats filled on 9/25. Seat counts change every minute, because the fetcher polls the registrar every 60 s." ] }, { "cell_type": "markdown", "id": "l9-16", "metadata": {}, "source": [ "### 1.4 Find this class\n", "\n", "How many sections are in the **ENGR** department? The department is at `.course.code.department`." ] }, { "cell_type": "code", "execution_count": 13, "id": "l9-17", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "55\n" ] } ], "source": [ "%%bash\n", "jq '[.[] | select(.course.code.department == \"ENGR\")] | length' sections.json" ] }, { "cell_type": "markdown", "id": "l9-18", "metadata": {}, "source": [ "Now find **ENGR 190BO**, this class, and print its title and seat counts.\n", "\n", "Hints: combine tests with `and`. The course number is `.course.code.courseNumber`, and it's stored as a number (`190`, not `\"190\"`). The `BO` is stored separately, in `.course.code.suffix`. Build a small object to print, as in the cheat sheet." ] }, { "cell_type": "code", "execution_count": 14, "id": "l9-19", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\"title\":\"AI-Amplified Prototyping\",\"filled\":17,\"total\":32}\n" ] } ], "source": [ "%%bash\n", "jq -c '.[] | select(.course.code.department == \"ENGR\" and .course.code.courseNumber == 190 and .course.code.suffix == \"BO\")\n", " | {title: .course.title, filled: .seatsFilled, total: .seatsTotal}' sections.json" ] }, { "cell_type": "markdown", "id": "l9-20", "metadata": {}, "source": [ "How many ENGR sections are there? Is there still room in ENGR 190BO?\n", "\n", "**Answer:** 55 ENGR sections on 9/28. ENGR 190BO is *AI-Amplified Prototyping*, 17 of 32 seats filled on 9/28, so yes, there's room. Test the department, the number, **and** the suffix: ENGR 190 alone matches three sections (`BI`, `BO`, and `BP`)." ] }, { "cell_type": "markdown", "id": "l9-27", "metadata": {}, "source": [ "## The network\n", "\n", "### 1.5 How much did the browser download?\n", "\n", "How many bytes does `$B/v4/sections` send back? The Toolbox shows two ways: `curl`'s `-w '%{size_download}'`, or `wc -c` on the file you saved." ] }, { "cell_type": "code", "execution_count": 15, "id": "l9-28", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2483499 bytes\n" ] } ], "source": [ "%%bash\n", "curl -s -o /dev/null -w '%{size_download} bytes\\n' $B/v4/sections" ] }, { "cell_type": "markdown", "id": "l9-29", "metadata": {}, "source": [ "How big is it? Hyperschedule's page downloads this whole list **once** when it opens. So when you type in the search box, which machine does the searching? When would this design be a bad idea?\n", "\n", "**Answer:** about 2.5 MB (2,483,498 bytes on 9/25). The search runs **in your browser**, over the copy it already downloaded, which is why it feels instant and makes no request per keystroke. It would be a bad choice on a slow phone connection, with far more data, or with data that some users aren't allowed to see." ] }, { "cell_type": "markdown", "id": "l9-30", "metadata": {}, "source": [ "### 1.6 Two hostnames, two companies\n", "\n", "Ask who serves `https://hyperschedule.io` (its `server:` header). Then ask the same about the API, `$B/v4/term/current`." ] }, { "cell_type": "code", "execution_count": 16, "id": "l9-31", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "server: GitHub.com\n", "server: cloudflare\n" ] } ], "source": [ "%%bash\n", "curl -sS -D - -o /dev/null https://hyperschedule.io | grep -i '^server'\n", "curl -sS -D - -o /dev/null $B/v4/term/current | grep -i '^server'" ] }, { "cell_type": "markdown", "id": "l9-32", "metadata": {}, "source": [ "Who serves each one? Which hostname delivers the app's *files* and which delivers its *data*?\n", "\n", "**Answer:** `hyperschedule.io` → `server: GitHub.com` (GitHub Pages serves the static HTML/JS/CSS). `banana.hyperschedule.io` → `server: cloudflare` (Cloudflare sits in front of the API server that serves the data)." ] }, { "cell_type": "markdown", "id": "l9-36", "metadata": {}, "source": [ "## Make it fail: four status codes\n", "\n", "Get Hyperschedule's API to return each of **200, 404, 401, and 400**. Here are its routes, from [the source](https://github.com/hyperschedule/hyperschedule/tree/d41a5e52e72398ba504ceddb5d179a9ff85fe601/backend/src/routes/v4):\n", "\n", "| Route | What it's for |\n", "|---|---|\n", "| `GET /v4/term/current` | the current term |\n", "| `GET /v4/sections/{term}` | every section in one term, e.g. `FA2026` |\n", "| `GET /v4/user/` | *your* account and saved schedules |\n", "| `GET /v4/calendar/{userId}/{scheduleId}` | a calendar feed for one saved schedule |\n", "\n", "### 1.7 Four codes\n", "\n", "Write one `curl` line per code. Use `-w '\\n[%{http_code}]\\n'` so you see the body **and** the code. The 200 is the easy one: it's your request from 1.1." ] }, { "cell_type": "code", "execution_count": 17, "id": "l9-37", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"year\": 2026,\n", " \"term\": \"FA\"\n", "}\n", "[200]\n", "\n", "[404]\n", "\n", "[401]\n", "{\n", " \"issues\": [\n", " {\n", " \"validation\": \"regex\",\n", " \"code\": \"invalid_string\",\n", " \"message\": \"Invalid\",\n", " \"path\": []\n", " }\n", " ],\n", " \"name\": \"ZodError\"\n", "}\n", "[400]\n" ] } ], "source": [ "%%bash\n", "curl -s -w '\\n[%{http_code}]\\n' $B/v4/term/current # 200\n", "curl -s -w '\\n[%{http_code}]\\n' $B/v4/sections/FA2099 # 404: a term in the future\n", "curl -s -w '\\n[%{http_code}]\\n' $B/v4/user/ # 401: no login cookie\n", "curl -s -w '\\n[%{http_code}]\\n' $B/v4/calendar/x/y # 400: not a valid user ID" ] }, { "cell_type": "markdown", "id": "l9-38", "metadata": {}, "source": [ "Which URL gave each code? Read the **400** body: what is it complaining about? Find the line that sent it in [`calendar.ts`](https://github.com/hyperschedule/hyperschedule/blob/d41a5e52e72398ba504ceddb5d179a9ff85fe601/backend/src/routes/v4/calendar.ts#L36). For the 401: what would your browser have sent that `curl` didn't?\n", "\n", "**Answer:**\n", "\n", "- **200**: `/v4/term/current`, or `/v4/sections/FA2026`.\n", "- **404**: `/v4/sections/FA2099`. A future term, or a term that doesn't parse, like `banana`, returns 404 (`courses.ts`).\n", "- **401**: `/v4/user/`. The route checks for a login token first, and `curl` sent none. A signed-in browser sends a **cookie** that it received after the college sign-in.\n", "- **400**: `/v4/calendar/x/y`. The body is a `ZodError` with `\"validation\": \"regex\"`: `x` doesn't match the pattern a user ID must have. It comes from the first `safeParse` check in `calendar.ts`, which returns 400 before the server ever looks in the database.\n", "\n", "Note the difference: 400 means *your request is malformed*, 404 means *well-formed, but no such thing*, and 401 means *who are you?*" ] }, { "cell_type": "markdown", "id": "l9-39", "metadata": {}, "source": [ "**✅ Checkpoint.** Compare your Part 1 answers with the key on the board. Then go on to Part 2." ] }, { "cell_type": "markdown", "id": "l9-40", "metadata": {}, "source": [ "---\n", "# Part 2 — Wordle\n", "\n", "**Your target:** \n", "\n", "This time, nobody gives you the URLs. You have the same tools as in Part 1, plus your browser's devtools. The questions tell you *what* to find, not *how*. Hints are folded up under each question; try without them first.\n", "\n", "Do **2.1–2.5**. If you finish early, there's more at the end.\n", "\n", "> **Spoiler warning.** Wordle's page asks for today's puzzle. In the Network tab, **don't click on today's puzzle request** unless you've already played. In this notebook, use **past dates only**." ] }, { "cell_type": "markdown", "id": "l9-41", "metadata": {}, "source": [ "### 2.1 Find the puzzle\n", "\n", "Open Wordle with devtools → **Network** open, then press **Play**. The page makes a lot of requests, and most of them are ads and analytics. How many requests did the page make in total? (The count is in the Network tab's bottom bar.) Find the **one** that delivers the puzzle.\n", "\n", "Then fetch a **past** day's puzzle from this notebook and pretty-print it.\n", "\n", "
Hint 1\n", "\n", "Set the Network filter to **Fetch/XHR**, which hides images, scripts, and styles. Then type wordle in the filter box.\n", "
\n", "\n", "
Hint 2\n", "\n", "Look for a .json URL with today's date in it. Keep everything up to the date, and swap the date for an old one, like 2026-09-20.json.\n", "
" ] }, { "cell_type": "code", "execution_count": 18, "id": "l9-42", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"id\": 1453,\n", " \"solution\": \"shone\",\n", " \"print_date\": \"2026-09-20\",\n", " \"days_since_launch\": 1919,\n", " \"editor\": \"Tracy Bennett\"\n", "}\n" ] } ], "source": [ "%%bash\n", "curl -s https://www.nytimes.com/svc/wordle/v2/2026-09-20.json | jq ." ] }, { "cell_type": "markdown", "id": "l9-43", "metadata": {}, "source": [ "What's the URL pattern? What fields does a puzzle have?\n", "\n", "**Answer:** `https://www.nytimes.com/svc/wordle/v2/YYYY-MM-DD.json`. Five fields: `id`, `solution`, `print_date`, `days_since_launch`, and `editor`. On 9/20/2026: `{\"id\": 1453, \"solution\": \"shone\", \"print_date\": \"2026-09-20\", \"days_since_launch\": 1919, \"editor\": \"Tracy Bennett\"}`. The page itself makes dozens of requests on load (about 60 in a headless browser on 9/25; fewer with an ad blocker), and only a handful of them are the game." ] }, { "cell_type": "markdown", "id": "l9-44", "metadata": {}, "source": [ "### 2.2 Who grades your guess?\n", "\n", "Clear the Network tab (🚫). Type a guess and press **Enter**. Watch the Network tab.\n", "\n", "How many requests did your guess cause? The tiles changed color anyway. So **which machine** decided they were green or yellow, and what did that machine need to know? Point to the evidence in your output from 2.1.\n", "\n", "This is the video's question: *which machine runs this code?*" ] }, { "cell_type": "markdown", "id": "l9-45", "metadata": {}, "source": [ "**Answer:** Zero requests. The tiles still turned green, yellow, and gray, so the grading ran **in the browser**. To grade you, the browser must already know the answer, and it does: the `solution` field is in the puzzle JSON it downloaded before you typed anything. Anyone can read it, with `curl` or in devtools." ] }, { "cell_type": "markdown", "id": "l9-46", "metadata": {}, "source": [ "### 2.3 Where's your game?\n", "\n", "Open devtools → **Application** → **Local Storage** → `https://www.nytimes.com`. Find the entry that holds your board. *(If you've played today, don't show your screen around.)*\n", "\n", "The page **also** asked the NYT's server for your game state. Find that request in the Network tab (filter for `state`), copy its URL, and `curl` it here with the status code.\n", "\n", "
Hint\n", "\n", "The URL contains svc/games/state. Use curl -s -w '\\n[%{http_code}]\\n' \"URL\", and keep the double quotes, because the URL has a ? in it.\n", "
" ] }, { "cell_type": "code", "execution_count": 19, "id": "l9-47", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\"error\":\"forbidden\"}\n", "[403]\n" ] } ], "source": [ "%%bash\n", "curl -s -w '\\n[%{http_code}]\\n' \"https://www.nytimes.com/svc/games/state/wordleV2/latests?puzzle_ids=1028\"" ] }, { "cell_type": "markdown", "id": "l9-48", "metadata": {}, "source": [ "What's in Local Storage? What did the state request return, and why? Which one survives switching laptops? Which part of Hyperschedule does this remind you of?\n", "\n", "**Answer:** Local Storage has `games-state-wordleV2/ANON`, which holds `boardState` (your guesses), `status`, `puzzleId`, and more. It lives only in this browser. The server's state endpoint returns **403 `{\"error\":\"forbidden\"}`** to `curl`: signed-in players' games are stored on the NYT's servers, and we're not signed in. This is exactly Hyperschedule: a **guest** schedule in Local Storage vs. a **signed-in** schedule in MongoDB behind `/v4/user/` (which gave you a 401). Only the server copy survives switching laptops.\n", "\n", "Side note: strictly, *\"we don't know who you are\"* is a **401**. NYT answers 403. Real APIs don't always use the codes by the book." ] }, { "cell_type": "markdown", "id": "l9-49", "metadata": {}, "source": [ "### 2.4 Make it fail — twice, differently\n", "\n", "Get **two 404s** from the puzzle URL whose **bodies look completely different**. What does the difference tell you about how many programs sit between you and the puzzle?\n", "\n", "
Hint\n", "\n", "Try a \"date\" that isn't a date at all. Then try a real date far in the future.\n", "
" ] }, { "cell_type": "code", "execution_count": 20, "id": "l9-50", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "404 page not found\n", "\n", "[404]\n", "{\"status\":\"ERROR\",\"errors\":[\"Not Found\"],\"results\":[]}\n", "[404]\n" ] } ], "source": [ "%%bash\n", "W=https://www.nytimes.com/svc/wordle/v2\n", "curl -s -w '\\n[%{http_code}]\\n' \"$W/not-a-date.json\"\n", "curl -s -w '\\n[%{http_code}]\\n' \"$W/2099-01-01.json\"" ] }, { "cell_type": "markdown", "id": "l9-51", "metadata": {}, "source": [ "**Answer:** `not-a-date.json` → a bare line of text, `404 page not found`. `2099-01-01.json` → structured JSON, `{\"status\":\"ERROR\",\"errors\":[\"Not Found\"],\"results\":[]}`. The first was turned away by a **router** that only checks whether the URL matches a pattern. That bare string is Go's standard 404, so the router never looked at the date. The second reached the **application**: it accepted the date, looked for that puzzle, and found nothing. Two different programs, told apart just by the shape of their errors. Same idea as Hyperschedule's 400 (malformed) vs. 404 (no such thing)." ] }, { "cell_type": "markdown", "id": "l9-58", "metadata": {}, "source": [ "### 2.5 Fill in the diagram\n", "\n", "Same boxes as the video, two apps. The Hyperschedule column is filled in from the video. Fill in Wordle's from what **you** found, and mark each box **[O]** observed (a command or devtools showed it) or **[G]** guessed (inferred)." ] }, { "cell_type": "markdown", "id": "l9-59", "metadata": {}, "source": [ "| Layer | Hyperschedule (from the video) | Wordle | [O] / [G] |\n", "|---|---|---|---|\n", "| The pixels | React components + CSS | the game page from `/games/wordle/index.html`, built by the `wordle.*.js` script | [O] |\n", "| Logic in the browser | search; guest schedule in Local Storage | **grading your guesses** (no request per guess); game state in Local Storage (`games-state-wordleV2/ANON`) | [O] |\n", "| The network | `hyperschedule.io` → GitHub Pages; `banana.hyperschedule.io` → Cloudflare | one hostname, `www.nytimes.com`, for page and data; `server: envoy` (a proxy); the page is cached (`x-nyt-edge-cache: HIT`) | [O] |\n", "| The API server | tinyhttp: `/v4/sections`, `/v4/user/` | `/svc/wordle/v2/{date}.json`; `/svc/games/state/…` for signed-in players (403 to us); a router in front (two kinds of 404) | [O] |\n", "| The database | MongoDB (we read the source) | something that stores puzzles by date and hands out non-sequential `id`s | **[G]** |\n", "| Below the bottom | the colleges' registrar, polled every 60 s | puzzle editors (`\"editor\": \"Tracy Bennett\"`) entering puzzles ahead of time | [G] |\n", "\n", "**Cheating:** Keep the answer on the server. Each guess becomes a `POST` to an API that grades it and returns only the colors. That changes the **logic in the browser** box (no more local grading), the **API server** box (a new grading endpoint, plus sessions so it knows your earlier guesses), and the **network** (one request per guess, which can't be cached). It costs servers, latency, and engineering time, to protect a free word game from people who are mostly just spoiling it for themselves. *Architecture is a series of decisions about what's worth paying for.*" ] }, { "cell_type": "markdown", "id": "l9-x0", "metadata": {}, "source": [ "---\n", "## If you finish early\n", "\n", "Four more, in any order. **A** and **B** go deeper into Wordle's data; **C** and **D** practice headers and loops." ] }, { "cell_type": "markdown", "id": "l9-52", "metadata": {}, "source": [ "### A. Is `id` the puzzle number?\n", "\n", "Fetch a few **consecutive** past days and print `print_date`, `id`, and `days_since_launch` for each one.\n", "\n", "
Hint\n", "\n", "jq -c '{print_date, id, days_since_launch}' prints one compact line per puzzle. Put three curl lines in the cell, or write a for loop.\n", "
" ] }, { "cell_type": "code", "execution_count": 21, "id": "l9-53", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\"print_date\":\"2026-09-14\",\"id\":2029,\"days_since_launch\":1913}\n", "{\"print_date\":\"2026-09-15\",\"id\":1966,\"days_since_launch\":1914}\n", "{\"print_date\":\"2026-09-16\",\"id\":1391,\"days_since_launch\":1915}\n", "{\"print_date\":\"2026-09-17\",\"id\":1130,\"days_since_launch\":1916}\n" ] } ], "source": [ "%%bash\n", "W=https://www.nytimes.com/svc/wordle/v2\n", "for d in 2026-09-14 2026-09-15 2026-09-16 2026-09-17; do\n", " curl -s \"$W/$d.json\" | jq -c '{print_date, id, days_since_launch}'\n", "done" ] }, { "cell_type": "markdown", "id": "l9-54", "metadata": {}, "source": [ "Which field counts puzzles? What do you think `id` is? Is that **observed** or **guessed**?\n", "\n", "**Answer:** `days_since_launch` goes up by one each day, so it's the puzzle number. `id` jumps around (2029, 1966, 1391, …). It's not in date order, which suggests a **database row ID** assigned when each puzzle was entered, and puzzles aren't entered in the order they're published. That's our best evidence of a database, and it's **guessed [G]**: we inferred it from a number. Nobody saw a database." ] }, { "cell_type": "markdown", "id": "l9-55", "metadata": {}, "source": [ "### B. Wordle #1\n", "\n", "Use `days_since_launch` to work out the date of the **first** Wordle, then fetch it. `date` can do date arithmetic: `date -d '2026-09-20 - 10 days' +%F`.\n", "\n", "Compare its fields with a recent puzzle's. What's missing, and what do you think happened?" ] }, { "cell_type": "code", "execution_count": 22, "id": "l9-56", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2021-06-19\n", "{\"id\":1,\"solution\":\"cigar\",\"print_date\":\"2021-06-19\"}\n", "[\"days_since_launch\",\"editor\",\"id\",\"print_date\",\"solution\"]\n" ] } ], "source": [ "%%bash\n", "date -d '2026-09-20 - 1919 days' +%F\n", "curl -s https://www.nytimes.com/svc/wordle/v2/2021-06-19.json | jq -c .\n", "curl -s https://www.nytimes.com/svc/wordle/v2/2026-09-20.json | jq -c 'keys'" ] }, { "cell_type": "markdown", "id": "l9-57", "metadata": {}, "source": [ "**Answer:** 2026-09-20 minus 1,919 days is **2021-06-19**: puzzle #1, `cigar`. It has only three fields, `id`, `solution`, and `print_date`, with no `days_since_launch` or `editor`. The NYT bought Wordle in early 2022 and added those fields afterward, and the old rows never got them. That's a **schema change** you can see from outside. You'll make one yourself in Lab 5." ] }, { "cell_type": "markdown", "id": "l9-60", "metadata": {}, "source": [ "### C. Who answers?\n", "\n", "Compare the headers of the puzzle JSON and the game page (`server`, `cache-control`, and anything that starts with `x-`). Is the page cached? Is the puzzle? Hyperschedule used two hostnames for files and data. Does Wordle?\n", "\n", "### D. A week in one command\n", "\n", "Write a `for` loop that prints the puzzle number and editor for each of the last seven days, **not including today**. `date -d \"-3 days\" +%F` gives the date three days ago." ] }, { "cell_type": "code", "execution_count": 23, "id": "l9-61", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cache-control: no-cache,max-age=0,s-maxage=300,stale-if-error=3600,stale-while-revalidate=60\n", "server: envoy\n", "x-nyt-edge-cache: HIT-HIT\n", "x-envoy-decorator-operation: games-puzzles.games.nyti.nyt.net:443/*\n", "cache-control: no-store, max-age=0\n", "server: envoy\n", "2026-09-14 #1913 Tracy Bennett\n", "2026-09-15 #1914 Tracy Bennett\n", "2026-09-16 #1915 Tracy Bennett\n", "2026-09-17 #1916 Tracy Bennett\n", "2026-09-18 #1917 Tracy Bennett\n", "2026-09-19 #1918 Tracy Bennett\n", "2026-09-20 #1919 Tracy Bennett\n" ] } ], "source": [ "%%bash\n", "curl -sS -D - -o /dev/null https://www.nytimes.com/games/wordle/index.html | grep -iE '^(server|cache-control|x-nyt-edge-cache)'\n", "curl -sS -D - -o /dev/null https://www.nytimes.com/svc/wordle/v2/2026-09-20.json | grep -iE '^(server|cache-control|x-envoy-decorator-operation)'\n", "\n", "# B, with fixed dates so this solution can't spoil a puzzle whenever it's re-run\n", "for d in 2026-09-14 2026-09-15 2026-09-16 2026-09-17 2026-09-18 2026-09-19 2026-09-20; do\n", " curl -s \"https://www.nytimes.com/svc/wordle/v2/$d.json\" | jq -r '\"\\(.print_date) #\\(.days_since_launch) \\(.editor)\"'\n", "done" ] }, { "cell_type": "markdown", "id": "l9-62", "metadata": {}, "source": [ "**Answer (C):** The page is `cache-control: …s-maxage=300,stale-if-error=3600…` with `x-nyt-edge-cache: HIT-HIT`, so a CDN serves it and would keep serving it for an hour during an outage (the same directive as Hyperschedule's). The puzzle JSON is `no-store`: never cached. Both come from `www.nytimes.com` behind `server: envoy`, a proxy. There's one hostname, but not one program: `x-envoy-decorator-operation: games-puzzles.games.nyti.nyt.net:443/*` names the internal service the proxy handed your request to. The split Hyperschedule made with two hostnames, the NYT makes behind one." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.25" } }, "nbformat": 4, "nbformat_minor": 5 }