Skip to content

Scraping part 3: dynamic pages and vibe coding

By Erik Hekman

Objectives

In this module, you will learn:

  • Why requests + BeautifulSoup sometimes returns "empty" pages
  • How to tell whether a page is static or dynamic
  • The options for dynamic pages, from best to last resort
  • How to use an AI assistant to write scrapers — and where it goes wrong

This part is mostly show-and-tell and discussion. The aim is that you know when your familiar tools stop working and what to reach for next — not that you install a browser automation stack under time pressure.

The core problem

requests downloads exactly the HTML the server sends — and nothing more. It does not run JavaScript. Many modern sites send a nearly-empty HTML shell and then build the real content in your browser with JavaScript: infinite scroll, "Load more" buttons, content that appears only after a click.

Take the PBS shows grid at pbs.org/shows/?genre=culture. It is full of shows in your browser, but scraping it with requests returns zero:

r = requests.get('https://www.pbs.org/shows/?genre=culture')
soup = BeautifulSoup(r.content, 'html.parser')
print(len(soup.select('a[href^="/show/"]')))   # -> 0  (!)

The whole grid is built by JavaScript after the page loads, and even more tiles appear only as you scroll. requests never sees any of it.

dynamic page
A page whose visible content is generated by JavaScript after the initial HTML loads. What you see in the browser is not what requests receives.

Is this page dynamic?

A 30-second check you can do live in the browser:

  1. Right-click → View Page Source. This is what requests sees.
  2. Press Ctrl/Cmd+F and search for a piece of text you can see on screen.
  3. If the text is not in the source, JavaScript added it later → dynamic.
  4. Compare with Inspect (the live DOM), where the text is present.

The options, best to last resort

1. Look for a hidden API (do this first)

Open DevTools → Network → Fetch/XHR, reload the page, and watch. Very often the page fetches its data from a clean JSON endpoint that you can call directly with requests. This is the fastest, most stable, and kindest approach — no browser needed.

2. Look for embedded structured data

Many pages ship their data inside a <script> tag as JSON — remember the schema.org/Recipe JSON-LD from part 1. If the data you want is already in the page source that way, you can grab it with requests and never touch a browser. Always view-source and look before reaching for Selenium.

3. Drive a real browser (Selenium / Playwright)

The last resort: automate an actual browser so JavaScript runs, then read the rendered page. Powerful, but slower, more fragile, and heavier to install and maintain.

The Selenium library

Selenium automates a real web browser from Python, so JavaScript runs and you can scroll and click just like a person. We also add webdriver-manager, which downloads the matching browser driver for you.

Installing

Install both into your virtual environment:

pip install selenium webdriver-manager

Selenium drives a real browser (Chrome in these examples), so you need Chrome installed on your machine — webdriver-manager takes care of the driver.

Forgetting something?

Update your requirements file with the new libraries!

Selenium demo: scrolling the PBS shows grid

The key insight: Selenium only replaces the "get the HTML" step. Once the page is rendered, you hand it to the same BeautifulSoup you already know — parsing does not change.

First, start a browser:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))

The PBS grid loads more tiles as you scroll, so we scroll to the bottom over and over until the page stops growing, then parse the result:

import time
from bs4 import BeautifulSoup

driver.get('https://www.pbs.org/shows/?genre=culture')
time.sleep(3)

last = driver.execute_script('return document.body.scrollHeight')
while True:
    driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')
    time.sleep(3)                                    # let JS load more tiles
    new = driver.execute_script('return document.body.scrollHeight')
    if new == last:                                  # nothing new -> done
        break
    last = new

soup = BeautifulSoup(driver.page_source, 'html.parser')
print(len(soup.select('a[href^="/show/"]')))         # now well over 0!

Clicking through a multi-step flow

Before that grid, PBS asks you to pick a local station — a few clicks we can also automate. The buttons have auto-generated class names like Button-module-scss-module__gW1A4G__white, which are not unique and change on every rebuild. So we target stable things a human can see too: the visible text and the accessibility aria-label.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select

# pick a state from the dropdown, then search
Select(driver.find_element(By.ID, 'state-select')).select_by_value('NY')
driver.find_element(By.XPATH, "//button[normalize-space()='Search State']").click()

# choose a station by its aria-label, then confirm
driver.find_element(By.XPATH, "//button[@aria-label='Select station Mountain Lake PBS']").click()
driver.find_element(By.XPATH, "//button[contains(normalize-space(), 'Confirm')]").click()
Select by what a human sees, not by hashed classes

Visible text (normalize-space()) and accessibility labels (aria-label) are stable and usually unique. Auto-generated CSS classes are neither — they are the first thing to break.

The full, commented version of this whole PBS walkthrough — station picker, infinite scroll, and saving each show — is in the workshop file part3_demo_selenium_pbs.py.

Exercise: click a button on a BBC Good Food recipe

Now try the other core move — clicking — yourself. Open a recipe you know from parts 1 and 2, e.g. Next level chilli con carne. It shows its Ingredients by default; the nutrition table sits behind a Nutrition tab. Your job: drive the browser to click that tab and then read the nutrition list that appears.

Your steps:

  1. Reuse the driver from the demo above (or start a fresh one) and driver.get(...) the recipe page.
  2. Find the Nutrition tab button. A robust way to locate a button by its text is an XPath: 
    from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)
button = wait.until(EC.element_to_be_clickable(
    (By.XPATH, "//button[normalize-space()='Nutrition']")))
  3. Scroll it into view and click it: 
    driver.execute_script('arguments[0].scrollIntoView({block: "center"});', button)
button.click()
  4. Hand the rendered page to BeautifulSoup and print the nutrition items (ul.nutrition-list li.nutrition-list__item) — the same parsing you already know.
You will have used both moves

scrollTo / scrollIntoView to reach content, and .click() to trigger it. Those two, plus waiting for elements to appear, cover the large majority of everyday dynamic-page scraping.

A cookie banner may be in the way

If a consent banner covers the page, click its Accept button first — the same technique, a different button: 

wait.until(EC.element_to_be_clickable(
    (By.XPATH, "//button[contains(., 'Accept')]"))).click()

Wrap-up

You can now: send a request, parse HTML, paginate, build a structured dataset, recognise a dynamic page, and choose the right tool for it — including an AI assistant you know how to double-check. Take these skills into your own project this afternoon.