Building a Web Monitor in Python with Requests and BeautifulSoup

Posted by

  

Have you ever found yourself obsessively refreshing a web page waiting for a product to come back in stock, or waiting for exam results to be published?

Or have you ever needed to monitor a website to make sure it was up & running (to intervene promptly in case of issues)?

Refreshing manually every day (or every hour!) is a waste of time. Today, we'll look at how to build a Web Content Monitor in Python: a lightweight script that periodically checks a specific HTML tag on a website and alerts you as soon as it detects a change.

šŸ”— Do you like Techelopment? Check out the website for all the details!

šŸ› ️ Tools of the Trade

For this project, we will use two of the most popular and loved libraries in the Python ecosystem:

  1. requests: The de facto standard library for making HTTP requests. We will use it to download the HTML code of the target web page.
  2. beautifulsoup4 (BS4): A fantastic HTML/XML parser that allows us to navigate the page structure and extract the exact content of the tag we care about.

If you haven't installed them yet, open your terminal and run:

pip install requests beautifulsoup4

šŸ’” How Does the Algorithm Work?

Our monitor relies on a very simple continuous loop:

  1. Download: We make a GET request to the target URL.
  2. Parsing: We parse the HTML with BeautifulSoup to find the desired tag (e.g., <div id="stock-status">).
  3. Comparison: We check if the extracted value is different from the one stored during the previous check.
  4. Notification: If a change occurred, we display an alert in the terminal (or send an email/Telegram message).
  5. Wait: We put the script on "pause" with a countdown before performing the next check.

šŸ’» The Final Code

Here is the complete script. You can save it in a file named web_monitor.py.


import time
import sys
from datetime import datetime
import requests
from bs4 import BeautifulSoup

# ==========================================
# MONITOR CONFIGURATION
# ==========================================
URL_TARGET = "https://www.example.com"

TAG_NAME = "span"
TAG_ATTRS = {"id": "logo"}  # Ex: {"class": "in-stock"} or {"id": "status"}

CHECK_INTERVAL = 300  # Interval in seconds (e.g., 300 sec = 5 minutes)

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}


def get_target_content(url, tag_name, attrs):
    """
    Performs the HTTP request and extracts text from the specified tag.
    """
    try:
        response = requests.get(url, headers=HEADERS, timeout=10)
        response.raise_for_status()

        soup = BeautifulSoup(response.text, "html.parser")
        element = soup.find(tag_name, attrs=attrs)

        if element:
            return element.get_text(strip=True)
        else:
            print(f"\n⚠️ [{datetime.now().strftime('%H:%M:%S')}] Tag not found on the page.")
            return None

    except requests.RequestException as e:
        print(f"\n❌ [{datetime.now().strftime('%H:%M:%S')}] Error during HTTP request: {e}")
        return None


def countdown(seconds):
    """
    Displays a countdown timer in MM:SS format on the same terminal line.
    """
    for remaining in range(seconds, 0, -1):
        mins, secs = divmod(remaining, 60)
        timer = f"{mins:02d}:{secs:02d}"
        # \r overwrites the current line in the terminal
        sys.stdout.write(f"\r⏳ Waiting for a new check in {timer} ")
        sys.stdout.flush()
        time.sleep(1)

    # Clear line before next output
    sys.stdout.write("\r" + " " * 50 + "\r")
    sys.stdout.flush()


def start_monitoring():
    """
    Main monitoring loop.
    """
    print(f"šŸš€ Monitor started for: {URL_TARGET}")
    print(f"⏱️ Check interval: {CHECK_INTERVAL} seconds\n")

    # Initial check to save baseline state
    last_state = get_target_content(URL_TARGET, TAG_NAME, TAG_ATTRS)

    if last_state is None:
        print("Unable to start monitor: initial element not found or HTTP error occurred.")
        return

    print(f"šŸ“Œ Initial state recorded: '{last_state}'\n")

    try:
        while True:
            # Start interactive countdown
            countdown(CHECK_INTERVAL)

            current_time = datetime.now().strftime('%d/%m/%Y %H:%M:%S')
            current_state = get_target_content(URL_TARGET, TAG_NAME, TAG_ATTRS)

            if current_state is None:
                continue

            # Check if state has changed
            if current_state != last_state:
                print("=" * 50)
                print(f"šŸ”” NOTIFICATION! CHANGE DETECTED [{current_time}]")
                print(f"šŸ”“ Previous state: {last_state}")
                print(f"šŸŸ¢ New state:      {current_state}")
                print("=" * 50 + "\n")

                last_state = current_state
            else:
                print(f"✅ [{current_time}] No change. Current state: '{current_state}'")

    except KeyboardInterrupt:
        print("\nšŸ›‘ Monitor stopped by user.")


if __name__ == "__main__":
    start_monitoring()

šŸ” How to Customize It for Your Target Site

To adapt this script to the page you want to monitor, follow these 3 steps:

  1. Open the page in Chrome or Firefox, right-click on the text you want to monitor, and select Inspect.
  2. Examine the highlighted HTML in the developer panel:
    • Identify the tag type (e.g., div, span, p, h1).
    • Look for a unique attribute, such as an id or a class.
  3. Update the TAG_NAME and TAG_ATTRS variables in the Python code based on what you identified in the previous step: 
    TAG_NAME = "div"
TAG_ATTRS = {"class": "availability-badge"}

šŸŽÆ Next Steps and Improvements

This script provides a solid starting point. If you want to take it further, here are a few upgrade ideas:

  • Push / Telegram Notifications: Instead of printing only to the terminal, use the Telegram API (python-telegram-bot) to send messages directly to your smartphone.
  • Cloud Execution: Deploy the script to a VPS or platforms like PythonAnywhere to run it 24/7 without keeping your home PC powered on.
  • JavaScript Support: If the target site loads data dynamic via AJAX/React, requests might not see the final content. In that case, consider using Playwright or Selenium.



Follow me #techelopment

Official site: www.techelopment.it
facebook: Techelopment
instagram: @techelopment
X: techelopment
Bluesky: @techelopment
telegram: @techelopment_channel
whatsapp: Techelopment
youtube: @techelopment