1. Loading and Parsing HTML Documents
Quick Overview of Functionality
The requests library is like our "messenger" who sets off to get the
HTML code of web pages. It makes HTTP requests and delivers pages like a pizza delivery person—just minus the "Margherita" and boxes.
BeautifulSoup, on the other hand, is our "chef" who effortlessly breaks down the received HTML into ingredients (tags, attributes, and text), so we can use them. It helps us find the necessary elements and save all the important info.
Using the requests Library
Now we’re ready to make our first HTTP request and get the HTML code of a page. Let’s load the page
example.com for practice. This site is an internet dinosaur and is perfect for getting started.
import requests
url = 'http://example.com'
response = requests.get(url)
# Let’s check if everything is okay
if response.status_code == 200:
print("Page loaded successfully!")
else:
print("Something went wrong. Error code:", response.status_code)
This program will send a request to the URL and output success or an error based on the response. If everything’s fine, we’ll have the HTML code of the page as text.
Tracking Error Codes
When automating scraping, you’ll often face situations where a page that should load doesn’t. So analyzing error codes is a mandatory part of any project that scrapes more than a couple of pages.
The thing is, website owners aren’t too thrilled when people scrape their data. First, it loads the site (especially when thousands of pages are scraped simultaneously). Second, it’s their data, and they make money from it. There are tons of ways to counter scraping: CAPTCHA, CloudFlare, etc.
For businesses, it’s ideal when you can scrape all your competitors, but no one can scrape you. Kind of like a cold war situation.
Using BeautifulSoup to Parse HTML
Once we get the HTML code, we can dive into it using BeautifulSoup. It’s like opening a book and reading its contents:
from bs4 import BeautifulSoup
# Passing the HTML content to BeautifulSoup
soup = BeautifulSoup(response.text, 'html.parser')
# Let’s see what’s inside
print(soup.prettify())
The prettify() method formats the HTML code nicely for you to study. In the next lesson, we’ll start digging into this HTML like kids digging into a sandbox. And we’ll go home tired, dirty, but happy :)
3. Practice: Loading and Analyzing HTML
To solidify our understanding, let’s do a practical exercise. We’ll try extracting the title and description from example.com. To do this, we’ll use our HTML knowledge and what we’ve learned about BeautifulSoup.
Extracting Data
# Extracting the page title
title = soup.title.string
print("Page title:", title)
# Extracting the main heading (h1)
main_heading = soup.h1.string
print("Main heading:", main_heading)
# Extracting paragraph text content
paragraph = soup.find('p').text
print("First paragraph:", paragraph)
In this example, we use the title attribute,
h1, and the find() method to pull
the necessary bits of info from the page. We’re becoming cyber detectives, examining clues at the crime scene!
4. Common Mistakes
Inevitably, while working with web scraping, you’ll run into common mistakes, like incorrect handling of HTTP requests, improper data extraction, or HTML parsing errors. Developing resilient and reliable scripts requires patience and practice. For instance, always check the status code (response.status_code) to ensure your request was successful. Incorrect use of the find() and find_all() methods can cause errors if you don’t consider the structure of the HTML pages. Always analyze the HTML before starting the parsing process.
Web scraping has tons of practical applications: from gathering data for analysis to automatically monitoring product prices. These skills can be handy in job interviews, where you might be asked about examples of project code. In real practice, for example, marketers use scraping to monitor competitors’ prices, while developers use it for integration with external websites.
Your knowledge of web scraping will also be useful in processing information for news aggregators and analytical systems. You can automate routine tasks by writing scripts that collect data from various sources on their own. Let’s continue building our virtual app and feel like true web masters!
GO TO FULL VERSION