1. Error Handling in Scripts
Web Scraping Problems
Imagine this: your script is sitting there, looking good, ready to work, but suddenly slips on a banana peel — ouch! Errors and crashes. How do you make sure it survives the harsh conditions of the Internet? Today, we’ll teach it two essential skills: patience and retrying. Yep, we’re setting up retries and timeouts.
Working with web scraping can be great until you find out your script suddenly stops working because of:
- Connection issues.
- Temporary server unavailability.
- Unpredictable changes in HTML structure.
Your script, like a Jedi, needs to be ready for surprises and know how to handle them. Sometimes, just waiting a minute and trying again solves the problem. That’s where our heroes come in — retries and timeouts!
Intro to Error Handling Mechanisms
First, let’s revisit the basics — error handling in Python. We use try-except blocks to manage errors and keep them from ruining our script.
import requests
try:
response = requests.get('https://example.com')
response.raise_for_status() # check request success
except requests.exceptions.RequestException as e:
print(f'An error occurred: {e}')
2. Setting Up Retries
Why Use Retries?
A script that gives up at the first hiccup is like a cat that’s afraid of rain. What you need is a script that stands strong through a few discomforts. That’s why we set up retries — it makes your script more confident.
How to Set Up Retries
Let’s figure out how we can organize retries. One of the simplest ways is by using the urllib3 library, which provides functionality for automatically retrying requests on errors.
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
import requests
session = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[500, 502, 503, 504])
session.mount('https://', HTTPAdapter(max_retries=retries))
try:
response = session.get('https://example.com')
response.raise_for_status()
print(response.content)
except requests.exceptions.RequestException as e:
print(f'An error occurred: {e}')
In this example, we created a session and applied the retry mechanism (Retry) to it. We said it should try up to 5 times for errors with codes 500, 502, 503, and 504. backoff_factor=1 means the time between retries will increase exponentially (1, 2, 4, 8... seconds).
3. Timeouts: Preventing Hangs
Let’s Skip the Endless Waiting
Timeouts are like alarms: they prevent your script from hanging while waiting for a server response. By setting a timeout, you’re telling your script, "Stop waiting! If the server doesn’t respond in time, move on!"
try:
response = requests.get('https://example.com', timeout=10)
response.raise_for_status()
print(response.content)
except requests.exceptions.Timeout:
print('Request timed out')
except requests.exceptions.RequestException as e:
print(f'An error occurred: {e}')
Why Bother?
Ever sat there waiting for your script to get a response from a server that’s as good as toast? Timeouts save you from unnecessary waiting and allow your code to recover quickly and move on. Don’t make your script feel like it could have gone out for a few smokes while waiting!
4. Examples of Setup
Building a Resilient Script with Retries
Now let’s put together our script, resilient like Iron Man’s armor. We’ll use both timeouts and retries.
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
import requests
def fetch_url(url):
session = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[500, 502, 503, 504])
session.mount('https://', HTTPAdapter(max_retries=retries))
try:
response = session.get(url, timeout=10)
response.raise_for_status()
return response.content
except requests.exceptions.Timeout:
print('Request timed out')
except requests.exceptions.RequestException as e:
print(f'An error occurred: {e}')
return None
content = fetch_url('https://example.com')
if content:
print('Successfully downloaded data!')
Using Timeouts to Prevent Script Hanging
We’ve already shown how to set timeouts. Let’s now make sure our script not only stands strong but also reacts smartly to long waits. Instead of "hanging," it just reminds its owner: "Hey, the server’s taking too long, I’m not sticking around!"
This simple and concise approach makes your code reliable and ready to tackle all the unpredictable situations the Internet can throw at it.
Practical Application
When you’re working on real-world scraping projects, you often face various server-side limitations. Retries and timeouts are your best buddies for minimizing failure risks. They help ensure smooth operation of your code, especially in automated data processing and when accurate results are a priority. These techniques can also boost your resume’s quality and the trust of clients relying on your data.
GO TO FULL VERSION