1. Extracting Table Data
What do tables and onions have in common? Yep, layers!
Tables in HTML are like a layered cake made of <table>, <tr> (rows), <th> (header cells), and <td> (regular cells). Each element plays its role in the data representation, and to extract information, we need to access each layer step by step.
Practice, practice, and more practice!
Let’s start with this simple HTML table:
<table>
<tr>
<th>Name</th>
<th>Age</th>
<th>City</th>
</tr>
<tr>
<td>Alice</td>
<td>29</td>
<td>Moscow</td>
</tr>
<tr>
<td>Bob</td>
<td>34</td>
<td>Saint Petersburg</td>
</tr>
</table>
Step 1: Finding the Table on the Page
Tables on web pages are defined by HTML tags <table>, and the data inside is organized into <tr> (rows) and <td> (data cells). In BeautifulSoup, the find method helps to locate the first table on the page, while find_all can fetch every table if there are multiple ones.
# Find the first table on the page
table = soup.find("table")
If you need to extract a specific table, you can refine the search by providing the table’s attributes, like id or class.
# Find table by class
table = soup.find("table", {"class": "table-class"})
Step 2: Extracting Data from the Table
Once loaded, you can begin extracting the data. Since table data is organized row by row, it makes sense to iterate through the rows and grab data from each cell.
Extracting Headers
Table headers are usually in the first row and wrapped in <th> tags. Using find_all you can gather all headers into a list.
# Extract headers from the first table row
headers = []
header_row = table.find("tr")
for th in header_row.find_all("th"):
headers.append(th.text.strip())
print("Table headers:", headers)
Extracting Data Rows
To extract all data rows (usually wrapped in <tr>, and each cell is in <td>), use a nested loop: first find all <tr> rows and then iterate over <td> cells in each row.
# Extract all data rows
data = []
rows = table.find_all("tr")[1:] # Skip the first header row
for row in rows:
row_data = []
for cell in row.find_all("td"):
row_data.append(cell.text.strip())
data.append(row_data)
print("Table data:", data)
This script cleanly loops through table rows and extracts the text from each cell. Isn’t it awesome? We all know that there’s no magic without the magic of loops!
2. Working with Lists
Lists: Tables’ Big Brothers
In life, there are only two infinite objects: tables and lists. Lists are represented by <ul> (unordered) and <ol> (ordered), with their items wrapped in <li>. Unlike tables, lists are simple and minimalistic—perfect candidates for quick and efficient data processing!
Extracting Data from Lists
Let’s check out this example of an HTML list:
<ul>
<li>Apple</li>
<li>Banana</li>
<li>Grapes</li>
</ul>
Now, let’s use our trusty BeautifulSoup to pull that data:
html = """
<ul>
<li>Apple</li>
<li>Banana</li>
<li>Grapes</li>
</ul>
"""
soup = BeautifulSoup(html, 'html.parser')
ul = soup.find('ul')
items = ul.find_all('li')
for item in items:
print(item.get_text())
That’s it! A simple yet effective approach you can use for more complex structures.
3. Example: Extracting and Processing Table Data
To put it all together, let’s try pulling data from a similar example with a more complex structure:
<table id="courses">
<tr>
<th>Course</th>
<th>Instructor</th>
</tr>
<tr>
<td>Python for Everyone</td>
<td>Guido van Rossum</td>
</tr>
<tr>
<td>Automation with Python</td>
<td>Eric Matthes</td>
</tr>
</table>
<ul class="technologies">
<li>Python</li>
<li>JavaScript</li>
<li>HTML & CSS</li>
</ul>
To extract data from both table and list, you can set up the following queries:
html = """
<table id="courses">
<tr>
<th>Course</th>
<th>Instructor</th>
</tr>
<tr>
<td>Python for Everyone</td>
<td>Guido van Rossum</td>
</tr>
<tr>
<td>Automation with Python</td>
<td>Eric Matthes</td>
</tr>
</table>
<ul class="technologies">
<li>Python</li>
<li>JavaScript</li>
<li>HTML & CSS</li>
</ul>
"""
soup = BeautifulSoup(html, 'html.parser')
# Extract data from the table
course_table = soup.find('table', id='courses')
course_rows = course_table.find_all('tr')
for row in course_rows:
cells = row.find_all(['th', 'td'])
for cell in cells:
print(cell.get_text())
print("---")
# Extract data from the list
tech_list = soup.find('ul', class_='technologies')
tech_items = tech_list.find_all('li')
for item in tech_items:
print(item.get_text())
This script handles both types of data. Notice how we use IDs and classes to precisely locate the elements. In practice, you may encounter more complex HTML structures, but the method remains the same: start at the top level, break down the elements, and finally extract the precious data!
4. Partial Conclusions and Common Errors
Working with tables and lists on web pages is like navigating a maze. Sometimes, you may run into imperfect data, like empty cells or missing elements. In such cases, it’s important to validate the data. Common mistakes include trying to access non-existent elements or using incorrect selectors. Remember, HTML may not always be clean and well-structured, so always consider exception handling and data validation.
Where Can This Be Useful?
The skills you’ve learned in this lecture will help you automate and process data from various online sources. For example, you can automate data collection from exchange rate tables, monitor prices in online stores, and even regularly analyze data from blogs and news websites. Mastering the art of extracting data from tables and lists unlocks a world of possibilities for automation and data analysis.
GO TO FULL VERSION