1. Opening Web Pages: First Steps
If you’ve ever dreamed of becoming the master of web pages, then you’re in the right lecture. Today we’ll be opening web pages using Selenium and learning how to interact with them. Imagine you’re conducting the browser like a maestro conducting an orchestra. The orchestra is the internet, and the sheet music is Python code. Let’s start our concert!
Before we can interact with a web page, we need to open it, right? It’s like entering a library before finding a book. In Selenium, opening web pages is done with a browser driver. Let’s assume for now that you already have Selenium and the corresponding browser driver installed (for example, ChromeDriver or GeckoDriver). If you’ve gone through the previous lecture, you know how to set it up. If not, go back and revisit it.
Let’s see how to start with simply opening a web page:
from selenium import webdriver
# Create an instance of the Chrome WebDriver
driver = webdriver.Chrome()
# Open a web page
driver.get('https://www.example.com')
# Print the page title in the console
print(driver.title)
# Close the browser
driver.quit()
What happens in the code:
- Import webdriver: This is the heart of Selenium, allowing you to control the browser.
- Create a driver: Here, we create a driver object for Chrome. It could be any browser for which you have the driver.
-
Open a page: The
get()method loads a web page by its URL. -
Get the title: Using
driver.title, we can print the title of the current page. -
Close the browser: The
quit()method ends the browser session.
And there you go, we’ve opened our first web page! This is the first step towards world domination... um, I mean, automation!
2. Navigation and Working with Tabs
Just opening a page isn’t enough. Often, we need to navigate between pages or even open multiple tabs. Imagine you’re a traveler who can instantly teleport between locations. With Selenium, this is totally doable.
Page Navigation
Moving from one page to another is as simple as opening:
# Navigate to another page
driver.get('https://www.example.org')
# Use the back() method to go back to the previous page
driver.back()
# Use the forward() method to move forward
driver.forward()
Working with Tabs
How about opening a new tab? It’s like taking your laptop with you to libraries in different cities!
# Open a new tab and switch to it
driver.execute_script("window.open('https://www.example.com', '_blank');")
driver.switch_to.window(driver.window_handles[1])
# Switch back to the original tab
driver.switch_to.window(driver.window_handles[0])
Explanation:
-
execute_script(): Used to run JavaScript code in the browser. Here, we open a new tab. -
switch_to.window(): This method lets you switch between tabs. We usewindow_handlesto access the list of all open tabs.
3. Interacting with Web Pages
Now that we know how to open and navigate pages, let’s move on to actual interaction — like a true programmer! It’s like learning to dance with every web page and doing it in style.
The Interaction Interface
Let’s go over an example of interacting with elements on a page. For instance, finding a text field and entering text:
# Find an element by its name
search_box = driver.find_element_by_name('q')
# Enter text into the found text field
search_box.send_keys('Selenium Python')
# Find the search button and click it
submit_button = driver.find_element_by_name('btnK')
submit_button.click()
Explanation:
-
find_element_by_name(): A method used to find an element by its name. Here we look for the search text field. -
send_keys(): Allows you to type text into a found element. -
click(): Simulates clicking on an element, like a button.
Additional Methods
For more complex interactions with elements, there are many methods available. Check out the Selenium documentation for a deeper dive. For now, we’ll cover a few basics.
-
find_element_by_id(): Finds an element by its ID. -
find_element_by_class_name(): Finds an element by its class. -
find_element_by_css_selector(): Uses CSS selectors to find elements.
Now you can open and explore pages like a true web traveler! Don’t be afraid to experiment and try new things. As the saying goes, "mistakes are ladders on the path to success," especially when you’re certain your code is reliable and tested!
Common Errors
Working with Selenium can feel a bit like walking through a minefield if you don’t know the common pitfalls. One of the most frequent mistakes is forgetting to end the browser session. This can lead to browser processes getting stuck in memory. Always use driver.quit() at the end of your script.
Another pain point for programmers in Selenium is changes to web page structures. If an element is found using a selector and the site developer changes the HTML structure, your script will suddenly stop working. It’s like having a door closed on you when you’re about to enter a room. It’s important to write flexible code and update it regularly to match changes on the site.
GO TO FULL VERSION