The basic use of Selenium in reptile series (xii)

Introduction to Selenium

With the development of network technology, most websites use dynamic loading technology, such as JavaScript dynamic rendering and Ajax dynamic loading

For crawling these sites, there are two general ideas:

• Analyzing Ajax requests and simulating them to get real data is a method that has been used many times in previous articles and won’t be covered here
• Using Selenium mock browser for dynamic rendering to capture the real data returned by the site is described in detail below

What exactly is Selenium? Selenium is simply a testing tool for Web applications

According to the official documentation, one of the biggest advantages of Selenium is that it can run directly in the browser, simulating real user behavior

But this is also its biggest drawback, as it is slow to run due to the need to simulate the real rendering process

Please refer to the official documentation for other details

Two, selenium use

0. Preparation

• To install selenium

pip install selenium
Copy the code

• Install the driver

When using Selenium, you must have the corresponding browser drive file in the Python installation directory, otherwise an exception will occur

Chrome driver download website is as follows: sites.google.com/a/chromium….

Because the above official website needs to climb over the wall to access, so the blogger also here simply to tell you about the method of installing drivers, the specific steps are as follows:

1. Open Chrome and enter Chrome :// Settings /help in the address box to view the version information of Chrome

   For example 70.0.3538.67

2. After remove the last part of the above information is attached to the chromedriver.storage.googleapis.com/LATEST_RELE…

   For example chromedriver.storage.googleapis.com/LATEST_RELE…

3. Visit the link above for the corresponding drive version information

   For example 70.0.3538.97

4. The above information is attached to the chromedriver.storage.googleapis.com/index.html?… “With a slash at the end

   For example chromedriver.storage.googleapis.com/index.html?…

5. Visit the above link and select the appropriate platform (Linux, MAC, and Win) to download the package

6. After the download is complete, decompress the decompressed file to the Python installation directory

1. Import modules

>>> from selenium import webdriver
Copy the code

2. Open the browser

>>> browser = webdriver.Chrome()
>>> type(browser)
# <class 'selenium.webdriver.chrome.webdriver.WebDriver'>
Copy the code

3. Visit the page

>>> browser.get('https://www.baidu.com')
>>> print(browser.current_url) The # current_URL property gets the URL of the current web page
# https://www.baidu.com/
>>> print(browser.page_source) The # page_source property gets the source code for the current web page
Copy the code

4. Find elements

Method one:

Here are several ways to try to match the input box:

>>> search_bar = browser.find_element_by_id('kw')
>>> search_bar = browser.find_element_by_css_selector('#kw')
>>> search_bar = browser.find_element_by_xpath('//*[@id="kw"]')
>>> type(search_bar)
# <class 'selenium.webdriver.remote.webelement.WebElement'>
Copy the code

Method 2:

>>> from selenium.webdriver.common.by import By
>>> element = browser.find_element(by,value)
Copy the code

• The value argument is the matching expression corresponding to the matching method
• The by argument specifies the matching method, whose optional values are listed below (similar to method 1).

Here’s an attempt to match the confirm button using several methods:

>>> from selenium.webdriver.common.by import By
>>> button = browser.find_element(By.ID,'su')
>>> button = browser.find_element(By.CSS_SELECTOR,'#su')
>>> button = browser.find_element(By.XPATH,'//*[@id="su"]')
>>> type(button)
# <class 'selenium.webdriver.remote.webelement.WebElement'>
Copy the code

Note:

For both methods, the WebElement object is returned on success, and NoSuchElementException is thrown on failure

When you need to find more than one element, you simply change element in the method to elements, and the list of matches is returned

5. Element interaction

>>> button.get_attribute('type')
# 'submit'
Copy the code

>>> search_bar.send_keys('Selenium') Enter content into the input box
>>> search_bar.clear() # Empty the input field
>>> search_bar.send_keys('Selenium')
>>> from selenium.webdriver.common.keys import Keys
>>> search_bar.send_keys(Keys.ENTER) Type ENTER into the input box
Copy the code

>>> button.click() # Click submit button, equivalent to search_bar.send_keys(keys.enter) above
Copy the code

6. Perform interactive actions

The common methods for attaching an action to an action chain are listed as follows:

The following example shows the scroll to next button and click the next button to turn the page

>>> from selenium.webdriver.common.action_chains import ActionChains
>>> target = browser.find_element_by_class_name('n')
>>> ActionChains(browser).move_to_element(target).click(target).perform()
Copy the code

7. Execute JavaScript

>>> js = "window.scrollTo(0,document.body.scrollHeight)"
>>> browser.execute_script(js)
Copy the code

8, wait

If the specified element is not found within the specified time, an exception will be thrown.

>>> from selenium.webdriver.support.wait import WebDriverWait
>>> from selenium.webdriver.support import expected_conditions as EC
>>> wait = WebDriverWait(browser,10)
>>> try:
	element = wait.until(EC.presence_of_element_located((By.CLASS_NAME,'n')))
except:
    browser.quit()
Copy the code

The other EXPECted_conditions methods are listed as follows:

9. Close the browser

>>> browser.quit()
Copy the code