I. Problem description

Some special web pages need to simulate clicking and sometimes dragging the scroll bar to display the complete content. For example, the web page of People’s Daily client is shown in the picture below:

You need to click to read the full text to display the complete content, and you need to drag the scroll bar to the image to display the content of the web page. If you need to obtain the content of this web page, you can use Python + Selenium to simulate the behavior of the browser to obtain the content.

Ii. Solutions

The installation and use of Selenium can be found in this article: juejin.cn/post/699697…

1. Use the JS script to drag the scroll bar to the bottom

# document. Body. ScrollHeight is used to obtain the height of the scroll bar
js = 'window.scrollTo(0, document.body.scrollHeight)'
driver.execute_script(js)
Copy the code

or

js = "var q=document.documentElement.scrollTop=10000"
driver.execute_script(js)
Copy the code

2. Use the js script to drag the scroll bar to the specified position

target = driver.find_element_by_class_name("class_name")
driver.execute_script("arguments[0].scrollIntoView();", target) Drag to the visible element
Copy the code

This method can be used to drag the scroll bar to the location of the element that you want to display.

For example, if you want to drag the scroll bar to the last image display, you can use the following method:

time.sleep(1)
Get the last image using xpath
target = driver.find_element_by_xpath('(//img)[last()]')
driver.execute_script("arguments[0].scrollIntoView();", target)  Drag to the visible element
Copy the code

For image sites, you can do this by scrolling down to load images.

3. Slowly drag the scroll bar using the JS script

Some web images require dragging the scroll bar to the image position to display the image content, you can use the following method to slowly drag the scroll bar:

js = "return document.body.scrollHeight"
Get the height of the scroll bar
new_height = driver.execute_script(js)
for i in range(0, new_height, 350):
    time.sleep(0.05)
    driver.execute_script('window.scrollTo(0, %s)' % i)
Copy the code

This is done by getting the height of the scroll bar and then dragging it 350 intervals at a time through range(start, stop, step).

4. How does Selenium determine whether an element exists

The first way: catch exceptions

def isElementExist(driver, class_name) :
    try:
        el = driver.find_element_by_class_name(class_name)
        return True
    except Exception as e:
        logging.error(e)
        return False
Copy the code

The second method: find_elements method

def is_element_exist(driver, class_name) :
    el = driver.find_elements_by_class_name(class_name)
    if len(el) > 0:
        return True
    else:
        return False
Copy the code

Reference Documents:

zhuanlan.zhihu.com/p/343516637

www.cnblogs.com/landhu/p/57…