Python3 Selenium3 Learning Selenium from the framework implementation code allows you to do more with less

Selenium is an overall project that covers a series of tools and libraries that support the automation of Web browsers. And when automation is performed, it behaves as if it were a real user.

Selenium has three versions, namely Selenium 1.0, Selenium2.0, Selenium3.0.

Selenium 1.0 mainly calls JS injection into the browser; Initially Selenium author Jason Huggins developed javascriptStrunner as a testing tool and showed it to several colleagues (the author is also an interesting soul). As the name suggests, this test tool is based on JavaScript. This tool is also the “precursor” of Selenium.

Selenium 2.0 is based on the apis provided by WebDriver to perform browser element manipulation. WebDriver is a testing framework and an integrated API library.

Selenium 3.0 is based on Selenium 2.0 and has little difference. This article will be described in Selenium 3.0 version.

“Selenium supports all the major browsers on the market, such as Chrom (ium), Firefox, Internet Explorer, Opera and Safari, through WebDriver,” the official introduction explains about the supported browsers.

Webdriver. py (from Selenium import webDriver)

import warnings

from selenium.webdriver.remote.webdriver import WebDriver as RemoteWebDriver
from .remote_connection import ChromeRemoteConnection
from .service import Service
from .options import Options


class WebDriver(RemoteWebDriver) :
    """ Controls the ChromeDriver and allows you to drive the browser. You will need to download the ChromeDriver executable  from http://chromedriver.storage.googleapis.com/index.html """

    def __init__(self, executable_path="chromedriver", port=0,
                 options=None, service_args=None,
                 desired_capabilities=None, service_log_path=None,
                 chrome_options=None, keep_alive=True) :
        """ Creates a new instance of the chrome driver. Starts the service and then creates new instance of chrome driver. :Args: - executable_path - path to the executable. If the default is used it assumes the executable is in the $PATH - port - port you would like the service to run, if left as 0, a free port will be found. - options - this takes an instance of ChromeOptions - service_args - List of args to pass to the driver service - desired_capabilities - Dictionary object with non-browser specific capabilities only, such as "proxy" or "loggingPref". - service_log_path - Where to log information from the driver. - chrome_options - Deprecated argument for options - keep_alive - Whether to configure ChromeRemoteConnection to use HTTP keep-alive. """
        if chrome_options:
            warnings.warn('use options instead of chrome_options',
                          DeprecationWarning, stacklevel=2)
            options = chrome_options

        if options is None:
            # desired_capabilities stays as passed in
            if desired_capabilities is None:
                desired_capabilities = self.create_options().to_capabilities()
        else:
            if desired_capabilities is None:
                desired_capabilities = options.to_capabilities()
            else:
                desired_capabilities.update(options.to_capabilities())

        self.service = Service(
            executable_path,
            port=port,
            service_args=service_args,
            log_path=service_log_path)
        self.service.start()

        try:
            RemoteWebDriver.__init__(
                self,
                command_executor=ChromeRemoteConnection(
                    remote_server_addr=self.service.service_url,
                    keep_alive=keep_alive),
                desired_capabilities=desired_capabilities)
        except Exception:
            self.quit()
            raise
        self._is_remote = False

    def launch_app(self, id) :
        """Launches Chrome app specified by id."""
        return self.execute("launchApp", {'id': id})

    def get_network_conditions(self) :
        return self.execute("getNetworkConditions") ['value']

    def set_network_conditions(self, **network_conditions) :
        self.execute("setNetworkConditions", {
            'network_conditions': network_conditions
        })

    def execute_cdp_cmd(self, cmd, cmd_args) :
        return self.execute("executeCdpCommand", {'cmd': cmd, 'params': cmd_args})['value']

    def quit(self) :
        try:
            RemoteWebDriver.quit(self)
        except Exception:
            # We don't care about the message because something probably has gone wrong
            pass
        finally:
            self.service.stop()

    def create_options(self) :
        return Options()
Copy the code

The comments indicate that this is “create a new instance of the Chrome driver, and create an instance of the Chrome driver.”

Only the parameters used in this article are listed here:

• Executable_path: path of executables. If default values are used, the executable is assumed to be in PATH; PATH is the root directory of the system environment

In Selenium automation, a necessary step is to start the service. Look at the init initialization method and you found the following code:

self.service = Service(
            executable_path,
            port=port,
            service_args=service_args,
            log_path=service_log_path)
self.service.start()
Copy the code

The above code instantiates the Service class and passes in parameters to start the Service. The primary parameter here is executable_path, which is the boot driver. Check out the Service class (Selenium. Service) :

from selenium.webdriver.common import service


class Service(service.Service) :
    """ Object that manages the starting and stopping of the ChromeDriver """

    def __init__(self, executable_path, port=0, service_args=None,
                 log_path=None, env=None) :
        """ Creates a new instance of the Service :Args: - executable_path : Path to the ChromeDriver - port : Port the service is running on - service_args : List of args to pass to the chromedriver service - log_path : Path for the chromedriver service to log to"""

        self.service_args = service_args or []
        if log_path:
            self.service_args.append('--log-path=%s' % log_path)

        service.Service.__init__(self, executable_path, port=port, env=env,
                                 start_error_message="Please see https://sites.google.com/a/chromium.org/chromedriver/home")

    def command_line_args(self) :
        return ["--port=%d" % self.port] + self.service_args
Copy the code

View base class start method (due to the base class for too long is not all, base class at selenium.webdriver.com mon import service) :

def start(self) :
        """ Starts the Service. :Exceptions: - WebDriverException : Raised either when it can't start the service or when it can't connect to the service """
        try: cmd = [self.path] cmd.extend(self.command_line_args()) self.process = subprocess.Popen(cmd, env=self.env, close_fds=platform.system() ! ='Windows',
                                            stdout=self.log_file,
                                            stderr=self.log_file,
                                            stdin=PIPE)
        except TypeError:
            raise
        except OSError as err:
            if err.errno == errno.ENOENT:
                raise WebDriverException(
                    "'%s' executable needs to be in PATH. %s" % (
                        os.path.basename(self.path), self.start_error_message)
                )
            elif err.errno == errno.EACCES:
                raise WebDriverException(
                    "'%s' executable may have wrong permissions. %s" % (
                        os.path.basename(self.path), self.start_error_message)
                )
            else:
                raise
        except Exception as e:
            raise WebDriverException(
                "The executable %s needs to be available in the path. %s\n%s" %
                (os.path.basename(self.path), self.start_error_message, str(e)))
        count = 0
        while True:
            self.assert_process_still_running()
            if self.is_connectable():
                break
            count += 1
            time.sleep(1)
            if count == 30:
                raise WebDriverException("Can not connect to the Service %s" % self.path)
Copy the code

Among the findings:

try: cmd = [self.path] cmd.extend(self.command_line_args()) self.process = subprocess.Popen(cmd, env=self.env, close_fds=platform.system() ! ='Windows',
                                      stdout=self.log_file,
                                      stderr=self.log_file,
                                      stdin=PIPE)
except TypeError:
            raise
        except OSError as err:
            if err.errno == errno.ENOENT:
                raise WebDriverException(
                    "'%s' executable needs to be in PATH. %s" % (
                        os.path.basename(self.path), self.start_error_message)
                )
            elif err.errno == errno.EACCES:
                raise WebDriverException(
                    "'%s' executable may have wrong permissions. %s" % (
                        os.path.basename(self.path), self.start_error_message)
                )
            else:
                raise
        except Exception as e:
            raise WebDriverException(
                "The executable %s needs to be available in the path. %s\n%s" %
                (os.path.basename(self.path), self.start_error_message, str(e)))
        count = 0
        while True:
            self.assert_process_still_running()
            if self.is_connectable():
                break
            count += 1
            time.sleep(1)
            if count == 30:
                raise WebDriverException("Can not connect to the Service %s" % self.path)
Copy the code

Start the child process to start the driver. Receives an exception thrown and an error when an exception occurs. Open the driver open the browser.

You know how Selenium started the service when an exception throw was detected. Next, look at the implementation process of the GET request URL. Check the webdriver base class (. Selenium webdriver. Remote. Webdriver), find the get method:

def get(self, url) :
    """ Loads a web page in the current browser session. """
    self.execute(Command.GET, {'url': url})

def execute(self, driver_command, params=None) :
        """ Sends a command to be executed by a command.CommandExecutor. :Args: - driver_command: The name of the command to execute as a string. - params: A dictionary of named parameters to send with the command. :Returns: The command's JSON response loaded into a dictionary object. """
        if self.session_id is not None:
            if not params:
                params = {'sessionId': self.session_id}
            elif 'sessionId' not in params:
                params['sessionId'] = self.session_id

        params = self._wrap_value(params)
        response = self.command_executor.execute(driver_command, params)
        if response:
            self.error_handler.check_response(response)
            response['value'] = self._unwrap_value(
                response.get('value'.None))
            return response
        # If the server doesn't send a response, assume the command was
        # a success
        return {'success': 0.'value': None.'sessionId': self.session_id}
Copy the code

The execute method is called and Command. Get and the URL are passed in. . See the Command GET class Command mand (selenium.webdriver.remote.com), the Command for the standard WebDriver Command constants; Find the GET constant:

GET = "get"
Copy the code

From the file, it should be the class file that executes the command mode. First, sort out the process:

• Start the service → call get method

The specific process of get method:

• The get method calls the execute method, passing in command-get and the URL. The Command value is a standard constant. In the execute method,

Execute is implemented as follows:

def execute(self, driver_command, params=None) :
        """ Sends a command to be executed by a command.CommandExecutor. :Args: - driver_command: The name of the command to execute as a string. - params: A dictionary of named parameters to send with the command. :Returns: The command's JSON response loaded into a dictionary object. """
        if self.session_id is not None:
            if not params:
                params = {'sessionId': self.session_id}
            elif 'sessionId' not in params:
                params['sessionId'] = self.session_id

        params = self._wrap_value(params)
        response = self.command_executor.execute(driver_command, params)
        if response:
            self.error_handler.check_response(response)
            response['value'] = self._unwrap_value(
                response.get('value'.None))
            return response
        # If the server doesn't send a response, assume the command was
        # a success
        return {'success': 0.'value': None.'sessionId': self.session_id}
Copy the code

The core code is as follows:

params = self._wrap_value(params)
response = self.command_executor.execute(driver_command, params)
if response:
    self.error_handler.check_response(response)
    response['value'] = self._unwrap_value(
        response.get('value'.None))
    return response
Copy the code

Main view:

self.command_executor.execute(driver_command, params)
Copy the code

Command_executor instantiates the derived webdriver class (Selenium import webDriver) as follows:

RemoteWebDriver.__init__(
                self,
                command_executor=ChromeRemoteConnection(
                    remote_server_addr=self.service.service_url,
                    keep_alive=keep_alive),
                desired_capabilities=desired_capabilities)
Copy the code

Check the ChromeRemoteConnection class (Selenium import remote_connection) :

from selenium.webdriver.remote.remote_connection import RemoteConnection


class ChromeRemoteConnection(RemoteConnection) :

    def __init__(self, remote_server_addr, keep_alive=True) :
        RemoteConnection.__init__(self, remote_server_addr, keep_alive)
        self._commands["launchApp"] = ('POST'.'/session/$sessionId/chromium/launch_app')
        self._commands["setNetworkConditions"] = ('POST'.'/session/$sessionId/chromium/network_conditions')
        self._commands["getNetworkConditions"] = ('GET'.'/session/$sessionId/chromium/network_conditions')
        self._commands['executeCdpCommand'] = ('POST'.'/session/$sessionId/goog/cdp/execute')
Copy the code

The execute method is implemented as follows:

def execute(self, command, params) :
        """ Send a command to the remote server. Any path subtitutions required for the URL mapped to the command should be included in the command parameters. :Args: - command - A string specifying the command to execute. - params - A dictionary of named parameters to send with the command as its JSON payload. """
        command_info = self._commands[command]
        assert command_info is not None.'Unrecognised command %s' % command
        path = string.Template(command_info[1]).substitute(params)
        if hasattr(self, 'w3c') and self.w3c and isinstance(params, dict) and 'sessionId' in params:
            del params['sessionId']
        data = utils.dump_json(params)
        url = '%s%s' % (self._url, path)
        return self._request(command_info[0], url, body=data)

    def _request(self, method, url, body=None) :
        """ Send an HTTP request to the remote server. :Args: - method - A string for the HTTP method to send the request with. - url - A string for the URL to send the request to. -  body - A string for request body. Ignored unless method is POST or PUT. :Returns: A dictionary with the server's parsed JSON response. """
        LOGGER.debug('%s %s %s' % (method, url, body))

        parsed_url = parse.urlparse(url)
        headers = self.get_remote_connection_headers(parsed_url, self.keep_alive)
        resp = None
        if body andmethod ! ='POST' andmethod ! ='PUT':
            body = None

        if self.keep_alive:
            resp = self._conn.request(method, url, body=body, headers=headers)

            statuscode = resp.status
        else:
            http = urllib3.PoolManager(timeout=self._timeout)
            resp = http.request(method, url, body=body, headers=headers)

            statuscode = resp.status
            if not hasattr(resp, 'getheader') :if hasattr(resp.headers, 'getheader'):
                    resp.getheader = lambda x: resp.headers.getheader(x)
                elif hasattr(resp.headers, 'get'):
                    resp.getheader = lambda x: resp.headers.get(x)

        data = resp.data.decode('UTF-8')
        try:
            if 300 <= statuscode < 304:
                return self._request('GET', resp.getheader('location'))
            if 399 < statuscode <= 500:
                return {'status': statuscode, 'value': data}
            content_type = []
            if resp.getheader('Content-Type') is not None:
                content_type = resp.getheader('Content-Type').split('; ')
            if not any([x.startswith('image/png') for x in content_type]):

                try:
                    data = utils.load_json(data.strip())
                except ValueError:
                    if 199 < statuscode < 300:
                        status = ErrorCode.SUCCESS
                    else:
                        status = ErrorCode.UNKNOWN_ERROR
                    return {'status': status, 'value': data.strip()}

                # Some of the drivers incorrectly return a response
                # with no 'value' field when they should return null.
                if 'value' not in data:
                    data['value'] = None
                return data
            else:
                data = {'status': 0.'value': data}
                return data
        finally:
            LOGGER.debug("Finished Request")
            resp.close()
Copy the code

Execute sends a request to the remote server. The _request method called in execute sends an HTTP request and returns the result, which is responded to by the browser.

The official instructions explain how the request works:

At its minimum, WebDriver talks to a browser through a driver.

Communication is two way: WebDriver passes commands to the browser through the driver, and receives information back via the same route.



The driver is specific to The browser, such as ChromeDriver for Google’s Chrome/Chromium, Thedriver runs on the same system as the browser. This may, or may not be, the same system where the tests themselves are executing.

This simple example above is direct communication. Communication to the browser may also be remote communication through Selenium Server or RemoteWebDriver. RemoteWebDriver runs on the same system as the driver and the browser.

Anyway, we talk to the browser through WebDriver, and the browser responds.

As you can see from the above example, using execute to send a request to a remote server will interact with the browser through WebDriver, and send a defined command constant to get some relevant information.

Because we’re instance webdriver instance in the code, to webdriver base class (. Selenium webdriver. Remote. Webdriver) in the query information, whether there is a correlation function can access to information. The following functions are found:

def title(self) :
    """Returns the title of the current page. :Usage: title = driver.title """
    resp = self.execute(Command.GET_TITLE)
    return resp['value'] if resp['value'] is not None else ""
@property
def current_url(self) :
    """ Gets the URL of the current page. :Usage: driver.current_url """
    return self.execute(Command.GET_CURRENT_URL)['value']
@property
def page_source(self) :
    """ Gets the source of the current page. :Usage: driver.page_source """
    return self.execute(Command.GET_PAGE_SOURCE)['value']
Copy the code

The above list is not complete, we simply try to use the above functions, the use method has been explained in the function. Try to get title, current_URL, page_source (page source) :

from selenium import webdriver
driver = webdriver.Chrome()
driver.get("http://www.csdn.net")
print(driver.title)
print(driver.current_url)
print('the author blog: https://blog.csdn.net/A757291228)
# Support original, reprint please paste the original link
# print(driver.page_source)
Copy the code

Page title and current url:



Try page_source:

from selenium import webdriver
driver = webdriver.Chrome()
driver.get("http://www.csdn.net")
print(driver.title)
print(driver.current_url)
print('the author blog: https://blog.csdn.net/A757291228)
# Support original, reprint please post a link
print(driver.page_source)
Copy the code

Successful acquisition:



Original is not easy, see here a thumbs-up support bai! thank you