The pros and cons of Python automated testing frameworks.

I have asked you about the advantages and disadvantages of Pytest and Robot Framework before. Since there is little information about this on the Internet, I have received your feedback and suggestions. Thank you very much.

Before comparing the pros and cons of a framework, what is the point of a framework? What are the essential features of a “good testing framework”?

What is a framework?

A Framework is a reusable design of a whole or part of a system that addresses the organization and operational control of code.

When we write automated scripts, we often need to read configuration files, read data files, send requests, log, connect and compare database data. Rewriting every method in every script is a lot of work and error-prone. So we need to extract the common methods, encapsulate them into separate modules, and put them in the common method package. In addition, we also need to store configuration files, data files and logs into different folders. This encapsulation of common methods and the design of how scripts and configuration files are organized is called a framework.

At the same time, in addition to completing the organization of code and configuration files, the general framework also provides operational control functions. Such as batch execution, modular execution, report generation, exception handling and so on.

The following three points can be summarized:

• Encapsulating common methods
• Organization of code and configuration files
• Executive control

What is a testing framework?

A complete test script (use case) generally consists of the following steps:

• Environment preparation or inspection
• Performing service Operations
• Assertion results
• Clean up the environment

And the test framework generally also need to complete the use case loading, batch execution, exception control, result output and other functions. Basic testing frameworks typically provide only execution control functionality.

Characteristics that the test framework should have

• Ease of use: easy to write use cases, execute use cases, generate reports and locate problems;
• Robustness: stability, such as timeout mechanism, etc.
• Extensibility: plug-ins;
• Flexibility: flexibility in use case organization or execution, Fixture functionality (different scopes of setUp and tearDown), and so on;
• Customization: convenient secondary development;

Pytest compared to the Robot Framework

Pytest framework features

Pytest is a powerful Python testing tool. It is easy to use, powerful, rich third-party plug-ins, efficient, scalable, and compatible. In fact, more and more projects are beginning to abandon Unittest, Nose, and the Robot Framework. Use Pytest instead, like Mozilla and Dropbox. Pytest provides richer functionality, including Assert rewriting, third-party plug-ins, and a fixture model that no other testing tool can match.

Pytest can automatically find test cases to execute and report test results. It has a rich base library, which can greatly improve the efficiency of user writing test cases. It has very strong scalability, users can write their own plug-ins, or install third-party plug-ins. Pytest can be easily used with other tools, such as continuous integration, Web automated testing, and so on.

Pytest also has the acceptance testing capabilities that Robot Framework is known for. One of the best features of Pytest is that it provides detailed failure information for test cases, enabling developers to quickly and accurately correct problems. It is compatible with the latest versions of Python and is also compatible with UnitTest, Doctest, and nose, right out of the box. Pytest also has various plug-ins available to add more functionality and variety to existing testing techniques and test cases. In fact, there are over 300 plug-ins available in its active community. The platform is designed to write simpler code with a lower error rate. You can use Pytest with graphical user interfaces such as Selenium and Splinter to make testing easier

Pytest has advantages over other testing frameworks:

• Simple tests can be simple to write
• Complex tests can also be written easily
• Simple and flexible, easy to use
• Support for parameterization
• Support simple unit tests, complex functional tests, selenium/ Appnium automation tests, Pytest+ Requests
• Pytest has many third-party plug-ins and can be customized with extensions, Useful examples are Pytest-Selenium (integrating Selenium), Pytest-allure-adaptor (generating nice Allure reports), and Pytest-Rerunfailures (cases) Repeated execution), Pytest-Xdist (multi-CPU distributed execution), etc
• Skip and XFail handling of test cases
• It integrates well with Jenkins
• Test code readability
• Easy to get started
• Assertion failures only use the native keyword assert, not self.assertequal (), or self.assertLessthan ()
• Pytest can run test cases written by UnitTest and Nose
• Both Python2 and Python3 can use the latest version regardless of a particular Python version
• Rapidly growing community development and maintenance

Robot Framework Framework (RF)

advantages

• Keyword driven, user-defined user keywords.
• Support for test log and report generation.
• Support system keyword development, good scalability.
• Supports database operations.

disadvantages

• Test cases are not neat to write.
• Specific grammar is required and expensive to learn
• Python2 is supported, but Python3 is not. This may cause different Python versions in the test team
• The interface is slow and often stuck
• Importing test libraries is sometimes an exception
• There are too many restrictions on the test team, which is not conducive to customizing functionality for specific businesses
• Problem debugging difficulty
• The output logs are complicated in structure and are mostly in English, which is difficult to understand
• Maintenance is difficult and over 2000 use cases are a nightmare
• Interface tests, rf is usually executed sequentially, but in interface tests there is a need for concurrent tests that include write concurrent keywords

*** Settings ***Library    RequestsLibraryLibrary    Collections*** Test Cases ***test_get_event_list    # query conference (GET Request) ${payload} = Create Dictionary eid = 1 the Create Session event http://127.0.0.1:8000/api ${r} = the GET Request event /get_event_list/ params=${payload} Should Be Equal As Strings ${r.status_code} 200 log ${r.json()} ${dict} Set ${MSG} Get From Dictionary ${dict} message Should Be Equal ${MSG} success ${sta} Get From Dictionary ${dict} status ${status} Evaluate int(200) Should Be Equal ${sta} ${status}Copy the code

Verdict: No, no one wants to write an interface use case like this.

(Article from Hogwarts Testing Institute)