How can I use Selenium WebDriver to modify HTTP request headers in JAVA?

One of the most common test automation challenges is how we modify the request header information in Selenium WebDriver. As an automation tester, you will encounter challenges in any programming language, including Java. Before solving the problem, we need to better understand the problem statement and figure out different possibilities to modify the request headers in Java when using Selenium WebDriver.

In this Selenium Java tutorial, we will learn how to modify the different options available for HTTP request headers in Java using Selenium WebDriver.

So, let’s get started!

What are HTTP headers

HTTP headers are an important part of the HTTP protocol. They define an HTTP message (request or response) and allow clients and servers to exchange optional metadata with that message. They consist of a case-insensitive header domain name and a colon, followed by a header value. A header field can be extended to multiple lines by prefacing each line with at least one space or horizontal TAB.

Header information can be grouped according to its context.

  1. Request headers. The HTTP request header is used to provide additional information about the resource being fetched and the requesting client.
  2. Response headers. The HTTP response header provides information about the response. The location header specifies the location of the resource, while the server header provides information about the server providing the resource.
  3. Representative header information. The HTTP presentation header is an important part of any HTTP response. They provide information about protocol elements, such as MIME types, character encodings, and so on. This makes them an important part of handling resources on the Internet.
  4. Payload head. The HTTP payload header contains data about the HTTP message payload, such as its length and encoding, but is independent of presentation.

Delve deeper into HTTP request headers

The HTTP request header is a communication mechanism that enables a browser or client to request a specific web page or data from a (web) server. When used in network communication or Internet browsing, HTTP request headers enable browsers and clients to communicate with the corresponding web server by sending a request.

The HTTP request header describes the request that a Web browser sends to load a page. It is also known as the client-to-server protocol. The header includes details of the client request, such as the type of browser and operating system the user is using, as well as other parameters needed to display the request content correctly on the screen.

The following is the main information contained in the HTTP request header.

  • IP address (source) and port number.
  • The URL of the requested page.
  • Web server or target web site (host).
  • The type of data the browser will accept (text, HTML, XML, and so on).
  • Browser type (Mozilla, Chrome, IE) to send compatible data.

In response, an HTTP response header containing the request data is sent back.

The need to change the HTTP request header

Can you guess why we need to change the request header once it has been set in the script?

Here are some situations where you might need to change the HTTP headers.

  1. Test control and/or test different variations by setting up appropriate HTTP headers.
  2. Test cases are needed when different aspects of the network application or even the server logic need to be thoroughly tested.
  3. Because HTTP request headers are used to enable specific parts of network application logic that are normally disabled in normal mode, you may need to modify the HTTP request headers from time to time based on testing.

Ideally, testing the visitor pattern on the web application under test, you may need to modify the HTTP request headers.

However, the functionality that Selenium RC once supported to modify HTTP request headers is now not handled by Selenium Webdriver.

This is why the question arises as to how we can change the request header when the test automation project is written in the Selenium framework and Java.

For a quick overview of developing Selenium test automation from scratch using Java, Selenium, JUnit 5, and Maven, see belowLambdaTest YouTube channelIn the video.

How do I modify header requests in Selenium Java projects

In this part of the Selenium Java tutorial, we look at the many ways to modify header requests in Java. In general, there are several possibilities that you can follow to modify the header request in a Java-Selenium project.

  1. Use a driver/library like REST Assured instead of Selenium.
  2. Use a reverse proxy, such as a browser mob-proxy or some other proxy mechanism.
  3. Use the Firefox browser extension, which will help modify the request header file.

Let’s explore each possibility one by one.

Use the REST assurance library to modify HTTP request headers

In addition to Selenium, REST Assured is available, which is a great tool for working with REST services in a simple way.

In any IDE, such as Eclipse, the prerequisites for configuring REST Assured with your project are fairly simple. Once you have Java, Eclipse, and TestNG set up, you need to download the required REST Assured JAR files.

Read – How to install TestNG in Eclipse. A step-by-step guide

After downloading the JAR file, you must create a project in Eclipse and add the downloaded JAR file to the properties section as an external JAR. This is also similar to how we added Selenium JAR files to our project. Once you’ve successfully set up your Java project with REST Assured libraries, you’re ready to go.

We intend to create a mechanism so that request headers can be customized. In order to realize this possibility, we first need to know the normal way to create a request header.

Let’s consider the following case.

  • We have a Java class called RequestHeaderChangeDemo, where we maintain the basic configuration
  • We have a test step file called TestSteps, where we will call the methods in the RequestHeaderChangeDemo Java class from which we will execute our test.

Take a look at the Following Java class called RequestHeaderChangeDemo.

BASE_URL is amazon (https://www.amazon.com), where the following four methods are applied.

  • authenticateUser
  • To get the product
  • Add product
  • Remove the product
public class RequestHeaderChangeDemo { private static final String BASE_URL = "https://amazon.com"; public static IRestResponse<Token> authenticateUser(AuthorizationRequest authRequest) { RestAssured.baseURI = BASE_URL; RequestSpecification request = RestAssured.given(); request.header("Content-Type", "application/json"); Response response = request.body(authRequest).post(Route.generateToken()); return new RestResponse(Token.class, response); } public static IRestResponse<Products> getProducts() { RestAssured.baseURI = BASE_URL; RequestSpecification request = RestAssured.given(); request.header("Content-Type", "application/json"); Response response = request.get(Route.products()); return new RestResponse(Products.class, response); } public static IRestResponse<UserAccount> addProduct(AddProductsRequest addProductsRequest, String token) { RestAssured.baseURI = BASE_URL; RequestSpecification request = RestAssured.given(); request.header("Authorization", "Bearer " + token) .header("Content-Type", "application/json"); Response response = request.body(addProductsRequest).post(Route.products()); return new RestResponse(UserAccount.class, response); } public static Response removeProduct(RemoveProductRequest removeProductRequest, String token) { RestAssured.baseURI = BASE_URL; RequestSpecification request = RestAssured.given(); request.header("Authorization", "Bearer " + token) .header("Content-Type", "application/json"); return request.body(removeProductRequest).delete(Route.product()); ,}}Copy the code

In the Java class file above, we repeatedly send the BASE_URL and header file in each successive method. An example is shown below.

1
2
3
4
RestAssured.baseURI = BASE_URL;
RequestSpecification request = RestAssured.given();
request.header(
"Content-Type"
.
"application/json"
);
Response response = request.body(authRequest).post(Route.generateToken());

The request.header method requests a JSON header. There is a lot of duplicate code, which reduces the maintainability of the code.

This can be avoided if we initialize the RequestSpecification objects in the constructor and make these methods non-static (that is, create instance methods).

Because instance methods in Java belong to the object of the class, not the class itself, the method can be called even after the object of the class is created. At the same time, we will also rewrite the instance methods.

Converting this method to an instance method provides the following benefits.

  • Authentication is done only once in a RequestSpecification object. You no longer need to create the same method for other requests.
  • The request header in the project can be modified flexibly.

So let’s take a look at what the Java class RequestHeaderChangeDemo and the test step file TestSteps look like when we use the instance method.

Java class of the RequestHeaderChangeDemo class with instance methods

public class RequestHeaderChangeDemo
{
    private final RequestSpecification request;
    public RequestHeaderChangeDemo(String baseUrl) 
    {
        RestAssured.baseURI = baseUrl;
        request = RestAssured.given();
        request.header("Content-Type", "application/json");
    }
 
    public void authenticateUser(AuthorizationRequest authRequest) 
    {
        Response response = request.body(authRequest).post(Route.generateToken());
        if (response.statusCode() != HttpStatus.SC_OK)
        throw new RuntimeException("Authentication Failed. Content of failed Response: " +             response.toString() + " , Status Code : " + response.statusCode());
        
        Token tokenResponse = response.body().jsonPath().getObject("$", Token.class);
        request.header("Authorization", "Bearer " + tokenResponse.token);
    }
 
    public IRestResponse<Products> getProducts() 
    {
        Response response = request.get(Route.products());
        return new RestResponse(Products.class, response);
    }
 
    public IRestResponse<UserAccount> addProduct(AddProductsRequest addProductsRequest) 
    {
        Response response = request.body(addProductsRequest).post(Route.products());
        return new RestResponse(UserAccount.class, response);
    }
 
    public Response removeProducts(RemoveProductRequest removeProductRequest)
    {
        return request.body(removeProductRequest).delete(Route.product());
    }
}

Copy the code

Code to rehearse

  1. We have created a constructor to initialize the RequestSpecification object that contains both BaseURL and Request Headers.
  2. Earlier, we had to pass tokens in each request header. Now, once we receive the TokenResponse in the authenticateUser() method, we put it into the same request instance. This enables the execution of the test step to move forward, rather than adding tokens for each request as previously. This makes the header available for subsequent calls to the server.
  3. The RequestHeaderChangeDemo Java class will now be initialized in the TestSteps file.

We changed the TestSteps file based on changes to the RequestHeaderChangeDemo Java class.

public class TestSteps { private final String USER_ID = " (Enter the user id from your test case )"; private Response response; private IRestResponse<UserAccount> userAccountResponse; private Product product; private final String BaseUrl = "https://amazon.com"; private RequestHeaderChangeDemo endPoints; @Given("^User is authorized$") public void authorizedUser() { endPoints = new RequestHeaderChangeDemo (BaseUrl); AuthorizationRequest authRequest = new AuthorizationRequest("(Username)", "(Password)"); endPoints.authenticateUser(authRequest); } @Given("^Available Product List$") public void availableProductLists() { IRestResponse<Products> productsResponse = endPoints.getProducts(); Product = productsResponse.getBody().products.get(0); } @When("^Adding the Product in Wishlist$") public void addProductInWishList() { ADDPROD code = new ADDPROD(product.code); AddProductsRequest addProductsRequest = new AddProductsRequest(USER_ID, code); userAccountResponse = endPoints.addProduct(addProductsRequest); } @Then("^The productis added$") public void productIsAdded() { Assert.assertTrue(userAccountResponse.isSuccessful()); Assert.assertEquals(201, userAccountResponse.getStatusCode()); Assert.assertEquals(USER_ID, userAccountResponse.getBody().userID); Asert.assertEquals(product.code, userAccountResponse.getBody().products.get(0).code); } @When("^Product to be removed from the list$") public void removeProductFromList() { RemoveProductRequest removeProductRequest = new RemoveProductRequest(USER_ID, product.code); response = endPoints.removeProduct(removeProductRequest); } @Then("^Product is removed$") public void productIsRemoved() { Assert.assertEquals(204, response.getStatusCode()); userAccountResponse = endPoints.getUserAccount(USER_ID); Assert.assertEquals(200, userAccountResponse.getStatusCode()); Assert.assertEquals(0, userAccountResponse.getBody().products.size()); }}Copy the code

Code to rehearse

Here’s what we did in our modified implementation.

  1. Initialize the RequestHeaderChangeDemo class object as an endpoint.
  2. BaseURL is passed into the first method (authorizedUser).
  3. In the authorizedUser method, we call authenticateUser, the constructor of the RequestHeaderChangeDemo class.
  4. Therefore, the same endpoint object is used by subsequent step definitions.

Modify the HTTP request header using a reverse Proxy, such as mob-proxy in the browser

As the name implies, we have the option of using a proxy when dealing with header changes in the Java-Selenium automated test suite. Because Selenium prohibits the injection of information between the browser and the server, agents can come to the rescue.

This approach is not advisable if the testing is done behind an enterprise firewall.

As a network infrastructure component, a proxy enables network traffic to pass through it by positioning itself between clients and servers. In the corporate world, proxies work similarly, moving traffic through it, allowing secure traffic and blocking potential threats. The agent has the ability to modify requests and responses, either partially or completely.

The core idea is to send authorization headers, bypassing the phase that includes the certificate dialog, also known as the basic authentication dialog. However, this can prove to be a tiring process, especially when test cases need to be reconfigured frequently.

This is where the browser mob-Proxy library comes in. When you make the agent configuration part of the Selenium Automated test suite, the agent configuration is in effect every time you execute the test suite.

Let’s see how we use the browser Mob-Proxy with a sample site with basic authentication. To solve this problem, we can narrow down two possible approaches.

  1. Add authorization headers to all requests, no conditions or exceptions.
  2. Add headers only to requests that meet certain criteria.

While we won’t solve the header management problem, we’ll show you how to solve the authorization problem with the help of the browser mob-Proxy authorization tool set.

In this part of the Selenium Java tutorial, we will focus only on the first approach (that is, adding authorization header information to all requests).

First, we add a dependency to BrowserMob-proxy in POP.xml

If you want to pass this method to all header requests, a specific proxy, in which case the forAllProxy method should be called, as shown in the figure below.

01
02
03
04
05
06
07
08
09
10
11
12
13
14
public
void
forAllProxy()
{

proxy =
new
BrowserMobProxyServer();

try
{

String authHeader =
"Basic "
+ Base64.getEncoder().encodeToString(
"webelement:click"
.getBytes(
"utf-8"
));

proxy.addHeader(
"checkauth"
, authfirstHeader);

}

catch
(UnsupportedEncodingException e)

{

System.err.println(
"the Authorization can not be passed"
);

e.printStackTrace();

}

proxy.start(
0
);
} 

public class caseFirstTest { WebDriver driver; BrowserMobProxy proxy; @BeforeAll public static void globalSetup() { System.setProperty("webdriver.gecko.driver", "(path of the driver)"); } @BeforeEach public void setUp() { setUpProxy(); FirefoxOptions Options = new FirefoxOptions(); Options.setProxy(ClientUtil.createSeleniumProxy(proxy)); driver = new FirefoxDriver(Options); } @Test public void testBasicAuth() { driver.get("https://webelement.click/stand/basic? lang=en"); Wait waiter = new FluentWait(driver).withTimeout(Duration.ofSeconds(50)).ignoring(NoSuchElementException.class); String greetings = waiter.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("(Mention the xpath)"))).getText(); Assertions.assertEquals("(message"); } @AfterEach public void tearDown() { if(driver ! = null) { driver.quit(); } if(proxy ! = null) { proxy.stop(); } } private void setUpProxy( { } }Copy the code

In the code above, the line starting with String authHeader indicates that we are creating header information, which will be added to the request. These requests are then delivered via a proxy we created in proxy.addheader (“checkauth”, authfirstHeader).

01
02
03
04
05
06
07
08
09
10
11
12
try
{

String authHeader =
"Basic "
+ Base64.getEncoder().encodeToString(
"webelement:click"
.getBytes(
"utf-8"
));

proxy.addHeader(
"checkauth"
, authfirstHeader);

}

catch
(UnsupportedEncodingException e)

{

.........................................................................................................................................................

.........................................................................................................................................................

...................................................................................................................................................... .

}

proxy.start(
0
);
}

Finally, we start the agent setting 0 to mark the start parameter, and the agent starts on the port.

Modify the HTTP request header using the Firefox extension

In this part of the Selenium Java tutorial, we take a look at how to modify the request header using the appropriate Firefox browser extension. The main drawback to this option is that it only works with Firefox (not other browsers like Chrome, Edge, etc.).

Perform the following steps to modify the HTTP request header using the Firefox browser extension.

  • Download the Firefox browser extension
  • Load the extension.
  • Set the extension’s preferences.
  • Set desired capabilities.
  • Prepare test automation scripts.

Let’s walk through each step.

1. Download the Firefox extension

Search for the Firefox extension with.*xpi and set it in the project.

2. Load the Firefox extension

Follow the code below to add the Firefox configuration file.

01
02
03
04
05
06
07
08
09
10
11
FirefoxProfile profile =
new
FirefoxProfile();
File modifyHeaders =
new
File(System.getProperty(
"user.dir"
) +
"/resources/modify_headers.xpi"
);
profile.setEnableNativeEvents(
false
);
try
{

profile.addExtension(modifyHeaders);
}
catch
(IOException e)
{

e.printStackTrace();
}

3. Set extension preferences

Once we load the Firefox extension into the project, we set preferences (that is, various inputs that need to be set before the extension is triggered). This is done using the profile.set preference method.

This method sets preferences for any given profile through the key setting parameter mechanism. The first argument here is the key that sets the value, and the second argument sets a corresponding integer value.

The following is a reference implementation.

1
2
3
4
5
6
7
profile.setPreference(
"modifyheaders.headers.count"
.
1
);
profile.setPreference(
"modifyheaders.headers.action0"
.
"Add"
);
profile.setPreference(
"modifyheaders.headers.name0"
.
"Value"
);
profile.setPreference(
"modifyheaders.headers.value0"
.
"numeric value"
);
profile.setPreference(
"modifyheaders.headers.enabled0"
.
true
);
profile.setPreference(
"modifyheaders.config.active"
.
true
);
profile.setPreference(
"modifyheaders.config.alwaysOn"
.
true
);

In the code above, we list the number of times we want to set instances of the header.

1
profile.setPreference(
"modifyheaders.headers.count"
.
1
);

Next, we specify that the action, header name, and header value contain the values dynamically received from the API call.

1
profile.setPreference(
"modifyheaders.headers.action0"
.
"Add"
);

For the remaining lines that implement.set preference, we enable all so that the extender can be loaded when WebDriver instantiates the Firefox browser, setting the extender to active mode with the HTTP header.

4. Set expectations

The desired capabilities in Selenium are used to set the browser, browser version, and platform type for automated testing.

Here’s how we set our expectations.

1
2
3
4
5
6
7
DesiredCapabilities capabilities =
new
DesiredCapabilities();
capabilities.setBrowserName(
"firefox"
);
capabilities.setPlatform(org.openqa.selenium.Platform.ANY);
capabilities.setCapability(FirefoxDriver.PROFILE, profile);
WebDriver driver =
new
FirefoxDriver(capabilities);
driver.get(
"url"
);

What if you want to change the HTTP headers with a Version of Firefox that is not installed on your local (or test) machine? This is where LambdaTest, the largest cloud-based automated testing platform, provides a faster cross-browser testing infrastructure.

LambdaTest gives you flexibility to modify HTTP headers for different browser and platform combinations. If you prefer to use a Firefox extension to modify HTTP headers, you can use LambdaTest to do the same on different versions of the Firefox browser.

5. Draft the entire test automation script

Once you’ve completed all of the above steps, we’ll start designing the entire test automation script.

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
public
void
startwebsite()
{

FirefoxProfile profile =
new
FirefoxProfile();

File modifyHeaders =
new
File(System.getProperty(
"user.dir"
) +
"/resources/modify_headers.xpi"
);

profile.setEnableNativeEvents(
false
);

try

{

profile.addExtension(modifyHeaders);

}

catch
(IOException e)

{

e.printStackTrace();

}

profile.setPreference(
"modifyheaders.headers.count"
.
1
);

profile.setPreference(
"modifyheaders.headers.action0"
.
"Add"
);

profile.setPreference(
"modifyheaders.headers.name0"
.
"Value"
);

profile.setPreference(
"modifyheaders.headers.value0"
.
"Numeric Value"
);

profile.setPreference(
"modifyheaders.headers.enabled0"
.
true
);

profile.setPreference(
"modifyheaders.config.active"
.
true
);

profile.setPreference(
"modifyheaders.config.alwaysOn"
.
true
);

DesiredCapabilities capabilities =
new
DesiredCapabilities();

capabilities.setBrowserName(
"firefox"
);

capabilities.setPlatform(org.openqa.selenium.Platform.ANY);

capabilities.setCapability(FirefoxDriver.PROFILE, profile);

WebDriver driver =
new
FirefoxDriver(capabilities);

driver.get(
"url"
);
}

Reading – Browser automation using Selenium and Java

conclusion

In this Selenium Java tutorial, we have explored three different ways to handle modifications to HTTP request headers. Selenium itself is a great tool that has been performing well in network automation testing.

source

However, the tool cannot change the request header. After exploring all three alternatives to modifying the request header in the Java Selenium project, we can vouch for the first one using REST Assured. However, you may want to try out the other options and post your observations and opinions in the comments section.

Published on Java Code Geeks by permission of LambdaTest, our JCG project partner. Click here to see the original article. How can I use Selenium WebDriver to modify HTTP request headers in JAVA?

The opinions expressed by Java Code Geeks are their own.