End-to-end testing using Cypress and Applitools

Today’s software applications are getting more complicated, thus every testing team needs to focus on expanding test coverage. To achieve this goal, it is important to use a combination of testing types, such as unit testing, integration testing, system testing, and end to end testing, depending on the software application’s complexity and requirements.

End to End (E2E) testing is designed to ensure that all components of a software application are working together correctly and that the system as a whole meets the desired functionality, performance, and reliability requirements.

Cypress is a popular open-source end-to-end testing framework for web applications. It is designed to make the testing process easier and more efficient for developers. One of the unique features of Cypress is that it runs the tests within the browser, which means that it can provide better control and visibility over the application under test.

In this blog on End to End testing, we will deep dive into performing Cypress End to End testing on a local Cypress grid and will explain how to start automating visual tests with Applitools Eyes and the Ultrafast Grid using Cypress in JavaScript.

What is End to End Testing?

End-to-end (E2E) testing is a software testing strategy that verifies an application’s complete flow from beginning to end.. It is a type of functional testing that tests the application’s behavior as a complete system, rather than testing individual components in isolation.

E2E testing simulates a real user scenario and covers all aspects of the application, including user interfaces, APIs, databases, and other integrations. It typically involves testing multiple components of an application to ensure that they work together as expected and fulfill the requirements of the business or end-users.

E2E testing is typically performed after other types of testing, such as unit testing and integration testing, have been completed. It is used to validate that the entire system works together seamlessly, and to identify any issues that may have been missed in earlier stages of testing.

Why is end to end testing necessary?

End-to-end testing (E2E testing) is a type of software testing that tests the entire system or application from start to finish, simulating real-world user scenarios.

Unit testing alone is not enough to ensure the quality and reliability of software. While unit testing is an important part of the testing process, it only verifies the behavior of individual components or modules of the software in isolation. It does not guarantee that the software will work correctly when integrated with other components or modules.

This is where integration testing enters  into the picture.Integration testing focuses on testing the interaction between two or more components of a system, to ensure that they work together correctly. However, even if all the individual components pass integration testing, there may still be issues with the overall system when all the components are put together. This is where end-to-end testing comes in – it tests the entire system from start to finish.

Cypress is a popular automation testing framework that is designed specifically for end-to-end testing. It runs tests directly in the browser, allowing it to provide an experience that is similar to how users interact with the application. This makes it easier to identify any issues that users might face, as the testing environment is as close to the real-world experience as possible.

To understand End to End testing, Let’s take a closer look at Mike Cohn’s test automation pyramid. We routinely do each level of testing listed in this pyramid while running automated Cypress testing.

Testing Pyramid Layers

Automation pyramid is a popular framework introduced by Mike Cohn that helps teams to plan and prioritize their testing efforts. It includes three levels of testing, which are:

1. Unit Tests: At the base of the pyramid are the unit tests, which test individual code components such as functions, methods, and classes. Unit tests are typically written by developers and are executed frequently during the development cycle. They are essential in ensuring that individual components of the application work as expected and can catch issues early in the development process.
2. Integration Tests: The middle layer of the pyramid consists of integration tests, which test how different components of the system work together. Integration tests ensure that the various parts of the application can communicate and interact with each other seamlessly. These tests are typically automated and are executed after the unit tests have passed.
3. End-to-End Tests: The top layer of the pyramid is end-to-end testing, which tests the entire application workflow from start to finish. These tests simulate real user scenarios and help ensure that the application functions as expected in a production environment. End-to-end tests are typically automated and are executed less frequently than the lower level tests.

Benefits of End to End Testing

There are several benefits of End to End testing. Some of the benefits of E2E testing include:

1. Increased Confidence: E2E testing provides a higher level of confidence in the software application by testing all components together. This testing approach ensures that all the components are integrated correctly and are working as expected.
2. Improved Quality: By testing the application from end-to-end,helps to identify and fix bugs earlier in the development process. This enhances the overall quality of the software.
3. Enhanced User Experience: E2E testing ensures that the application is working as expected for the end-user. This helps to provide a better user experience and can lead to increased customer satisfaction.
4. Time and Cost Savings: E2E testing helps to identify issues early in the development cycle, which can save time and money by reducing the need for costly rework later in the process.
5. Better Collaboration: E2E testing promotes better collaboration between different teams working on the same application. This testing approach helps to identify issues that may be caused by a lack of communication between teams.
6. Increased Productivity: By automating the testing process, E2E testing can help to increase productivity by reducing the time and effort required to manually test the application.
7. Faster Time-to-Market: By catching defects earlier in the development process, end-to-end testing can help to reduce delays and accelerate the time-to-market of the application.

Frameworks for End to End testing

There are several popular frameworks for end-to-end testing, including:

Cypress

Cypress is a JavaScript-based end-to-end testing framework that provides a simple and intuitive API for testing web applications. Cypress supports modern web development technologies like React, Angular, Vue.js, and more. It provides a built-in test runner, and it runs tests in the browser, which makes it fast and reliable.

Cypress runs tests inside the browser; it also provides detailed information about what’s happening at every step of the test, including network requests, console output, and DOM changes. This makes it easier to identify and troubleshoot issues, and helps ensure that the application is working as intended.

Cypress Trends on GitHub

The following information is taken from the official website of Cypress GitHub repository:

• Stars: 43.3k
• Forks: 2.8k
• Used By: 797k
• Releases: 303
• Contributors: 427

WebdriverIO

WebdriverIO is a popular open-source testing framework for Node.js that allows developers to automate web applications in a simple and efficient way. It uses the WebDriver API to communicate with browsers and supports a variety of testing frameworks, including Mocha, Jasmine, and Cucumber.

.WebdriverIO Trends on GitHub

The following information is taken from the official website of WebdriverIO GitHub repository:

• Stars: 8.1k
• Forks: 2.3k
• Used By: 50.5k
• Releases: 305
• Contributors: 491

Nightwatch.js

Nightwatch.js is an open-source Node.js based end-to-end testing framework used to automate browser testing. It provides a simple and easy-to-use syntax for writing automated tests in JavaScript and allows you to run tests in real web browsers like Chrome, Firefox, and Safari.

Nightwatch.js uses the WebDriver protocol to communicate with the browser and control its behavior. It also includes a powerful built-in assertion library that makes it easy to write test assertions and helps you quickly identify issues with your web application.

Nightwatch.js Trends on GitHub

The following information is taken from the official website of Nightwatch.js GitHub repository:

• Stars: 11.4k
• Forks: 1.1k
• Used By: 142k
• Releases: 219
• Contributors: 112

Protractor

Protractor is an open-source end-to-end testing framework for Angular and AngularJS applications. It is built on top of WebDriverJS and uses Jasmine syntax for writing test scripts. Protractor is designed to simulate user interactions with the application and to verify that the application behaves as expected.

Protractor Trends on GitHub

The following information is taken from the official website of Protractor GitHub repository:

• Stars: 8.8k
• Forks: 2.4k
• Used By: 1.9m
• Contributors: 250

TestCafe

TestCafe is an open-source end-to-end testing framework that allows you to automate web testing without using browser plugins. TestCafe is built on top of Node.js and provides a simple and powerful API for testing web applications.

TestCafe Trends on GitHub

The following information is taken from the official website of TestCafe GitHub repository:

• Stars: 9.6k
• Forks: 677
• Used By: 12.3k
• Releases: 390
• Contributors: 117

Benefits End to End Testing Using Cypress

Here are some of the features of Cypress End to End testing:

1. Easy Setup: Cypress has a simple setup process that doesn’t require any additional drivers or libraries. You can get started with Cypress by installing a single package.
2. Automatic Waiting: Cypress automatically waits for elements to appear and become intractable before executing commands. This ensures that the tests are not affected by the timing of the application’s response.
3. Real-time Reloads: Cypress provides real-time reloads, which means that as you make changes to your code or tests, the application will automatically reload, and the tests will be re-run.
4. Interactive Debugging: Cypress provides an interactive test runner, which allows you to debug your tests by stepping through them, setting breakpoints, and viewing the application’s state at any point in time.
5. Time Travel: Cypress allows you to go back and forth in time to see what happened during the execution of a test. This feature is useful for debugging and understanding the behavior of your application.
6. Cross-browser Testing: Cypress allows you to run your tests on multiple browsers and viewports simultaneously. This helps you ensure that your application works correctly across different environments.
7. Network Traffic Control: Cypress allows you to control the network traffic of your application. You can stub, spy, and mock network requests to simulate different scenarios.
8. Automatic screenshots and videos: Cypress automatically takes screenshots and records videos of your tests, which makes it easy to see what went wrong when a test fails.

Set up Cypress For End to End Testing

To create a new project for Cypress automated testing, follow the steps listed below.

Step 1: Generate package.json.

• Create a project, let’s name it as cypress_applitools
• Use the npm init command to create a package.json file

Step 2: Install Cypress.

Install Cypress by running the command in the newly created folder:

npm install cypress –save-dev

OR

yarn add cypress –dev

Above command will install Cypress locally as a dev dependency for your project.

As shown below, Cypress version 12.11.0 is reflected after installation. The newest Cypress version at the time this blog was being written was 12.11.0.

Below is a diagram of Cypress’s default folder layout. The “e2e” folder is where test cases can be created.

About Project structure of Cypress

Cypress has built a default folder hierarchy when it opens for the first time, as can be seen in the screenshots.Each of these files and folders that Cypress created is described in detail below.

• e2e: All test cases are stored under this folder. This folder contains the actual test files, written in JavaScript, that define the tests to be run.
• Fixtures: This folder contains any data files that are needed for the tests, such as JSON or CSV files.
• Support: There are two files inside the support folder: commands.js and e2e.js
  • command.js: Is the file where your frequently used functions and unique commands are added. It has functions like the login function that you may use in various tests. You can alter some of the functions Cypress generated for you right here.
  • e2e.js: This file is executed before each and every spec file. This file is an excellent location for global configuration and behavior that alters Cypress in the same way as before or before. It just imports commands.js by default, but you can import or need more files to keep things organized.
• Node_Modules: The node_modules directory will have all the node packages installed and all test files will have access to them. When you install Node packages using NPM, they are downloaded and installed in the node_modules directory, which is located in the root directory of your project
• cypress.config.json: cypress.config.json is a configuration file used by Cypress to override the default configuration settings for a project. It is similar to cypress.json, but it is intended to be used as a per-environment configuration file.

Some examples of configuration options that can be set in cypress.config.json include:

• baseUrl: The base URL for the application being tested.
• testFiles: A list of test files to include or exclude from the test suite.
• video: Configuration options for Cypress video recording.
• screenshots: Configuration options for Cypress screenshots.

Basic constructs of Cypress

Cypress used Mocha’s syntax for developing test cases. Key constructs that are frequently used in Cypress test development are listed below.

• describe(): This method is used in Cypress (using Mocha’s syntax) to group together related test cases. It takes two arguments. It takes two arguments: A string that describes the group of test cases (e.g. “Login Page Tests”) and another argument, a callback function that contains the individual test cases (using the it() method).
• it(): This method is used to define an individual test case.It requires two arguments: a string that specifies the test scenario and a callback function that has the test code itself.
• before():  This method is used to run a the code under before() block before any test case .The before() method takes one argument: a callback function that contains the setup code to be executed before any of the test cases
• after(): This method is used to run a cleanup once all the test cases are executed. The after() method takes one argument: a callback function that contains the cleanup code to be executed after all the test cases
• beforeEach(): This method is used to run a the code under beforeEach() block beforeEach.The beforeEach() method takes one argument: a callback function that contains the code to be executed before each test case
• afterEach(): This method is used is used to run a cleanup function after each test case.The afterEach() function takes one argument: a callback function that contains the cleanup code to be executed after each test case
• .only(): It is used to run a specified suite or test exclusively, ignoring all other tests and suites.This can be useful when you’re debugging a specific test case or working on a specific suite of tests, and you want to focus on that specific test case or suite without running any others.
• .skip(): It is used to skip a specified suite or test, effectively ignoring it during test execution. This can be useful when you’re working on a test suite or test case that isn’t ready to be run yet, or when you want to temporarily disable a test without deleting it.

Demonstration: Cypress End to End Testing

Let’s create a new folder under the e2e folder named  “cypress_applitools” to perform Cypress End to End testing.

Example : Cypress End to End Testing

Create the first spec with the name applitoolLogin.cy.js under the folder cypress_applitools

Write the below script in the applitoolLogin.cy.js which covers login with valid and invalid credentials and verify the text after clicking on the learning center icon.

Code snippet attached below

/// <reference types=”cypress” />

describe(‘Login into applitools site with valid and invalid data’, () => {

 before(() => {

   cy.visit(‘https://applitools.com/’)

 })

 it(‘Login into applitools site with valid crediential’, () => {

   cy.viewport(1280, 720)

   cy.get(‘[id=”login-item”]’).click()

   cy.get(‘#email’).type(‘[email protected]’)

   cy.get(‘#password’).type(‘Test@123’)

   cy.get(‘[type=”submit”]’).click()

 })

 it(‘Click on learning center icon and verify the text and logout’, () => {

   cy.viewport(1280, 720)

   cy.get(‘[data-test=”learning-center-button”]’).click()

   cy.contains(‘Learning Center’)

   cy.get(‘[data-test=”modal-x”]’).click()

   cy.get(‘[id=”dropdown-1-opener”]’).click()

   cy.get(‘[data-test=”menu-item-text”]’).eq(2).click()

 })

 it(‘Login into applitools Site with invalid crediential ‘, () => {

   cy.viewport(1280, 720)

   cy.get(‘#email’).type(‘[email protected]’)

   cy.get(‘#password’).type(‘Test132’)

   cy.get(‘[type=”submit”]’).click()

   cy.contains(‘Incorrect username or password.’)

 })

})

Code Walkthrough 

The test script performs the following steps:

1. Navigates to the Applitools website.
2. Logs into the site with valid credentials.
3. Clicks on the Learning Center icon, verifies the text, closes the modal, and clicks on the second menu item to logout.
4. Logs into the site with invalid credentials and checks for the expected error message.

The before() hook runs once before all the tests and visits the Applitools website. 

The first test logs in to the site with valid credentials by filling the email and password fields and clicking on the submit button. 

The second test clicks on the Learning Center icon, verifies the text, closes the modal, and clicks on the second menu item to logout from the application.

The third test logs in to the site with invalid credentials by filling the email and password fields with invalid values and clicking on the submit button. It then checks whether the error message is “Incorrect username or password.” is displayed on the page.

Execute the test case on local Cypress Grid

The command yarn run cypress open is used to launch the Cypress Test Runner, which provides a graphical user interface (GUI) for running and managing Cypress tests. 

When you run this command, Cypress will open a new window that shows a list of all the test files present in the project’s integration folder. 

You can select a test file to run and click the “Run” button to execute the test. The Cypress Test Runner will then open a new browser window and run the test in that window.

yarn run cypress open

After running the above command  the Cypress test runner will be launched with the current test cases. When running the test case, you can choose which browser to use. 

Clicking on the first .spec, the first test case starts executing. Below is a screenshot of when the test case ran successfully.

Set up Cypress End to End Testing on Applitools cloud

Pre-request :

You’ll need a few things to run this quickstart: A free account with Applitools, which you can create.

1. A free account with Applitools
2. Version 16 or later of Node.js.

To perform end-to-end testing of a Cypress application using the Applitools cloud, you can use the Applitools Eyes Cypress SDK. The SDK allows you to integrate Applitools visual testing into your existing Cypress tests, and then view the results in the Applitools cloud.

Before doing set up in Applitools cloud let first see how we can integrate Applitools with Cypress 

Applitools/Cypress Integration

Applitools integrates with Cypress and offers a general-purpose library that can be embedded within your project, whether it be the Angular, Vue, or React app.

The Applitools Eyes Cypress SDK is a Cypress plugin, After installation, it extends the main cy object with a few new commands.Three new primary methods are added: To begin the test, type cy.eyesOpen. To take screenshots (for each test step), type cy.eyesCheckWindow. To end the test, use cy.eyesClose. Here’s a diagram:

Three new primary methods are added: To begin the test, type cy.eyesOpen. To take screenshots (for each test step), type cy.eyesCheckWindow. To end the test, use cy.eyesClose. 

Step-up Applitools for Cypress test case execution

Step 1: Create project and generate package.json.

• Create a project, let’s name it as cypress_applitools
• npm init command is used in the terminal to create a package.json file

Step 2: Install Cypress.

To install Cypress, run this command in the newly created folder:

npm install cypress –save-dev

OR

yarn add cypress –dev

Above command will install cypress locally as a dev dependency.

Step 3 :Install Applitools Eyes Cypress SDK

Run below command 

npm install @applitools/eyes-cypress –save-dev

Step 4: Configure the Applitools Eyes Cypress SDK by issuing the command

npx eyes-setup

Step 5: Configure Eyes-Cypress plugin

Add plugin in cypress.config.js file

const { defineConfig } = require(“cypress”);

const eyesPlugin = require(‘@applitools/eyes-cypress’)

module.exports = eyesPlugin(defineConfig({

 e2e: {

   testIsolation: false,

   setupNodeEvents(on, config) {

     // implement node event listeners here

   },

 },

}));

Step 6: Configure custom commands

Add below line under cypress/support/e2e.js

import ‘@applitools/eyes-cypress/commands’

Step 7: Set APPLITOOLS_API_KEY

In mac run export APPLITOOLS_API_KEY=<your-api-key> in local terminal

In Window run set APPLITOOLS_API_KEY=<your-api-key> 

This command is used in the terminal to set the environment variable APPLITOOLS_API_KEY to the API key provided by Applitools. This API key is required to authenticate and access the Applitools services, such as Applitools Eyes

Step 8: Execute the test case on Applitool locally 

We are executing below test cases in Applitool cloud locally

/// <reference types=”cypress” />

describe(‘Login into applitools site with valid crediential’, () => {

 beforeEach(() => {

   cy.eyesOpen({

     appName: ‘AppTool!’

   });

 })

 afterEach(() => {

   cy.eyesClose();

 });

 it(‘Login into applitools site with valid crediential’, () => {

   cy.visit(‘https://demo.applitools.com/’)

   cy.eyesCheckWindow(‘Applitools Main Page’);

   cy.get(‘#username’).type(‘andy’)

   cy.get(‘#password’).type(‘i<3pandas’)

   cy.get(‘#log-in’).click()

   cy.eyesCheckWindow(‘Logged-In Page’);

 })

})

Code Walkthrough 

Here’s how above the test case works:

1. The beforeEach hook opens the Applitools Eyes session with the appName set to “AppTool!”.
2. The afterEach hook closes the Applitools Eyes session.
3. The it block describes the actual test scenario.
4. The cy.visit command visits the Applitools demo site.
5. The cy.eyesCheckWindow command takes a visual snapshot of the Applitools main page and logs it to the Applitools dashboard as the “Applitools Main Page” checkpoint.
6. The cy.get commands locate the username and password fields, and the cy.type commands fill them in with valid credentials.
7. The cy.get command locates the login button, and the cy.click command clicks on it to submit the form.
8. The cy.eyesCheckWindow command takes a visual snapshot of the logged-in page and logs it to the Applitools dashboard as the “Logged-In Page” checkpoint.

Execute the Test case in Applitool Cloud

Run the command ‘npx cypress open’

Test case start executing in local terminal

As we run the above command, test cases start executing in the Applitools cloud in Chrome browser and we can see the execution report in Applitools dashboard. Below the Screenshot attached.

Applitools Cloud provides an easy way to perform cross-browser testing for your web applications. Cross-browser testing is the process of testing your application on different browsers and browser versions to ensure that it works correctly and looks the same across all platforms.

Cross Browser testing in Applitools Cloud

The Ultrafast Grid is a cloud-based infrastructure provided by Applitools that enables you to run visual tests across the browsers and devices. It allows you to execute visual tests in parallel across different environments, making it faster and more efficient than running tests on your local machine.

About Ultrafast Grid

Applitools Ultrafast Grid is a cloud-based testing platform that allows you to run visual tests in parallel on multiple browsers and devices.Ultrafast Grid supports a wide range of browsers and browser versions, including Chrome, Firefox, Safari, Edge, Internet Explorer, and more. You can also test on mobile devices and tablets, including iOS and Android.

How Ultrafast Grid work

1. You write your visual tests using a test framework like Selenium or Cypress. These tests are written in code and are designed to simulate user interactions with your application’s user interface.
2. You configure your test framework to use the Applitools SDK, which allows you to perform visual testing using the Ultrafast Grid.
3. When you run your tests, the Applitools SDK captures screenshots of your application’s user interface and sends them to the Ultrafast Grid for analysis.
4. The Ultrafast Grid uses a cloud infrastructure to execute your tests across multiple browsers and devices in parallel. This allows you to test your application’s visual appearance and behavior across a wide range of environments.
5. As your tests run, the Ultrafast Grid uses AI-powered algorithms to perform smart visual validation of your application’s user interface. It can detect visual differences and anomalies that might be missed by traditional testing methods.
6. The results of your tests are sent back to the Applitools Dashboard, where you can view and analyze them. The dashboard provides a visual representation of your application’s user interface across different browsers and devices, along with detailed information about any visual differences that were detected.

Key features of Ultrafast Grid:

1. Support for a wide range of browsers and devices: The Ultrafast Grid supports a variety of browsers and devices, including desktop browsers, mobile browsers, and real mobile devices. This enables you to test your application’s visual appearance and behavior across different environments, without having to manually set up and maintain all the configurations yourself.
2. Parallel test execution: The Ultrafast Grid allows you to run visual tests in parallel, which can significantly reduce the time it takes to execute a large suite of tests. You can specify the number of concurrent tests to run, and the Ultrafast Grid will automatically distribute the tests across multiple environments.
3. Smart visual validation: Applitools uses AI-powered algorithms to perform smart visual validation of your application’s user interface. This means that it can detect visual differences and anomalies that might be missed by traditional testing methods, such as pixel-level differences or dynamic content changes.
4. Integration with popular testing frameworks: Applitools integrates with popular testing frameworks like Selenium, Cypress, and Appium, making it easy to add visual testing to your existing test suite.
5. Easy setup and configuration: Setting up and configuring the Ultrafast Grid with Applitools is straightforward, and can be done through a simple configuration file. You can specify the browsers and devices you want to test on, as well as other parameters like viewport size and concurrency level.

Set up applitools.config.js 

To run the test cases across the browsers we have to create applitools.config.js in the root folder.The applitools.config.js file  is typically used to configure Applitools’ test runner, which allows you to run visual tests across multiple browsers and devices in parallel.

You can define the different configurations that you want to run your tests against, including the browser type and version, viewport size, and other settings

module.exports = {

   testConcurrency: 1,

   apiKey: ‘<apiKey>’,

   browser: [

       {width: 800, height: 600, name: ‘chrome’},

       {width: 1600, height: 1200, name: ‘firefox’},

       {width: 1024, height: 768, name: ‘safari’},

       {deviceName: ‘Pixel 2’, screenOrientation: ‘portrait’},

       {deviceName: ‘Nexus 10’, screenOrientation: ‘landscape’},

   ],

   batchName: ‘Cypress JavaScript with the Ultrafast Grid’,

}

Let’s break down each of the properties in the above  configuration file on multiple browsers and devices in parallel.

1. testConcurrency: This property sets the maximum number of tests that can run concurrently.
2. apiKey: This property is the API key used to authenticate with the Applitools Eyes server.
3. browser: This property defines the browsers and devices on which you want to run your visual tests. In this case, we have specified three desktop browsers (chrome, firefox, and safari) and two mobile devices (Pixel 2 and Nexus 10).
4. batchName: This property is the name of the batch of tests that you want to run. A batch is a group of tests that are run together and can be used to organize your tests and view their results.

Execute Test case In Cross Browser

Run the below command 

npx  cypress open OR    npx cypress run

As we execute the above command, the test case starts executing in Applitools Cloud in 5 different browsers.

Test-results can be seen in APplitools Dashboard.

Summary

Cypress is a popular open-source JavaScript testing framework for web applications. It is designed to make testing web applications easier and more efficient by providing a simple and intuitive API for writing tests. 

Applitools is a cloud-based visual testing platform that allows you to easily automate visual testing of your web application. By combining Cypress and Applitools, you can create powerful end-to-end tests that verify both the functionality and the appearance of your application.