Test Architects Hub

Alright, ladies and gentlemen, time for something exciting! While I’ve been trying to find some time to write a bunch of materials about the Playwright framework, its developers released a new feature: Playwright UI mode. While still being sceptical in nature, I’ve given it a try during the last couple of days and must say that it does solve some problems of a test automation engineer.

“What is it? How can it help me?” – I’m going to answer these questions in today’s blog post.

Intro: What is Playwright?

Before we move on to the UI mode feature, I thought it would be a good idea to explain what Playwright actually is. Chances always are that some readers of this blog post aren’t familiar with Playwright or even Web Applications test automation.

Basically, Playwright is a set of tools that allows interaction with browser engines and helps to automate functional testing of web applications. It does that on different platforms, browsers and also in different languages. Which makes it a cross-everything framework. The popularity of Playwright has been rising over the past few years and some people even tend to call it the “Selenium killer”. Which I think is not that far from the truth. But we will just have to wait and see.

There are a few pillars of Playwright that I identified for myself over the last few months:

• more reliable and faster test execution due to the Web Socket protocol used;
• smart waits and events that take care of tricky scenarios and reduce test flakiness;
• light-weight and isolated contexts that improve overall tests stability;
• a bunch of extra perks like Inspector, Codegen and Trace Viewer (my favourite) that aren’t mandatory but can bring a lot of conveniences.

Overall, I found Playwright an easy tool to start working with, if you already have some experience with writing web UI tests. Before, I was mostly using Playwright with the Java binding and did have some struggles with it (especially when using Gradle instead of Maven). However, in this article, I decided to challenge myself, to explore the new tech stack and, hence, proceed with the Node.js version.

Generating a demo project

The good thing about Node.js is that it’s extremely easy to get started. Assuming that there are Node.js and npm installed on your machine, just navigate to the place where you want your first Playwright project to be and execute the following command:

npm init playwright@latest

It will walk you through the initialisation steps such as choosing a language (JS or TS), tests folder location, adding GitHub Actions workflow and installing Playwright browsers. In the end, if everything goes well, you should receive a project template that has everything (even demo tests) to explore and run.

Here’s how a freshly baked generated Playwright project looks in Visual Studio Code:

As can be seen from the picture above, the init command did pretty good work by creating the basic project structure, adding the playwright.config file with basic options and explanations, generating the package.json file and even quite an extensive test suite with demo tests for the Demo ToDo website.

I find it useful because right before that I’d tried to do it manually for the first time, messed up with installing browsers, tests location and, as a result, my VS Code wasn’t able to discover tests properly.

Okay, shall I run the tests? Yes, but let’s do it differently – it’s time to give the UI mode a try.

UI mode: is it indeed a game-changer?

So, we have the tests, we have Node.js Playwright installed… Where is this UI mode hiding from us?

Well, it’s actually pretty simple. The following command (in fact, the --ui flag) does the trick and starts the UI mode window:

npx playwright test --ui

For those of you who have some experience with Playwright already, the left panel might remind the Visual Studio Code’s Test Explorer window. And the rest of the window does look pretty much like the Trace Viewer – another tool from Playwright for viewing recorded trace files.

The difference, however, is that the UI mode allows us to execute tests right from its interface. At that, tests can be run all at once or any particular test can be run independently (see button #1):

When a test is being executed, UI mode displays its actions in real time which is quite cool to observe. Usually, in most of the testing frameworks is rather a post-factum event or consists of watching unpleasant console output logs that still have to be analysed later. I was only able to achieve similar observability with Report Portal, but that is honestly a too heavy-weight solution for such everyday problems.

In this regard, I think Playwright is defining a new reality where running and watching tests doesn’t have to be cumbersome anymore. This is the merit of a holistic ecosystem, I believe, where a bunch of different tools are well-integrated with each other.

The buttons #2 and #3 in the picture above are no less interesting! In my modest opinion, they both work better together:

• Button #2 opens a test code in the VS Code editor, so you can start making changes right away, without looking for a required test;
• Button #3 starts watching for a particular test (or even all tests in a suite). When pressed, UI mode will start tracking changes to a test/suite and will re-run the item it watches right after you save changes to that file in VS Code.

For example, in the picture below, I decided to watch the “should clear text input field when an item is added” test. After I changed the button’s text from “Enter” to “Hi there” at line 46 and saved the changes in VS Code, UI mode almost instantly restarted the test. Of course, it failed with a timeout, since there’s no such locator on the page:

I consider this feature one of the most useful ones since the workflow of writing, debugging and analysing tests is almost seamless. I can change things, see them rerunning immediately, passing or failing and get extensive insights into what might be wrong (including network communication, console, screenshots and even pointing at the actual test code).

If the above-mentioned is not enough, snapshots can be opened in a bigger window. Since it all happens in a browser, all dev tools are available for debugging and troubleshooting:

It’s extremely useful.

What else is in the stash?

Another interesting area to explore in UI mode is Search and Filters (top left corner). Let’s imagine that you’d like to run tests in the 3 most popular browser engines: Chromium, Firefox and Webkit.

In order to achieve that in Playwright Node.js, the following projects need to be added to the playwright.config.ts file:

projects: [
  {
    name: 'chromium',
    use: { ...devices['Desktop Chrome'] },
  },
  {
    name: 'firefox',
    use: { ...devices['Desktop Firefox'] },
  },
  {
    name: 'webkit',
    use: { ...devices['Desktop Safari'] },
  },
]

When UI mode starts, it is able to read the defined projects from the config file and display them in the Filters section:

The good thing about having projects in the UI mode is that it’s really easy to run and rerun tests in multiple browsers, see the difference, possible weak places, and debug and fix tests on the fly. For me, as an engineer, this UI mode’s feature saves decent time on switching browsers manually.

If you noticed in the previous picture, there’re also different filters: passed, failed and skipped tests. I played with them a bit and found filtering by failed tests very convenient. Basically, this filter lets you display only failed tests, work on them and then later rerun only previously failed tests. I do think it speeds the debugging process up.

As for skipped tests, there’s nothing really special about this filter. It obviously displays tests that are marked as skipped, and the UI mode itself doesn’t have the functionality to run those skipped tests if they’re triggered manually. This feature might seem redundant but IntelliJ IDEA, for instance, does allow running JUnit 5 tests annotated with @Disabled, if they’re triggered from the IDE’s UI. Which is quite convenient if you need to debug them locally but still keep them ignored on the CI.

And, last but not least – the Output window:

For those technical geeks like myself, it opens the view where Playwright’s console output is displayed. But having in mind all those perks that we’ve just walked through, there is no real need to use this Output view.

Conclusions, pros & cons

1. I did see value in the Playwright UI mode. It’s not a redundant feature that will be just one of many. On the contrary – I foresee it being used by most of the engineers that work with the Playwright Node.js version. First of all, because it’s well integrated with VS Code, so jumping between the IDE and UI mode is easy, fun and efficient. Second of all, it does save time that would be otherwise spent on routine actions instead of actual test development.
2. I did notice some visual bugs, and some UI glitches when using filters. As I understood, UI mode is still in the preview stage, so these things can happen. I’m also pretty sure that the Playwright team will definitely listen to feedback (maybe, even from this article) and fix issues quickly.
3. There’re a couple of features that I would love to see in the UI mode. For example, to have a button that generates an HTML test report based on what’s currently executed in the Live Test Explorer panel. This kind of report is generated when executing tests from VS Code, and it’s pretty nice, so why not express this functionality to the UI mode as well?
4. Having lifecycle methods in the Live Test Explorer would also be nice. Currently, actions from methods like test.beforeEach(), test.afterEach() are included in the UI mode in actions of every nested test. But they can have some important code, test engineers would also need to change that code and possibly watch such lifecycle methods in the UI mode as well.
5. It’s not yet clear whether conditionally skipped tests are gonna be highlighted somehow. They’re currently marked as skipped and aren’t run but the skip condition is not displayed anywhere, so no clue whether it’s a bug in the UI mode or a conscious decision of a test creator.
6. The search field in the Filter section is supposed to search through tags as well, and it does. But it would also be nice to have available test tags presented as checkboxes (like UI mode already as “passed”, “failed”, “chromium”, etc.). It would really help any engineer who’s looking into tests using UI mode to understand what tags are there in the project and quickly filter tests out by a required tag and run them. For example, all @e2e tests, all @regression, or all @smoke tests.
7. When exploring the options, I tried to add a custom annotation as shown in the official example. Unfortunately, Playwright UI mode went crazy and refused to display any tests after saving the file in VS Code. I’ll try to reach the team regarding this because it seems to be a useful option. Well, if it’s not gonna be supported by the UI mode any time soon, at least it shouldn’t get broken if people have their custom annotations in their tests.

That’s it for today, guys and gals. Stay in touch for the next materials!