Test Architects Hub

A few days ago JetBrains released their freshly-baked Aqua IDE for test automation. I can’t argue that this kind of news is eye-catching. So, I decided to spend a few hours and try it out. You know – the very first questions I had were: “What’s so special about it that the good old IntelliJ IDEA can’t provide? Why, for God’s sake, produce another piece of software that will sit on your laptop and consume system resources?” Having these thoughts in mind, I did install Aqua and gave it a try as a user that already has experience in test automation using various languages and frameworks.

Since I’m not that proficient in JavaScript and its ecosystem, I decided not to evaluate that part of Aqua IDE. Instead, I concentrated on Aqua’s abilities to solve my current tasks and challenges which are:

• writing automated tests for REST backend microservices (Java/Kotlin, JUnit 5, RestAssured);
• finding a good approach to automating frontend testing (Java + frontend testing framework which is yet to be decided).

After reading a couple of blog posts, I understood that Aqua inherited lots of IntelliJ IDEA features like auto-completion, inspections, smart navigation, usage search, etc. Thus, I decided to omit them in my ad-hoc testing intentionally and concentrate on some fresh/unique features instead.

---

  Important note!   Please keep in mind that Aqua is still in the Early Access Preview (EAP) stage, hence, it might have issues or raw features that require feedback from the community. This review is not to blame but to evaluate it for potential future usage or sharing feedback about improvements or missing features.

---

Aqua IDE installation and first look

After a very straightforward installation (Mac), I ended up on the start screen with a few options to continue: Projects, Customize, Plugins and Learn. My primary goal was to understand if I can use this new IDE for my own challenges. For start, I just decided to explore the Projects tab and proceed to the creation of appropriate types of projects.

Basically, there are 2 ways to proceed with the creation of a project: 1) set it up by yourself; 2) use one of the generators provided by Aqua. There are several of them to choose from. But, honestly, the purpose of all provided generators in terms of test automation was not completely clear:

Basically, it’s a bit modified and limited Intellij IDEA’s new project dialogue, with a few differences, however. For instance, the Selenium generator is a new option. Since I couldn’t find any suitable generator for my backend tests, I just proceeded to the New Project tab…

Creating a project for automated REST API tests

For the start, I created a new Java project with the Maven build system. Despite the quite convenient configurator, there was no way to select a test framework (for instance, JUnit, Cucumber or TestNG). It was a bit of a surprise for me as I was expecting Aqua to provide me with a starting test automation toolkit but nothing of that happened.

Strange, but okay, can live without that. Let the show begin!

Surprisingly, Aqua threw an exception immediately after I hit the “Create” button. And then continued throwing other exceptions every time I was creating new projects. Since it’s an EAP build, I paid no attention to that and just submitted a report to the developers:

After a new project was generated, I went to check its pom.xml and sample code. Unfortunately, pom.xml didn’t have any dependencies that would help me to start writing tests faster (like RestAssured or JUnit). Basically, there was nothing except the compiler version and source encoding. Again, quite a strange outcome taking into account that Aqua is supposed to be a test automation-focused IDE. It was promised that one of Aqua’s pillars is Web API tests:

If you check the UI tests column on the left, there’s the zero-configuration project wizard. Why there is no similar wizard for Web API tests – is a big mystery to me. Yeah, there is the built-in HTTP client. However, it’s almost identical to the one we already have in our IntelliJ IDEA. Only GraphQL and WebSocket support was added.

So, my conclusion at this stage was that Aqua does not really simplify anything for a Java backend test automation engineer setup-wise. Perhaps, there’re other perks that might balance this impression?

Writing a minimalistic backend test

Remember, I checked that “Add sample code” checkbox in the New project dialogue? In fact, the auto-generated code consisted only of the “Hello world” example in the Main class. Come on, guys, seriously? That’s the IDE created for automated testing. Why there was nothing generated for a backend test automation project?

Okay, never mind. I manually added the JUnit 5 and AssertJ dependencies to see if I can run a test. After writing the @Test annotation in my test class, Aqua couldn’t figure out that there is already JUnit 5 dependency added. It just suggested adding it again to the classpath. After doing that, another JUnit 4 (!) dependency appeared in my pom.xml which wasn’t obviously the expected outcome. After removing that duplicate, Aqua seemed to sort things out and added the Import option to the context menu:

On the contrary, the RestAssured import went well, and I was able to create a simple test with a GET request:

OpenAPI support saves the world?

Looking for something special in this test automation-sharpened IDE, I decided to try its OpenAPI (Swagger) support feature. First, I installed the required OpenAPI Specifications plugin. Then generated code for an example API spec downloaded from the OpenAPI GitHub repository. Believe me or not, there was even a test class generated with a few dummy JUnit 4 tests inside. It would have been an improvement but… it’s exactly the same plugin and exactly the same feature that we already have in IntelliJ IDEA. So, what’s the point?

My conclusion for this part: Aqua IDE in its EAP form does not really bring any noticeable improvements for a Java backend test automation engineer.

Alrighty then, the easy part is done. Let’s see what Aqua brings to the table in terms of front-end UI automation.

Creating a project for web UI test automation

Taking into account my recent exploration of the Playwright framework (probably, worth an article), I was really hoping to see that framework support in Aqua. Unfortunately, it’s not there yet but the official website says that it’s coming soon. So, I just proceeded to the Selenium wizard. And… it’s a huge difference, I’d say:

First of all, I was given a choice to select the desired test framework. JUnit seemed like a good option despite I had pretty extensive experience with TestNG in the past and even wrote a quiz (brain crash test, actually) about it. Pay attention that like for the backend project, I chose to generate sample code as well.

Second of all, the next setup screen offered me different related third-party dependencies to add to my project! This is what I was dreaming about a few paragraphs back:

For the sake of the demo, I chose a few dependencies from each chapter: Allure (for custom reports), Selenide (wrapper on top of Selenium) and AssertJ (fluent assertions). I was about to tick the WebDriverManager as well but then recalled that Selenide should probably be using it under the hood anyway (I was right). After that, Aqua generated a new project for me.

I was pleasantly surprised that this time the project had much more stuff that is needed for a test automation engineer to actually start working on tests:

• pom.xml contained all pre-selected dependencies. Moreover, tweaks required for the Allure framework to work properly were also added to my pom.xml in the scope of the Maven Surefire plugin;
• a simple Page Object class was generated with a few Selenide elements defined (JetBrain’s website main page was taken as an example);
• a simple test class with 3 tests and setup methods was generated.

It’s worth mentioning here that one of the newly added distinctive Aqua features is its ability to generate Page Object classes from the Project context menu (New -> Selenium Page Object):

My expectations from this feature, however, diverged from what I really got. Despite a quite versatile creation dialogue and specified page URL, the generated Page Object class was… just empty. Well, there were a few Selenide imports added but nothing more than that. It might be okay for an engineer with some experience but if Aqua targets different seniority levels (I hope it does), then a generated Page Object class has to have at least some basic structure.

I mean, in most cases a Page Object describes the behaviour of a page (methods) and its state (class fields, fields, properties, etc.). Then why wouldn’t Aqua IDE generate a few of these just as an example, perhaps even deriving some methods or variable names from the provided page URL? So that test automation starters can benefit from that example and start extending it? But as it is now, this feature makes no sense either for beginners (due to incompleteness) or for skilled professionals (who can make Page Object designs without examples).

Okay, it is what it is., not a big deal. What about some crash tests?

I started all available tests for the first time without even looking into the code. As a result, Selenide spun up a Chrome driver via the WebDriverManager, and all 3 tests were run in a new browser window very quickly. Allure JSON files were also generated in the allure-results folder. Basically, within a few clicks, I was able to create a project and start writing actual tests almost instantly. That was quite impressive!

Tearing the Web Inspector apart

I had a haunting feeling that everything couldn’t be so cool regarding web UI tests. Since the Web Inspector feature is one of the most eye-catching ones in the Aqua portfolio, I decided to play around with it and see how it helps (or doesn’t) with designing Page Object models. One of the auto-generated tests was about clicking in the Search field on the JetBrains website, and I decided to check in the Web Inspector can easily find the same.

Looking ahead, I’ll say – yes but with some struggles. Here is the final result:

The biggest and most annoying issue is that I was not able to click on the Search icon and go to the appropriate element in the DOM. The only way was to expand all those nested elements one by one and click on each row with the hope that it was going to be the one I needed. For some reason, typing the “search” keyword in the Web Inspector search field didn’t trigger any actual search, and hovering the cursor over the Search icon on the JetBrains website didn’t work either. It might be okay-ish on simple websites but if there’re lots of elements in the DOM, Web Inspector in its current form only complicates things.

I have to admit, however, that the final result was quite good. The Web Inspector generated the following snippet for the Search element:

public SelenideElement siteHeaderMobileSearchActionButton = $("button[data-test='site-header-mobile-search-action']");

Compared to the originally generated element locator, the difference was only in the value of the data-test attribute because the website accidentally decided to show me its mobile version.

What about the opposite?

To make experimenting even more fun, I generated a new (empty) Page Object class from the context menu (New -> Selenium Page Object) for the Amazon main page. It was promised that the Aqua auto-completion feature should work when typing locations in code when the Web Inspector is opened. After that, I started typing the “search” word and hoping to find the desired search-related elements on the Amazon web page. Here is what happened:

As you can see, Aqua was able to suggest at least 3 relevant locators:

•  select[id=’searchDropdownBox]  which locates the “All” categories drop-down menu on the left of the search box input field;
•  input[id=’twotabsearchtextbox’]  which locates the main search input field;
•  div[class^=’nav-search-submit’]  which locates the search submit icon.

Navigating through the auto-complete suggestions also led to highlighting the appropriate elements on the Amazon web page in the Web Inspector window. Which I personally find very convenient. It’s even weirder now that clicking on or hovering the mouse over elements in Web Inspector earlier didn’t really show them in the Locators Evaluator DOM.

Aqua IDE: Conclusions

1. Aqua has some potential if the developers polish it in a way that it removes the hassle from its users and starts solving their problems. The biggest benefit that I personally see is that this IDE can handle the boring part that consists of adding dependencies manually, configuring Maven plugins, and generating rich examples. So, that test automation engineers can concentrate on writing actual tests quickly.
2. If there will be no more changes regarding the backend API tests, then it doesn’t make sense to switch from IntelliJ IDEA (or even any other IDE) to Aqua. IMHO, backend test automation is the same versatile and interesting and deserves the same cool setup wizard as for Selenium UI tests. But as you have seen already, most of the setup work I did manually, so, this is a no-go for me.
3. I wasn’t able to assess the Python (due to lack of time) or JavaScript (due to lack of experience) parts of Aqua. After a brief look, I noticed that lots of options are either absent in the wizard (like no way to create a Selenium project with Python) or do not really reflect testing needs in the domain (like a bunch of frontend developer frameworks in the wizard that does not simplify any testing tasks). Perhaps, this is yet to come in the next builds of Aqua, and I’ll try to keep an eye on that.
4. I saw Scala, Ruby and a few other languages in the New Project window. The purpose of adding them is not clear to me if there will be no smart wizard built around them. Having support for many languages is not a primary goal of a test automation IDE that Aqua is trying to be. The primary goal (as for me) is solving testers’ needs, so they can start working faster, writing tests faster and concentrating on the framework architecture.
5. And last but not least: Aqua has other built-it perks like the database client, Docker containers support, some integration with the test management systems. But I consider all of them secondary or auxiliary (perhaps, it’s a better word). Most of the time, test automation engineers are busy coding things, and any IDE first should make that process convenient.

This is it for now. If you liked the article, please share it with friends. Another material is already being written, so stay in tune for future updates! Thanks!