Test Architects Hub

Among the various activities that test engineers perform, making proof of concepts (PoC) is essential. We create PoCs when there’s uncertainty about which process to follow, what tool or framework to use, or even when it’s an excellent time to try something new.

Although it may appear spontaneous and ad hoc, creating a PoC is actually an important activity that can influence company-wide decisions. If not executed properly and important points aren’t taken into account, teams might start using inefficient tools or frameworks and following processes that don’t solve their problems.

In this article, I’ll attempt to systematize the process of creating a PoC for selecting a testing tool or framework.

Table of contents:

What are the goals of PoC?

A proper proof of concept should have a goal. A PoC without a goal is simply a dubiously efficient use of time, playing with a tool without a challenging context. The outcome of such a PoC can still provide insights, but mostly in terms of a broad outlook.

Over the years, I was able to identify several main goals of a PoC:

• Proving that a chosen tool/framework can solve a specific problem or challenge;
• Assessing how well a selected tool/framework fits into the existing workflow and identifying necessary adjustments;
• Identifying knowledge gaps and learning needs for test engineers who will work with the tool/framework;
• Determining the cost efficiency of using a tool/framework, also known as the Return on Investment (ROI).

Ideally, at the end of a PoC, all the above-mentioned goals should be achieved, and you or other decision-makers should have enough insights to make a “go/no-go” decision.

It’s possible that a chosen tool may fail to fulfil some of the goals. In such a scenario, you must evaluate whether any trade-off can be made with the tool or consider switching to an alternative option.

Stages of making a PoC

1. Defining the problem/challenge you are trying to solve

Usually, the fact of thinking about making a proof of concept means we’re not completely happy about something we currently have. Not happy to the extent that we start seeking a potential solution and don’t mind giving it a try.

Having a well-described challenge, in this case, helps you and everyone around you understand the context of PoC and its potential outcome. In the end, this information will be essential for determining whether a PoC was successful or not.

For instance, you are unsatisfied with the end-2-end test coverage of a web application your team is developing and you want to introduce automated UI testing. A typical challenge would be to find a framework that is easy to set up, minimizes test flakiness and allows tests to run seamlessly on different browsers with minimal maintenance.

2. Researching available solutions in the market

Good, if you have a potential solution in mind or have heard about one. Often, however, it’s not the case, and some research has to be done to gather information about potential tools and frameworks to be tried in proof of concept.

The good news is that there is a very high chance someone else has already faced the same or similar challenge, come up with a decent solution and even posted an article about that. So, don’t hesitate to spend a day or two surfing the internet for other people’s experiences and researching available solutions on the market.

There are various testing tools and frameworks available there: paid, pay-per-usage, open-source/free, etc. It is highly likely there is a good tool among them that can partially or completely solve your problem.

However, be cautious of sponsored articles or blogs from tool vendors where their products are usually praised and presented as an all-in-one solution for any occasion. I have nothing against paid tools but advocate against buying a pig in a poke, without a proper assessment.

The objective of this stage is to select a few top options for further evaluation and eliminate those variants that are clearly unsuitable.

3. Comparing the selected options and choosing one to do PoC with

If at this point you have selected several alternatives, it’s time to compare them and choose one to proceed with. There can be various criteria for comparison, and they may also vary depending on your specific situation. In the following sections of this article, I will share the most generic ones that I was able to identify through the years and still use nowadays.

The objective of this stage is to narrow down the available options and select the most efficient one to proceed with for the PoC. The second-best option then serves as a backup in case something goes terribly wrong, and the defined goals of the PoC cannot be achieved. That can happen, for example, if a chosen tool fails to solve your problem or proves to be way too expensive, so the game isn’t worth the candle.

4. Defining the scope of PoC (what should be done)

Writing a few tests using a new tool or a framework doesn’t really showcase enough. Well, except maybe the case when you are selecting an assertion library for your tests. Although I’d question if it’s worthy of a whole PoC.

The closer a PoC aligns with a real testing workflow and conditions, the higher the chance to identify something that won’t work for you. Taking our UI tests example, while it may be really enjoyable to write locators with a new framework, people might still face challenges when executing their tests remotely and lack insights in the report to debug and troubleshoot failures efficiently.

Defining the scope of a proof of concept helps to ensure you don’t overlook anything essential for making a decision. To simplify this process, I usually describe the scope in the form of user stories:

• As an engineer, I’d like to write functional tests for [feature/page name] that cover happy and unhappy flows;
• I’d like these tests to be integrated into my team’s CI pipeline and executed remotely on our cloud infrastructure;
• I’d like these tests to be run in a headless mode but still provide detailed insights about test execution, enabling the team to rewind and understand what’s wrong.

And so on…

5. Making a PoC and documenting findings, key results and observations 

As wise people say, “Good planning is halfway to success”. At this point, you should have everything prepared, and it is now time to proceed with the proof of concept. 

While working on a PoC may seem similar to your daily routine in many cases, it is important to take notes and document your observations, even if everything seems to be clear. Time flies, and people tend to forget things, but these notes will always be at hand for your own or other decision-makers’ reference.

I usually recommend taking one step forward and wrapping notes into a form of a page on your internal wiki portal. Along with making a PoC, observations on such pages become documents that can later be reviewed and discussed.

6. Presenting to a team and gathering feedback

While a PoC can be made alone or by a few colleagues, reviewing its results with a group of interested people is always a good idea. Such reviewers can give valuable feedback and point at nuances or use cases you could have missed.

I’ve seen situations when reviewing a PoC by a group of peers highlighted critical issues, and people even had to redo it using a totally different tool than was initially planned. However, you shouldn’t be afraid of presenting PoC results to the team. Think of it as a “quality gate” that only robust and suitable tools/frameworks can pass through.

7. Fixing comments and making a “go/no-go” decision  

If reviewers’ comments have been addressed and the goals of the proof of concept have been achieved, it’s time to wrap up and make a decision.

Here are a few questions that I ask myself at this stage:

• Did the tool solve my problem? Did it introduce new issues?
• Did I need to implement workarounds or hacks to make the tool work for me?
• Are my tests now more robust or more flaky?
• Can other engineers use this framework efficiently? Can it be reused by other teams?
• Does it seamlessly integrate into our CI and testing pipeline?
• Can we afford the tool or is it too expensive and we need to explore alternatives?

Document the answers to these questions and make a decision accordingly. If the decision is positive, the tool/framework can proceed to the trial phase, and the team (or teams) can start using it in their daily work.

If a decision is negative, the backup option can be re-considered then, taking into account the insights discovered during the PoC. It is also possible that the proof of concept has revealed aspects that weren’t thought about before. In such cases, the process may need to be restarted from stage 2 (or even 1) with adjusted requirements and a new scope for the PoC.

Criteria for assessing a tool/framework

As mentioned earlier (in stage 3), comparing selected testing tools/frameworks is a crucial part of a PoC. Below, I listed the main criteria that I use on a regular basis:

1. The convenience of using a tool/framework. If a tool tends to solve problems but has a steep learning curve, causes frustration during usage, and maintenance of it becomes hell, this tool might not be the right choice for you. A proper tool/framework should have an understandable API and/or intuitive controls and offer a convenient way to do routine actions (write tests, build locators, assert results, etc.).
2. Help in reducing the flakiness of tests. Introducing a tool or a framework that will make tests (more) brittle, and prone to flakiness is a very questionable idea. That is why, during a PoC we should seek tools that enhance the robustness of our test frameworks, or (in the worst-case scenario) don’t exacerbate their flakiness.
3. CI & cloud friendliness. I’ve seen several situations when people made PoCs, got extremely excited about their new instrument and then had to implement lots of dirty hacks and workarounds to make that tool work in their CI pipeline properly. Unfortunately, we can’t afford that “luxury”, that’s why assessing how well your tool fits into the existing pipeline is a must. Luckily, modern frameworks offer integrations with many popular CI systems and cloud services.
4. The convenience of debugging/troubleshooting failed tests. It is important to assess how your tool helps when something goes wrong. The most obvious example is a “red” pipeline with failed tests. How does a tool help to understand what failed then? Is there any reporting or logging available? What data is logged and how can it be accessed? Does it help a team to troubleshoot faster? All these questions are important to answer when doing a PoC.
5. Return on Investment (ROI). Introducing a new tool or approach almost always incurs costs, whether in the form of time, resources or financial investments. If a tool is free and open-source, people still might need time to acquire a new skill or complete extra training. Additional support from the company’s DevOps engineers might be necessary, and their time also costs money. In the case of a paid tool, on top of the price, it might also mean a long-time financial commitment for your company. All these cases must be considered, and after all, the benefit derived from using a tool/framework should outweigh the associated costs.
6. Compliance requirements. They are important considerations in certain cases when evaluating a tool/framework. These requirements may include aspects such as the storage and retention of (test) data, collection of usage statistics, adherence to GDPR compliance, and more. While this criterion is more relevant for B2B and SAAS testing products, it still should be taken into account when assessing even free open-source tools to avoid any potential reputational damage in future. By evaluating compliance requirements, you can ensure that the chosen tool or framework aligns with your organisation’s data protection and privacy standards.
7. And last but not least, Security. Over the years, I’ve been trying to advocate for the following principle – any new introduction should not degrade your organisation’s security robustness. That’s why paying attention to the tool’s security hardening must be one of the top priorities when doing a selection. Ideally, good PoC candidates should implement best security practices, support the same authentication methods used by your company, offer access roles management and show commitment to fixing security vulnerabilities quickly.

Of course, the list provided above is not exhaustive, and you might have your own (extra) assessment criteria specific to your situation and requirements. However, it serves as a solid foundation to start with and then dig deeper, tailor and expand it.

“Don’ts” when making a PoC

As in most things in our life, it’s possible to make mistakes here as well. If it happens, we should not be afraid of failure – it is important to learn from it and move on. But if given the opportunity, it is better to learn from the mistakes of others, benefiting from their experiences and insights.

In this chapter, I have compiled some of the most obvious “don’ts” that you should avoid when making a PoC:

• Don’t start without a clear understanding of the problem you’re trying to solve. If in doubt, align with colleagues. Alignment fosters clarity. Misalignment creates false expectations.
• Don’t assume that people around will share the same level of excitement about yet another new framework. Try to remain unbiased, be open-minded and document well.
• Don’t blindly trust what tool vendors say on their websites. Even if we put aside the commercial part, none of the vendors knows your exact challenge.
• Don’t spend ages doing a PoC. If you struggle, your fellow colleagues will most likely struggle to use it as well. Revise what was wrong and move on. Ask a teammate, when in doubt.
• Don’t over-rely on someone’s pet open-source projects. They might become abandoned and you’ll be left on your own. Always evaluate this risk and its impact.

Some final words (instead of conclusions)

As we’ve seen, proof of concepts can be quite comprehensive activities on different organizational levels: within a single team, affecting multiple teams or even an entire company. Sometimes, making a PoC may be tricky, and it’s easy to miss some important key points.

I wanted to streamline this process a bit and make it more structured, and I hope I was able to achieve it in today’s article. This is not the end, however, and other materials about PoCs are yet to come. Stay in touch to make sure you won’t miss any of them!

Cheers!