Test Architects Hub

The popularity of different artificial intelligence (AI) tools has been rising daily. AI draws pictures, generates texts, writes music, and reads things aloud – it penetrates into our lives more and more. Whether we want that or not. One such tool, ChatGPT is well-known for its ability to generate different answers and even programming code based on the huge database of texts and its machine learning algorithms.

I decided to give ChatGPT a try and pretended to be a novice QA to see what pieces of advice it is able to give me to start my Junior QA career. The essential goal was to understand whether its answers are too dumb, overly generalized, far from reality or they do make some sense. If this experiment works, I’m gonna consider making it a regular “Talks with AI” series in my blog. Let’s see…

Ohh, it’s been a long time since I was looking for my first job. What are beginners facing these days? Just to break some ice and get the conversation started, I asked ChatGPT a simple but important question:

Well, it does make sense, doesn’t it? ChatGPT’s answer contains high-level steps of what would be needed to start looking for a job in the software testing industry. The items on the list feel a bit generalized and obvious. But okay, acceptable. Let’s see what it’s gonna do with a bit trickier question:

Again, not that far from reality. I know a few cases when people did follow some tips from the above answer and managed to get their first job as a Junior QA. I’m a bit surprised, however, by the absence of crowd-testing platforms (like Utest, Testlio, Test IO, etc.) in the answer. They’ve been quite a good jump-start for many beginners over the past few years.

Now, let’s “brainstorm” some skillset:

Yes and no. The question I asked was assuming the tools and technologies that a Junior QA is required to have to get a job. Instead, ChatGPT compiled a list of 6 random categories and specified popular keywords in each. I wonder how TestComplete ended up in category 2, but the most popular programming language which takes ~31% of the job market (JavaScript) is missing from category 3.

If we challenge the answer: you are highly unlikely to be rejected as a Junior QA applicant, if you don’t know TestRail, Zephyr, TestComplete or even Travis CI. That’s why this answer is a bit misleading, I’d say since it enumerates things that are not essential to become a Junior QA.

Possibly, one of the reasons this answer isn’t really relevant is that ChatGPT’s training data cuts off in 2021, and its historical data about software testing might be even more outdated. It could be completely unaware of the latest trends.

Okay, may I have some deliverables please:

Damn, not bad I’d say (if we turn a blind eye to a bad tools selection)!

It is concise, it is in a logical order, and the Experience section contains good examples of typical Junior QA’s responsibilities. Just fill it in, export it to PDF, write a nice motivational cover letter and start sending straight ahead!

Before I even thought to experiment with ChatGPT, a few friends of mine had posted flattering reviews about this AI tool saying that it can generate nice business emails, cover letters and other template-based forms of communication. Seems to be working!

Alright, the time for easy questions is over:

Honestly, the answer is so generic that I can’t even argue with that. However, I meant other things as “domains” such as web testing, mobile testing, backend testing, etc. Perhaps, I could be more specific in my questions as the AI does not have the same context I do and cannot (?) assume things accurately.

Let’s go to some painful questions:

What a politically correct answer! I think I cracked it – ChatGPT AI is like a psychologist that never gives you a direct answer but rather “speaks” vaguely, righteously, so you cannot really disagree with anything.

What about a slightly provocative question:

I’m now recalling that my first trainer in software testing was saying exactly the same 15 years ago. I suppose the same answer is gonna be still valid 15 years from now. So, false alarm everyone, no need to panic…

Assuming that I got hired as a Junior QA, what do you, ChatGPT, suggest to me in a real situation:

Overall it’s not a bad answer at all as it provides a good breakdown of activities and ideas on where to start. Of course, implementation of those suggestions is up to a Junior QA who’s already supposed to be curious, eager to ask and learn and have a fundamental knowledge of software testing practices. Good for you, AI!

But here is the fly in the ointment: when I asked a more specific question about a typical CI/CD pipeline for a micro-service later, the answer was quite irrelevant. As well as answers to my attempts to clarify the most popular testing frameworks and libraries that might be useful to learn for a QA beginner. The more concrete questions I was asking, the more mediocre answers I was getting. This is probably expected (and I suspect, intentionally programmed).

In the end, I asked ChatGPT to draw me a bug, and here is what happened:

Not that I was expecting to see a masterpiece but it’s clearly a rabbit’s head, not a bug 😀

Conclusions

• apparently, AI cannot answer questions that require concrete answers (like required tools and technologies) well. If asked such questions, it usually provides irrelevant answers that can be misleading sometimes.
• ChatGPT, however, was quite good at answering questions about generic software testing topics like levels of testing or what to do and where to run on your first working day.
• while some of its answers did make sense, the overall answering style looked still too… artificial. I didn’t have high expectations, however. If I were a person that would be starting their career as a Junior QA, such “conversation” with ChatGPT would help to clarify some basic steps and things to do. However, I don’t recommend relying on AI and advise talking to a professional or getting a mentor instead.

That’s it for today, folks! Stay tuned for further articles. The next one is going to be technical.