Issue #230 : Software Testing Notes

When automation needs testing too

Hello there! 👋

Welcome to the 230th edition of Software Testing Notes, a weekly newsletter featuring must-read content on Software Testing. I hope this week has been good for you so far.

There’s a lot of AI and automation in this issue, but the more interesting question is what happens when we stop taking automated results at face value.

Here’s what caught my eye this week:

• What happens when AI tries to design your tests? One experiment finds that generating test cases is easy. Deciding what actually needs testing is where things get interesting.

• Can you trust a passing test? A green result, a broken Cypress session, and an LLM judge that couldn’t say “maybe” all offer good reasons to look a little closer.

• Is your test strategy actually a strategy? Jesper Ottosen has a useful approach for keeping testing connected to business goals without creating another giant process document.

• Performance tests can be the problem. An investigation involving JMeter, Kafka, and Hibernate shows why adding more servers isn’t always the answer.

• Should QA own more security testing? From testing authorization with Postman to connecting Playwright, ZAP, Cursor, and MCP, there are some interesting ideas about where QA’s role could go next.

And more on accessibility evidence, Playwright 1.63, AI and tester jobs, and building AI-powered QA workflows.

Let’s get into it.

---

✨ Featured

500+ testing tools. One place.

Looking for a better tool for your next testing problem?

TestDevTools is a curated collection of 500+ tools and resources for software testers and developers, organized by category so you can find what you need faster. Less searching. More testing.

---

📚 Testing

I Tried Automating the Requirement-to-Test-Case Workflow. Here’s What Broke by Gayathri Bolineni

Generating test cases from requirements is the easy part. Gayathri Bolineni’s experiment gets more interesting when the you have to deal with missing information, edge cases, expected results, and more.

Taming the test strategy dragon by Jesper Ottosen

If your test strategy has turned into a 30-page checklist, it may have forgotten what a strategy is for. Jesper Ottosen explores a simpler approach: use a few principles to connect business objectives to testing decisions, giving teams something they can actually use.

AI and Testers and Jobs by Peter Walen

AI can write test cases, analyse requirements, and automate plenty of the work testers do today. Peter Walen asks the more uncomfortable question: if that is the work you bring to the table, what happens when AI gets good at it? His answer is to become better at questioning assumptions, understanding risk, and finding problems that nobody thought to specify.

My ideal AI QA Workflow — more than test case generation by Adrian How

Generating a test case is hardly the end goal of AI-powered QA. Adrian How sketches out a much more useful workflow where AI moves from Jira requirements to BDD scenarios, repository-aware API automation, test execution, and a merge request, with humans still deciding whether the whole thing makes sense.

---

⚙️ Test Automation

When a Test Passes, How Do We Know We Can Trust It? by Sri Ramya

I have shared many articles in the past about idea of just having green test. Here is a another one, Sri Ramya argues that a PASS should be treated as evidence to investigate, not a certificate that the software works: did the intended scenario actually run, did the assertion test the right behaviour, and do we have enough evidence to believe the result?

My restored Cypress session was lying to me by Marcelo C.

There’s a particularly nasty kind of test failure where the test isn't flaky, your safety net is broken. Marcelo C. found that a weak cy.session() validation was happily accepting an expired authentication session, and the fix comes with a great testing lesson.

Playwright 1.63: Stop Shared Tests Colliding by Alex Khvastovich

Parallel tests are great until two of them decide they both own the same account. Playwright 1.63 introduces test locks, letting conflicting tests wait for shared resources while unrelated tests keep running, alongside some useful improvements to locators, reporting, and accessibility snapshots.

My LLM Judge Never Said “Maybe” - Here’s What That Was Actually Telling Me by Pratham M.

An LLM judge that confidently says “good” or “bad” may be missing the most interesting cases, the ones in between. Pratham M. found that a AI model struggled to recognise borderline results without thinking enabled, and the experiment uncovered a second problem worth worrying about. Read this article to find out.

---

💨 Performance Testing

Performance Engineering in Practice: From Analysis to Optimization

Sometimes the thing failing your performance test isn't the server. This investigation uncovers problems ranging from JMeter authentication and unordered pagination to Kafka partitioning and Hibernate N+1 queries, showing why performance test engineers need to scrutinise the workload and application code before reaching for more infrastructure.

---

🌞 Accessibility Testing

What Evidence Should an Accessibility Audit Report Include?

“Fails WCAG 2.1” isn't particularly useful to the developer who has to fix it. A good accessibility finding needs to explain who is affected, how to reproduce the problem, what criterion it violates, and how to fix it, with retesting evidence showing that the user experience really improved.

---

🔒 Security Testing

Security and Authentication Testing QA Can Actually Own by Julia Pottinger

You don't need to be a security specialist to find some seriously important security bugs. Julia Pottinger shows how QA can use familiar tools such as Postman, Burp Suite, OWASP ZAP, and browser developer tools to probe authorization, sessions, secrets, and injection.

Reframing Quality: Cursor + Playwright + OWASP ZAP + MCP — What If QA is Asked to Own Security Testing? (untested POC) by Ferawati Hartanti Pratiwi

What if your browser tests could double as a security testing pipeline? Ferawati Hartanti Pratiwi demos how you can route traffic with Playwright through OWASP ZAP, send the findings into Cursor via MCP, and turn the resulting analysis into actionable engineering work, an interesting look at where QA and security tooling could start to overlap.

---

🛠️ Resources & Tools

eval-harness — An LLM evaluation harness, built from scratch to bring QA rigor to non-deterministic AI outputs — the first of a three-project, six-month plan moving from AI Quality into agentic GenAI engineering.

LocaleSweep — Localization QA for Flutter — automated. One function call. Every locale, viewport, text scale, and brightness. Screenshots + reports.

---

📝 Software Testing Creators

Do you create content about software testing? Add your blog, website, or newsletter here, and I’ll add it to the list.

---

📨 Have Something to Share?

If you’ve written an article, tutorial, built a useful tool, or found something the testing community should know about, send it my way via DM on X or submit it here.

And if you spot a mistake or have an idea for improving the newsletter, just hit reply.

---

🔎 Other Interesting Stuff

• How to End Your Workday: The Hemingway Principle

• AI Coding will Prevent Expertise

• The Art of Simplifying Decisions.

⭐ Last Week’s Most Read

1. QA in a Vibe-Coded World: What the Discipline Needs Right Now by Joshua Bihun

2. Why Testing AI Agents Is More Conversation Than Code by Gayathri Bolineni

3. Are Test Management Tools Becoming Obsolete? by Emna Ayadi

---

😂 And Finally

Bugs have entered the chat

---

Thanks to everyone for subscribing and reading!

Happy Testing!
Pritesh
@priteshusdadiya