Hello there! 👋
Welcome to the 228th edition of Software Testing Notes, a weekly newsletter featuring must-read content on Software Testing. I hope this week has been good for you so far.
AI is making it easier to generate code and tests, but that also raises a harder question: are we testing the right things? This week, we look at what green tests can miss, why our assumptions need testing too, and how testing can tell us more about the behaviour and business impact of the systems we build.
📚 Testing
GIGO: We Still Need To Test Our Assumptions by Paul Grizzaffi
AI coding tools can turn bad assumptions into bad software remarkably quickly, so it’s worth testing the requirements before handing them to an agent. Paul Grizzaffi argues that the important question isn’t whether AI understands the ticket, but whether the ticket describes the right behaviour in the first place.
Stop Reporting Test Activity: Show the Business Impact of Testing Instead by Richard Forjoe
Test counts and execution numbers show how much testing was done, but not whether it made a difference to the business. Richard Forjoe explains how Business Impact Reporting connects testing to business goals, risks, outcomes, and the cost of poor quality.
Apparently We Need a Testing Mindset After All by Keith Klain
It’s difficult to objectively judge something you created yourself, and AI systems can have the same problem. Keith Klain looks at problems with LLM-as-a-Judge systems, including self-preference, reliability, and bias when the model creating something is closely related to the model judging it.
What Is the Right Question a QA Should Ask During Refinement — Especially in the AI Era by Marina Jordão
“What happens when things go wrong?” is a simple question that can uncover missing requirements, users, dependencies, and unclear definitions of done. Marina Jordão explains why this matters even more when AI coding agents can quickly build exactly what a ticket says, including the things nobody meant to specify.
Post-Release Data Validation: A New Type of Testing? by Sergei Terentev
Production data can reveal problems that application tests and database checks never catch, especially in complex legacy systems. Sergei Terentev shows how simple, read-only SQL checks can find duplicates, broken relationships, unexpected values, and unusual record counts.
🔍 Software Testing
⚙️ Automation
Testability Is a Feature. Does Your Code Agent Know About It? by Gil Zilberfeld
AI coding agents can write working software without making it easy to test or troubleshoot. Gil Zilberfeld argues that testability should be part of the requirements, with clear system structure, useful logs, and enough information to understand what the code is doing.
Playwright UI Mode: Time Travel Debugging in TypeScript by Pramod Dutta
Debugging browser tests often means going through logs, screenshots, and repeated test runs to figure out what happened. Pramod Dutta shows how Playwright UI Mode lets you step through the test, inspect the page and network activity, pick locators, and see failures as they happen.
How to Test System Clipboard in Playwright (Without Boilerplate) by Sergey Oleksenko
Testing the system clipboard in Playwright can get messy once permissions, async updates, and JSON parsing are involved. Sergey Oleksenko shows how a reusable clipboard fixture can handle that setup and make the tests easier to write and read.
Why I Built TestFly: A Java SDK That Doesn’t Hide Selenium by Hakan GUL
Selenium tests can spend more time dealing with waits, retries, screenshots, reporting, and driver setup than describing what the application should actually do. Hakan GUL built TestFly as a simpler Java API around Selenium that handles much of this infrastructure while still exposing the underlying WebDriver when you need it.
Your Regression Suite Passed - So Why Did the System Behave Differently? by Mohit Mongia
A regression suite can pass thousands of tests while the system slowly starts behaving differently from before. Mohit Mongia explains why checking whether tests passed isn’t enough, and why teams also need to check whether the system is behaving as expected.
Mobile App Testing Like a Human using Gen AI by Amrit Bhardwaj
Appium and Selenium tests can become expensive to maintain as mobile and web UIs change. Amrit Bhardwaj describes a goal-driven LLM testing system that uses the current UI, past interactions, feedback, and loop detection to decide what to do next.
Your Test Suite Shouldn’t Break Because Someone Renamed a Div by Surya Kulshreshtha
UI tests become fragile when selectors depend too much on how the page is built rather than what users see and do. Surya Kulshreshtha describes a layered approach that starts with more reliable selectors such as test IDs and ARIA roles.
🔍 Test Automation
💨 Performance
The Day I Performance Tested a Video Stream and Realized It Wasn’t About the Video by Priya Raman
Performance testing a video streaming migration doesn’t mean sending traffic to every request made by the browser. Priya Raman explains how understanding the system architecture helps you focus on the APIs that actually matter, such as authentication, token generation, metadata, and application integration.
🔍 Performance Testing
🌞 Accessibility
WCAG 2.3.1 Flash Test: How to Test Websites for Seizure Hazards
Some accessibility problems only appear when users interact with dynamic content, making them easy for automated checks to miss. This approach records the browser while triggering animations and interactive states, then measures luminance and red flashes to check against WCAG requirements.
🔍 Accessibility Testing
🔒 Security
I fault-injected two AI agent frameworks. One recovered — the other charged the card and said 'done' by Ashwin Govind Ugale
The same AI agent can handle a failed tool call correctly in one framework and charge a customer anyway in another, even with the same model and prompt. Ashwin Govind Ugale tested payment failures in two agent frameworks and found that one continued incorrectly 22% of the time when it received a requires_action status, while the other recovered every time.
🔍 Security Testing
🛠️ Resources & Tools
TestFly — Java Test Automation SDK — web, API, mobile, and AI/MCP-powered test automation without hiding Selenium
ForkablePlaywrightSelfHealer — Self-healing Playwright tests in Python. Plain English to UI + API tests that repair their own locators. Heuristic-first, LLM-assisted, fully offline.
📝 List of Software Testers
Do you create content around Software Testing ? Submit yours blog details here and I will add it to the list.
🎁 Bonus Content
📌 OTHER INTERESTING STUFF
⭐ LAST WEEK’S MOST READ
Implementing Page Objects in Playwright by T.J. - Maher, Jr.
More about the testing, less about testers by Jesper Ottosen
😂 And Finally,
AI: 1000 lines of code in seconds. QA: 1000 reasons it won't work in production 🤣
📨 Send Me Your Articles, Tutorials, Tools!
Wrote something? Send links via Direct Message on Twitter @thetestingkit (details here). If you have any suggestions for improvement or corrections, feel free to reply to this email.
Thanks to everyone for subscribing and reading!
Happy Testing!
Pritesh(@priteshusdadiya)