Hello there! 👋
Welcome to 232nd edition edition of Software Testing Notes. This week, there is a common thread across many of the things I’ve been reading: AI can generate tests, but it still does not understand everything that makes a test valuable.
We look at what happens when AI agents are tested in the real world, what QA should focus on when AI writes the scripts, and why a green test can still miss the problem that matters.
A few other things caught my attention too:
Why visual bugs can slip past perfectly green functional tests.
Why
Sleep(5)might be one of the most expensive lines in your test suite.How Playwright fixtures can make complex automation easier to manage.
What accessibility testing can reveal that automated checks cannot.
Why certificate pinning creates some interesting security scenarios for QA.
There is a lot to explore this week, so grab a coffee, pick a few links that catch your eye, and dig in. I hope you find something that makes you rethink how you test, automate, or build software.
✨ Featured
Turn the Web Into A Data Pipeline. Generate Your API, in Minutes.
If your team is still patching selectors and rotating proxies, you’re maintaining infrastructure your competitors stopped owning.
Scraper Studio - from Bright Data turns any public website into a hosted data API from a single prompt:
Self-healing scrapers that detect site changes and patch themselves
Proxies, unblocking, and parsing handled - not your problem
Build from Claude Code, Cursor, or Codex via CLI - or visually in the control panel
Scale production data pipelines without owning the stack
5,000 free credits every month
No sales call. No demo. Just spin up a scraper and watch it run.
📚 Testing
Turns Out Testing is Hard by Keith Klain
AI agents can pass plenty of tests and still fail when their output meets the real world. Keith Klain looks at many open-source LLM agent projects and shows why testing needs to cover the whole chain, from model output to tools, APIs, permissions, state, and external services.
If developers have autonomy, mastery and purpose, what do you need testers for? by Joep Schuurkes
If developers can test their own code, what exactly is the tester's role? Joep Schuurkes makes the case for treating testing as an activity rather than a job title, while exploring where testers add perspective, experience, and critical distance that development teams can otherwise lose.
The Visual Bugs Hiding in Plain Sight by Parnika Zade
A page can pass every functional check and still look broken to the person using it. Parnika Zade explores the visual regressions that hide across browsers, devices, screen sizes, and checkout flows, and why automation alone cannot tell you what the customer actually sees.
What Should QA Do While AI Writes The Tests? by Nirawit Best
AI can write a test in seconds, but it still cannot decide which failures matter most to your product. Nirawit Best explores how QA can shift from writing every test to using product knowledge, production history, and business risk to decide what AI should actually test.
⚙️ Automation Testing
How well do agents use test/verification techniques? by Dan Luu
Tell an AI coding agent to use TDD, fuzzing, property-based testing, or formal verification and you might expect better code. Dan Luu's experiments suggest the opposite can happen, with agents often applying these techniques superficially or incorrectly and sometimes doing better without the extra instructions.
As a QA Engineer, How Do You Decide What to Automate?
More automated tests do not automatically mean better testing. This guide looks at business risk, frequency, stability, repeatability, and maintenance cost to help QA teams decide which tests deserve automation and which ones are better left alone.
Advanced Playwright Fixtures: Worker-Scoped, Parameterized, and Auto-Cleanup Patterns by Pramod Dutta
Playwright fixtures become much more interesting when your tests need shared resources, multiple roles, and automatic cleanup. Pramod Dutta shows how worker-scoped and parameterized fixtures can manage database connections, authentication, test data, and other setup without turning your test suite into a maintenance problem.
The Most Expensive Line in Your Test Suite Is Sleep(5) by Muskan Agarwal
Sleep(5) looks harmless until you multiply those five seconds across thousands of tests. Muskan Agarwal explains why fixed waits make suites slower and flaky, and how condition-based polling can wait for the application to actually reach the state your test needs.
Parameterized tests and test organization with Swift Testing by Natascha Fadeeva
Writing the same test again and again for different inputs gets old quickly. Natascha Fadeeva shows how Swift Testing's parameterized tests, suites, tags, and Cartesian arguments can keep large test suites compact while still giving each input its own useful failure result.
Get Started with UI Path Mobile Automation by Bibhu D
Bibhu D walks through connecting UiPath Studio, Mobile Device Manager, Appium, and BrowserStack to run Android and iOS automation on cloud devices.
🌞 Accessibility Testing
Why TalkBack keeps switching accents by Craig Abbott
Ever heard TalkBack suddenly switch from one English accent to another? Craig Abbott digs into how Chrome's language detection can override a generic lang="en" and shows why regional language tags such as en-GB and en-US matter, along with a simple reminder that automated checks will not catch every screen reader problem.
How many pages is enough in an accessibility audit? by Karl Groves
How many pages do you really need to test in a huge website? Karl Groves challenges the idea that accessibility audits should be driven by page counts and explains how sampling templates, components, workflows, states, and high-risk functionality can give you a much more meaningful picture.
🔒 Security Testing
When a Green Test Protects the Wrong Authorization Rule by Ivan Rossouw
A green security test can still be protecting the wrong thing. Ivan Rossouw shows how Blazor authorization tests can enforce the intended permission boundaries, with structural checks that catch missing policies and prevent old exceptions from quietly becoming security gaps.
Certificate Pinning - The MOST Important Security Measure for Network Traffic In Your App by Philipp Lackner
Certificate pinning changes what an Android app is willing to trust on the network, but that also creates new things for QA to break. Philipp Lackner explains how pinning works and what happens when certificates, keys, or pins change, giving testers concrete scenarios around invalid certificates, expired pins, and man-in-the-middle attacks..
🛠️ Resources & Tools
AutoMancer — Bring older Windows applications back to life. AutoMancer enables modern testing patterns for UI automation with a C# class library and CLI.
Convoy — Semantic end-to-end agent testing for iOS, Android, and web. You describe what should happen in plain language. Convoy dumps the live accessibility tree, normalizes it to a platform-agnostic element table, asks Jev which control matches the intent, and taps it.
📝 Software Testing Creators
Do you create content about software testing? Add your blog, website, or newsletter here, and I’ll add it to the list.
📨 Have Something to Share?
If you’ve written an article, tutorial, built a useful tool, or found something the testing community should know about, send it my way via DM on X or submit it here.
And if you spot a mistake or have an idea for improving the newsletter, just hit reply.
🔎 Other Interesting Stuff
⭐ Last Week’s Most Read
Built My Playwright Framework for AI — Not Just With AI by Dipen Chavan
End-to-end (UI) Test Automation Stats that Matter by Courtney Zhan
Your Regression Suite Is a Graveyard and You're Still Adding Headstones by Nahla Davies
😂 And Finally
World will never not need Testers 😄
Thanks to everyone for subscribing and reading!
Happy Testing!
Pritesh
@priteshusdadiya