218: Balancing test coverage with test costs - Nicole Tietz-Sokolskaya

Nicole is a software engineer and writer, and recently wrote about the trade-offs we make when deciding which tests to write and how much testing is enough.

We talk about:
  • Balancing schedule vs testing
  • How much testing is the right about of testing
  • Should code coverage be measured and tracked
  • Good refactoring can reduce code coverage
  • Is it worth testing error conditions?
  • Are rare error codes ok to just monitor?
  • API drift and autospec
  • Mitigating risk
  • Deciding what to test and what not to test
  • Focus testing on key money-making features 
  • If there's a bug in this part of the code, how much business impact is there?
  • Performance testing needs to approximately match real world workloads
  • Cost of a service breaking vs the cost of creating, maintaining, and running tests
  • Keeping test suites quick to minimize getting distracted


Sponsored by Mailtrap.io
  • An Email Delivery Platform that developers love. 
  • An email-sending solution with industry-best analytics, SMTP, and email API, SDKs for major programming languages, and 24/7 human support. 
  • Try for Free at MAILTRAP.IO
Sponsored by PyCharm Pro
The Complete pytest Course
  • For the fastest way to learn pytest, go to courses.pythontest.com
  • Whether your new to testing or pytest, or just want to maximize your efficiency and effectiveness when testing.

Creators and Guests

Brian Okken
Brian Okken
Software Engineer, also on Python Bytes and Python People podcasts
Nicole Tietz-Sokolskaya
Nicole Tietz-Sokolskaya
Nicole is a software engineer and writer. She works as a principal software engineer at Remesh where her main responsibilities are performance, security, and backend systems using Python and Rust. She writes frequently about myriad topics on her blog. Outside of computers, she's kept busy with her family and all of life's other responsibilities.
218: Balancing test coverage with test costs -  Nicole Tietz-Sokolskaya
Broadcast by