You must be signed in to see that page
Discuss Agilepublic

Agile Product Management

A place for product managers to learn and share tips on working with agile development teams

This is a public Discussion Area  publicRSS

Question

    Dealing with massive QA effort
    Question posted 12/11/07 by Catherine Connor , tagged Challenges
    2011 Views, 10 Comments
    Title:
    Dealing with massive QA effort
    Question:

    1.     QA is the largest part of our typical dev cycle because the product is massive. Even only adding a small number of features requires testing the entire application which normally takes months. How do we get around that?

    Comments

    • posted 12/11/07 by Catherine Connor
      The only way to get around that is to automate testing - from unit testing, to component testing, to system testing, and do continuous system builds that create a full build of all integrated components on a regular basis. It is a hard task for an older product  where developers who wrote the code may no longer be there to write unit tests to test their code, but this is the ticket to adapting QA testing practices to agile.

      Reply to this Comment

      Attach File

    • posted 12/13/07 by Bob Cotton
      I would agree with Catherine here. Automation is the key.

      Here at Rally we leverage test automation heavily to cut down on our manual testing load. New features are manually tested, but we leverage automation for the rest.

      Its never too late to start. Any automation you do today is less manual testing you have to do tomorrow.

      Reply to this Comment

      Attach File

    • posted 12/18/07 by DanNeumann
      Bob, it sounds like you're saying that within the iteration that a new feature is built, automated test scripts are created. There is some manual testing of the new feature as it is developed, but the acceptance criteria would mandate that the feature have an automated set of tests before exiting the iteration. Is that right? 

      Reply to this Comment

      Attach File

    • posted 12/18/07 by Bob Cotton
      That's correct.

      During iteration planning, the team decides which acceptance criteria needs to be automated (through the GUI). Some things can stand with just a unit test. Others are far to expensive to automation, and those get added to the manual tests. The team (product owners, developers and testers) make this decision together.

      Either way there is always exploratory testing that gets done prior to acceptance.

      Reply to this Comment

      Attach File

    • posted 12/18/07 by Richard Leavitt
      Bob - Two questions I'd love to get your answers to show us an example:

      Q1: can you describe the number of tests, of various types, currently in use at Rally and how many are added by a typical scrum team each release (e.g. Unit Tests with JUnit and ?, Functional Tests with Fitnesse, GUI Tests with ?, Performance/Load Testing, Manual Tests, etc) ?

      Q2: Of the entire active test suite, what % of tests are currently manual and how often do you execute them?

      Reply to this Comment

      Attach File

    • posted 12/20/07 by Craig Langenfeld
      This is a great conversation and always a hot topic when we provide Rally Tool Implementations. I have encouraged many of the teams that I've recently worked with to join in.

      Bob, it would be very helpful if you could answer Richard's questions.

      Reply to this Comment

      Attach File

    • posted 12/20/07 by pete skrebetz

      Automating tests is 'the right thing' to do, but reality shows that the number of automated tests is always significantly smaller than the number of manual tests.   Tracking the execution effort is another challenge in itself.  Getting a clear picture of how many tests have been executed against the requirements and how much of each requirement has been tested continues to be a challenge for us. 

      Reply to this Comment

      Attach File

    • posted 12/20/07 by Tom Burket
      With respect to Rally, we have commented to company representatives that we have a significant challenge in Rallly with the "massive QA effort" (to use the initial phrase at the start of the thread) and the regression testing from one project to another and/or one release to another. That is, if we have hundreds of test cases developed in a project, what is the way to manage them and run them during the next release? There are many sub-topics under that general theme, independently of whether the tests are automated or manual.

      Reply to this Comment

      Attach File

    • posted 12/20/07 by Richard Leavitt
      Tom is right on about the multiple facets of managing test initiatives. Rally is continuing to make investments in Test Management and connectors to commercial and open source testing tools.   We believe we can make a significant contribution to quality management over the coming year, both manual and automated.  We will start to share the items currently under development in our Backlog area.  MEanwhile, there are several discussion threads going on with Rally users,including scheduling tests and reporting results and the addition of test suites .  Helping manage and report on the software build process and the tests they run is also getting a lot of attention.  We're looking forward to rolling the highest priority features out through the next few releases.

      Reply to this Comment

      Attach File

    • posted 2/28/08 by Pres Hatt
      In QA the metrics are always important. I'm in the process of becoming a Rally Business Partner but here are some numbers that may be of interest to the hive. At an eCommerce site ~13 QA people created fully functional WinRunner tests for a complete redesign of the site. ~600 automated tests were ready before the first code drop. Each automated test case was "monitored" meaning the execution and output was manually verified then the test case was certified as "Runs Once" and put in the regression suite. From first code drop the completely new site was pushed live in ~10 days. Within 3 months there were 3 distict sites using the same code base and ~3000 fully automated regression tests run over night (within hours). [next] A start-up company had 8 people in India and 5 people in CA create and maintain ~1200 WinRunner tests for daily regression testing of a ~500 page J2EE site. Over ~8 months the site went through 2 complete UI redesigns and we ported all the tests from each UI design to the other. [next] An individual contributor created WinRunner regression tests for a fairly stable existing eCommerce site starting with 0 automated tests and few poor manual tests. Over a 2 year period, created ~2000 tests all of which ran in WinRunner automation mode. ~600 of those tests were in the regression suite which ran under TestDirector on 1 WinRunner machine in ~4 hours. My point is ... with the proper tools most of the UI testing can be done with automation ... more importantly Quality can not be tested into a product. All of the functionality of Rally assists the QA people do the most important QA job ... deciding what to test with the close cooperation of development and the business proponent. These numbers should resonate with the other comments about automation being an important part of the QA process.  :-) 

      Reply to this Comment

      Attach File