October 17, 2010
Cost of Testing – a critical way of measuring project success
Testing is tradationally a quality control activity i.e. we start testing after almost 75-80% coding has been completed. However a big difference between the manufactuting industry and the IT industry is that Quality Control is statistical in nature and only a fraction of the manufactured goods are verified against standards – in the IT industry we do check almost 100% of the code for both completeness and correctness. As such testing costs spiral upwards and it also adds to the delay in time to market. As such most testing organizations or Testing Center of Excellence teams in IT companies are seen more as necessary evils rather than value adding partners. To change this perception it is very important to ascertain the cost of testing accurately and measuring the Testing Cost.
The million dollar question then is what should be accounted as activities building to the cost of testing. And how to keep reducing the same continuously keeping the quality output unaffected. Here is a bit of a brain dump
Methodology to calculate cost of testing (CoT)
- Define the scope for which CoT is to be calculated. This needs to be either a release , a time period (in case of operations or maintainence projects) or an iteration. The reason why we need defined start and end points is that we need an accurate input of the development size and effort for which testing is being conducted.
- Estimate the size of the testing required for the scope mentioned above. – Es
- Test management (This includes activites like Test strategy preparation, preparing test plans and resource management) – Tm
Test Analysis ( This the analysis of the business requirements to prepare of test scripting) – Ta
Test Scripting (This includes writing the test scenarios and test cases) -Ts
Data Setup (Time required to setup the test environment) -Ds
Test Execution =Te
Automation scripting (any time ivested in automating regression test scripts) -Ta
- List down the hardware and software costs
Cost of the test environment – Hc
Cost of software for test management (in case applicable) – Sc
Disk space usage for test artifact storage – and covert to storage cost Dc
- List out the defects (functional only) ideintified during the test pahse. -Q
- Test Effort = (Tm + Ts + Te + Ta)
Test Effort Cost = (Tm + Ts + Te + Ta) X Average Daily rate/cost of resources.
Infrastructure Costs = (Hc + Sc + Dc)
Total Cost of Testing = Test Effort Cost + Test Infrastructure Cost
Cost of Testing (CoT) = Total Cost of Testing / Development SizeThis now gives us an indicator of the cost of verifying and validating per unit of development size. There are no industry bench marks available but based on my experience and chats with other industry collegues in a typicall offshore environment this should be around $20 – $30 per function point.
- Another interesting metric is to find out the defect detection cost (DDc)
- Once we have these metrics – we need to publish the same every month and try to reduce the CoT and DDC by improving our test scripting productivity – this measure is arrived at by
Test Script productivity = No of Test Cases / No of Defects
What this does is focus the tester on writing test cases which will capture defects. The focus of testing now changes from completeness to correctness. This can be a starting point of changing the perception of Testing from a “Necessary Evil” to a “Value Added Partner”