Saturday, September 5, 2009

State-of-Art Testing

State of the Art Testing : Software testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required result. Software testing is an art of observation, experience, experiment conducted to provide stakeholders with information about quality of the product or service under test, basically AUT. This art also provides an independent view of the software to allow the business to appreciate and understand the risks at implementation of the software or service.


What does State of the Art Testing comprise of ?
It comprises many important aspects relevant to the guidelines, e.g.:
  • Test Management (aka - Planning and Organization)
  • Process models for testing - Unit, Integration, System, and Acceptance
  • Testing Phases
  • Methods for test case design
  • Testing from the point of view of the user/customer
  • Sequence testing procedure
  • Testing real-time aspects
  • Regression testing
  • Auomation Test Tools
Test Management (aka - Planning and Organization): As the name suggests test management comprises of test case management. It often includes Requirements documents and functional specifications which help in generating Requirement test matrix. Basically it provides information about how much of requirement and specifications are covered by available test cases.
First step of test management is to define the Test plan in association with product requirements and functional specifications. Test plan is followed by designing and developing test cases or use cases. Each test case or use case should have a priority defined. Every test change must be versioned to get a comprehensive view of overall history of a particular test case or use case. Test management's ultimate goal should be able to generate metrics or reports displaying various data related to quality assurance of the given product.


Process models for testing - Unit, Integration, System, and Acceptance: Let us understand what is test process. It comprises of planning, specification, execution, recording and checking for completion. Why improve test process. To decrease cost and time of testing and to increase quality of product. Software test model includes test designing, test plan, test case generation, test execution, test result analysis and test reuse. These are laid out in the whole life cycle of software development process. Using software test models improves reliability and facilitates reliability measurements. These are divided as follows:


Process Improvement Models: For example: Six Sigma, TQM


Software Process Improvement Models: For example: Bootstrap, CMM(I), SPICE

Test Process Improvement Models: For example: TPI, TIM, TMM, Test certification standards. 

In counterpoint, emerging software disciplines like extreme programming, Agile software development model, test-driven software development model are more popular now a days in web application testing. In these test process models, unit tests are written first. Testing is done at following levels:
  • Unit Testing: Tests minimal software component or module. Each unit is tested to verify that the detailed design for Unit has been correctly implemented. In an Oops environment, this is usually at class level and minimal unit test includes constructors and destructor. 
  • Integration Testing: Exposes defects in interfaces and interaction between integrated components or modules.
  • System Testing: Tests a completely integrated system to verify that it does meet the requirements. Acceptance Testing: Finally an acceptance testing is conducted from an end user's perspective to validate whether or not to accept the product.


Testing Phases and Testing Methods: Unit Testing, Integration Testing, System Testing (explained above) are few of the testing phases including ones mentioned below:
  • Alpha Acceptance Testing: Try it out with in-house “users”
  • Installation Testing: Can users install it and does it work in their environment?
  • Beta Acceptance Testing: Try it out with real users
Quality assurance testing methods are usually divided into Black box and White Box testing methods.

Black box testing means testing product without any knowledge of internal implementation and includes cross-browser compatibility testing, exploratory testing, boundary value testing, pair testing, model-based testing, specification or requirement based testing.

White box testing means testing with knowledge and access of internal implementations,algorithms including code implementation and includes API testing, Code Coverage, Static testing and code completion evaluation testing.
Methods For Test Case Design: Design of test case or use case plays pivotal role in ensuring higher quality of product. These methods provide developers with a systematic approach to testing more important, method provide mechanism that can help to ensure completeness of tests and provide the highest likelihood for uncovering errors in software. Again its divided into Black box test case design and white box test case design.
Black box test case design: Usually these are designed based upon requirements or specifications documents and at software interface levels.

White box test case design: These are used to detect errors by means of execution-oriented test cases. In white box testing strategy QA investigates the internal structure of the object to be assessed in order to specify execution-oriented test cases on the basis of the program logic. In this connection the specifications have to be taken into consideration, though. In a test case design, the portion of the assessed object which is addressed by the test cases is taken into consideration. The considered aspect may be a path, a statement, a branch, and a condition. The test cases are selected in such a manner that the correspondingly addressed portion of the assessed object is increased.

An open source tool TestLink can be used to design these test cases. More information about Test Link can be found here.

Testing From The Point Of View Of The User/Customer: A well-performing applications is one that lets the end user carry out a given task without undue perceived delay or irritations and process of ensuring this referred as Testing from an end user's perspective. It includes, identifying and escalating most common operations or transactions performed by end user. Various ways are available to find out most common operations, end user's feedback, evaluation about a particular feature or product. 
Quality assurance testers needs to think and act like end user, based on requirements or specifications documents, regularly meeting with field representatives, product managers also help contribute to test like an end user. Also it is important to have domain expertise for given product. Also quality assurance team needs to have expertise on internal implementation and overall architecture of product to think from an end user's perspective. With a performance application users are never greeted with a blank screen durin logons and can accomplish what they set up to accomplish without letting their attention wander. Casual visitors browsing a web application or web site can find what they are looking for purchase it without experiencing too much frustration. Do your own analysis by asking questions to yourself like would your potential customer's life be improved with your product or service? Does your customer find a certain product or service inferior?

Sequence Testing Procedure: As per web definition squence testing procedure means specific order of related actions or steps that comprise a test procedure or test run. Dynamic test sequence compaction is an effective means of reducing test application time and often results in higher fault coverages and reduced test generation time as well. Three simulation-based techniques for dynamic compaction of test sequences are described. The first technique uses a fault simulator to remove test vectors from the test sequence generated by a test generator if the vectors are not needed to detect the target fault, considering that the circuit state may be known. The second technique uses genetic algorithms to fill the unspecified bits in a partially-specified test sequence in order to increase the number of faults detected by the sequence. The third technique uses test sequences provided by the test generator as seeds in a genetic algorithm, and better sequences are evolved that detect more faults. Significant improvements in test set size, fault coverage, and test generation time have been obtained over previous approaches using combinations of the three techniques.

Testing Real-Time Aspects: Evaluation of a system (or its components) at its normal operating frequency, speed, or timing. real-time systems, that is, systems which operate in an environment with strict timing constraints. Examples of such systems are many: embedded systems (e.g., automotive, avionic and robotic controllers, mobile phones), communication protocols, multimedia systems, and so on. When testing real-time systems, one must pay attention to two important facts. First, it is not sufficient to check whether the SUT produces the correct outputs; it must also be checked that the timing of the outputs is correct. Second,the timing of the inputs determines which outputs will be produced as well as the timing of these outputs. Real time and performance tests related to different aspects of nonfunctional system requirements. Reliability is understood as the ability of a system or component to perform its required functions under stated conditions for a specified period of time. Here is small diagram visualized real time and performance testing.


Regression Testing: Best explained here


Automated Testing Tools: A number of test tools are available. These tools provide automated functional and performance testing for environments including various technologies such as JAVA, SOAP, CORBA, HTML etc.The accurate or correctness testing tool usually had limitations or risks of implementations. Good testing also requires a tester's creativity, experience and intuition, together with proper techniques.


Summary: State of the Art Testing approach results of expectations and variances for length, occurrences, time to first occurrence, etc., for arcs, events, and states. Probability of occurrence in a test for arcs, events, and states. Complexity measures such as source entropy.


Refs: wikipedia.org / blogpost.com / qality_assurance_news /