Thursday, December 17, 2009

A day without work notebook or laptop :-)

How would a day be without a notebook or laptop :-) My machine froze on me this morning, driver just died and now it just not coming up!! waiting to get a replacement. while waiting on machine, thinking how could be a day with no laptop or notebook ???

Friday, November 13, 2009

Domain Based Testing

Domain Based Testing:  Usually carried out based on two concepts - domain analysis and domain modelling. Its intent is to focus on test rather than the action of the test.
It wraps browser object into page object that provides API that returns the Html element object based on what you can do at the current page.
For ex:

login_page = LoginPage.new(@browser)
login_page.username.type('username')
login_page.password.type('password')
login_page.logon_button.click

It does require designing, implementing page, most common actions, verifications on page and Html elements. But it does pay off. With shared page classes, each tests only need to write code that is specific about that test case. This makes it easy to understand and maintain. Also, it provides an ease to find the code and change it when the element id or name changes.

Tuesday, October 20, 2009

Clearing IE cache and cookies before running Selenium tests

When using FireFox, it is quite easy to setup a profile to be used with your Selenium tests. FireFox allows configuration which can delete all cookies and cached files when shutting down. Internet Explorer neither allows a profile like FireFox nor deletes cookies and cache unless you delete them by using the Tools –> Internet Options menu.

Deleting cache and cookies might be required for some web applications when testing via Selenium. Here is a easy way to incorporate documented Windows calls to delete cache and cookies via Ruby.

The following script uses the system function to get access to the Windows command prompt. Run the following ruby script on a Windows based ruby installation to clear all cookies and cached files.

#This script is meant for Ruby on Windows only for
#clearing all cache and cookies in Internet Explorer.
#Run this script from the command prompt like this
#C:\Ruby>ruby clear-ie-cache-cookies.rb

#To clear IE7 temporary Internet files
system('RunDll32.exe InetCpl.cpl,ClearMyTracksByProcess 8')

#To clear IE7 browsing cookies
system('RunDll32.exe InetCpl.cpl,ClearMyTracksByProcess 2')

#To clear IE7 browsing history
system('RunDll32.exe InetCpl.cpl,ClearMyTracksByProcess 1')

#To clear IE7 form data
system('RunDll32.exe InetCpl.cpl,ClearMyTracksByProcess 16')

#To clear IE7 remembered passwords for filling web login forms
system('RunDll32.exe InetCpl.cpl,ClearMyTracksByProcess 32')

#To clear or delete all IE7 browsing history – all of the above!
system('RunDll32.exe InetCpl.cpl,ClearMyTracksByProcess 255')

Wednesday, October 7, 2009

Running Selenium tests written in Ruby on Windows

Here is a follow-up to my earlier post about installing and using Ruby and Selenium on Windows.

Recording a Selenium test using Selenium IDE for FireFox.

I used the Selenium IDE add-on with FireFox to record a quick example test in Ruby. I visited the Yahoo homepage and typed a search for Selenium RC. Here is the code in Ruby

require "selenium"
require "test/unit"

class SeleniumRubyWindowsTest < Test::Unit::TestCase
def setup
@verification_errors = []
if $selenium
@selenium = $selenium
else
@selenium = Selenium::SeleniumDriver.new("localhost", 4444, "*chrome", "http://www.yahoo.com/", 10000);
@selenium.start
end
@selenium.set_context("SeleniumRubyWindowsTest")
end

def teardown
@selenium.stop unless $selenium
assert_equal [], @verification_errors
end

def test_SeleniumRubyWindowsTest
@selenium.open "/"
assert_equal "Yahoo!", @selenium.get_title
@selenium.type "p_13838465-p", "Selenium RC"
@selenium.click "search-submit"
@selenium.wait_for_page_to_load "30000"
assert_equal "Selenium RC - Yahoo! Search Results", @selenium.get_title
end
end


Save this file as SeleniumRubyWindowsTest.rb


Now that a test is ready to be run, its time to start up the Selenium server.



Running the Selenium server


If you have followed the earlier post about installing Ruby, then you open a command window and go to the directory where you copied the selenium jars. In short go to the directory where Ruby Gems are installed -

<GEM_INSTALL_DIR>\Selenium-1.1.16\lib\selenium\openqa


In this location, using the Java JRE installed in your system, start the Selenium server like this


<GEM_INSTALL_DIR>\Selenium-1.1.16\lib\selenium\openqa java –jar selenium-server.jar


See the screen shot


clip_image002


Now your Selenium Server should start running at the default port of 4444. See screen shot.



clip_image002[5]



Running your first Selenium test written in Ruby

The Selenium test has already been recorded using the Seleniun IDE add-on for FireFox - SeleniumRubyWindowsTest.rb



  • Ensure that your Selenium server is still running

  • Run this test by going to C:\Ruby and typing the following command

  • C:\Ruby>ruby SeleniumRubyWindowsTest.rb

  • Selenium RC window should launch in Internet Explorer

  • A separate FireFox window should launch and show the web pages as per the test SeleniumRubyWindowsTest.rb



Running tests in Internet Explorer on Windows XP and Windows Vista

I have tried the above test using Internet Explorer 7 on Windows XP as well as Internet Explorer 8 on Windows Vista. There were no problems in running the test on IE7 with Win XP. Offcourse you must turn off your popup blocker before running tests. Extra steps are required to run on Windows Vista, as described in the next pargraph.



Code change required to make the example test run in IE is as simple as changing the following line



@selenium = Selenium::SeleniumDriver.new("localhost", 4444, "*chrome", http://www.yahoo.com/, 10000); 

to


@selenium = Selenium::SeleniumDriver.new("localhost", 4444, "*iexplore", http://www.yahoo.com/, 10000); 


Steps to run the same test on Windows Vista and IE8

However, you need to take a few more steps to run the tests without any problems on Windows Vista and IE8.



  • Create a Desktop shortcut for running the Selenium Server.

  • Start the Selenium server by right clicking the shortcut, and then selecting Run as Administrator option

  • Once server starts running, you can use the same steps as described above for running your test from the command line.


One more tip – IE8 has a popup blocker turned on by default. Turn it off before you start running your tests.



Thats all there is to using Ruby and Selenium on your Windows machine! Happy testing :)

Installing Ruby and Selenium on Windows

Though Ruby is primarily used on *nix and Mac OS X, I wanted to run it on Windows. So I went about setting up a ruby instance on Windows with the aim of writing web application tests using Selenium.

Aim: To set up Ruby and Selenium on Windows to be able to run Selenium based tests against a web application

There is a lot of information out there, but its in bits and pieces for Windows. So here is an attempt to make this a one stop resource for setting up Ruby and Selenium on Windows.

Pre-requisites:

Since Selenium server needs to be started from the provided jar, you will need to have a JRE on your Windows machine.

Getting and installing Ruby on Windows:
  • Go to the following site http://rubyforge.org/frs/?group_id=167

  • Download the latest Ruby Installer for windows. Example - ruby186-27_rc2.exe

  • Run this file to install Ruby.

  • When installing you will get to see options that need to be installed. Ensure that you select ‘Enable RubyGems’ to be installed. See screen shot. Click Next.

clip_image002
  • Install Ruby in a top level directory like C:\Ruby as per screen shot. Click Next.

clip_image002[5]

  • Accept defaults on rest of the screens and complete the installation.

Installing the Ruby Selenium Gem on Windows
  • Go to the following site http://selenium.rubyforge.org/

  • Navigate to the Downloads section (http://selenium.rubyforge.org/download.html) or go to the latest builds here http://selenium.rubyforge.org/builds/ and download the latest version.

    Example – Selenium-1.1.16.gem

  • Save this file to the “C:\Ruby” directory where you installed Ruby for Windows

  • Now start a command prompt and go to C:\Ruby

  • Type the command C:\Ruby>gem install Selenium

  • Wait for the Selenium gem to be installed.

  • Note that Selenium should be with uppercase S and not lowercase s. See screen shot.

clip_image002[7]

Installing Selenium Remote Control for Windows

Selenium Remote Control (RC) is required so that the Selenium Jetty Server can start running and then tests written in Ruby can be run against the web application under test using Selenium Ruby Client driver.

Tests can be executed against any Windows browser like Internet Explorer and FireFox.

  • Go to the following site and download the latest Selenium RC driver http://seleniumhq.org/download/

  • Example – Selenium RC 1.0.1

  • Extract this zip file

  • Copy only the following two folders to the C:\Ruby folder. Refer screen shot.

      1. selenium-server-1.0.1

      2. selenium-ruby-client-driver-1.0.1

    clip_image002[9]

    • Now go to the folder C:\Ruby\selenium-server-1.0.1

    • Copy the selenium-server.jar file from here to the following location

      C:\Ruby\lib\ruby\gems\1.8\gems\Selenium-1.1.16\lib\selenium\openqa

    • In short locate the directory where Ruby Gems are installed <GEM_INSTALL_DIR>\Selenium-1.1.16\lib\selenium\openqa

    You are done with all installations! Now the next step is to run the Selenium RC via the Jetty Server bundled with Selenium. I will post about this later on.

    Wednesday, September 30, 2009

    QA/Testing - Interview Questions

    Automated testing vs Manual testing ?

    Modern web applications are characterized by a multitude of elements and possible interactions within the GUI which all have to be considered during testing.
    Besides the possibility of reducing the testing effort, there is another reason to choose automated web testing: Especially extensive GUIs with lots of elements can push the testers to their limits when tested manually. Entering the same or similar data in hundreds of input masks for example is not an intellectual challenge but requires full concentration and can lead to a decreasing motivation and inattentiveness.

    How would you decide what to automate or not to automate ?
    1. Automate only that which needs automating.
    2. Design and build for maintainability.
    3. Whether or not to automate: rules of thumb
    1. GUIs are difficult to automate. Despite software vendors telling you how easy it is to use record-and-playback functionality, graphical interfaces are notoriously difficult to automate with any sort of maintainability. Application interfaces tend to become static quicker than Web pages and are thus a bit easier to automate. Also, I have found that using Windows hooks is more reliable than the DOM interface. Keys to look for when deciding to automate a GUI is how static it is (the less it changes, the easier it will be to automate) and how close the application is tied to Windows standard libraries (custom objects can be difficult to automate).
    2. If possible, automate on the command-line or API level. Removing the GUI interface dramatically helps reliability in test scripts. If the application has a command-line interface, not only does it lend itself to reliable automation but is also somewhat data driven, another green light to go forward with automation.
    3. Automate those things that a human cannot do. If you suspect a memory leak by performing a certain function but can’t seem to reproduce it in a reasonable amount of time, automate it. Also particularly interesting to automate are time-sensitive actions (requiring precise timing to capture a state change, for example) and very rapid actions (e.g., loading a component with a hundred operations a second).
    4. Stick with the golden rule in automation: do one thing, and do it well. A robust test case that does a single operation will pay off more than a test case that covers more but requires heavy maintenance. If you design your test cases (or library functions, preferably) to do single actions, and you write them robustly, pretty soon you can execute a series of them together to perform those broad test cases that you would have written into a single test case.
    Given a product to test with no product specifications or functional specification, how would you go about testing ?
    Go through UI. Look at various flows. Run through application - various transactions and actions. Note down the behavior noticed for these transactions and actions. These can be your expected behavior. Meet with development teams, product management team, user experience team. Learn more about most common operations. 


    How would you test a calculator ?
    Divide your test cases into positive test cases and negative test cases.


    Tell me test scenarios for testing traffic lights ?
    Divide your test cases into positive test cases and negative test cases. These kind of questions are more to learn about your thought process and imagination.


    Given a website with 2 inputs, what test cases you will perform ?
    Same as above. Divide test cases in positive and negative test cases.


    what is Boundary value analysis ?
    Boundary value analysis is a software testing design technique in which tests are designed to include representatives of boundary values. Values on the edge of an equivalence partition or at the smallest value on either side of an edge. The values could be either input or output ranges of a software component. Since these boundaries are common locations for errors that result in software faults they are frequently exercised in test cases.


    What matrix you can think of ?
    Bugs - Major/show stopper, open P0/P1, fixed/verified, open bugs etc
    Automation results - Pass/Fail/Not executed/blocked.
    Load and performance - Expected benchmark/actual benchmark, history data/stats, results per package or functions, memeory leaks, information about CPU/disk etc
    Msll: Test repository results etc


    What should QA dashboard look like ? What all it should be on QA dashboard ?
    Please refer to my blog about QA dashboad.

    Thursday, September 24, 2009

    JAVA / J2EE / SWING Interview Questions W/ Solutions

    Sharing Java interview questions being asked for mid-level to senior-level test engineering positions at well-established companies:
    1. What's the difference between a String and a StringBuffer?
    2. What's the difference between EJB and Hibernate?
    3. What's the difference between final, finally, and finalize?
    4. Difference between ArrayList and HashMap ?
    5. Expalin Swing Action architecture ?
    6. How will you communicate between two Applets ?
    7. How would you detect and minimize memory leaks in Java ? (Very important question)
    8. Explain SOA ?
    9. Explain Web and EJB containers ?
    10. Which Java pattern you have used, explain ?
    11. How to get a pop-up window when clicking on a button ?
    12. Given following code what will be output ?
    13. Explain exception handling in java ?
    14. Garbage collection - Can you force it ? (Very important)
    15. What is RMI ?
    16. How do you check that a date given as a combination of day, month and year is a valid date? For instance a date 2008-02-31 (as in yyyy-mm-dd) would be invalid date ?
    Solution:
        Date parseDate(String maybeDate, String format, boolean lenient) {
        Date date = null;
        // test date string matches format structure using regex
        // - weed out illegal characters and enforce 4-digit year
        // - create the regex based on the local format string
        String reFormat = Pattern.compile("d+|M+").matcher(Matcher.quoteReplacement(format)).replaceAll("\\\\d{1,2}");
        reFormat = Pattern.compile("y+").matcher(reFormat).replaceAll("\\\\d{4}");
        if ( Pattern.compile(reFormat).matcher(maybeDate).matches() ) {
          // date string matches format structure, 
          // - now test it can be converted to a valid date
          SimpleDateFormat sdf = (SimpleDateFormat)DateFormat.getDateInstance();
          sdf.applyPattern(format);
          sdf.setLenient(lenient);
          try { date = sdf.parse(maybeDate); } catch (ParseException e) { }
        } 
        return date;
      } 
      // used like this:
      Date date = parseDate( "21/5/2009", "d/M/yyyy", false);
    OR
    package <>
    import java.text.ParseException;
    import java.text.SimpleDateFormat;


    public class ValidateDate {
      public static boolean isValidDateStr(String date, String format) {
       try {
        SimpleDateFormat sdf = new SimpleDateFormat(format);
        sdf.setLenient(false);
        sdf.parse(date);
       } catch (ParseException e) {
        System.out.println(e.getMessage());
        return false;
       } catch (IllegalArgumentException e) {
        System.out.println(e.getMessage());
        return false;
       }
       return true;
      }
      public static void main(String[] args) {
       System.out.println(" 1900-12-13 valid ? "
         + ValidateDate.isValidDateStr("1900-12-13", "yyyy-MM-dd"));
       // "1990-12/13" throws a ParseException
       System.out.println(" 1900-12/13 valid ? "
         + ValidateDate.isValidDateStr("1900-12/13", "yyyy-MM-dd"));
       // "1990-13-12" throws a IllegalArgumentException
       System.out.println(" 1900-13-12 valid ? "
         + ValidateDate.isValidDateStr("1900-13-12", "yyyy-MM-dd"));
       /*
        * output : 1900-12-13 valid ? true 1900-12/13 valid ? false 1900-13-12
        * valid ? false
        */
       //LEAP YEAR
       System.out.println("2009-02-29 valid ? "
         + ValidateDate.isValidDateStr("2009-02-29", "yyyy-MM-dd"));    //EPOCH TIME
       System.out.println("1969-01-01 valid ? "
         + ValidateDate.isValidDateStr("1969-01-01", "yyyy-MM-dd"));
      }
    }

    How to test Private or Protected methods ?
    Solution: You can not test Private methods they have to be made either protected or public. Though using Goovy framework you could achieve it, look for the same on web. Protected methods are for sub-classing so sub-class it and have a public method that will "exercise" the protected method in the unit test of that subclass. thats just a quick thought.

    Monday, September 21, 2009

    Security Testing - Web 2.0 and Web Applications

    Web application security is not like regression testing that you come up with test scenarios once and keep running these from time to time or build to build. Rather security testing is a process to determine :confidentiality, integrity, authentication, authorization, availability, and non-repudiation. Despite of the fact that you have best of best firewall, still security attacks on web applications is a huge concern and something that needs to be treated an important aspect of quality assurance or testing of web 2.0 - new web apps.

    What kind of security attacks possibly could be detected and needs to be taken care of as part of testing: Vulnerability: Cause of vulnerability can be bugs in application or script code or presence of viruses.
    URL manipulation: When web application communicates between client and server through a regular means - browser, changing information in URL could led to unintended behavior on server side.
    XSS - Cross-site scripting: Standard issue when one tries to insert client-side script and when such insertion is viewable to user then one must test for XSS attacks.
    SQL Injection: Insertion of SQL query which is then executed by server.

    Test Approach To Perform:
    URL Manipulation through HTTP GET Methos: In HTTP GET methods, information is sent in the form of querystring. One should modify parameter value in querystring to check if server accepts it.

    SQL Injection: These are very critical as attacker get vitalinformation from server database. To check these, look at the snippet of code where direct MySql queries are executed by accepting some user inputs.If user input data is crafted in SQL queries to query the database, attacker can inject SQL statements or part of SQL statements as user inputs to extract vital information from database. Even if attacker is successful to crash the application, from the SQL query error shown on browser, attacker can get the information they are looking for. Special characters from user inputs should be handled/escaped properly in such cases.

    XSS - Cross-Site Scripting: As the name suggests one must test applications for XSS using any HTML or any script

    Sunday, September 13, 2009

    File Upload Issue - Solution For Selenium Limitation


    Selenium, a web application test tool, has one limitation which is the File dialog which gets launched by clicking the Browse button to select a file for uploading. Selenium is unable to do anything once the File dialog opens up.


    Here is the solution code for above mentioned issue in Python as follows:
    def upload_file ( self, sel, im, locator, value ):
            # move the focus to the appropriate field
            print "locator : " + locator
            sel.focus( locator )
            for i in range ( 0, len(value) ):
                o = ord( value[i] )
                print value[i] + " : " + str ( o )
                if ( o >= ord('a') and o <= ord('z') ) or \
                   ( o >= ord('0') and o <= ord('9') ):
                    o = o - 32
                    sel.key_press_native( o )
                elif ( o >= ord('A') and o <= ord('Z') ):
                    sel.key_down_native( 16 )
                    sel.key_press_native( o )
                    sel.key_up_native( 16 )
                elif o == ord('.') or o == ord('\\') or o == ord(' ') or o == ord('/'):
                    sel.key_press_native( o )
                elif o == ord('_'):
                    sel.key_down_native( 16 )
                    sel.key_press_native( ord('-') )
                    sel.key_up_native( 16 )
                elif o == ord(':') :
                    sel.key_down_native( 16 )
                    sel.key_press_native( ord(';') )
                    sel.key_up_native( 16 )
                else:
                    print "Don't know how to handle *" + value[i] + "*"
    In Java as follows:
    While automating using Selenium is stuck when simulate the click to upload file input. I have to launch the concurrent thread that do the file name input. Thread Code(Note file name should be given in Java format like 'c:/boot.ini):
    import javax.swing.*;
    import java.awt.event.KeyEvent;
    import java.awt.*;
    /**
     * @author Bogdan Gusiev
     *         Date 29.03.2009
     */
    public class FileChooserThread extends Thread {
        public FileChooserThread(String file) {
            super(new FileRunner(file));
        }
    }
    class FileRunner implements Runnable {
        private String fullName;
        public FileRunner(String fileName) {
            this.fullName = fileName;
        }
        public void run() {
            try {
                Thread.sleep(1000);
                Robot robot = new Robot(); //input simulation class
                for (char c : fullName.toCharArray()) {
                    if (c == ':') {
                        robot.keyPress(KeyEvent.VK_SHIFT);
                        robot.keyPress(KeyEvent.VK_SEMICOLON);
                        robot.keyRelease(KeyEvent.VK_SHIFT);
                    } else if (c == '/') {
                        robot.keyPress(KeyEvent.VK_BACK_SLASH);
                    } else {
                        robot.keyPress(KeyStroke.getKeyStroke(
                                       Character.toUpperCase(c), 0).getKeyCode());
                    }
                }
                robot.keyPress(KeyEvent.VK_ENTER);
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        }
    }
    Here is Selenium call method:
    protected void chooseFile(String element, String fileName) {
               Number positionLeft = selenium.getElementPositionLeft(element);
               Number positionTop = selenium.getElementPositionTop(element);
               new FileChooserThread(fileName).start(); //launch input thread.
               //this method will held current thread while FileChooser gives the file name
               selenium.clickAt("file", positionLeft + "," + positionTop);
               return fileName;
    }

    Selenium Interview Questions and Answers

    Sharing Selenium interview questions being asked (to me at various interviews) for Mid-level to Senior-level Test Engineering positions at well-established companies.

    Describe some problems that you had with Selenium tool ?
    Solution: The one big limitation in Selenium - Select a file in a File Input field for sake of uploading.
    XPaths makes tests slow
    Having Ids for elements make tests faster but the UI code might not have Ids for all elements
    XPaths are brittle - ie flaky because if the UI developer changes anything like adding or removing a DIV, then the XPath becomes invalid and has to be updated in the tests.
    *Also look at my blog above which talks more about File Upload issue with Selenium *

    What are the limitations of using Selenium with Flex based UIs ?
    Solution: There is a relatively new open source project called FlexUISelenium located here, which allows Flex based UIs to be tested by Selenium. FlexUISelenium is an extension to the Selenium RC client driver that enables the Selenium RC client drivers to interact (and test) the Flex UI components and methods of the Flex application. FlexUISelenium works on the same lines as the Selenium Flash UI testing concept make calls to the ActionScript via JavaScript from Selenium.
    Pros:

    • With FlexUISelenium the developers do not need to expose every method to be tested by using the ExernalInterface technique.
    • Just adding the SeleniumFlexAPI.swc library to the Flex application while building automatically exposes all methods to be called from external JavaScript.

    Cons:

    • The FlexUISelenium project is quiet new and has some issues as reported on the project home page.
    • Its not production ready yet and hence not very reliable.
    • The developers have to include an external library which is otherwise not required for production builds.
    • How would you test Pop-up window using Selenium ?
    • Solution: First steps would be to determine whether this is an alert dialog -created using JavaScript's alert() or a bona-fide pop up window probably using open(). Since they both are treated differently its important to understand what kind of test case it is and what exactly interviewee asking.


    What is difference between Borland Silk test and Selenium?
    Solution: Selenium is completely free test automation tool, while Silk Test is not. Only web applications can be testing using Selenium testing suite. However, Silk Test can be used for testing client server applications. Selenium supports following web browsers: Internet Explorer, Firefox, Safari, Opera or Konqueror on Windows, Mac OS X and Linux. However, Silk Test is limited to Internet Explorer and Firefox. Silk Test uses 4Test scripting language. However, Selenium test suite has the flexibility to use many languages like Java, .Net, Perl, PHP, Python, and Ruby.

    What is Selenium RC (Remote Control)? 
    Solution: Selenium RC allows the test automation expert to use a programming language for maximum flexibility and extensibility in developing test logic. For example, if the application under test returns a result set and the automated test program needs to run tests on each element in the result set, the iteration / loop support of programming language’s can be used to iterate through the result set, calling Selenium commands to run tests on each item. Selenium RC provides an API and library for each of its supported languages. This ability to use Selenium RC with a high level programming language to develop test cases also allows the automated testing to be integrated with the project’s automated build environment.
    More info can be found here

    How have you used Selenium for your own most recent project ?
    Solution: Make sure you explain about whether you used standard Selenium API to automate tests, if extended Selenium as per your needs then how have you done that, besides that which test runner you used with Selenium, also explain about paradigm you followed for automation using Selenium.

    Monday, September 7, 2009

    Quality Assurance Dashboard

    Quality Assurance Dashboard: A QA dashboard is a reporting tool, to manage and track given project. It will provide a high level summary of white-box and black-box testing status, major show-stopper bugs,  scorecards, performance metrics, and reports that are distributed to both client and server side stakeholders. The portal provides a single source for critical information with on-demand availability, access, and visibility into vital program activities. It also provides one-line summary about a particular BUILD or Release Candidate status in Green or Red bar - Green (looks good) and Red (Major blocks - needs to be fixed).

    The dashboard also provides a breakdown of performance metrics by area of responsibility to distinguish between the metrics within the scope of projects' responsibility and control, and those that are affected by other organizations or agencies. The Dashboard provides online access to a drilldown to the status of the individual performance metrics, including trend analysis, by functional area and service area. 

    The dashboard also answers the following main questions:


    • Status of testing
    • POA for testing for given release candidate or build for given day
    • When will it be finished
    • Why is it taking so long
    • Have you tested (specific item) yet ?
    Dashboard becomes effective because 
    • Provides management and all stakeholders a detailed test status reports
    • Provides visualization of overall testing in terms of charts or graphs or data or stats


    Here is an example 
    HTML template of a QA dashboard. A good start for folks looking to develop your own dashboard.



    Sunday, September 6, 2009

    Test Link - Tutorial With Example

    Test Link: An open source test management tool. It enables creation and organization of test cases and helps manage into test plan. Allows execution of test cases from test link itself. One can easily track test results dynamically, generate reports, generate test metrics, prioritize test cases and assign unfinished tasks.

    Its a web based tool with GUI, which provides an ease to develop test cases, organize test cases into test plans, execute these test cases and generate reports.

    Test link exposes API, written in PHP, can help generate quality assurance dashboards. The functions like AddTestCaseToTestPlan, AssignRequirements,CreateTestCase etc. helps create and organize test cases per test plan. Functions like GetTestCasesForTestPlan, GetLastExecutionResult allows one to create quality assurance dashboard.

    How to use Test Link: Example of TestLink simple work-flow:


    1. Initial step would be to create a new Test Project and assign QA testers or engineers with tasks.


    2.Import Software Requirements and for part of these requirements generates empty Test Cases. Reorganize them into Test Suites.


    3.Create a content for empty test cases using test specifications that are being organized into Test suites.


    4.Create “Regression testing” and assigns to applicable test cases.


    5.Create a Test Plan, Build and link all Test Cases in Test Suite to this Test Plan. Assign resources to this test plan.


    6.Assume QA got there first Build or Release Candidate from development team, execute and record the testing with the result.


    7.Assume QA get new Build or Release Candidate with fixes for blocking issues, verify these fixes for blocking issues, execute regression tests. 


    8.Manager (Test or Engineering) and other project related stakeholders want to see results and status of testing. Then in such a case, these stakeholders including managers can create accounts or use default Guest account to view test results for a particular Build. An overall report gets generated for automated test suites, as a Guest manager is able to view test results at a higher level in graphical format.


    9.Suppose new changes happens to existing functionality, its very easy to modify existing test plan and add new test cases or enhance/modify existing test cases and attach them to a particular test plan.


    10.Test suites continue to execute as usual by generating various reports.


    11.For new project again QA creates a new Test , follows above steps to implement TestLink for there project.


    Find more about TestLink

    Saturday, September 5, 2009

    State-of-Art Testing

    State of the Art Testing : Software testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required result. Software testing is an art of observation, experience, experiment conducted to provide stakeholders with information about quality of the product or service under test, basically AUT. This art also provides an independent view of the software to allow the business to appreciate and understand the risks at implementation of the software or service.


    What does State of the Art Testing comprise of ?
    It comprises many important aspects relevant to the guidelines, e.g.:
    • Test Management (aka - Planning and Organization)
    • Process models for testing - Unit, Integration, System, and Acceptance
    • Testing Phases
    • Methods for test case design
    • Testing from the point of view of the user/customer
    • Sequence testing procedure
    • Testing real-time aspects
    • Regression testing
    • Auomation Test Tools
    Test Management (aka - Planning and Organization): As the name suggests test management comprises of test case management. It often includes Requirements documents and functional specifications which help in generating Requirement test matrix. Basically it provides information about how much of requirement and specifications are covered by available test cases.
    First step of test management is to define the Test plan in association with product requirements and functional specifications. Test plan is followed by designing and developing test cases or use cases. Each test case or use case should have a priority defined. Every test change must be versioned to get a comprehensive view of overall history of a particular test case or use case. Test management's ultimate goal should be able to generate metrics or reports displaying various data related to quality assurance of the given product.


    Process models for testing - Unit, Integration, System, and Acceptance: Let us understand what is test process. It comprises of planning, specification, execution, recording and checking for completion. Why improve test process. To decrease cost and time of testing and to increase quality of product. Software test model includes test designing, test plan, test case generation, test execution, test result analysis and test reuse. These are laid out in the whole life cycle of software development process. Using software test models improves reliability and facilitates reliability measurements. These are divided as follows:


    Process Improvement Models: For example: Six Sigma, TQM


    Software Process Improvement Models: For example: Bootstrap, CMM(I), SPICE

    Test Process Improvement Models: For example: TPI, TIM, TMM, Test certification standards. 

    In counterpoint, emerging software disciplines like extreme programming, Agile software development model, test-driven software development model are more popular now a days in web application testing. In these test process models, unit tests are written first. Testing is done at following levels:
    • Unit Testing: Tests minimal software component or module. Each unit is tested to verify that the detailed design for Unit has been correctly implemented. In an Oops environment, this is usually at class level and minimal unit test includes constructors and destructor. 
    • Integration Testing: Exposes defects in interfaces and interaction between integrated components or modules.
    • System Testing: Tests a completely integrated system to verify that it does meet the requirements. Acceptance Testing: Finally an acceptance testing is conducted from an end user's perspective to validate whether or not to accept the product.


    Testing Phases and Testing Methods: Unit Testing, Integration Testing, System Testing (explained above) are few of the testing phases including ones mentioned below:
    • Alpha Acceptance Testing: Try it out with in-house “users”
    • Installation Testing: Can users install it and does it work in their environment?
    • Beta Acceptance Testing: Try it out with real users
    Quality assurance testing methods are usually divided into Black box and White Box testing methods.

    Black box testing means testing product without any knowledge of internal implementation and includes cross-browser compatibility testing, exploratory testing, boundary value testing, pair testing, model-based testing, specification or requirement based testing.

    White box testing means testing with knowledge and access of internal implementations,algorithms including code implementation and includes API testing, Code Coverage, Static testing and code completion evaluation testing.
    Methods For Test Case Design: Design of test case or use case plays pivotal role in ensuring higher quality of product. These methods provide developers with a systematic approach to testing more important, method provide mechanism that can help to ensure completeness of tests and provide the highest likelihood for uncovering errors in software. Again its divided into Black box test case design and white box test case design.
    Black box test case design: Usually these are designed based upon requirements or specifications documents and at software interface levels.

    White box test case design: These are used to detect errors by means of execution-oriented test cases. In white box testing strategy QA investigates the internal structure of the object to be assessed in order to specify execution-oriented test cases on the basis of the program logic. In this connection the specifications have to be taken into consideration, though. In a test case design, the portion of the assessed object which is addressed by the test cases is taken into consideration. The considered aspect may be a path, a statement, a branch, and a condition. The test cases are selected in such a manner that the correspondingly addressed portion of the assessed object is increased.

    An open source tool TestLink can be used to design these test cases. More information about Test Link can be found here.

    Testing From The Point Of View Of The User/Customer: A well-performing applications is one that lets the end user carry out a given task without undue perceived delay or irritations and process of ensuring this referred as Testing from an end user's perspective. It includes, identifying and escalating most common operations or transactions performed by end user. Various ways are available to find out most common operations, end user's feedback, evaluation about a particular feature or product. 
    Quality assurance testers needs to think and act like end user, based on requirements or specifications documents, regularly meeting with field representatives, product managers also help contribute to test like an end user. Also it is important to have domain expertise for given product. Also quality assurance team needs to have expertise on internal implementation and overall architecture of product to think from an end user's perspective. With a performance application users are never greeted with a blank screen durin logons and can accomplish what they set up to accomplish without letting their attention wander. Casual visitors browsing a web application or web site can find what they are looking for purchase it without experiencing too much frustration. Do your own analysis by asking questions to yourself like would your potential customer's life be improved with your product or service? Does your customer find a certain product or service inferior?

    Sequence Testing Procedure: As per web definition squence testing procedure means specific order of related actions or steps that comprise a test procedure or test run. Dynamic test sequence compaction is an effective means of reducing test application time and often results in higher fault coverages and reduced test generation time as well. Three simulation-based techniques for dynamic compaction of test sequences are described. The first technique uses a fault simulator to remove test vectors from the test sequence generated by a test generator if the vectors are not needed to detect the target fault, considering that the circuit state may be known. The second technique uses genetic algorithms to fill the unspecified bits in a partially-specified test sequence in order to increase the number of faults detected by the sequence. The third technique uses test sequences provided by the test generator as seeds in a genetic algorithm, and better sequences are evolved that detect more faults. Significant improvements in test set size, fault coverage, and test generation time have been obtained over previous approaches using combinations of the three techniques.

    Testing Real-Time Aspects: Evaluation of a system (or its components) at its normal operating frequency, speed, or timing. real-time systems, that is, systems which operate in an environment with strict timing constraints. Examples of such systems are many: embedded systems (e.g., automotive, avionic and robotic controllers, mobile phones), communication protocols, multimedia systems, and so on. When testing real-time systems, one must pay attention to two important facts. First, it is not sufficient to check whether the SUT produces the correct outputs; it must also be checked that the timing of the outputs is correct. Second,the timing of the inputs determines which outputs will be produced as well as the timing of these outputs. Real time and performance tests related to different aspects of nonfunctional system requirements. Reliability is understood as the ability of a system or component to perform its required functions under stated conditions for a specified period of time. Here is small diagram visualized real time and performance testing.


    Regression Testing: Best explained here


    Automated Testing Tools: A number of test tools are available. These tools provide automated functional and performance testing for environments including various technologies such as JAVA, SOAP, CORBA, HTML etc.The accurate or correctness testing tool usually had limitations or risks of implementations. Good testing also requires a tester's creativity, experience and intuition, together with proper techniques.


    Summary: State of the Art Testing approach results of expectations and variances for length, occurrences, time to first occurrence, etc., for arcs, events, and states. Probability of occurrence in a test for arcs, events, and states. Complexity measures such as source entropy.


    Refs: wikipedia.org / blogpost.com / qality_assurance_news /