Document Sample
testing Powered By Docstoc

Bugs can broadly under go following stages: 1) Open stage: A defect originated by a tester. 2) Assign: Raised defect assigned to Developer. 3) Resolved stage: Developer provides the code fix for the bug and makes it resolved. 4) Closed stage: Tester re-tests the bug and closes it, if it‘s working. Otherwise he will re-

open the defect and it will go back to stage 1. There is no standard that bug life cycle should have the above stages. You can define your own sub stages according to your convenience.

 Bug Life Cycle & Guidelines In this tutorial you will learn about Bug Life Cycle & Guidelines, Introduction, Bug Life Cycle, The different states of a bug, Description of Various Stages, Guidelines on deciding the Severity of Bug, A sample guideline for assignment of Priority Levels during the product test phase and Guidelines on writing Bug Description. Introduction: Bug can be defined as the abnormal behavior of the software. No software exists without a bug. The elimination of bugs from the software depends upon the efficiency of testing done on the software. A bug is a specific concern about the quality of the Application under Test (AUT). Bug Life Cycle: In software development process, the bug has a life cycle. The bug should go through the life cycle to be closed. A specific life cycle ensures that the process is standardized. The bug attains different states in the life cycle. The life cycle of the bug can be shown diagrammatically as follows:

The different states of a bug can be summarized as follows: 1. New 2. Open 3. Assign 4. Test 5. Verified 6. Deferred 7. Reopened 8. Duplicate 9. Rejected, 10. Closed Description of Various Stages: 1. New: When the bug is posted for the first time, its state will be ―NEW‖. This means that the bug is not yet approved. 2. Open: After a tester has posted a bug, the lead of the tester approves that the bug is genuine and he changes the state as ―OPEN‖. 3. Assign: Once the lead changes the state as ―OPEN‖, he assigns the bug to corresponding developer or developer team. The state of the bug now is changed to ―ASSIGN‖. 4. Test: Once the developer fixes the bug, he has to assign the bug to the testing team for next round of testing. Before he releases the software with bug fixed, he changes the state of bug to ―TEST‖. It specifies that the bug has been fixed and is released to testing team. 5. Deferred: The bug, changed to deferred state means the bug is expected to be fixed in next releases. The reasons for changing the bug to this state have many factors. Some of them are priority of the bug may be low, lack of time for the release or the bug may not have major effect on the software. 6. Rejected: If the developer feels that the bug is not genuine, he rejects the bug. Then the state of the bug is changed to ―REJECTED‖. 7. Duplicate: If the bug is repeated twice or the two bugs mention the same concept of the bug, then one bug status is changed to ―DUPLICATE‖. 8. Verified: Once the bug is fixed and the status is changed to ―TEST‖, the tester tests the bug. If the bug is not present in the software, he approves that the bug is fixed and changes the status to ―VERIFIED‖. 9. Reopened: If the bug still exists even after the bug is fixed by the developer, the tester changes the status to ―REOPENED‖. The bug traverses the life cycle once again.

10. Closed: Once the bug is fixed, it is tested by the tester. If the tester feels that the bug no longer exists in the software, he changes the status of the bug to ―CLOSED‖. This state means that the bug is fixed, tested and approved. While defect prevention is much more effective and efficient in reducing the number of defects, most organization conducts defect discovery and removal. Discovering and removing defects is an expensive and inefficient process. It is much more efficient for an organization to conduct activities that prevent defects.  Guidelines on deciding the Severity of Bug: Indicate the impact each defect has on testing efforts or users and administrators of the application under test. This information is used by developers and management as the basis for assigning priority of work on defects. A sample guideline for assignment of Priority Levels during the product test phase includes: 1. Critical / Show Stopper — An item that prevents further testing of the product or function under test can be classified as Critical Bug. No workaround is possible for such bugs. Examples of this include a missing menu option or security permission required to access a function under test. . 2. Major / High — A defect that does not function as expected/designed or cause other functionality to fail to meet requirements can be classified as Major Bug. The workaround can be provided for such bugs. Examples of this include inaccurate calculations; the wrong field being updated, etc. . 3. Average / Medium — The defects which do not conform to standards and conventions can be classified as Medium Bugs. Easy workarounds exists to achieve functionality objectives. Examples include matching visual and text links which lead to different end points. . 4. Minor / Low — Cosmetic defects which do not affect the functionality of the system can be classified as Minor Bugs.  Guidelines on writing Bug Description: Bug can be expressed as ―Result followed by the action‖. That means, the unexpected behavior occurring when a particular action takes place can be given as bug description.

1. Be specific. State the expected behavior which did not occur - such as after pop-up did not appear and the behavior which occurred instead. 2. Use present tense. 3. Don‘t use unnecessary words. 4. Don‘t add exclamation points. End sentences with a period. 5. DON‘T USE ALL CAPS. Format words in upper and lower case (mixed case). 6. Mention steps to reproduce the bug compulsorily.  How to Report Bugs Effectively? Introduction Anybody who has written software for public use will probably have received at least one bad bug report. Reports that say nothing ("It doesn't work!"); reports that make no sense; reports that don't give enough information; reports that give wrong information. Reports of problems that turn out to be user error; reports of problems that turn out to be the fault of somebody else's program; reports of problems that turn out to be network failures. There's a reason why technical support is seen as a horrible job to be in, and that reason is bad bug reports. However, not all bug reports are unpleasant: I maintain free software, when I'm not earning my living, and sometimes I receive wonderfully clear, helpful, informative bug reports. In this essay I'll try to state clearly what makes a good bug report. Ideally I would like everybody in the world to read this essay before reporting any bugs to anybody. Certainly I would like everybody who reports bugs to me to have read it. In a nutshell, the aim of a bug report is to enable the programmer to see the program failing in front of them. You can either show them in person, or give them careful and detailed instructions on how to make it fail. If they can make it fail, they will try to gather extra information until they know the cause. If they can't make it fail, they will have to ask you to gather that information for them. In bug reports, try to make very clear what are actual facts ("I was at the computer and this happened") and what are speculations ("I think the problem might be this"). Leave out speculations if you want to, but don't leave out facts. When you report a bug, you are doing so because you want the bug fixed. There is no point in swearing at the programmer or being deliberately unhelpful: it may be their fault and your problem, and you might be right to be angry with them, but the bug will get fixed faster if you help them by supplying all the information they need. Remember also that if the program is free, then the author is providing it out of kindness, so if too many people are rude to them then they may stop feeling kind. "It doesn't work." Give the programmer some credit for basic intelligence: if the program really didn't work at all, they would probably have noticed. Since they haven't noticed, it must be working for them.



numbers, and even unexplained delays are all just as important as fingerprints at the scene of a crime. Keep them! If you are using UNIX, the program may have produced a core dump. Core dumps are a particularly good source of clues, so don't throw them away. On the other hand, most programmers don't like to receive huge core files by e-mail without warning, so ask before mailing one to anybody. Also, be aware that the core file contains a record of the complete state of the program: any "secrets" involved (maybe the program was handling a personal message, or dealing with confidential data) may be contained in the core file. "So then I tried.” There are a lot of things you might do when an error or bug comes up. Many of them make the problem worse. A friend of mine at school deleted all her Word documents by mistake, and before calling in any expert help, she tried reinstalling Word, and then she tried running Defrag. Neither of these helped recover her files, and between them they scrambled her disk to the extent that no Undelete program in the world would have been able to recover anything. If she'd only left it alone, she might have had a chance. Users like this are like a mongoose backed into a corner: with its back to the wall and seeing certain death staring it in the face, it attacks frantically, because doing something has to be better than doing nothing. This is not well adapted to the type of problems computers produce. Instead of being a mongoose, is an antelope. When an antelope is confronted with something unexpected or frightening, it freezes. It stays absolutely still and tries not to attract any attention, while it stops and thinks and works out the best thing to do. (If antelopes had a technical support line, it would be telephoning it at this point.) Then, once it has decided what the safest thing to do is, it does it. When something goes wrong, immediately stop doing anything. Don't touch any buttons at all. Look at the screen and notice everything out of the ordinary, and remember it or write it down. Then perhaps start cautiously pressing "OK" or "Cancel", whichever seems safest. Try to develop a reflex reaction - if a computer does anything unexpected, freeze. If you manage to get out of the problem, whether by closing down the affected program or by rebooting the computer, a good thing to do is to try to make it happen again. Programmers like problems that they can reproduce more than once. Happy programmers fix bugs faster and more efficiently. "I think the tachyon modulation must be wrongly polarized." It isn't only non-programmers who produce bad bug reports. Some of the worst bug reports I've ever seen come from programmers, and even from good programmers.





window, and FooApp crashed." This is longer and more repetitive, but also clearer and less easy to misunderstand. Read what you wrote. Read the report back to yourself, and see if you think it's clear. If you have listed a sequence of actions which should produce the failure, try following them yourself, to see if you missed a step.

Summary   The first aim of a bug report is to let the programmer see the failure with their own eyes. If you can't be with them to make it fail in front of them, give them detailed instructions so that they can make it fail for themselves. In case the first aim doesn't succeed, and the programmer can't see it failing them, the second aim of a bug report is to describe what went wrong. Describe everything in detail. State what you saw, and also state what you expected to see. Write down the error messages, especially if they have numbers in. When your computer does something unexpected, freeze. Do nothing until you're calm, and don't do anything that you think might be dangerous. By all means try to diagnose the fault yourself if you think you can, but if you do, you should still report the symptoms as well. Be ready to provide extra information if the programmer needs it. If they didn't need it, they wouldn't be asking for it. They aren't being deliberately awkward. Have version numbers at your fingertips, because they will probably be needed. Write clearly. Say what you mean, and make sure it can't be misinterpreted. Above all, be precise. Programmers like precision.

    

A Bug's Life Cycle The status and resolution field define and track the life cycle of a bug.
STATUS RESOLUTION The status field indicates the general health of a bug. The resolution field indicates what happened to this Only certain status transitions are allowed. bug. UNCONFIRMED This bug has recently been added to the database. Nobody has validated that this bug is true. Users who have the "can confirm" permission set may confirm this bug, changing its state to NEW. Or, it may be directly resolved and marked RESOLVED. NEW This bug has recently been added to the assignee's list of bugs and must be processed. Bugs in this state may be accepted, and become ASSIGNED, passed on to someone else, and remain NEW, or resolved and marked RESOLVED. ASSIGNED This bug is not yet resolved, but is assigned to the proper person. From here bugs can be given to another person and become NEW or resolved and become RESOLVED. NEEDINFO More information from the reporter is needed to proceed further in fixing this bug. REOPENED This bug was once resolved, but the No resolution yet. All bugs which are in one of these "open" states have the resolution set to blank. All other bugs will be marked with one of the following resolutions.

resolution was deemed incorrect. For example, a WORKSFORME bug is REOPENED when more information shows up and the bug is now reproducible. From here bugs are either marked ASSIGNED or RESOLVED. RESOLVED A resolution has been taken, and it is awaiting verification by QA. From here bugs are either re-opened and become REOPENED, are marked VERIFIED, or are closed for good and marked CLOSED. VERIFIED QA has looked at the bug and the resolution and agrees that the appropriate resolution has been taken. Bugs remain in this state until the product they were reported against actually ships, at which point they become CLOSED. CLOSED The bug is considered dead, the resolution is correct. Any zombie bugs who choose to walk the earth again must do so by becoming REOPENED. FIXED A fix for this bug is checked into the tree and tested. WONTFIX The problem described is a bug which will never be fixed. This should be reserved for "unfixable" things, otherwise use NOTGNOME or NOTABUG. LATER The problem described is a bug which will not be fixed in this version of the product. REMIND The problem described is a bug which will probably not be fixed in this version of the product, but might still be. DUPLICATE The problem is a duplicate of an existing bug. Marking a bug duplicate requires the bug# of the duplicating bug and will at least put that bug number in the description field. INCOMPLETE All attempts at reproducing this bug were futile, or not enough information was available to reproduce the bug. Reading the code produces no clues as to why this behavior would occur. If more information appears later, please reopen the bug. NOTGNOME The bug report describes a problem in software not produced by the GNOME project. It should be reported elsewhere. This is also a suitable resolution for bugs that appear to have been introduced by someone creating a distribution of GNOME. NOTABUG The bug report describes behavior which is the correct behavior of the software or was reported in error.


Software Quality Assurance (SQA) consists of the software engineering processes and methods used to ensure quality. SQA encompasses the entire software development process, which may include processes such as reviewing requirements documents, source code control, code reviews, change management, configuration management, release management and of course, software testing.

Software quality assurance is related to the practice of quality assurance in product manufacturing. There are, however, some notable differences between software and a manufactured product. These differences all stem from the fact that the manufactured product is physical and can be seen whereas the software product is not visible. Therefore its function, benefit and costs are not as easily measured. Software QA involves the entire software development PROCESS - monitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with. It is oriented to 'prevention'.  How can new Software QA processes be introduced in an existing organization? A lot depends on the size of the organization and the risks involved. For large organizations with high-risk (in terms of lives or property) projects, serious management buy-in is required and a formalized QA process is necessary. Where the risk is lower, management and organizational buy-in and QA implementation may be a slower, step-at-a-time process. QA processes should be balanced with productivity so as to keep bureaucracy from getting out of hand. For small groups or projects, a more ad-hoc process may be appropriate, depending on the type of customers and projects. A lot will depend on team leads or managers, feedback to developers, and ensuring adequate communications among customers, managers, developers, and testers. The most value for effort will often be in (a) requirements management processes, with a goal of clear, complete, testable requirement specifications embodied in requirements or design documentation, or in 'agile'-type environments extensive continuous coordination with end-users, (b) design inspections and code inspections, and (c) post-mortems/retrospectives. Other possibilities include incremental self-managed team approaches such as 'Kaizen' methods of continuous process improvement, the Deming-Shewhart PlanDo-Check-Act cycle, and others. Software ‘Quality’:

  




Quality software is reasonably bug-free, delivered on time and within budget, meets requirements and/or expectations, and is maintainable. However, quality is obviously a subjective term. It will depend on who the 'customer' is and their overall influence in the scheme of things. A wide-angle view of the 'customers' of a software development project might include end-users, customer acceptance testers, customer contract officers, customer management, the development organization's management/accountants/testers/salespeople, future software maintenance engineers, stockholders, magazine columnists, etc. Each type of 'customer' will have their own slant on 'quality' - the accounting department might define quality in terms of profits while an enduser might define quality as user-friendly and bug-free.  Good Code:


also various theories and metrics, such as McCabe Complexity metrics. It should be kept in mind that excessive use of standards and rules can stifle productivity and creativity. 'Peer reviews', 'buddy checks' code analysis tools, etc. can be used to check for problems and enforce standards. For C and C++ coding, here are some typical ideas to consider in setting rules/standards; these may or may not apply to a particular situation:               Minimize or eliminate use of global variables. Use descriptive function and method names - use both upper and lower case, avoid abbreviations, use as many characters as necessary to be adequately descriptive (use of more than 20 characters is not out of line); be consistent in naming conventions. Use descriptive variable names - use both upper and lower case, avoid abbreviations, use as many characters as necessary to be adequately descriptive (use of more than 20 characters is not out of line); be consistent in naming conventions. Function and method sizes should be minimized; less than 100 lines of code is good, less than 50 lines is preferable. Function descriptions should be clearly spelled out in comments preceding a function's code. Organize code for readability. Use white space generously - vertically and horizontally Each line of code should contain 70 characters max. One code statement per line. Coding style should be consistent through out a program (e.g., use of brackets, indentations, naming conventions, etc.) In adding comments, err on the side of too many rather than too few comments; a common rule of thumb is that there should be at least as many lines of comments (including header blocks) as lines of code. No matter how small, an application should include documentation of the overall program function and flow (even a few paragraphs is better than nothing); or if possible a separate flow chart and detailed program documentation. Make extensive use of error handling procedures and status and error logging. For C++, to minimize complexity and increase maintainability, avoid too many levels of inheritance in class hierarchies (relative to the size and complexity of the application). Minimize use of multiple inheritances, and minimize use of operator overloading (note that the Java programming language eliminates multiple inheritance and operator overloading.) For C++, keep class methods small, less than 50 lines of code per method is preferable. For C++, make liberal use of exception handlers  Good Design:

 


  

The program should act in a way that least surprises the user It should always be evident to the user what can be done next and how to exit The program shouldn't let the users do something stupid without warning them.  Good Software QA engineer:


A good QA, test, or QA/Test (combined) manager should:          be familiar with the software development process be able to maintain enthusiasm of their team and promote a positive atmosphere, despite what is a somewhat 'negative' process (e.g., looking for or preventing problems) be able to promote teamwork to increase productivity be able to promote cooperation between software, test, and QA engineers have the diplomatic skills needed to promote improvements in QA processes have the ability to withstand pressures and say 'no' to other managers when quality is insufficient or QA processes are not being adhered to have people judgment skills for hiring and keeping skilled personnel Be able to communicate with technical and non-technical people, engineers, managers, and customers. Be able to run meetings and keep them focused.  Role of Documentation in QA:

Critical. (Note that documentation can be electronic, not necessarily paper, may be embedded in code comments, etc.) QA practices should be documented such that they are repeatable. Specifications, designs, business rules, inspection reports, configurations, code changes, test plans, test cases, bug reports, user manuals, etc. should all be documented in some form. There should ideally be a system for easily finding and obtaining information and determining what documentation will have a particular piece of information. Change management for documentation should be used if possible.    SE, CMM, CMMI, ISO, IEEE, ANSI:



Level 1 - characterized by chaos, periodic panics, and heroic efforts required by individuals to successfully complete projects. Few if any processes in place; successes may not be repeatable. Level 2 - software project tracking, requirements management, realistic planning, and configuration management processes are in place; successful practices can be repeated. Level 3 - standard software development and maintenance processes are integrated throughout an organization; a Software Engineering Process Group is is in place to oversee software processes, and training programs are used to ensure understanding and compliance. Level 4 - metrics are used to track productivity, processes, and products. Project performance is predictable, and quality is consistently high. Level 5 - the focus is on continouous process improvement. The impact of new processes and technologies can be predicted and effectively implemented when required. Perspective on CMM ratings: During 1997-2001, 1018 organizations were assessed. Of those, 27% were rated at Level 1, 39% at 2, 23% at 3, 6% at 4, and 5% at 5. (For ratings during the period 1992-96, 62% were at Level 1, 23% at 2, 13% at 3, 2% at 4, and 0.4% at 5.) The median size of organizations was 100 software engineering/maintenance personnel; 32% of organizations were U.S. federal contractors or agencies. For those rated at Level 1, the most problematical key process area was in Software Quality Assurance.




 




Testing is a process used to help identify the correctness, completeness and quality of developed computer software. With that in mind, testing can never completely establish the correctness of computer software. In other words Testing is nothing but CRITICISM or COMPARISION.Here comparison in the sense comparing the actual value with expected one. There are many approaches to software testing, but effective testing of complex products is essentially a process of investigation, not merely a matter of creating and following rote procedure. One definition of testing is "the process of questioning a product in order to evaluate it", where the "questions" are things the tester tries to do with the product, and the product answers with its behavior in reaction to the probing of the tester. Although most of the intellectual processes of testing are nearly identical to that of review or inspection, the word testing is connoted to mean the dynamic analysis of the product—putting the product through its paces. The quality of the application can and normally does vary widely from system to system but some of the common quality attributes include reliability, stability, portability, maintainability and usability. Refer to the ISO standard ISO 9126 for a more complete list of attributes and criteria.

Testing helps is verifying and Validating if the Software is working as it is intended to be working. Thins involves using Static and Dynamic methodologies to Test the application. Because of the fallibility of its human designers and its own abstract, complex nature, software development must be accompanied by quality assurance activities. It is not unusual for developers to spend 40% of the total project time on testing. For life-critical software (e.g. flight control, reactor monitoring), testing can cost 3 to 5 times as much as all other activities combined. The destructive nature of testing requires that the developer discard preconceived notions of the correctness of his/her developed software.

In general, software engineers distinguish software faults from software failures. In case of a failure, the software does not do what the user expects. A fault is a programming error that may or may not actually manifest as a failure. A fault can also be described as an error in the correctness of the semantic of a computer program. A fault will become a failure if the exact computation conditions are met, one of them being that the faulty portion of computer software executes on the CPU. A fault can also turn into a failure when the software is ported to a different hardware platform or a different compiler, or when the software gets extended. Software testing is the technical investigation of the product under test to provide stakeholders with quality related information. Software testing may be viewed as a sub-field of Software Quality Assurance but typically exists independently (and there may be no SQA areas in some companies). In SQA, software process specialists and auditors take a broader view on software and its development. They examine and change the software engineering process itself to reduce the amount of faults that end up in the code or deliver faster. Regardless of the methods used or level of formality involved the desired result of testing is a level of confidence in the software so that the organization is confident that the software has an acceptable defect rate. What constitutes an acceptable defect rate depends on the nature of the software. An arcade video game designed to simulate flying an airplane would presumably have a much higher tolerance for defects than software used to control an actual airliner. A problem with software testing is that the number of defects in a software product can be very large, and the number of configurations of the product larger still. Bugs that occur infrequently are difficult to find in testing. A rule of thumb is that a system that is expected to function without faults for a certain length of time must have already been tested for at least that length of time. This has severe consequences for projects to write long-lived reliable software. A common practice of software testing is that it is performed by an independent group of testers after the functionality is developed but before it is shipped to the customer. This practice often results in the testing phase being used as project buffer to compensate for project delays. Another practice is to start software testing at the same moment the project starts and it is a continuous process until the project finishes.

Another common practice is for test suites to be developed during technical support escalation procedures. Such tests are then maintained in regression testing suites to ensure that future updates to the software don't repeat any of the known mistakes. It is commonly believed that the earlier a defect is found the cheaper it is to fix it. Time Detected Time Introduced Requirements Architecture Construction Requirements 1 Architecture 3 1 Construction 5-10 10 1 System Test 10 15 10 PostRelease 10-100 25-100 10-25

In counterpoint, some emerging software disciplines such as extreme programming and the agile software development movement, adhere to a "test-driven software development" model. In this process unit tests are written first, by the programmers (often with pair programming in the extreme programming methodology). Of course these tests fail initially; as they are expected to. Then as code is written it passes incrementally larger portions of the test suites. The test suites are continuously updated as new failure conditions and corner cases are discovered, and they are integrated with any regression tests that are developed. Unit tests are maintained along with the rest of the software source code and generally integrated into the build process (with inherently interactive tests being relegated to a partially manual build acceptance process). The software, tools, samples of data input and output, and configurations are all referred to collectively as a test harness . 


The separation of debugging from testing was initially introduced by Glenford J. Myers in his 1978 book the "Art of Software Testing". Drs. Dave Gelperin and William C. Hetzel classified in 1988 the phases and goals in software testing as follows: until 1956 it was the debugging oriented period, where testing was often associated to debugging: there was no clear difference between testing and debugging. From 1957-1978 there was the demonstration oriented period where debugging and testing was distinguished now - in this period it was shown, that software satisfies the requirements. The time between 1979-1982 is announced as the destruction oriented period, where the goal was to find errors. 1983-1987 is classified as the evaluation oriented period: intention here is that during the software lifecycle a product evaluation is provided and measuring quality. From 1988 on it was seen as prevention oriented period where tests were to demonstrate that software satisfies its specification, to detect faults and to prevent faults.

 Software Testing Fundamentals:
Testing Objectives:


error. 2. A good test case is one that has a high probability of finding an as yet undiscovered error. 3. A successful test is one that uncovers an as yet undiscovered error.

Testing should systematically uncover different classes of errors in a minimum amount of time and with a minimum amount of effort. A secondary benefit of testing is that it demonstrates that the software appears to be working as stated in the specifications. The data collected through testing can also provide an indication of the software's reliability and quality. But, testing cannot show the absence of defect -- it can only show that software defects are present . Testing involves operation of a system or application under controlled conditions and evaluating the results (e.g., 'if the user is in interface A of the application while using hardware B, and does C, then D should happen'). The controlled conditions should include both normal and abnormal conditions. Testing should intentionally attempt to make things go wrong to determine if things happen when they shouldn't or things don't happen when they should. It is oriented to 'detection'. Organizations vary considerably in how they assign responsibility for QA and testing. Sometimes they're the combined responsibility of one group or individual. Also common are project teams that include a mix of testers and developers who work closely together, with overall QA processes monitored by project managers. It will depend on what best fits an organization's size and business structure.

Software Testing is the process used to help identify the correctness , completeness , security, and quality of developed computer software. Testing is a process of technical investigation, performed on behalf of stakeholders, that is intended to reveal quality-related information about the product with respect to the context in which it is intended to operate. This includes, but is not limited to, the process of executing a program or application with the intent of finding errors. Quality is not an absolute; it is value to some person. An important point is that software testing should be distinguished from the separate discipline of Software Quality Assurance (SQA), which encompasses all business process areas, not just testing.   Some Recent Major Computer System Failures Caused By Software Bugs:



A September 2006 news report indicated problems with software utilized in a state government's primary election, resulting in periodic unexpected rebooting of voter checking machines, which were separate from the electronic voting machines, and resulted in confusion and delays at voting sites. The problem was reportedly due to insufficient testing. In August of 2006 a U.S. government student loan service erroneously made public the personal data of as many as 21,000 borrowers on it's web site, due to a software error. The bug was fixed and the government department subsequently offered to arrange for free credit monitoring services for those affected. A software error reportedly resulted in over billing of up to several thousand dollars to each of 11,000 customers of a major telecommunications company in June of 2006. It



 







 

was reported that the software bug was fixed within days, but that correcting the billing errors would take much longer. News reports in May of 2006 described a multi-million dollar lawsuit settlement paid by a healthcare software vendor to one of its customers. It was reported that the customer claimed there were problems with the software they had contracted for, including poor integration of software modules, and problems that resulted in missing or incorrect data used by medical personnel. In early 2006 problems in a government's financial monitoring software resulted in incorrect election candidate financial reports being made available to the public. The government's election finance reporting web site had to be shut down until the software was repaired. Trading on a major Asian stock exchange was brought to a halt in November of 2005, reportedly due to an error in a system software upgrade. The problem was rectified and trading resumed later the same day. A May 2005 newspaper article reported that a major hybrid car manufacturer had to install a software fix on 20,000 vehicles due to problems with invalid engine warning lights and occasional stalling. In the article, an automotive software specialist indicated that the automobile industry spends $2 billion to $3 billion per year fixing software problems. Media reports in January of 2005 detailed severe problems with a $170 million highprofile U.S. government IT systems project. Software testing was one of the five major problem areas according to a report of the commission reviewing the project. In March of 2005 it was decided to scrap the entire project. In July 2004 newspapers reported that a new government welfare management system in Canada costing several hundred million dollars was unable to handle a simple benefits rate increase after being put into live operation. Reportedly the original contract allowed for only 6 weeks of acceptance testing and the system was never tested for its ability to handle a rate increase. Millions of bank accounts were impacted by errors due to installation of inadequately tested software code in the transaction processing system of a major North American bank, according to mid-2004 news reports. Articles about the incident stated that it took two weeks to fix all the resulting errors, that additional problems resulted when the incident drew a large number of e-mail phishing attacks against the bank's customers, and that the total cost of the incident could exceed $100 million. A bug in site management software utilized by companies with a significant percentage of worldwide web traffic was reported in May of 2004. The bug resulted in performance problems for many of the sites simultaneously and required disabling of the software until the bug was fixed. According to news reports in April of 2004, a software bug was determined to be a major contributor to the 2003 Northeast blackout, the worst power system failure in North American history. The failure involved loss of electrical power to 50 million customers, forced shutdown of 100 power plants, and economic losses estimated at $6 billion. The bug was reportedly in one utility company's vendor-supplied power monitoring and management system, which was unable to correctly handle and report on an unusual confluence of initially localized events. The error was found and corrected after examining millions of lines of code. In early 2004, news reports revealed the intentional use of a software bug as a counterespionage tool. According to the report, in the early 1980's one nation surreptitiously allowed a hostile nation's espionage service to steal a version of sophisticated industrial software that had intentionally-added flaws. This eventually resulted in major industrial disruption in the country that used the stolen flawed software. A major U.S. retailer was reportedly hit with a large government fine in October of 2003 due to web site errors that enabled customers to view one another‘s' online orders. News stories in the fall of 2003 stated that a manufacturing company recalled all their transportation products in order to fix a software problem causing instability in certain




 



 

 


circumstances. The company found and reported the bug itself and initiated the recall procedure in which a software upgrade fixed the problems. In August of 2003 a U.S. court ruled that a lawsuit against a large online brokerage company could proceed; the lawsuit reportedly involved claims that the company was not fixing system problems that sometimes resulted in failed stock trades, based on the experiences of 4 plaintiffs during an 8-month period. A previous lower court's ruling that "...six miscues out of more than 400 trades do not indicate negligence." was invalidated. In April of 2003 it was announced that a large student loan company in the U.S. made a software error in calculating the monthly payments on 800,000 loans. Although borrowers were to be notified of an increase in their required payments, the company will still reportedly lose $8 million in interest. The error was uncovered when borrowers began reporting inconsistencies in their bills. News reports in February of 2003 revealed that the U.S. Treasury Department mailed 50,000 Social Security checks without any beneficiary names. A spokesperson indicated that the missing names were due to an error in a software change. Replacement checks were subsequently mailed out with the problem corrected, and recipients were then able to cash their Social Security checks. In March of 2002 it was reported that software bugs in Britain's national tax system resulted in more than 100,000 erroneous tax overcharges. The problem was partly attributed to the difficulty of testing the integration of multiple systems. A newspaper columnist reported in July 2001 that a serious flaw was found in off-theshelf software that had long been used in systems for tracking certain U.S. nuclear materials. The same software had been recently donated to another country to be used in tracking their own nuclear materials, and it was not until scientists in that country discovered the problem, and shared the information, that U.S. officials became aware of the problems. According to newspaper stories in mid-2001, a major systems development contractor was fired and sued over problems with a large retirement plan management system. According to the reports, the client claimed that system deliveries were late, the software had excessive defects, and it caused other systems to crash. In January of 2001 newspapers reported that a major European railroad was hit by the aftereffects of the Y2K bug. The company found that many of their newer trains would not run due to their inability to recognize the date '31/12/2000'; the trains were started by altering the control system's date settings. News reports in September of 2000 told of a software vendor settling a lawsuit with a large mortgage lender; the vendor had reportedly delivered an online mortgage processing system that did not meet specifications, was delivered late, and didn't work. In early 2000, major problems were reported with a new computer system in a large suburban U.S. public school district with 100,000+ students; problems included 10,000 erroneous report cards and students left stranded by failed class registration systems; the district's CIO was fired. The school district decided to reinstate it's original 25-year old system for at least a year until the bugs were worked out of the new system by the software vendors. A review board concluded that the NASA Mars Polar Lander failed in December 1999 due to software problems that caused improper functioning of retro rockets utilized by the Lander as it entered the Martian atmosphere. In October of 1999 the $125 million NASA Mars Climate Orbiter spacecraft was believed to be lost in space due to a simple data conversion error. It was determined that spacecraft software used certain data in English units that should have been in metric units. Among other tasks, the orbiter was to serve as a communications relay for the Mars Polar Lander mission, which failed for unknown reasons in December 1999. Several investigating panels were convened to determine the process failures that allowed the error to go undetected. Bugs in software supporting a large commercial high-speed data network affected 70,000 business customers over a period of 8 days in August of 1999. Among those affected



 




 




was the electronic trading system of the largest U.S. futures exchange, which was shut down for most of a week as a result of the outages. In April of 1999 a software bug caused the failure of a $1.2 billion U.S. military satellite launch, the costliest unmanned accident in the history of Cape Canaveral launches. The failure was the latest in a string of launch failures, triggering a complete military and industry review of U.S. space launch programs, including software integration and testing processes. Congressional oversight hearings were requested. A small town in Illinois in the U.S. received an unusually large monthly electric bill of $7 million in March of 1999. This was about 700 times larger than its normal bill. It turned out to be due to bugs in new software that had been purchased by the local power company to deal with Y2K software issues. In early 1999 a major computer game company recalled all copies of a popular new product due to software problems. The company made a public apology for releasing a product before it was ready. The computer system of a major online U.S. stock trading service failed during trading hours several times over a period of days in February of 1999 according to nationwide news reports. The problem was reportedly due to bugs in a software upgrade intended to speed online trade confirmations. In April of 1998 a major U.S. data communications network failed for 24 hours, crippling a large part of some U.S. credit card transaction authorization systems as well as other large U.S. bank, retail, and government data systems. The cause was eventually traced to a software bug. January 1998 news reports told of software problems at a major U.S. telecommunications company that resulted in no charges for long distance calls for a month for 400,000 customers. The problem went undetected until customers called up with questions about their bills. In November of 1997 the stock of a major health industry company dropped 60% due to reports of failures in computer billing systems, problems with a large database conversion, and inadequate software testing. It was reported that more than $100,000,000 in receivables had to be written off and that multi-million dollar fines were levied on the company by government agencies. A retail store chain filed suit in August of 1997 against a transaction processing system vendor (not a credit card company) due to the software's inability to handle credit cards with year 2000 expiration dates. In August of 1997 one of the leading consumer credit reporting companies reportedly shut down their new public web site after less than two days of operation due to software problems. The new site allowed web site visitors instant access, for a small fee, to their personal credit reports. However, a number of initial users ended up viewing each others' reports instead of their own, resulting in irate customers and nationwide publicity. The problem was attributed to "...unexpectedly high demand from consumers and faulty software that routed the files to the wrong computers." In November of 1996, newspapers reported that software bugs caused the 411 telephone information system of one of the U.S. RBOC's to fail for most of a day. Most of the 2000 operators had to search through phone books instead of using their 13,000,000-listing database. The bugs were introduced by new software modifications and the problem software had been installed on both the production and backup systems. A spokesman for the software vendor reportedly stated that 'It had nothing to do with the integrity of the software. It was human error.' On June 4 1996 the first flight of the European Space Agency's new Ariane 5 rocket failed shortly after launching, resulting in an estimated uninsured loss of a half billion dollars. It was reportedly due to the lack of exception handling of a floating-point error in a conversion from a 64-bit integer to a 16-bit signed integer. Software bugs caused the bank accounts of 823 customers of a major U.S. bank to be credited with $924,844,208.32 each in May of 1996, according to newspaper reports. The American Bankers Association claimed it was the largest such error in banking history. A


bank spokesman said the programming errors were corrected and all funds were recovered. Software bugs in a Soviet early-warning monitoring system nearly brought on nuclear war in 1983, according to news reports in early 1999. The software was supposed to filter out false missile detections caused by Soviet satellites picking up sunlight reflections off cloud-tops, but failed to do so. Disaster was averted when a Soviet commander, based on what he said was a '...funny feeling in my gut', decided the apparent missile attack was a false alarm. The filtering software code was rewritten.  Does every software project need testers?

While all projects will benefit from testing, some projects may not require independent test staff to succeed. Which projects may not need independent test staff? The answer depends on the size and context of the project, the risks, the development methodology, the skill and experience of the developers, and other factors. For instance, if the project is a short-term, small, low risk project, with highly experienced programmers utilizing thorough unit testing or test-first development, then test engineers may not be required for the project to succeed. In some cases an IT organization may be too small or new to have a testing staff even if the situation calls for it. In these circumstances it may be appropriate to instead use contractors or outsourcing, or adjust the project management and development approach (by switching to more senior developers and agile test-first development, for example). Inexperienced managers sometimes gamble on the success of a project by skipping thorough testing or having programmers do post-development functional testing of their own work, a decidedly high risk gamble. For non-trivial-size projects or projects with non-trivial risks, a testing staff is usually necessary. As in any business, the use of personnel with specialized skills enhances an organization's ability to be successful in large, complex, or difficult tasks. It allows for both a) deeper and stronger skills and b) the contribution of differing perspectives. For example, programmers typically have the perspective of 'what are the technical issues in making this functionality work?'. A test engineer typically has the perspective of 'what might go wrong with this functionality, and how can we ensure it meets expectations?' Technical people who can be highly effective in approaching tasks from both of those perspectives are rare, which is why, sooner or later, organizations bring in test specialists.    Why does software have bugs?


Miscommunication or no communication - as to specifics of what an application should or shouldn't do (the application's requirements). Software complexity - the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development. Multitiered applications, client-server and distributed applications, data communications, enormous relational databases, and sheer size of applications have all contributed to the exponential growth in software/system complexity. Programming errors - programmers, like anyone else, can make mistakes.



changing requirements (whether documented or undocumented) - the end-user may not understand the effects of changes, or may understand and request them anyway redesign, rescheduling of engineers, effects on other projects, work already completed that may have to be redone or thrown out, hardware requirements that may be affected, etc. If there are many minor changes or any major changes, known and unknown dependencies among parts of the project are likely to interact and cause problems, and the complexity of coordinating changes may result in errors. Enthusiasm of engineering staff may be affected. In some fast-changing business environments, continuously modified requirements may be a fact of life. In this case, management must understand the resulting risks, and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control – Time pressures - scheduling of software projects is difficult at best, often requiring a lot of guesswork. When deadlines loom and the crunch comes, mistakes will be made. egos - people prefer to say things like: 'no problem' 'piece of cake' 'I can whip that out in a few hours' 'it should be easy to update that old code' instead of: 'that adds a lot of complexity and we could end up making a lot of mistakes' 'we have no idea if we can do that; we'll wing it' 'I can't estimate how long it will take, until I take a close look at it' 'we can't figure out what that old spaghetti code did in the first place' If there are too many unrealistic 'no problem's', the Result is bugs. Poorly documented code - it's tough to maintain and modify code that is badly written or poorly documented; the result is bugs. In many organizations management provides no incentive for programmers to document their code or write clear, understandable, maintainable code. In fact, it's usually the opposite: they get points mostly for quickly turning out code, and there's job security if nobody else can understand it ('if it was hard to write, it should be hard to read'). software development tools - visual tools, class libraries, compilers, scripting tools, etc. often introduce their own bugs or are poorly documented, resulting in added bugs. 


editRoles in software testing:

Software testing can be done by software testers. Until the 1950s the term software tester was used generally, but later it was also seen as a separate profession. Regarding the periods and the different goals in software testing (see D. Gelperin and W.C. Hetzel) there have been established different roles: test lead/manager, tester, test designer, test automater/automation developer, test administrator, Participants of testing team 1.Tester 2.Developer 3.customer 4.Information Service Management He is the person who collects information from the customer 5.Senior Organization Management 6.Quality team.  Walk through:

A 'walkthrough' is an informal meeting for evaluation or informational purposes. Little or no preparation is usually required.  Inspection:

An inspection is more formalized than a 'walkthrough', typically with 3-8 people including a moderator, reader, and a recorder to take notes. The subject of the inspection is typically a document such as a requirements spec or a test plan, and the purpose is to find problems and see what's missing, not to fix anything. Attendees should prepare for this type of meeting by reading thru the document; most problems will be found during this preparation. The result of the inspection meeting should be a written report. Thorough preparation for inspections is difficult, painstaking work, but is one of the most cost effective methods of ensuring quality. Employees who are most skilled at inspections are like the 'eldest brother' in the parable in 'Why is it often hard for organizations to get serious about quality assurance?' . Their skill may have low visibility but they are extremely valuable to any software development organization, since bug prevention is far more cost-effective than bug detection.       What are 5 common problems in the software development process?

Poor requirements - if requirements are unclear, incomplete, too general, and not testable, there will be problems. Unrealistic schedule - if too much work is crammed in too little time, problems are inevitable. Inadequate testing - no one will know whether or not the program is any good until the customer complains or systems crash. Futurities - requests to pile on new features after development is underway; extremely common. Miscommunication - if developers don't know what's needed or customers have erroneous expectations, problems are guaranteed.  What are 5 common solutions to software development problems?


  


Solid requirements - clear, complete, detailed, cohesive, attainable, testable requirements that are agreed to by all players. Use prototypes to help nail down requirements. In 'agile'-type environments, continuous close coordination with customers/end-users is necessary. Realistic schedules - allow adequate time for planning, design, testing, bug fixing, retesting, changes, and documentation; personnel should be able to complete the project without burning out. Adequate testing - start testing early on, re-test after fixes or changes, plan for adequate time for testing and bug-fixing. 'Early' testing ideally includes unit testing by developers and built-in testing and diagnostic capabilities. Stick to initial requirements as much as possible - be prepared to defend against excessive changes and additions once development has begun, and be prepared to explain consequences. If changes are necessary, they should be adequately reflected in related schedule changes. If possible, work closely with customers/end-users to manage expectations. This will provide them a higher comfort level with their requirements decisions and minimize excessive changes later on. communication - require walkthroughs and inspections when appropriate; make extensive use of group communication tools - groupware, wiki's, bug-tracking tools and change management tools, intranet capabilities, etc.; insure that

information/documentation is available and up-to-date - preferably electronic, not paper; promote teamwork and cooperation; use prototypes and/or continuous communication with end-users if possible to clarify expectations. ………………………………………………………………………………………………………………… DOCUMENTATION

 Test Strategy: How we plan to cover the product so as to develop an adequate assessment of quality. A good test strategy is: Specific Practical Justified The purpose of a test strategy is to clarify the major tasks and challenges of the test project. Test Approach and Test Architecture are other terms commonly used to describe what I’m calling test strategy. Example of a poorly stated (and probably poorly conceived) test strategy: "We will use black box testing, cause-effect graphing, boundary testing, and white box testing to test this product against its specification."
Test Plan - Why

  

Identify Risks and Assumptions up front to reduce surprises later. Communicate objectives to all team members. Foundation for Test Spec, Test Cases, and ultimately the Bugs we find. Failing to plan = planning to fail.

Test Plan - What


Derived from Test Approach, Requirements, Project Plan, Functional Spec., and Design Spec.

    

Details out project-specific Test Approach. Lists general (high level) Test Case areas. Include testing

Risk Assessment.

Include preliminary Test Schedule Lists Resource requirements.


Test Plan

The test strategy identifies multiple test levels, which are going to be performed for the project. Activities at each level must be planned well in advance and it has to be formally documented. Based on the individual plans only, the individual test levels are carried out. Entry means the entry point to that phase. For example, for unit testing, the coding must be complete and then only one can start unit testing. Task is the activity that is performed. Validation is the way in which the progress and correctness and compliance are verified for that phase. Exit tells the completion criteria of that phase, after the validation is done. For example, the exit criterion for unit testing is all unit test cases must pass.  Unit Test Plan {UTP}

The unit test plan is the overall plan to carry out the unit test activities. The lead tester prepares it and it will be distributed to the individual testers, which contains the following sections.


What is to be tested?

The unit test plan must clearly specify the scope of unit testing. In this, normally the basic input/output of the units along with their basic functionality will be tested. In this case mostly the input units will be tested for the format, alignment, accuracy and the totals. The UTP will clearly give the rules of what data types are present in the system, their format and their boundary conditions. This list may not be exhaustive; but it is better to have a complete list of these details.
 Sequence of Testing:

The sequences of test activities that are to be carried out in this phase are to be listed in this section. This includes whether to execute positive test cases first or negative test cases first, to execute test cases based on the priority, to execute test cases based on test groups etc. Positive test cases prove that the system performs what is supposed to do; negative test cases prove that

the system does not perform what is not supposed to do. Testing the screens, files, database etc., are to be given in proper sequence.

Basic Functionality of Units How the independent functionalities of the units are tested which excludes any communication between the unit and other units. The interface part is out of scope of this test level. Apart from the above sections, the following sections are addressed, very specific to unit testing.
     
Unit Testing Tools Priority of Program units Naming convention for test cases Status reporting mechanism Regression test approach ETVX criteria

Integration Test Plan The integration test plan is the overall plan for carrying out the activities in the integration test level, which contains the following sections. What is to be tested? This section clearly specifies the kinds of interfaces fall under the scope of testing internal, external interfaces, with request and response is to be explained. This need not go deep in terms of technical details but the general approach how the interfaces are triggered is explained. Sequence of Integration When there are multiple modules present in an application, the sequence in which they are to be integrated will be specified in this section. In this, the dependencies between the modules play a vital role. If a unit B has to be executed, it may need the data that is fed

by unit A and unit X. In this case, the units A and X have to be integrated and then using that data, the unit B has to be tested. This has to be stated to the whole set of units in the program. Given this correctly, the testing activities will lead to the product, slowly building the product, unit by unit and then integrating them. System Test Plan {STP} The system test plan is the overall plan carrying out the system test level activities. In the system test, apart from testing the functional aspects of the system, there are some special testing activities carried out, such as stress testing etc. The following are the sections normally present in system test plan. What is to be tested? This section defines the scope of system testing, very specific to the project. Normally, the system testing is based on the requirements. All requirements are to be verified in the scope of system testing. This covers the functionality of the product. Apart from this what special testing is performed are also stated here. Functional Groups and the Sequence The requirements can be grouped in terms of the functionality. Based on this, there may be priorities also among the functional groups. For example, in a banking application, anything related to customer accounts can be grouped into one area, anything related to inter-branch transactions may be grouped into one area etc. Same way for the product being tested, these areas are to be mentioned here and the suggested sequences of testing of these areas, based on the priorities are to be described. Acceptance Test Plan {ATP} The client at their place performs the acceptance testing. It will be very similar to the system test performed by the Software Development Unit. Since the client is the one who decides the format and testing methods as part of acceptance testing, there is no specific clue on the way they will carry out the testing. But it will not differ much from the system testing. Assume that all the rules, which are applicable to system test, can be implemented to acceptance testing also.
Since this is just one level of testing done by the client for the overall product, it may include test cases including the unit and integration test level details.

A sample Test Plan Outline along with their description is as shown below:

Test Plan Outline

1. BACKGROUND – This item summarizes the functions of the application system and the tests to be performed. 2. INTRODUCTION 3. ASSUMPTIONS – Indicates any anticipated assumptions which will be made while testing the application. 4. TEST ITEMS - List each of the items (programs) to be tested. 5. FEATURES TO BE TESTED - List each of the features (functions or requirements) which will be tested or demonstrated by the test. 6. FEATURES NOT TO BE TESTED - Explicitly lists each feature, function, or requirement which won't be tested and why not. 7. APPROACH - Describe the data flows and test philosophy. Simulation or Live execution, Etc. This section also mentions all the approaches which will be followed at the various stages of the test execution. 8. ITEM PASS/FAIL CRITERIA Blanket statement - Itemized list of expected output and tolerances 9. SUSPENSION/RESUMPTION CRITERIA - Must the test run from start to completion? Under what circumstances it may be resumed in the middle? Establish check-points in long tests. 10. TEST DELIVERABLES - What, besides software, will be delivered? Test report Test software 11. TESTING TASKS Functional tasks (e.g., equipment set up) Administrative tasks 12. ENVIRONMENTAL NEEDS Security clearance Office space & equipment Hardware/software requirements 13. RESPONSIBILITIES Who does the tasks in Section 10? What does the user do? 14. STAFFING & TRAINING 15. SCHEDULE 16. RESOURCES

17. RISKS & CONTINGENCIES 18. APPROVALS The schedule details of the various test pass such as Unit tests, Integration tests, System Tests should be clearly mentioned along with the estimated efforts. Traceability means that you would like to be able to trace back and forth how and where any workproduct fulfills the directions of the preceeding (source-) product. The matrix deals with the where, while the how you have to do yourself, once you know the where. Take e.g. the Requirement of UserFriendliness (UF). Since UF is a complex concept, it is not solved by just one design-solution and it is not solved by one line of code. Many partial design-solutions may contribute to this Requirement and many groups of lines of code may contribute to it. A Requirements-Design Traceability Matrix puts on one side (e.g. left) the sub-requirements that together are supposed to solve the UF requirement, along with other (sub-)requirements. On the other side (e.g. top) you specify all design solutions. Now you can connect on the crosspoints of

the matrix, which design solutions solve (more, or less) any requirement. If a design solution
does not solve any requirement, it should be deleted, as it is of no value. Having this matrix, you can check whether any requirement has at least one design solution and by checking the solution(s) you may see whether the requirement is sufficiently solved by this (or the set of) connected design(s). If you have to change any requirement, you can see which designs are affected. And if you change any design, you can check which requirements may be affected and see what the impact is. In a Design-Code Traceability Matrix you can do the same to keep trace of how and which code solves a particular design and how changes in design or code affect each other. Demonstrates that the implemented system meets the user requirements. Serves as a single source for tracking purposes. Identifies gaps in the design and testing. Prevents delays in the project timeline, which can be brought about by having to backtrack to fill the gaps.


Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that looks at the code directly and as such comes under the heading of white box testing. Code coverage techniques were amongst the first techniques invented for systematic software testing. The first published reference was by Miller and Maloney in Communications of the ACM in 1963. There are a number of different ways of measuring code coverage, the main ones being:    

Statement Coverage - Has each line of the source code been executed and tested? Condition Coverage - Has each evaluation point (such as a true/false decision) been
executed and tested? Path Coverage - Has every possible route through a given part of the code been executed and tested? Entry/Exit Coverage - Has every possible call and return of the function been executed and tested?

Safety critical applications are often required to demonstrate that testing achieves 100% of
some form of code coverage. Some of the coverage criteria above are connected; for instance, path coverage implies condition, statement and entry/exit coverage. Statement coverage does not imply condition coverage, as the code (in the C programming language) below shows:

void foo(int bar) { printf("This is "); if (bar < 0) { printf("not "); } printf("a positive integer.\n"); return; }

If the function "foo" were called with variable "bar = -1", statement coverage would be achieved. Condition coverage, however, would not. Full path coverage, of the type described above, is usually impractical or impossible. Any module with a succession of n decisions in it can have up to 2n paths within it; loop constructs can result in an infinite number of paths. Many paths may also be infeasible, in that there is no input to the program under test that can cause that particular path to be executed. However, a generalpurpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem). Techniques for practical path coverage testing instead attempt to identify classes of code paths that differ only in the number of loop executions, and to achieve "basis path" coverage the tester must cover all the path classes.

Usually the source code is instrumented and run through a series of tests. The resulting output is then analysed to see what areas of code have not been exercised, and the tests are updated to include these areas as necessary. Combined with other code coverage methods the aim is to develop a rigorous yet manageable set of regression tests. Code coverage is ultimately expressed as a percentage, as in "We have tested 67% of the code." The meaning of this depends on what form(s) of code coverage have been used, as 67% path coverage is more comprehensive than 67% statement coverage. The value of code coverage as a measure of test quality is debated. What should be done after a bug is found? The bug needs to be communicated and assigned to developers that can fix it. After the problem is resolved, fixes should be re-tested, and determinations made regarding requirements for regression testing to check that fixes didn't create problems elsewhere. If a problem-tracking system is in place, it should encapsulate these processes. A variety of commercial problemtracking/management software tools are available Complete information such that developers can understand the bug, get an idea of it's severity, and reproduce it if necessary.                             Bug identifier (number, ID, etc.) Current bug status (e.g., 'Released for Retest', 'New', etc.) The application name or identifier and version The function, module, feature, object, screen, etc. where the bug occurred Environment specifics, system, platform, relevant hardware specifics Test case name/number/identifier One-line bug description Full bug description Description of steps needed to reproduce the bug if not covered by a test case or if the developer doesn't have easy access to the test case/test script/test tool Names and/or descriptions of file/data/messages/etc. used in test File excerpts/error messages/log file excerpts/screen shots/test tool logs that would be helpful in finding the cause of the problem Severity estimate (a 5-level range such as 1-5 or 'critical'-to-'low' is common) Was the bug reproducible? Tester name Test date Bug reporting date Name of developer/group/organization the problem is assigned to Description of problem cause Description of fix Code section/file/module/class/method that was fixed Date of fix Application version that contains the fix Tester responsible for retest Retest date Retest results Regression testing requirements Tester responsible for regression tests Regression testing results


How can it be known when to stop testing?

This can be difficult to determine. Many modern software applications are so complex, and run in such an interdependent environment, that complete testing can never be done. Common factors in deciding when to stop are:       Deadlines (release deadlines, testing deadlines, etc.) Test cases completed with certain percentage passed Test budget depleted Coverage of code/functionality/requirements reaches a specified point Bug rate falls below a certain level Beta or alpha testing period ends

What if there isn't enough time for thorough testing?



Which functionality is most important to the project's intended purpose? Which functionality is most visible to the user? Which functionality has the largest safety impact? Which functionality has the largest financial impact on users? Which aspects of the application are most important to the customer? Which aspects of the application can be tested early in the development cycle? Which parts of the code are most complex, and thus most subject to errors? Which parts of the application were developed in rush or panic mode? Which aspects of similar/related previous projects caused problems? Which aspects of similar/related previous projects had large maintenance expenses? Which parts of the requirements and design are unclear or poorly thought out? What do the developers think are the highest-risk aspects of the application? What kinds of problems would cause the worst publicity? What kinds of problems would cause the most customer service complaints? What kinds of tests could easily cover multiple functionalities? Which tests will have the best high-risk-coverage to time-required ratio?

Risk Analysis: A risk is a potential for loss or damage to an Organization from materialized threats. Risk Analysis attempts to identify all the risks and then quantify the severity of the risks.A threat as we have seen is a possible damaging event. If it occurs, it exploits vulnerability in the security of a computer based system.
Risk Identification: 1. Software Risks: Knowledge of the most common risks associated with Software development, and the platform you are working on. 2. Business Risks: Most common risks associated with the business using the Software 3. Testing Risks: Knowledge of the most common risks associated with Software Testing for the platform you are working on, tools being used, and test methods being applied. 4. Premature Release Risk: Ability to determine the risk associated with releasing unsatisfactory or untested Software Predicts. 5. Risk Methods: Strategies and approaches for identifying risks or problems associated with implementing and operating information technology , products and process; assessing their likelihood, and initiating strategies to test those risks.


How does a client/server environment affect testing? Client/server applications can be quite complex due to the multiple dependencies among clients, data communications, hardware, and servers, especially in multi-tier systems. Thus testing requirements can be extensive. When time is limited (as it usually is) the focus should be on integration and system testing. Additionally, load/stress/performance testing may be useful in determining client/server application limitations and capabilities. There are commercial tools to assist with such testing. How is testing affected by object-oriented designs? Well-engineered object-oriented design can make it easier to trace from code to internal design to functional design to requirements. While there will be little affect on black box testing (where an understanding of the internal design of the application is unnecessary), white-box testing can be oriented to the application's objects. If the application was well-designed this can simplify test design.

Test Management
1. Why test - what is Testing? Testing is a process used to help identify the correctness, completeness and quality of developed computer software. 2.

System Testing myths and legends - What are they?
Myth1: There is no need to test Myth2: If testing must be done; two weeks at the end of the project is sufficient for testing Myth3: Re-testing is not necessary Myth4: Any fool can test Myth5: The last thing you want is users involved in test Myth6: The V-model is too complicated


What are the Concepts for Application Test Management?

         

Testing should be pro-active following the V-model Test execution can be a manual process Test execution can be an automated process It is possible to plan the start date for testing It is not possible to accurately plan the end date of testing Ending testing is through risk assessment A fool with a tool is still a fool Testing is not a diagnosis process Testing is a triage process Testing is expensive


Not testing, can be more expensive

What Test Principles do you Recommend? • Test involvement early in the lifecycle - Test Architect Signs off Requirements - Test Architect Signs off Use Cases • Fail Fast - Identify failures early via core test scripts • All Test Phases have equal value - Each Test Phase has its own value add • RACI chart everything • Testing is a pro-active activity - Plan the Test - Test the Plan • Finding defects is good - Ignorance of faults in a non-conformant system is no excuse


Test Analysts - What is their Value Add?

     

Understand the system under test Document Assumptions Create and execute repeatable tests Value add through negative testing Contribute to Impact Analysis when assessing Changes

Contribute to the risk assessment when considering to end testing

What do Test Analysts Need?

   

Education Test Environment Test Tools


Requirements Traceability - What is this about?

    

Tracing requirements to test cases Tracing test cases to requirements Should be a feature of the Test Asset Management tool Automatic on-demand process

Pie chart reporting

What is involved in the Application Test Lifecycle?

     

Unit testing Module testing Component testing Component integration testing Subsystem testing System testing

       

Functional testing Technical integration testing System integration testing Non-functional testing Integration testing Regression testing Model Office testing

User Acceptance testing

How to manage Risk Mitigation?

    

Identify risks before the adversity affects the project Analyze risk data for interpretation by the project team Plan actions for probability, magnitude & consequences Track risks and actions, maintaining a risk register

Control risk action plan, correct plan deviations

10. What should the Test Team do?

    

Programme Management Strong

Change Management

Strict Configuration Control Pro Active Scope Creep Management Inclusion in the decision making process

11. What are the Test Team Deliverables

    

Test Plans Test Script Planner Test Scripts Test Execution Results

Defect Reports


MANUAL TESTING is a part of software testing that requires operator input, analysis or evaluation. It means a value derived by adding several qualities and dividing the sum by the number of these quantities. Manual Testing is self doing testing means we test our software without help of tools. The basic meaning of Manual Testing is do all things with urself. The following are the suitable steps for Manual Testing: 1. First generate the test data. 2. Then make test Script. 3. After that check them, they are running or not. Or in other words: 1. Give input first. 2. Raw application with partial documentation. 3. Then study questionare, interview, demo & training - to get knowledge transfer. 4. Now prepare test plans and test cases. 5. After preparing test cases then run test cases. 6. Then send report. 7. So there are two cases when we check report: (a) If report is rejected then update Tests and again follow step no. 4, 5 and 6. (b) If report is accepted then we get application with guaranteed Quality. 8. And finally we get the output. So we know that Manual Testing is a very important part of the software testing. It plays an important role in it. After applying these steps we can get guaranteed QUALITY SOFTWARE. MANUAL TESTING starts with careful planning of the TEST PROCESS and TEST CASES. Or in other words we can say that Manual Testing can be defined as developing and executing tests that rely primarily on DIRECT HUMAN INTERACTION throughout the entire test cycle , especially in terms of evaluating correctness and ascertaining test status . MANUAL TESTING can include the use of tools to track and monitor the defects. Manual Testing is done by the software tester who carries out all the steps on the tested application, step by step and tells whether a particular step was running successfully or whether it failed. Manual Testing is very useful in the starting phase of the Software Development Life Cycle.

………………………………………………………………………………………………………………… SOFTWARE RELEASE LIFE CYCLE What is the 'software life cycle'? The life cycle begins when an application is first conceived and ends when it is no longer in use. It includes aspects such as initial concept, requirements analysis, functional design, internal design, documentation planning, test planning, coding, document preparation, integration, testing, maintenance, updates, retesting, phase-out, and other aspects.

A software release refers to the distribution, whether public or private, of an initial or new and upgraded version of a computer software product. Each time a software program or system is changed, the programmers and company doing the work decide on how to distribute the program or system, or changes to that program or system. Software patches are one method of distributing the changes, as are downloads and compact discs .

Software release stages The software release life cycle is composed of different stages that describe the stability of a piece of software and the amount of development it requires before final release. Each major version of a product usually goes through a stage when new features are added, or the alpha stage; a stage when it is being actively debugged, or the beta stage; and finally a stage when all important bugs have been removed, or the stable stage. Intermediate stages may also be

recognized. The stages may be formally announced and regulated by the project's developers, but sometimes the terms are used informally to describe the state of a product. Conventionally, code names are often used by many companies for versions prior to the release of the product, though the actual product and features are rarely secret.

edit Software release stages
edit Pre-alpha
Sometimes a build known as pre-alpha is issued, before the release of an alpha or beta. In contrast to alpha and beta versions, the pre-alpha is usually not "feature complete". At this stage designers are still determining exactly what functionalities the product should and should not have. This can be equivalent to Prototype in hardware.

edit Alpha
The alpha version of a product still awaits full debugging or full implementation of all its functionality but satisfies a majority of the software requirements . It often lacks features promised in the final release but demonstrates the feasibility and basic structure of the software. As the first major stage in the release lifecycle, it is named after the Greek letter alpha, the first letter in the Greek alphabet. The alpha build of the software is usually the first build delivered to the software testers . In the first phase of alpha testing, developers test the software using white box techniques . Additional inspection is then performed using black box or grey box techniques. This is usually done by another dedicated testing team sometimes concurrently. Moving to black box testing is often known as the second stage of alpha testing.

edit Beta
A beta version or beta release usually represents the first version of a computer program that implements all features in the initial requirements analysis . It is likely to be useful for internal demonstrations and previews to select customers, but unstable and not yet ready for release. Some developers refer to this stage as a preview, as a technical preview (TP) or as an early access. As the second major stage in the release lifecycle, following the alpha stage, it is named after the Greek letter beta, the second letter in the Greek alphabet. Often this stage begins when the developers announce a feature freeze on the product, indicating that no more feature requirements will be accepted for this version of the product. Only software issues, or bugs and unimplemented features will be addressed. Beta versions stand at an intermediate step in the full development cycle. Developers release either a closed beta or an open beta; closed beta versions are released to a select group of individuals for a user test, while open betas are to a larger community group, usually the general

public. The testers report any bugs that they found and sometimes minor features they would like to see in the final version. An example of a major public beta test was when Microsoft started releasing regular Windows Vista Community Technology Previews (CTP) to beta testers starting in January 2005. The first of these was build 5219. Subsequent CTPs introduced most of the planned features, as well as a number of changes to the user interface, based in large part on feedback from beta testers. Windows Vista was deemed feature complete with the release of build 5308 CTP, released on February 22, 2006, and much of the remainder of work between that build and the final release of the product focused on stability, performance, application and driver compatibility, and documentation. When a beta becomes available to the general public it is often widely used by the technologically savvy and those familiar with previous versions as though it were the finished product. Usually developers of freeware or open-source betas release them to the general public while proprietary betas go to a relatively small group of testers. Recipients of highly proprietary betas may have to sign a non-disclosure agreement. A release is called feature complete when the product team agrees that functional requirements of the system are met and no new features will be put into the release, but significant software bugs may still exist. Companies with a formal software process will tend to enter the beta period with a list of known bugs that must be fixed to exit the beta period, and some companies make this list available to customers and testers. As the internet has allowed for rapid and inexpensive distribution of software, companies have begun to take a more flexible approach to use of the word "beta". Netscape Communications was infamous for releasing alpha level versions of its Netscape web browser as public beta releases. In February 2005, ZDNet published an article about the recent phenomenon of a beta version often staying for years and being used as if it were in productionlevel [1]. It noted that Gmail and Google News , for example, had been in beta for a long period of time and were not expected to drop the beta status despite the fact that they were widely used; however, Google News did leave beta in January 2006. This technique may also allow a developer to delay offering full support and/or responsibility for remaining issues. In the context of Web 2.0, people even talk of perpetual betas to signify that some software is meant to stay in beta state.

edit Origin of 'alpha' and 'beta'
The term beta test applied to software comes from an early IBM hardware product test convention dating back to punched card tabulating and sorting machines. Hardware first went through an alpha test for preliminary functionality and manufacturing feasibility. Then came a beta test to verify that it actually correctly performed the functions it was supposed to, and then a c test to verify safety. With the advent of programmable computers and the first shareable software programs, IBM used the same terminology for testing software. Beta tests were conducted by people or groups other than the developers. As other companies began developing software for their own use, and for distribution to others, the terminology stuck and now is part of our common vocabulary.

edit Release candidate

Microsoft Corporation often uses the term release candidate. During the 1990s, Apple Computer used the term "golden master" for its release candidates, and the final golden
master was the general availability release. Other terms include gamma (and occasionally also delta, and perhaps even more Greek letters) for versions that are substantially complete, but still under test, and omega for final testing of versions that are believed to be bug-free, and may go into production at any time. Gamma, delta, and omega are, respectively, the third, fourth, and last letters of the Greek alphabet. Some users disparagingly refer to release candidates and even final "point oh" releases as "gamma test" software, suggesting that the developer has chosen to use its customers to test software that is not truly ready for general release. Often, beta testers, if privately selected, will be billed for using the release candidate as though it were a finished product. A release is called code complete when the development team agrees that no entirely new source code will be added to this release. There may still be source code changes to fix defects. There may still be changes to documentation and data files, and to the code for test cases or utilities. New code may be added in a future release.

edit Gold/general availability release
The gold or general availability release version is the final version of a particular product. It is typically almost identical to the final release candidate, with only last-minute bugs fixed. A gold release is considered to be very stable and relatively bug-free with a quality suitable for wide distribution and use by end users. In commercial software releases, this version may also be signed (used to allow end-users to verify that code has not been modified since the release). The expression that a software product "has gone gold" means that the code has been completed and "is being mass-produced and will be for sale soon." Other terms for the version include gold master, gold release, or gold build. The term gold anecdotally refers to the use of "gold master disc" which was commonly used to send the final version to manufacturers who use it to create the mass-produced retail copies. It may in this context be a hold-over from music production. In some cases, however, the master disc is still actually made of gold, for both aesthetic appeal and resistance to corrosion.

Microsoft and others use the term "release to manufacturing" (RTM) to refer to this version (for commercial products, like Windows XP, as in, "Build 2600 is the Windows XP RTM release"),
and "release to Web" (RTW) for freely downloadable products.

edit Box Copy

A box copy is the final product, printed on a disc that is included in the actual release, complete with disc graphic art. This term is used mostly by reviewers to differentiate from gold master discs. A box copy does not necessarily come enclosed in the actual boxed product - it refers to the disc itself.

edit Stable/unstable
In open source programming, version numbers or the terms stable and unstable commonly distinguish the stage of development. The term stable refers to a version of software that is substantially identical to a version that has been through enough real-world testing to reasonably assume there are no showstopper problems, or at least that any problems are known and documented. On the other hand, the term unstable does not necessarily mean that there are problems - rather, that enhancements or changes have been made to the software that have not undergone rigorous testing and that more changes are expected to be imminent. Users of such software are advised to use the stable version if it meets their needs, and to only use the unstable version if the new functionality is of interest that exceeds the risk that something might simply not work right. In the Linux kernel, version numbers take the form of three numbers, separated by a decimal point. Prior to the 2.6.x series, an even second number was used to represent a stable release and an odd second number used to represent an unstable release. As of the 2.6.x series, the even or odd status of the second number no longer holds any significance. The practice of using even and odd numbers to indicate the stability of a release has been used by many other open and closed source projects.

editA sample testing cycle
Although testing varies between organizations, there is a cycle to testing: 1. Requirements Analysis: Testing should begin in the requirements phase of the software development life cycle. During the design phase, testers work with developers in determining what aspects of a design are testable and under what parameter those tests work. 2. Test Planning: Test Strategy, Test Plan(s), Test Bed creation. 3. Test Development: Test Procedures, Test Scenarios, Test Cases, Test Scripts to use in testing software. 4. Test Execution: Testers execute the software based on the plans and tests and report any errors found to the development team. 5. Test Reporting: Once testing is completed, testers generate metrics and make final reports on their test effort and whether or not the software tested is ready for release. 6. Retesting the Defects

Software Testing Life Cycle:

The test development life cycle contains the following components:

Requirements Use Case Document

Test Plan
Test Case Test Case execution Report Analysis Bug Analysis Bug Reporting

Typical interaction scenario from a user's perspective for system requirements studies or testing. In other words, "an actual or realistic example scenario". A use case describes the use of a system from start to finish. Use cases focus attention on aspects of a system useful to people outside of the system itself.

 

Users of a program are called users or clients. Users of an enterprise are called customers, suppliers, etc.

Use Case: A collection of possible scenarios between the system under discussion and external actors, characterized by the goal the primary actor has toward the system's declared responsibilities, showing how the primary actor's goal might be delivered or might fail. Use cases are goals (use cases and goals are used interchangeably) that are made up of scenarios. Scenarios consist of a sequence of steps to achieve the goal, each step in a scenario is a sub (or

use case) or an autonomous action that is at the lowest level desired by our use case decomposition. This hierarchical relationship is needed to properly model the requirements of a system being developed. A complete use case analysis requires several levels. In addition the level at which the use case is operating at it is important to understand the scope it is addressing. The level and scope

are important to assure that the language and granularity of scenario steps remain consistent within the use case. There are two scopes that use cases are written from: Strategic and System. There are also three levels: Summary, User and Sub-function. Scopes: Strategic and System

Strategic Scope: The goal (Use Case) is a strategic goal with respect to the system. These goals are goals of value to the organization. The use case shows how the

system is used to benefit the organization.,/p>

These strategic use cases will eventually use some of the same lower level (subordinate) use cases.

System Scope: Use cases at system scope are bounded by the system under development. The goals represent specific functionality required of the system. The majority of the use cases are at system scope. These use cases are often steps in strategic level use cases Levels: Summary Goal , User Goal and Sub-function.

Sub-function Level Use Case: A sub goal or step is below the main level of interest to the user. Examples are "logging in" and "locate a device in a DB". Always at System Scope.

User Level Use Case: This is the level of greatest interest. It represents a user task or elementary business process. A user level goal addresses the question "Does your job performance depend on how many of these you do in a day". For example "Create Site View" or "Create New Device" would be user level goals but "Log In to System" would not. Always at System Scope.
Summary Level Use Case:

Written for either strategic or system scope. They represent collections of User Level Goals. For example summary goal "Configure Data Base" might include as a step, user level goal "Add Device to

database". Either at System of Strategic Scope.

Test Documentation Test documentation is a required tool for managing and maintaining the testing process. Documents produced by testers should answer the following questions:

  

What to test? Test Plan How to test? Test Specification What are the results? Test Results Analysis Report

Bug Life cycle: In entomology(the study of real, living Bugs), the term life cycle refers to the various stages that an insect assumes over its life. If you think back to your high school biology class, you will remember that the life cycle stages for most insects are the egg, larvae, pupae and adult. It seems appropriate, given that

software problems are also called bugs, that a similar life cycle system is

used to identify their stages of life. Figure 18.2 shows an example of the simplest, and most optimal, software bug life cycle.

This example shows that when a bug is found by a Software Tester, its logged and assigned to a programmer to be fixed. This state is called open state. Once the programmer fixes the code , he assigns it back to the tester and the bugs enters the resolved state. The tester then performs a regression test to confirm that the bug is indeed fixed and, if it closes it out. The bug then enters its final state, the closed state. In some situations though, the life cycle gets a bit more complicated. In this case the life cycle starts out the same with the Tester opening the bug and assigning to the programmer, but the programmer doesn‘t fix it. He doesn‘t think its bad enough to fix and assigns it to the project manager to decide. The Project Manager agrees with the Programmer and places the Bug in the resolved state as a ―wont-fix‖ bug. The tester disagrees, looks for and finds a more obvious and general case that demonstrates the bug, reopens it, and assigns it to the Programmer to fix. The programmer fixes the bg, resolves it as fixed, and assign it to the Tester. The tester confirms the fix and closes the bug.

You can see that a bug might undergo numerous changes and iterations over its life, sometimes looping back and starting the life all over again. Figure below takes the simple model above and adds to it possible decisions, approvals, and looping that can occur in most projects. Of course every software company and project will have its own

system, but this figure is fairly generic and

should cover most any bug life cycle that you‘ll encounter

The generic life cycle has two additional states and extra connecting lines. The review state is where Project Manager or the committee, sometimes called a change Control Board, decides whether the bug should be fixed. In some projects all bugs go through the review state before they‘re assigned to the programmer for fixing. In other projects, this may not occur until near the end of the project, or not at all. Notice that the review state can also go directly to the closed state. This happens if the review decides that the bug shouldn‘t be fixed – it could be too minor is really not a problem, or is a testing error. The other is a deferred. The review may determine that the bug should be considered for fixing at sometime in the future, but not for this release of the software. The additional line from resolved state back to the open state covers the situation where the tester finds that the bug hasn‘t been fixed. It gets reopened and the bugs life cycle repeats. The two dotted lines that loop from the closed and the deferred state back to the open state rarely occur but are important enough to mention. Since a Tester never gives up, its possible that a bug was thought to be fixed, tested and closed could reappear. Such bugs are often called Regressions. It‘s possible that a deferred bug could later be proven serious enough to fix immediately. If either of these occurs, the bug is reopened and started through the process again. Most Project teams adopt rules for who can change the state of a bug or assign it to someone else.For example, maybe only the Project Manager can decide to defer a bug or only a tester is permitted to close a bug. What‘s important is that once you log a bug, you follow it through its life cycle, don‘t lose track of it, and prove the necessary information to drive it to being fixed and closed. Bug Report - Why

 

Communicate bug for reproducibility, resolution, and regression. Track bug status (open, resolved, closed).


Ensure bug is not forgotten, lost or ignored.

Used to back create test case where none existed before.

Testing Start Process When Testing should start: Testing early in the life cycle reduces the errors. Test deliverables are associated with every phase of development. The goal of Software Tester is to find bugs, find them as early as possible, and make them sure they are fixed.
The number one cause of Software bugs is the Specification. There are several reasons specifications are the largest bug producer. In many instances a Spec simply isn‘t written. Other reasons may be that the spec isn‘t thorough enough, its constantly changing, or it‘s not communicated well to the entire team. Planning software is vitally important. If it‘s not done correctly bugs will be created. The next largest source of bugs is the Design, That‘s where the programmers lay the plan for their Software. Compare it to an

architect creating the blue print for the building, Bugs occur here for

the same reason they occur in the specification. It‘s rushed, changed, or not well communicated.

Coding errors may be more familiar to you if you are a programmer. Typically these can be traced to the Software complexity, poor documentation, schedule pressure or just plain dump mistakes.

It‘s important to note that many bugs that appear on the surface to be programming errors can really be traced to specification. It‘s quite common to hear a programmer say, ― oh, so that‘s what its supposed to do. If someone had told me that I wouldn‘t have written the code that way.‖ The other category is the catch-all for what is left. Some bugs can blamed for false positives, conditions that were thought to be bugs but really weren‘t. There may be duplicate bugs, multiple ones that resulted from the square root cause. Some bugs can be traced to Testing errors. Costs: The costs re logarithmic- that is, they increase tenfold as time increases. A bug found and fixed during the early stages when the specification is being written might cost next to nothing, or 10 cents in our example. The same bug, if not found until the

software is coded and tested,

might cost $1 to $10. If a customer finds it, the cost would easily top $100.

………………………………………………………………………………………………………………… VALIDATION AND VERIFICATION Verification is one aspect of testing a product's fitness for purpose. Validation is the complementary aspect. Often one refers to the overall checking process as V & V.   Validation: "Are we building the right product?", i.e., does the product do what the user really requires? Verification: "Are we building the product right?", i.e., does the product conform to the specifications?

The verification process consists of static and dynamic parts. E.g., for a software product one can inspect the source code (static) and run against specific test cases (dynamic). Validation usually can only be done dynamically, i.e., the product is tested by putting it through typical usages and atypical usages ("Can we break it?"). Software testing is used in association with Verification and Validation. Verification is the checking of or testing of items, including software, for conformance and consistency with an associated specification. Software testing is just one kind of verification, which also uses techniques as reviews, inspections, walk-through. Validation is the process of checking what has been specified is what the user wanted actually.   Validation: Are we doing the right job? Verification: Are we doing the job right?

In order to achieve consistency in the Testing style, it is imperative to have and follow a set of testing principles. This enhances the efficiency of testing within SQA team members and thus contributes to increased productivity. The purpose of this document is to provide overview of the testing, plus the techniques.

Verification and Validation Verification typically involves reviews and meetings to evaluate documents, plans, code, requirements, and specifications. This can be done with checklists, issues lists, walkthroughs, and inspection meetings. Validation typically involves actual testing and takes place after verifications are completed. The term 'IV & V' refers to Independent Verification and Validation. Formal verification In the context of hardware and software systems, formal verification is the act of proving or disproving the correctness of a system with respect to a certain formal specification or property, using formal methods of mathematics . Explanation

Software testing alone cannot prove that a system does not have a certain defect. Neither can
it prove that it does have a certain property. Only the process of formal verification can prove that a system does not have a certain defect or does have a certain property. It is impossible to prove or test that a system has "no defect" since it is impossible to formally specify what "no defect" means. All that can be done is prove that a system does not have any of the defects that can be thought of, and has all of the properties that together make it functional and useful. Usage Formal verification can be used for example for systems such as cryptographic protocols , combinatorial circuits, digital circuits with internal memory, and software expressed as source code. The verification of these systems is done by providing a formal proof on an abstract mathematical model of the system, the correspondence between the mathematical model and the nature of the system being otherwise known by construction. Examples of mathematical objects often used to model systems are: finite state machines , labelled transition systems , Petri nets, timed automata, hybrid automata, process algebra, formal semantics of programming languages such as operational semantics , denotational semantics , axiomatic semantics and Hoare logic. Approaches to formal verification There are roughly two approaches to formal verification. The first approach is model checking, which consists of a systematically exhaustive exploration of the mathematical model (this is only possible for a model that is finite). Usually this consists of exploring all states and transitions in the model, by using smart and domain-specific abstraction techniques to consider whole groups of states in a single operation and reduce computing time. Implementation techniques include state space enumeration, symbolic state space enumeration, abstract interpretation, symbolic simulation, abstraction refinement.

The second approach is logical inference. It consists of using a formal version of mathematical reasoning about the system, usually using theorem proving software such as the HOL theorem prover or Isabelle theorem prover. This is usually only partially automated and is driven by the user's understanding of the system to validate. The properties to be verified are often described in temporal logics , such as linear temporal logic (LTL) or computational tree logic (CTL). Validation and Verification Verification is one aspect of testing a product's fitness for purpose. Validation is the complementary aspect. Often one refers to the overall checking process as V & V.   Validation: "Are we building the right product?", i.e., does the product do what the user really requires? Verification: "Are we building the product right?", i.e., does the product conform to the specifications?


Program verification Program verification is the process of formally proving that a computer program does exactly what is stated in the program specification it was written to realize. This is a type of formal verification which is specifically aimed at verifying the code itself, not an abstract model of the program.

functional programming languages, some programs can be verified by equational reasoning, usually together with induction. Code in an imperative language could be proved correct by use of Hoare logic



Unit testing tests the minimal software component and sub-component or modules by
the programmers.

   

Integration testing exposes defects in the interfaces and interaction between
integrated components (modules). Functional testing tests the product according to programmable work. System testing tests an integrated system to verify/validate that it meets its requirements. Acceptance testing can be conducted by the client. It allows the end-user or customer or client to decide whether or not to accept the product. Acceptance testing may be performed after the testing and before the implementation phase.

UNIT TESTING In computer programming, unit testing is a procedure used to validate that individual modules or units of source code are working properly. More technically one should consider that a unit is the smallest testable part of an application. In a Procedural Design a unit may be an individual program, function, procedure, web page, menu etc. But in Object Oriented Design, the smallest unit is always a Class; which may be a base/super class, abstract class or derived/child class. A unit test is a test for a specific unit. Ideally, each test case is independent from the others; mock objects and test harnesses can be used to assist testing a module in isolation. Unit testing is typically done by the developers and not by end-users. Benefit The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. Unit testing provides a strict, written contract that the piece of code must satisfy. As a result, it affords several benefits. Facilitates change Unit testing allows the programmer to refractor code at a later date, and make sure the module still works correctly (i.e. regression testing). The procedure is to write test cases for all functions and methods so that whenever a change causes a regression, it can be quickly identified and fixed. This provides the benefit of encouraging programmers to make changes to the code since it is easy for the programmer to check if the piece is still working properly. Good unit test design produces test cases that cover all paths through the unit with attention paid to loop conditions. In continuous unit testing environments, through the inherent practice of sustained maintenance, unit tests will continue to accurately reflect the intended use of the executable and code in the face of any change. Depending upon established development practices and unit test coverage up-to-the-second accuracy can be maintained. Simplifies integration

Unit testing helps to eliminate uncertainty in the units themselves and can be used in a bottomup testing style approach. By testing the parts of a program first and then testing the sum of its parts, integration testing becomes much easier. A heavily debated matter exists in assessing the need to perform manual integration testing. While an elaborate hierarchy of unit tests may seem to have achieved integration testing, this presents a false sense of confidence since integration testing evaluates many other objectives that can only be proven through the human factor. Some argue that given a sufficient variety of test automation systems, integration testing by a human test group is unnecessary. Realistically, the actual need will ultimately depend upon the characteristics of the product being developed and its intended uses. Additionally, the human or manual testing will greatly depend on the availability of resources in the organization. Documentation Unit testing provides a sort of "living document". Clients and other developers looking to learn how to use the module can look at the unit tests to determine how to use the module to fit their needs and gain a basic understanding of the API. Unit test cases embody characteristics that are critical to the success of the unit. These characteristics can indicate appropriate/inappropriate use of a unit as well as negative behaviors that are to be trapped by the unit. A unit test case, in and of itself, documents these critical characteristics, although many software development environments do not rely solely upon code to document the product in development. On the other hand, ordinary narrative documentation is more susceptible to drifting from the implementation of the program and will thus become outdated (e.g. design changes, feature creep, relaxed practices to keep documents up to date). Separation of interface from implementation Because some classes may have references to other classes, testing a class can frequently spill over into testing another class. A common example of this is classes that depend on a database: in order to test the class, the tester often writes code that interacts with the database. This is a mistake, because a unit test should never go outside of its own class boundary. As a result, the software developer abstracts an interface around the database connection, and then implements that interface with their own mock object. By abstracting this necessary attachment from the code (temporarily reducing the net effective coupling), the independent unit can be more thoroughly tested than may have been previously achieved. This results in a higher quality unit that is also more maintainable. In this manner, the benefits themselves begin returning dividends back to the programmer creating a seemingly perpetual upward cycle in quality. Limitations of unit testing Unit testing will not catch every error in the program. By definition, it only tests the functionality of the units themselves. Therefore, it will not catch integration errors, performance problems or

any other system-wide issues. In addition, it may not be easy to anticipate all special cases of input the program unit under study may receive in reality. Unit testing is only effective if it is used in conjunction with other software testing activities . It is unrealistic to test all possible input combinations for any non-trivial piece of software. Like all forms of software testing, unit tests can only show the presence of errors; it cannot show the absence of errors. To obtain the intended benefits from unit-testing, a rigorous sense of discipline is needed throughout the software development process. It is essential to keep careful records, not only of the tests that have been performed, but also of all changes that have been made to the sourcecode of this or any other unit in the software. Use of a so-called "version control system" is essential: if a later version of the unit fails a particular test that it had previously passed, the version-control software can provide list of the source-code changes (if any) that have been applied to the unit since that time. Applications Extreme Programming The cornerstone of Extreme Programming (XP) is the unit test. XP relies on an automated unit testing framework. This automated unit testing framework can be either third party, e.g. xUnit, or created within the development group. Extreme Programming uses the creation of unit tests for Test Driven Development. The developer writes a unit test that exposes either a software requirement or a defect. This test will fail because either the requirement isn't implemented yet, or because it intentionally exposes a defect in the existing code. Then, the developer writes the simplest code to make the test, along with other tests, pass. All classes in the system are unit tested. Developers release unit testing code to the code repository in conjunction with the code it tests. XP's thorough unit testing allows the benefits mentioned above, such as simpler and more confident code development and refactoring, simplified code integration, accurate documentation, and more modular designs. These unit tests are also constantly run as a form of regression test. What is Extreme Programming and what's it got to do with testing? Extreme Programming (XP) is a software development approach for small teams on risk-prone projects with unstable requirements. It was created by Kent Beck who described the approach in his book 'Extreme Programming Explained' Testing ('extreme testing') is a core aspect of Extreme Programming. Programmers are expected to write unit and functional test code first - before writing the application code. Test code is under source control along with the rest of the code. Customers are expected to be an integral part of the project team and to help developed scenarios for acceptance/black box testing. Acceptance tests are preferably automated, and are modified and rerun for each of the frequent development iterations. QA and test personnel are also required to be an integral part of the project team. Detailed requirements documentation is

not used, and frequent re-scheduling, re-estimating, and re-prioritizing is expected. For more info on XP and other 'agile' software development approaches (Scrum, Crystal, etc.) Techniques Unit testing is commonly automated, but may still be performed manually. As with many reputable standards bodies, the IEEE prescribes neither an automated nor a manual approach. A manual approach to unit testing may employ a step-by-step instructional document. Nevertheless, the objective in unit testing is to isolate a unit and validate its correctness. Automation is efficient for achieving this, and enables the many benefits listed in this article. Conversely, if not planned carefully, a careless manual unit test case may execute as an integration test case that involves many software components, and thus preclude the achievement of most if not all of the goals established for unit testing. Under the automated approach, to fully realize the effect of isolation, the unit or code body subjected to the unit test is executed within a framework outside of its natural environment, that is, outside of the product or calling context for which it was originally created. Testing in an isolated manner has the benefit of revealing unnecessary dependencies between the code being tested and other units or data spaces in the product. These dependencies can then be eliminated through refactoring, or if necessary, re-design. Using a unit testing framework, the developer codifies criteria into the unit test to verify the correctness of the unit under test. During execution of the unit test(s), the framework logs test cases that fail any criterion. Many frameworks will also automatically flag and report in a summary these failed test cases. Depending upon the severity of a failure, the framework may halt subsequent testing. As a consequence, unit testing is traditionally a motivator for programmers to create decoupled and cohesive code bodies. This practice promotes healthy habits in software development. Design patterns, unit testing, and refactoring often work together so that the most ideal solution may emerge. Unit testing frameworks Unit testing frameworks, which help simplify the process of unit testing, have been developed for a wide variety of languages . It is generally possible to perform unit testing without the support of specific framework by writing client code that exercises the units under test and uses assertion, exception, or early exit mechanisms to signal failure. This approach is valuable in that there is a negligible barrier to the adoption of unit testing. However, it is also limited in that many advanced features of a proper framework are missing or must be hand-coded. To address this issue the D programming language offers direct support for unit testing. ERROR HANDLING TESTING

Usage: It determines the ability of applications properly

system to process the incorrect transactions

 

Errors encompass all unexpected conditions. In some system approx. 50% of programming effort will be devoted to handling error condition.

Objective: Determine Application system recognizes all expected error conditions

 

Determine Accountability of processing errors has been assigned and procedures provide a high probability that errors will be properly corrected Determine During correction process reasonable control is maintained over errors.

How to Use A group of knowledgeable people is required to anticipate what can go wrong in the application system.

 

It is needed that all the application knowledgeable people assemble to integrate their knowledge of user area, auditing and error tracking. Then logical test error conditions should be created based on this assimilated information.

When to Use

Throughout SDLC.

 

Impact from errors should be identified and should be corrected to reduce the errors to acceptable level. Used to assist in error management process of system development and maintenance.


Create a set of erroneous transactions and enter them into the application system then find out whether the system is able to identify the problems..


Using iterative testing enters transactions and trap errors. Correct them. Then enter transactions with errors, which were not present in the system earlier.


It involves testing of all the functions performed by the people while preparing the data and using these data from automated Objective:


Verify manual support documents and procedures are correct.

  

Determine Manual support responsibility is correct Determine Manual support people are adequately trained. Determine Manual support and automated segment are properly interfaced.

How to Use Process evaluated in all segments of SDLC.

  

Execution of the can be done in conjunction with normal system testing. Instead of preparing, execution and entering actual test transactions the clerical and supervisory personnel can use the results of processing from application system. To test people it requires testing the interface between the people and application system. When to Use Verification that manual systems function properly should be conducted throughout the SDLC.

 

Should not be done at later stages of SDLC. Best done at installation stage so that the clerical people do not get used to the actual system just before system goes to production.


Provide input personnel with the type of information they would normally receive from their customers and then have them transcribe that information and enter it in the



Users can be provided a series of test conditions and then asked to respond to those conditions. Conducted in this manner, manual support testing is like an examination in

which the users are asked to obtain the answer from the procedures and manuals available to them.

Usage: To ensure interconnection between application functions correctly. Objective: Determine Proper parameters and data are correctly passed between the applications

 

Documentation for involved

system is correct and accurate.

Ensure Proper timing and coordination of functions exists between the application system.

How to Use Operations of multiple systems are tested.


Multiple systems are run from one another to check that they are acceptable and processed properly.

When to Use When there is change in parameters in application system

 

The parameters, which are erroneous then risk associated to such parameters, would decide the extent of testing and type of testing. Intersystem parameters would be checked / verified after the change or new application is placed in the production.

Example Develop test transaction set in one application and passing to another system to verify the processing.

 

Entering test transactions in live production environment and then using integrated test facility to check the processing from one system to another. Verifying new changes of the parameters in the system, which are being tested, are corrected in the document.


Time consuming and tedious if test automation not done

 

Cost may be expensive if system is run several times iteratively.

Usage: Control is a

management tool to ensure that processing is performed in accordance

to what management desire or intents of management. Objective: Accurate and complete data

   

Authorized transactions Maintenance of adequate audit trail of information. Efficient, effective and economical process. Process meeting the needs of the user.

How to Use To test controls risks must be identified.

 

Testers should have negative approach i.e. should determine or anticipate what can go wrong in the application system. Develop risk matrix, which identifies the risks, controls; segment within application system in which control resides.

When to Use Should be tested with other

system tests.

Example file reconciliation procedures work


Manual controls in place.




To ensure that the processing of new application (new version) is consistent with respect to the processing of previous application version.


 

Conducting redundant processing to ensure that the new version or application performs correctly. Demonstrating consistency and inconsistency between 2 versions of the application.

How to Use

 

Same input data should be run through 2 versions of same application system. Parallel testing can be done with whole system or part of system (segment).

When to Use

 

When there is uncertainty regarding correctness of processing of new application where the new and old version are similar. In financial applications like banking where there are many similar applications the processing can be verified for old and new version through parallel testing


 

Operating new and old version of a

payroll system to determine that the paychecks

from both systems are reconcilable. Running old version of application to ensure that the functions of old system are working fine with respect to the problems encountered in the new system.

Whichever title you choose (for us volume test) here we are talking about realistically exercising an application in order to measure the service delivered to users at different levels of usage. We are particularly interested in its behavior when the maximum number of users are concurrently active and when the

database contains the greatest data volume.

The creation of a volume test environment requires considerable effort. It is essential that the correct level of complexity exists in terms of the data within the database and the range of transactions and data used by the scripted users, if the tests are to reliably reflect the to be production environment. Once the test environment is built it must be fully utilised. Volume tests offer much more than simple service delivery measurement. The exercise should seek to answer the following questions: What service level can be guaranteed. How can it be specified and monitored?

Are changes in user behaviour likely? What impact will such changes have on resource consumption and service delivery?

Which transactions/processes are resource hungry in relation to their tasks?

What are the resource bottlenecks? Can they be addressed?

How much spare capacity is there?

The purpose of volume testing is to find weaknesses in the large amount of data during extended time periods.

system with respect to its handling of

The purpose of stress testing is to find defects of the

system capacity of handling large numbers

of transactions during peak periods. For example, a script might require users to login and proceed with their daily activities while, at the same time, requiring that a series of workstations emulating a large number of other systems are running recorded scripts that add, update, or delete from the database.

System performance is generally assessed in terms of response time and throughput rates under differing processing and configuration conditions. To attack the performance problems, there are several questions should be asked first: How much application logic should be remotely executed?

How much updating should be done to the database server over the network from the client

How much data should be sent to each in each transaction?

According to Hamilton [10], the performance problems are most often the result of the client or server being configured inappropriately.

The best strategy for improving client-sever performance is a three-step process [11]. First, execute controlled performance tests that collect the data about volume, stress, and loading tests. Second, analyze the collected data. Third, examine and tune the database queries and, if necessary, provide temporary

data storage on the client while the application is executing.

INTEGRATION TESTING Integration testing (sometimes called Integration and Testing, abbreviated I&T) is the phase of software testing in which individual software modules are combined and tested as a group. It follows unit testing and precedes system testing. Integration testing takes as its input modules that have been unit tested, groups them in larger aggregates, applies tests defined in an integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing. Purpose The purpose of integration testing is to verify functional, performance and reliability requirements placed on major design items. These "design items", i.e. assemblages (or groups of units), are exercised through their interfaces using black box testing, success and error cases being simulated via appropriate parameter and data inputs. Simulated usage of shared data areas and inter-process communication is tested and individual subsystems are exercised through their input interface. Test cases are constructed to test that all components within assemblages interact correctly, for example across procedure calls or process activations, and this is done after testing individual modules, i.e. unit testing. The overall idea is a "building block" approach, in which verified assemblages are added to a verified base which is then used to support the integration testing of further assemblages. The different types of integration testing are big bang, top-down, bottom-up, and back bone.

Big Bang: In this approach all or most of the developed modules are coupled together to form a complete software systems or major part of the system and then used for integration testing. The Big Bang method is very effective for saving time in the integration testing process. However, if the test cases and their results are not recorded properly, the entire integration process will be more complicated and may prevent the testing team from achieving the goal of integration testing. Bottom Up: All the bottom or low level modules, procedures or functions are integrated and then tested. After the integration testing of lower level integrated modules, the next level of modules will be formed and can be used for integration testing. This approach is helpful only when all or most of the modules of the same development level are ready. This method also helps to determine the levels of software developed and makes it easier to report testing progress in the form of a percentage. Limitations Any conditions not stated in specified integration tests, outside of the confirmation of the execution of design items, will generally not be tested. Integration tests can not include systemwide (end-to-end) change testing. SYSTEM TESTING

System testing is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic. [1] As a rule, system testing takes, as its input, all of the "integrated" software components that have successfully passed integration testing and also the software system itself integrated with any applicable hardware system(s). The purpose of integration testing is to detect any inconsistencies between the software units that are integrated together (called assemblages) or between any of the assemblages and the hardware. System testing is a more limiting type of testing; it seeks to detect defects both within the "inter-assemblages" and also within the system as a whole.

editTesting the whole system
System testing is actually done to the entire system against the Functional Requirement Specification(s) (FRS) and/or the System Requirement Specification (SRS). Moreover, the System testing is an investigatory testing phase, where the focus is to have almost a destructive attitude and test not only the design, but also the behavior and even the believed expectations of the customer. It is also intended to test up to and beyond the bounds defined in the software/hardware requirements specification(s). One could view System testing as the final destructive testing phase before Acceptance testing.

editTypes of system testing
The following examples are different types of testing that should be considered during System testing:                    

Functional testing User interface testing Usability testing Compatibility testing Model based testing Error exit testing User help testing Security testing Capacity testing Performance testing Sanity testing Smoke testing Exploratory testing Ad hoc testing Regression testing Reliability testing Recovery testing Installation testing Maintenance testing Accessibility testing, including compliance with:  Americans with Disabilities Act of 1990  Section 508 Amendment to the Rehabilitation Act of 1973  Web Accessibility Initiative (WAI) of the World Wide Web Consortium

Although different testing organizations may prescribe different tests as part of System testing, this list serves as a general framework or foundation to begin with. ACCEPTANCE TESTING In engineering and its various sub disciplines , acceptance testing is performed by the customer on a system prior to the customer accepting delivery or accepting transfer of ownership of that system. For systems, it is performed exclusively through black-box testing. Acceptance testing is also performed on lots (mechanical parts) or batches (chemicals) of manufactured or produced items . Depending on the engineering sub discipline, it may also be referred to as functional testing, black-box testing, beta testing, user acceptance testing (UAT), release acceptance, QA testing, application testing, confidence testing, end user testing, final testing, validation testing, factory acceptance testing or site or field acceptance testing. 'Factory acceptance testing'

is run within the manufacturer's facilities whereas the site testing or field testing is run within the user's environment.

Acceptance testing generally involves running a suite of tests on the completed system. Each individual test, known as a case, exercises a particular operating condition of the user's environment or feature of the system, and will result in a pass or fail Boolean outcome. There is generally no degree of success or failure. The test environment is usually designed to be identical, or as close as possible, to the anticipated user's environment, including extremes of such. These test cases must each be accompanied by test case input data or a formal description of the operational activities (or both) to be performed—intended to thoroughly exercise the specific case—and a formal description of the expected results.

The acceptance test suite is run against the supplied input data or using an acceptance test script to direct the testers. Then the results obtained are compared with the expected results. If there is a correct match for every case, the test suite is said to pass. If not, the system may either be rejected or accepted on conditions previously agreed between the sponsor and the manufacturer. The objective is to provide confidence that the delivered system meets the business requirements of both sponsors and users. The acceptance phase may also act as the final quality gateway, where any quality defects not previously detected may be uncovered. A principal purpose of acceptance testing is that, once completed successfully, and provided certain additional (contractually agreed) acceptance criteria are met, the sponsors will then sign off on the system as satisfying the contract (previously agreed between sponsor and manufacturer), and deliver final payment.

editAcceptance Testing in Extreme Programming
Acceptance testing is a term used in agile software development methodologies, particularly Extreme Programming, referring to the functional testing of a user story by the software development team during the implementation phase. The customer specifies scenarios to test when a user story has been correctly implemented. A story can have one or many acceptance tests, what ever it takes to ensure the functionality works. Acceptance tests are black box system tests. Each acceptance test represents some expected result from the system. Customers are responsible for verifying the correctness of the acceptance tests and reviewing test scores to decide which failed tests are of highest priority. Acceptance tests are also used as regression tests prior to a production release. A user story is not considered complete until it has passed its acceptance tests. This means that new


edit List of Acceptance Testing Frameworks
      

Avignon Functional testing tool for HTTP, .NET WinForm and Swing applications. eTester Automated functional testing tool for web applications. Fit FitNesse, a fork of Fit JWebUnit Selenium (software) Watir



It should be noted that although both Alpha and Beta are referred to as testing it is in fact use emersion. The rigors that are applied are often unsystematic and many of the basic tenets of testing process are not used. The Alpha and Beta period provides insight into environmental and utilization conditions that can impact the software. After modifying software, either for a change in functionality or to fix defects, a regression test re-runs previously passing tests on the modified software to ensure that the modifications haven't unintentionally caused a regression of previous functionality. Regression testing can be performed at any or all of the above test levels. These regression tests are often automated. At SDEI, 3 levels of software testing is done at various SDLC phases    Unit Testing: in which each unit (basic component) of the software is tested to verify that the detailed design for the unit has been correctly implemented Integration testing: in which progressively larger groups of tested software components corresponding to elements of the architectural design are integrated and tested until the software works as a whole. System testing: in which the software is integrated to the overall product and tested to show that all requirements are met

A further level of testing is also done, in accordance with requirements:

 

Acceptance testing: upon which the acceptance of the complete software is based. The clients often do this. Regression testing: is used to refer the repetition of the earlier successful tests to ensure that changes made in the software have not introduced new bugs/side effects.

The Verification activities fall into the category of Static Testing. During static testing, you have a checklist to check whether the work you are doing is going as per the set standards of the organization. These standards can be for Coding, Integrating and Deployment. Review's, Inspection's and Walkthrough's are static testing methodologies.

Dynamic Testing involves working with the software, giving input values and checking if the output is as expected. These are the Validation activities. Unit Tests, Integration Tests, System Tests and Acceptance Tests are few of the Dynamic Testing methodologies. As we go further, let us understand the various Test Life Cycle's and get to know the Testing Terminologies. To understand

software testing, various methodologies, tools and techniques, you can download the Software Testing Guide Book from here.
more of Difference Between Static and Dynamic Testing: Please refer the definition of to observe the differnce between the static testing and dynamic testing.

Static Testing

editWhite-box and black-box testing
White box and black box testing are terms used to describe the point of view a test engineer
takes when designing test cases. Black box being an external view of the test object and white box being an internal view. Software testing is partly intuitive, but largely systematic. Good testing involves much more than just running the program a few times to see whether it works. Thorough analysis of the program under test, backed by a broad knowledge of testing techniques and tools are prerequisites to systematic testing. Software Testing is the process of executing software in a controlled manner; in order to answer the question ―Does this software behave as specified?‖ Software testing is used in association with Verification and Validation. Verification is the checking of or testing of items, including software, for conformance and consistency with an associated specification. Software testing is just one kind of verification, which also uses techniques as reviews, inspections, walk-through. Validation is the process of checking what has been specified is what the user wanted actually.   Validation: Are we doing the right job? Verification: Are we doing the job right?

In order to achieve consistency in the Testing style, it is imperative to have and follow a set of testing principles. This enhances the efficiency of testing within SQA team members and thus contributes to increased productivity. The purpose of this document is to provide overview of the testing, plus the techniques. In recent years the term grey box testing has come into common usage. The typical grey box tester is permitted to set up or manipulate the testing environment, like seeding a database, and can view the state of the product after his actions, like performing a SQL query on the database to be certain of the values of columns. It is used almost exclusively of client-server testers or others who use a database as a repository of information, but can also apply to a tester who has to manipulate XML files (DTD or an actual XML file) or configuration files directly. It can also be used of testers who know the internal workings or algorithm of the software under test and can write tests specifically for the anticipated results. For example, testing a data warehouse implementation involves loading the target database with information, and verifying the correctness of data population and loading of data into the correct tables. FUNCTIONAL TESTING BLACK BOX TESTING

Black box testing attempts to derive sets of inputs that will fully exercise all the functional
requirements of a system. It is not an alternative to white box testing. This type of testing attempts to find errors in the following categories:

1. incorrect or missing functions, 2. interface errors, 3. errors in data structures or external database access, 4. performance errors, and 5. initialization and termination errors.

Tests are designed to answer the following questions:

1. How is the function's validity tested? 2. What classes of input will make good test cases? 3. Is the system particularly sensitive to certain input values? 4. How are the boundaries of a data class isolated? 5. What data rates and data volume can the

system tolerate?

6. What effect will specific combinations of data have on system operation?

White box testing should be performed early in the testing process, while black box testing tends to

be applied during later stages. Test cases should be derived which

1. reduce the number of additional test cases that must be designed to achieve reasonable testing, and 2. tell us something about the presence or absence of classes of errors, rather than an error associated only with the specific test at hand.

Equivalence Partitioning

This method divides the input domain of a program into classes of data from which test cases can be derived. Equivalence partitioning strives to define a test case that uncovers classes of errors and thereby reduces the number of test cases needed. It is based on an evaluation of equivalence classes for an input condition. An equivalence class represents a set of valid or invalid states for input conditions.

Equivalence classes may be defined according to the following guidelines:

1. If an input condition specifies a range, one valid and two invalid equivalence classes are defined. 2. If an input condition requires a specific value, then one valid and two invalid equivalence classes are defined. 3. If an input condition specifies a member of a set, then one valid and one invalid equivalence class are defined. 4. If an input condition is boolean, then one valid and one invalid equivalence class are defined.

Boundary Value Analysis

This method leads to a selection of test cases that exercise boundary values. It complements equivalence partitioning since it selects test cases at the edges of a class. Rather than focusing on input conditions solely, BVA derives test cases from the output domain also. BVA guidelines include:

1. For input ranges bounded by a and b, test cases should include values a and b and just above and just below a and b respectively. 2. If an input condition specifies a number of values, test cases should be developed to exercise the minimum and maximum numbers and values just above and below these limits. 3. Apply guidelines 1 and 2 to the output. 4. If internal data structures have prescribed boundaries, a test case should be designed to exercise

the data structure at its boundary.

Cause-Effect Graphing Techniques

Cause-effect graphing is a technique that provides a concise representation of logical conditions and corresponding actions. There are four steps:

1. Causes (input conditions) and effects (actions) are listed for a module and an identifier is assigned to each. 2. A cause-effect graph is developed. 3. The graph is converted to a decision table. 4. Decision table rules are converted to test cases.

Black box testing

Black box testing takes an external perspective of the test object to derive test cases. These tests can be functional or non-functional, though usually functional. The test designer selects valid and invalid input and determines the correct output. There is no knowledge of the test object's internal structure. This method of test design is applicable to all levels of software testing: unit, integration, functional testing, system and acceptance. The higher the level, and hence the bigger and more complex the box, the more we're forced to use black box testing to simplify. While this method can uncover unimplemented parts of the specification, you can't be sure that all existent paths are tested.

editTest design techniques
Typical black box test design techniques include:        

Equivalence partitioning Boundary value analysis Decision table testing Pair wise testing State transition tables Use case testing
Cross-functional testing Functional testing

edit User input validation

User input must be validated to conform to expected values. For example, if the software program is requesting input on the price of an item, and is expecting a value such as 3.99, the software must check to make sure all invalid cases are handled. A user could enter the price as "1" and achieve results contrary to the design of the program. Other examples of entries that be entered and cause a failure in the software include: "1.20.35", "Abc", "0.000001", and "999999999". These are possible test scenarios that should be entered for each point of user input. Other domains, such as text input, need to restrict the length of the characters that can be entered. If a program allocates 30 characters of memory space for a name, and the user enters 50 characters, a buffer overflow condition can occur. Typically when invalid user input occurs, the program will either correct it automatically, or display a message to the user that their input needs to be corrected before proceeding.

edit Hardware
Functional testing devices like power supplies, amplifiers, and many other simple function electrical devices is common in the electronics industry. Automated functional testing of specified characteristics is used for production testing, and part of design validation. GLASS BOX TESTING What is black box/glass box testing?

Black-box and glass-box are test design methods. Black-box test design treats the system as a
"black-box", so it doesn't explicitly use knowledge of the internal structure. Black-box test design is usually described as focusing on testing functional requirements. Glass-box test design allows one to peek inside the "box", and it focuses specifically on using internal knowledge of the software to guide the selection of test data. Glass box testing requires the intimate knowledge of program internals, while black box testing is based solely on the knowledge of the system requirements. Being primarily concerned with program internals, it is obvious in SE literature that the primary effort to develop a testing methodology has been devoted to glass box tests. However, since the importance of black box testing has gained general acknowledgement, also a certain number of useful black box testing techniques were developed. It is important to understand that these methods are used during the test design phase, and their influence is hard to see in the tests once they're implemented. Glass box testing definition Software testing approaches that examine the program structure and derive test data from the program logic. Structural testing is sometimes referred to as clear-box testing since white boxes are considered opaque and do not really permit visibility into the code.

Types of Glass Box testing Static and Dynamic Analysis static analysis techniques do not necessitate the execution of the software, dynamic analysis is what is generally considered as ``testing``, i.e. it involves running the system. o Statement Coverage Testing performed where every statements is executed at least once. Branch Coverage Running a series of tests to ensure that all branches are tested at least once. Path Coverage Testing all paths. All-definition-use-path Coverage All paths between the definition of a variable and the use of that definition are now identified and tested.


o o

Tradeoffs of Glass box testing Advantages o o o o o o Disadvantages o o expensive miss cases omitted in the code forces test developer to reason carefully about implementation approximates the partitioning done by execution equivalence reveals errors in "hidden" code: beneficent side-effects optimizations (e.g. chartable that changes reps when size > 100)

Combining glass-box and black-box Most common approach in practice int abs(int x) { // effects: return -x if x, x otherwise if (x < -1) return -x; else

return x; } black-box subdomains: x < 0, x >= 0 glass-box subdomains: x < -1, x >= -1 intersect the subdomains, giving: x < -1, x = -1, x >= 0 testing these subdomains reveals the error intuition: good probability of error if either of the following occurs: o o program executes two inputs the same which have different specs program executes differently under two inputs with same specs

Glass-Box Testing In this testing technique, you use the code at hand to determine a test suite. Ideally, you want test data that exercises all possible paths through your code; however, this isn't always possible, and we can approximate by ensuring that each path is visited at least once. As an example, consider the following program:

#include <iostream.h> int main(void) { int nDependents, Exemption; float Income, TaxSubTotal, TaxTotal; cout << "Welcome to the Elbonian tax calculator. Enter your yearly income: "; cin >> Income; // first if - check income if (Income < 0) { cout << "You cannot have a negative income.\n"; return 0; } cout << "Enter the number of dependents you have, including yourself: "; cin >> nDependents; // second if - check dependents

if (nDependents <= 0) { cout <<"You must have at least one dependent.\n"; return 0; } // third if (else-if) - compute tax subtotal if (Income < 10000) TaxSubTotal = .02 * Income; else if (Income < 50000) TaxSubTotal = 200 + .03 * (Income - 10000); else TaxSubTotal = 1400 + .04 * (Income - 50000); Exemption= nDependents * 50; TaxTotal=TaxSubTotal - Exemption; // last if - check negative tax if (TaxTotal<0) //In case of negative tax TaxTotal=0; cout << "$S$S$S$S$S$S$S$S$S$S$S$S$S$S$S$S$S$ \n"; cout << "Elbonian Tax Collection Agency \n"; cout << "Tax Bill \n"; cout << " Citizen's Income: " << Income <<'\n'; cout << " Tax Subtotal: " << TaxSubTotal << '\n'; cout << "Number of Dependents: " << nDependents << '\n'; cout << " Tax Exepmtion: " << Exemption << '\n'; cout << " Final Tax Bill: " << TaxTotal << '\n'; cout << "$S$S$S$S$S$S$S$S$S$S$S$S$S$S$S$S$S$ \n"; }

There are four if statements in this code and no loops (these are the source of multiple paths in your code). They form a total of eight paths through the program (verify this!):

First if

Second if


Last if


Income < 0

doesn't matter

doesn't matter

doesn't matter

negative income error

Income >= 0

NDependents <=

doesn't matter

doesn't matter

invalid dependents error


Income >= 0 NDependents > 0 Income < 10000

TaxTotal < 0

bracket 1 negative tax

Income >= 0 NDependents > 0 10000 <= Income < 50000 TaxTotal < 0

bracket 2 negative tax

Income >= 0 NDependents > 0 Income >= 50000

TaxTotal < 0

bracket 3 negative tax

Income >= 0 NDependents > 0 Income < 10000

TaxTotal >= 0

bracket 1

Income >= 0 NDependents > 0 10000 <= Income < 50000 TaxTotal >= 0

bracket 2

Income >= 0 NDependents > 0 Income >= 50000

TaxTotal >= 0

bracket 3

To test this, we must come up with eight sets of data (income and number of dependents), one to test each possible path. Ranges for income are fairly evident for each case; we need only select an appropriate number of dependents for each case. The following table shows one test suite, and the expected results (TaxTotal):



Expected Result


Doesn't matter

negative income error



invalid dependents error



0 (bracket 1, negative tax)



0 (bracket 2, negative tax)



0 (bracket 3, negative tax)



130 (bracket 1)



300 (bracket 2)



3350 (bracket 3)

To complete the glass-box testing process, you would test your program with all eight input sets, and verify that the output matches the expected result. If all outputs match, then the code passes the test. WHITE BOX TESTING
White box testing is a test case design method that uses the control structure of the procedural design to derive test cases. Test cases can be derived that

1. guarantee that all independent paths within a module have been exercised at least once, 2. exercise all logical decisions on their true and false sides, 3. execute all loops at their boundaries and within their operational bounds, and 4. exercise internal data structures to ensure their validity. The Nature of Software Defects Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed. General processing tends to be well understood while special case processing tends to be prone to errors. We often believe that a logical path is not likely to be executed when it may be executed on a regular basis. Our unconscious assumptions about control flow and data lead to design errors that can only be detected by path testing. Typographical errors are random. Basis Path Testing This method enables the designer to derive a logical complexity measure of a procedural design and use it as a guide for defining a basis set of execution paths. Test cases that exercise the basis set are guaranteed to execute every statement in the program at least once during testing.

Flow Graphs Flow graphs can be used to represent control flow in a program and can help in the derivation of the basis set. Each flow graph node represents one or more procedural statements. The edges between nodes represent flow of control. An edge must terminate at a node, even if the node does not represent any useful procedural statements. A region in a flow graph is an area bounded by edges and nodes. Each node that contains a condition is called a predicate node. Cyclomatic complexity is a metric that provides a quantitative measure of the logical complexity of a program. It defines the number of independent paths in the basis set and thus provides an upper bound for the number of tests that must be performed.

The Basis Set An independent path is any path through a program that introduces at least one new set of processing statements (must move along at least one new edge in the path). The basis set is not unique. Any number of different basis sets can be derived for a given procedural design. Cyclomatic complexity, V(G), for a flow graph G is equal to 1. The number of regions in the flow graph. 2. V(G) = E - N + 2 where E is the number of edges and N is the number of nodes. 3. V(G) = P + 1 where P is the number of predicate nodes. Deriving Test Cases 1. From the design or

2. Determine the cyclomatic complexity of this flow graph. Even without a flow graph, V(G) can be determined by counting the number of conditional statements in the code. 3. Determine a basis set of linearly independent paths. Predicate nodes are useful for determining the necessary paths. 4. Prepare test cases that will force execution of each path in the basis set. Each test case is executed and compared to the expected results. Automating Basis Set Derivation The derivation of the flow graph and the set of basis paths is amenable to automation. A software tool to do this can be developed using a data structure called a graph matrix. A graph matrix is a square matrix whose size is equivalent to the number of nodes in the flow graph. Each row and column correspond to a particular node and

source code

, derive a flow graph.



corresponds to the connections (edges) between nodes. By adding a link weight to each matrix entry, more information about the control flow can be captured. In its simplest form, the link weight is 1 if an edge exists and 0 if it does not. But other types of link weights can be represented: ï¿½ the probability that an edge will be executed,

ï¿½ the processing time expended during link traversal, ï¿½ the memory required during link traversal, or ï¿½ the resources required during link traversal. Graph theory algorithms can be applied to these graph matrices to help in the analysis necessary to produce the basis set.

Loop Testing This white box technique focuses exclusively on the validity of loop constructs. Four different classes of loops can be defined: 1. 2. 3. 4. simple loops, nested loops, concatenated loops, and unstructured loops.

Simple Loops The following tests should be applied to simple loops where n is the maximum number of allowable passes through the loop: 1. 2. 3. 4. skip the loop entirely, only pass once through the loop, m passes through the loop where m < n, n - 1, n, n + 1 passes through the loop.

Nested Loops The testing of nested loops cannot simply extend the technique of simple loops since this would result in a geometrically increasing number of test cases. One approach for nested loops: 1. Start at the innermost loop. Set all other loops to minimum values. 2. Conduct simple loop tests for the innermost loop while holding the outer loops at their minimums. Add tests for out-of-range or excluded values. 3. Work outward, conducting tests for the next loop while keeping all other outer loops at minimums and other nested loops to typical values. 4. Continue until all loops have been tested. Concatenated Loops Concatenated loops can be tested as simple loops if each loop is independent of the others. If they are not independent (e.g. the loop counter for one is the loop counter for the other), then the nested approach can be used. Unstructured Loops This type of loop should be redesigned not tested!!! Other White Box Techniques Other white box testing techniques include: 1. Condition testing exercises the logical conditions in a program. 2. Data flow testing selects test paths according to the locations of definitions and uses of variables in the program.

White box testing (a.k.a. clear box testing, glass box testing or structural testing) uses an internal perspective of the system to design test cases based on internal structure. It requires programming skills to identify all paths through the software. The tester chooses test case inputs to exercise paths through the code and determines the appropriate outputs. In electrical hardware, testing every node in a circuit may be probed and measured; an example is in-circuit testing (ICT). Since the tests are based on the actual implementation, if the implementation changes, the tests probably will need to also. For example ICT needs updates if component values change, and needs modified/new fixture if the circuit changes. This adds financial resistance to the change process, thus buggy products may stay buggy. Automated optical inspection (AOI) offers similar component level correctness checking without the cost of ICT fixtures, however changes still require test updates. While white box testing is applicable at the unit, integration and system levels of the software testing process, it's typically applied to the unit. So while it normally tests paths within a unit, it can also test paths between units during integration, and between subsystems during a system level test. Though this method of test design can uncover an overwhelming number of test cases, it might not detect unimplemented parts of the specification or missing requirements. But you can be sure that all paths through the test object are executed. Typical white box test design techniques include:  

Control flow testing
Data flow testing

Software Testing - White Box Testing Strategy An introduction to White Box Testing Strategy and various types of testing that comes under white box testing strategy. What is a White Box Testing Strategy? White box testing strategy deals with the internal logic and structure of the code. White box testing is also called as glass, structural, open box or clear box testing. The tests written based on the white box testing strategy incorporate coverage of the code written, branches, paths, statements and internal logic of the code etc. In order to implement white box testing, the tester has to deal with the code and hence is needed to possess knowledge of coding and logic i.e. internal working of the code. White box test also needs the tester to look into the code and find out which unit/statement/chunk of the code is malfunctioning. Advantages of White box testing are: i) As the knowledge of internal coding structure is prerequisite, it becomes very easy to find out which type of input/data can help in testing the application effectively. ii) The other advantage of white box testing is that it helps in optimizing the code iii) It helps in removing the extra lines of code, which can bring in hidden defects. Disadvantages of white box testing are:

i) As knowledge of code and internal structure is a prerequisite, a skilled tester is needed to carry out this type of testing, which increases the cost. ii) And it is nearly impossible to look into every bit of code to find out hidden errors, which may create problems, resulting in failure of the application. Types of testing under White/Glass Box Testing Strategy: Unit Testing: The developer carries out unit testing in order to check if the particular module or unit of code is working fine. The Unit Testing comes at the very basic level as it is carried out as and when the unit of the code is developed or a particular functionality is built. Static and dynamic Analysis: Static analysis involves going through the code in order to find out any possible defect in the code. Dynamic analysis involves executing the code and analyzing the output. Statement Coverage: In this type of testing the code is executed in such a manner that every statement of the application is executed at least once. It helps in assuring that all the statements execute without any side effect. Branch Coverage: No software application can be written in a continuous mode of coding, at some point we need to branch out the code in order to perform a particular functionality. Branch coverage testing helps in validating of all the branches in the code and making sure that no branching leads to abnormal behavior of the application. Security Testing: Security Testing is carried out in order to find out how well the system can protect itself from unauthorized access, hacking – cracking, any code damage etc. which deals with the code of application. This type of testing needs sophisticated testing techniques. Mutation Testing: A kind of testing in which, the application is tested for the code that was modified after fixing a particular bug/defect. It also helps in finding out which code and which strategy of coding can help in developing the functionality effectively. Besides all the testing types given above, there are some more types which fall under both Black box and White box testing strategies such as: Functional testing (which deals with the code in order to check its functional performance), Incremental integration testing (which deals with the testing of newly added code in the application), Performance and Load testing (which helps in finding out how the particular code manages resources and give performance etc.) etc. GREY BOX TESTING In recent years the term grey box testing has come into common usage. The typical grey box tester is permitted to set up or manipulate the testing environment, like seeding a database, and can view the state of the product after his actions, like performing a SQL query on the database to be certain of the values of columns. It is used almost exclusively of client-server testers or others who use a database as a repository of information, but can also apply to a tester who has to manipulate XML files (DTD or an actual XML file) or configuration files directly. It can also be used of testers who know the internal workings or algorithm of the software under test and can write tests specifically for the anticipated results. For example, testing a data warehouse implementation involves loading the target database with information, and verifying the correctness of data population and loading of data into the correct tables.

………………………………………………………………………………………………………………… DIFFERENT TYPES OF TESTING EXPLORATORY TESTING Exploratory testing is an approach in software testing with simultaneous learning, test design and test execution. While the software is being tested, the tester learns things that together with experience and creativity generates new good tests to run. History Exploratory testing has been performed for a long time, and has similarities to ad hoc testing. In the early 1990s, ad hoc was too often synonymous with sloppy and careless work. As a result, a group of test methodologists (now calling themselves the Context-Driven School) began using the term "exploratory" seeking to emphasize the dominant thought process involved in unscripted testing, and to begin to develop the practice into a teachable discipline. This new terminology was first published by Cem Kaner in his book Testing Computer Software. Exploratory testing can be as disciplined as any other intellectual activity. Description Exploratory testing seeks to find out how the software actually works, and to ask questions about how it will handle difficult and easy cases. The testing is dependent on the testers skill of inventing test cases and finding defects. The more the tester knows about the product and different test methods, the better the testing will be. To further explain, comparison can be made to the antithesis scripted testing, which basically means that test cases are designed in advance, including steps to reproduce and expected results. These tests are later performed by a tester who compares the actual result with the expected. When performing exploratory testing, there are no exact expected results; it is the tester that decides what will be verified, critically investigating the correctness of the result. In reality, testing almost always is a combination of exploratory and scripted testing, but with a tendency towards either one, depending on context. The documentation of exploratory testing ranges from documenting all tests performed to just documenting the bugs. During pair testing, two persons create test cases together; one performs

them, and the other documents. Session-based testing is a method specifically designed to make exploratory testing auditable and measurable on a wider scale. Benefits and drawbacks The main advantage of exploratory testing is that less preparation is needed, important bugs are found fast, and is more intellectually stimulating than scripted testing. Disadvantages are that the tests can't be reviewed in advance (and by that prevent errors in code and test cases), and that it can be difficult to show exactly which tests have been run. When repeating exploratory tests, they will not be performed in the exact same manner, which can be an advantage if it is important to find new errors; or a disadvantage if it is more important to know that exact things are functional. Usage Exploratory testing is extra suitable if requirements and specifications are incomplete, or if there is lack of time. The method can also be used to verify that previous testing has found the most important defects. It is common to perform a combination of exploratory and scripted testing where the choice is based on risk. An example of exploratory testing in practice is Microsoft‘s verification of Windows compatibility. REGRESSION TESTING
Usage: All aspects of system remain functional after testing.


Change in one segment does not change the functionality of other segment.

Objective: Determine System documents remain current

 

Determine System test data and test conditions remain current Determine Previously tested system functions properly without getting effected though changes are made in some other segment of application system.

How to Use Test cases, which were used previously for the already tested segment is, re-run to ensure that the results of the segment tested currently and the results of same segment tested earlier are same.

 

Test automation is needed to carry out the test transactions (test condition execution) else the process is very time consuming and tedious. In this case of testing cost/benefit should be carefully evaluated else the efforts spend on testing would be more and payback would be minimum.

When to Use

When there is high risk that the new changes may effect the unchanged areas of application system.

 

In development process: Regression testing should be carried out after the pre-determined changes are incorporated in the application system. In Maintenance phase : regression testing should be carried out if there is a high risk that loss may occur when the changes are made to the system

Example Re-running of previously conducted tests to ensure that the unchanged portion of system functions properly.


Reviewing previously prepared system documents (manuals) to ensure that they do not get effected after changes are made to the application system. Disadvantage Time consuming and tedious if test automation not done

Regression testing is any type of software testing which seeks to uncover regression bugs. Regression bugs occur whenever software functionality that previously worked as desired stops working or no longer works in the same way that was previously planned. Typically regression bugs occur as an unintended consequence of program changes. Common methods of regression testing include re-running previously run tests and checking whether previously fixed faults have reemerged. Experience has shown that as software is developed, this kind of reemergence of faults is quite common. Sometimes it occurs because a fix gets lost through poor revision control practices (or simple human error in revision control), but just as often a fix for a problem will be "fragile" i.e. if some other change is made to the program, the fix no longer works. Finally, it has often

been the case that when some feature is redesigned, the same mistakes will be made in the redesign that were made in the original implementation of the feature. Therefore, in most software development situations it is considered good practice that when a bug is located and fixed, a test that exposes the bug is recorded and regularly retested after subsequent changes to the program. Although this may be done through manual testing procedures using programming techniques, it is often done using automated testing tools. Such a 'test suite' contains software tools that allows the testing environment to execute all the regression test cases automatically; some projects even set up automated systems to automatically re-run all regression tests at specified intervals and report any regressions. Common strategies are to run such a system after every successful compile (for small projects), every night, or once a week. Regression testing is an integral part of the extreme programming software development methodology. In this methodology, design documents are replaced by extensive, repeatable, and automated testing of the entire software package at every stage in the software development cycle. Types of regression    Local - changes introduce new bugs Unmasked - changes unmask previously existing bugs Remote - Changing one part breaks another part of the program. For example, Module A writes to a database. Module B reads from the database. If changes to what Module A writes to the database break Module B, it is remote regression.

There's another way to classify regression.   New feature regression - changes to code that is new to release 1.1 break other code that is new to release 1.1. Existing feature regression - changes to code that is new to release 1.1 break code that existed in release 1.0.

Mitigating regression risk     Complete test suite repetition Regression test automation (GUI, API, CLI) Partial test repetition based on traceability and analysis of technical and business risks Customer or user testing o Beta - early release to both potential and current customers o Pilot - deploy to a subset of users o Parallel - users use both old and new systems simultaneously Use larger releases. Testing new functions often covers existing functions. The more new features in a release, the more "accidental" regression testing. Emergency patches - these patches are released immediately, and will be included in future maintenance releases.

 


Regression testing can be used not only for testing the correctness of a program, but it is also often used to track the quality of its output. For instance in the design of a compiler, regression testing should track the code size, simulation time, and compilation time of the test suites. Whenever a new build comes, regression process will come into the picture. SOAK TESTING Soak testing involves testing a system with a significant load extended over a significant period of time, to discover how the system behaves under sustained use. For example, in software testing, a system may behave exactly as expected when tested for 1 hour. However, when it is tested for 3 hours, problems like 'memory leak' may cause the system to fail or behave randomly.

ALL-PAIRS TESTING All-pairs testing or pair wise testing is a combinatorial software testing method that, for each pair of input parameters to a system (typically, a software algorithm) tests all possible discrete combinations of those parameters. Using carefully chosen test vectors, this can be done much faster than an exhaustive search of all combinations of all parameters, by "parallelizing" the tests of parameter pairs. The number of tests is typically O(nm), where n and m are the number of possibilities for each of the two parameters with the most choices. The reasoning behind all-pairs testing is this: the simplest bugs in a program are generally triggered by a single input parameter. The next simplest category of bugs consists of those dependent on interactions between pairs of parameters, which can be caught with all-pairs testing. Bugs involving interactions between three or more parameters are progressively less common, whilst at the same time being progressively more expensive to find by exhaustive testing, which has as its limit the exhaustive testing of all possible inputs. Many testing methods regard all-pairs testing of a system or subsystem as a reasonable costbenefit compromise between often computationally infeasible higher-order combinatorial testing methods, and less exhaustive methods which fail to exercise all possible pairs of parameters. Because no testing technique can find all bugs, all-pairs testing is typically used together with other quality assurance techniques such as unit testing, symbolic execution, fuzz testing, and code review. PSEUDO LOCALIZATION Pseudo localization is a software testing method that is used to test internationalization aspects of software. Specifically, it brings to light potential difficulties with localization by replacing localizable text (particularly in a graphical user interface) with text that imitates the most problematic characteristics of text from a wide variety of languages, and by forcing the

application to deal with similar input text. If used properly, it provides a cheap but effective sanity test for localizability that can be helpful in the early stages of a software project. If software is not designed with localizability in mind, certain problems can occur when the software is localized. Text in a target language may tend to be significantly longer than the corresponding text in the original language of the program, causing the ends of text to be cut off if insufficient space is allocated. Words in a target language may be longer, causing awkward line breaks. In addition, individual characters in a target language may require more space, causing for example modified characters to be cut off vertically. Even worse, characters of a target language may fail to render properly (or at all) if support for an appropriate font is not included (this is a larger problem for legacy software than for newer programs). On the input side, programmers may make inappropriate assumptions about the form that user input can take. For small changes to mature software products, for which a large amount of target text is already available, directly testing several target languages may be the best option. For newer software or for larger UI changes, however, waiting for text to be translated can introduce a significant lag into the testing schedule. In addition, it may not be cost-effective to translate UI text early in the development cycle, as it might change and need to be retranslated. Here, pseudo localization can be the best option, as no real translation is needed. Typically, pseudo localized text for a program will be generated and used as if it were for a real locale. Pseudo localized text should be longer than the original text (perhaps twice as long), contain longer unbroken strings of characters to test line breaking, and contain characters from different writing systems . A tester will then inspect each element of the UI to make sure everything is displayed properly. To make it easier for the tester to find his or her way around the UI, the text may include the original text, or perhaps characters that look similar to the original text. For example, the string Edit program settings might be replaced with [YxĤ8z* jQ ^dЊÚk&d== εÐiţ_Þr0ģЯãm səTτıИğ§] (The brackets on either side of the text make it easier to spot text that is cut off). This type of transformation can be performed by a simple tool and does not require a human translator, resulting in time and cost savings. RISK-BASED TESTING Risk-based testing (RBT) is a type of software testing that prioritizes the features and functions to be tested based on priority/importance and likelihood or impact or failure. In theory, since there is an infinite number of a possible test, any set of tests must be a subset of all possible tests. Test techniques such as boundary value analysis and state transition testing aim to find the areas most likely to be defective. So by using test techniques, a software test engineer is already selecting tests based on risk.

Types of Risks This section lists some common risks. Business or Operational   High use of a sybsystem, function or feature Criticality of a subsystem, function or feature, including unacceptability of failure

Technical   Geographic distribution of development team Complexity of a subsystem or function

External   History Most people consider James Bach to be the "father" of risk-based testing. He described the idea in a paper entitled The Challenge of Good Enough Software in October 1995 appearing in the journal American Programmer CHARACTERIZATION TEST In computer programming, a characterization test is a mean to describe (characterize) the actual comportment of an existing piece of software. The goal of Characterization Tests is to help developers verify that the modifications made to a reference version of a software system did not modify its behavior in unwanted or undesirable ways. They enable, and provide a safety net for, extending and refactoring code that does not have adequate unit test. Characterization Tests are a mean to automate testing of legacy code. Unlike Regression Tests , to which they are very close, they do not verify the correct behavior of the code, but the behavior that was observed when they have been written. Characterization tests are, essentially, change detectors. It is up to the person analyzing the results to determine if the detected change was expected and/or desirable, or unexpected and/or undesirable. This term was coined by Michael Feathers in its book working effectively with legacy code (ISBN 0-13-117705-2). One of the interesting aspects of characterization tests is that, since they are based on existing code, it's possible to generate some characterization tests automatically. An automated characterization test tool (ACTT) would exercise the existing code with a wide range of relevant Sponsor or executive preference Regulatory requirements

and/or random input values, record the output values (or state changes) and generate a set of characterization tests. When the generated tests are executed against a new version of the code, they will produce one or more failures/warnings if that version of the code is modified in a way that changes a previously established behavior. SESSION-BASED TESTING Session-based testing is more closely related to Exploratory testing than to Scenario testing, but the concepts can be used in conjunction. It is a controlled and improved ad-hoc testing that is able to use the knowledge gained as a basis for ongoing, product sustained improvement. The improved test results can help put the case for change throughout development, and ways in which the testing team can build on the initial processes to arrive at a better overall testing. Session-based testing can be used to introduce measurement and control to an immature test process, and can form a foundation for significant improvements in productivity and error detection. Session-based testing is ideal when formal requirements are non present, incomplete, or changing rapidly. The main tool in session-based testing is the charter or test point. Charters and test points list what should be tested and provide a record of what was tested. They should take between two and four hours to execute. Testers using Session-based testing plan their testing daily and new revelations on what needs to be tested are added as tests are executed. AD HOC TESTING There are two divergent views of ad hoc testing. To one camp it is software testing performed without planning and documentation. The tests are intended to be run only once, unless a defect is discovered. Ad hoc testing is a part of exploratory testing, being the least formal of test methods. In this view, ad hoc testing has been criticized because it isn't structured, but this can also be a strength: important things can be found quickly. It is performed with improvisation, the tester seeks to find bugs with any means that seem appropriate. It contrasts to regression testing that looks for a specific issue with detailed reproduction steps, and a clear expected result. Ad hoc testing is most often used as a complement to other types of testing. To the Context-Driven School of testing, ad hoc testing is synonymous with exploratory testing, which is simultaneous learning, test design and test execution. The main advantages of this approach is that more time is spent testing and less time is expended in preparation. Important bugs are found fast, and it is a more intellectually stimulating activity than "scripted" testing. In this view, when executing ad hoc tests, the tester discovers what needs to be tested, understands the system using heuristic methods and other skills, creates notes to determine what needs to be tested, and tests what is currently testable while pushing forward additional testing activities as required or revealed by the system under test. Documentation is created after the tests have been run describing what was tested and what still needs to be tested. Testing is driven by the needs of the system under test, not some document that will be changing repeatedly in the future.

FUZZ TESTING Fuzz testing or fuzzing is a software testing technique. The basic idea is to attach the inputs of a program to a source of random data ("fuzz"). If the program fails (for example, by crashing, or by failing built-in code assertions), then there are defects to correct. The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior. Fuzz testing was developed at the University of Wisconsin-Madison in 1989 by Professor Barton Miller and the students in his graduate Advanced Operating Systems class. Their work can be found at Uses Fuzz testing is often used in large software development projects that perform black box testing. These usually have a budget to develop test tools, and fuzz testing is one of the techniques which offers a high benefit to cost ratio. Fuzz testing is also used as a gross measurement of a large software system's quality. The advantage here is that the cost of generating the tests is relatively low. For example, third party testers have used fuzz testing to evaluate the relative merits of different operating systems and application programs. Fuzz testing is thought to enhance software security and software safety because it often finds odd oversights and defects which human testers would fail to find, and even careful human test designers would fail to create tests for. However, fuzz testing is not a substitute for exhaustive testing or formal methods: it can only provide a random sample of the system's behavior, and in many cases passing a fuzz test may only demonstrate that a piece of software handles exceptions without crashing, rather than behaving correctly. Thus, fuzz testing can only be regarded as a proxy for program correctness, rather than a direct measure, with fuzz test failures actually being more useful as a bug-finding tool than fuzz test passes as an assurance of quality. Fuzz testing methods As a practical matter, developers need to reproduce errors in order to fix them. For this reason, almost all fuzz testing makes a record of the data it manufactures, usually before applying it to the software, so that if the computer fails dramatically, the test data is preserved. If the fuzz stream is pseudo-random number generated it may be easier to store the seed value to reproduce the fuzz attempt. Modern software has several different types of inputs:

   

Event driven inputs are usually from a graphical user interface, or possibly from a mechanism in an embedded system. Character driven inputs are from files, or data streams such as sockets. Database inputs are from tabular data, such as relational databases . Inherited program state such as environment variables

There are at least two different forms of fuzz testing:    Valid fuzz attempts to assure that the random input is reasonable, or conforms to actual production data. Simple fuzz usually uses a pseudo random number generator to provide input. A combined approach uses valid test data with some proportion of totally random input injected.

By using all of these techniques in combination, fuzz-generated randomness can test the undesigned behavior surrounding a wider range of designed system states. Fuzz testing may use tools to simulate all of these domains. Advantages and disadvantages The main problem with fuzzing to find program faults is that it generally only finds very simple faults. The problem itself is exponential and every fuzzer takes shortcuts in order to find something interesting in a timeframe that a human cares about. A primitive fuzzer may have poor code coverage; for example, if the input includes a checksum which is not properly updated to match other random changes, only the checksum validation code will be verified. Code coverage tools are often used to estimate how "well" a fuzzer works, but these are only guidelines to fuzzer quality. Every fuzzer can be expected to find a different set of bugs. On the other hand, bugs found using fuzz testing are frequently severe, exploitable bugs that could be used by a real attacker. This has become even more true as fuzz testing has become more widely known, as the same techniques and tools are now used by attackers to exploit deployed software. This is a major advantage over binary or source auditing, or even fuzzing's close cousin, fault injection, which often rely on artificial fault conditions that are difficult or impossible to exploit. Event-driven fuzz Normally this is provided as a queue of data structures. The queue is filled with data structures that have random values. The most common problem with an event-driven program is that it will often simply use the data in the queue, without even crude validation. To succeed in a fuzz-tested environment, software must validate all fields of every queue entry, decode every possible binary value, and then ignore impossible requests.

One of the more interesting issues with real-time event handling is that if error reporting is too verbose, simply providing error status can cause resource problems or a crash. Robust error detection systems will report only the most significant or most recent error over a period of time. Character-driven fuzz Normally this is provided as a stream of random data. The classic source in UNIX is the random data generator. One common problem with a character driven program is a buffer overrun, when the character data exceeds the available buffer space. This problem tends to recur in every instance in which a string or number is parsed from the data stream and placed in a limited-size area. Another is that decode tables or logic may be incomplete, not handling every possible binary value.

Database fuzz The standard database scheme is usually filled with fuzz that is random data of random sizes. Some IT shops use software tools to migrate and manipulate such databases. Often the same schema descriptions can be used to automatically generate fuzz databases. Database fuzz is controversial, because input and comparison constraints reduce the invalid data in a database. However, often the database is more tolerant of odd data than its client software, and a general-purpose interface is available to users. Since major customer and enterprise management software is starting to be open-source, database-based security attacks are becoming more credible. A common problem with fuzz databases is buffer overflow. A common data dictionary, with some form of automated enforcement is quite helpful and entirely possible. To enforce this, normally all the database clients need to be recompiled and retested at the same time. Another common problem is that database clients may not understand the binary possibilities of the database field type, or, legacy software might have been ported to a new database system with different possible binary values. A normal, inexpensive solution is to have each program validate database inputs in the same fashion as user inputs. The normal way to achieve this is to periodically "clean" production databases with automated verifiers. Summary:    Black box testing - not based on any knowledge of internal design or code. Tests are based on requirements and functionality. White box testing - based on knowledge of the internal logic of an application's code. Tests are based on coverage of code statements, branches, paths, conditions. Unit testing - the most 'micro' scale of testing; to test particular functions or code modules. Typically done by the programmer and not by testers, as it requires detailed




 



  

 

    

knowledge of the internal program design and code. Not always easily done unless the application has a well-designed architecture with tight code; may require developing test driver modules or test harnesses. Incremental integration testing - continuous testing of an application as new functionality is added; requires that various aspects of an application's functionality be independent enough to work separately before all parts of the program are completed, or that test drivers be developed as needed; done by programmers or by testers. Integration testing - testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server and distributed systems. Functional testing - black-box type testing geared to functional requirements of an application; this type of testing should be done by testers. This doesn't mean that the programmers shouldn't check that their code works before releasing it (which of course applies to any stage of testing.) System testing - black-box type testing that is based on overall requirements specifications; covers all combined parts of a system. End-to-end testing - similar to system testing; the 'macro' end of the test scale; involves testing of a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate. Sanity testing or smokes testing - typically an initial testing effort to determine if a new software version is performing well enough to accept it for a major testing effort. For example, if the new software is crashing systems every 5 minutes, bogging down systems to a crawl, or corrupting databases, the software may not be in a 'sane' enough condition to warrant further testing in its current state. Regression testing - re-testing after fixes or modifications of the software or its environment. It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Automated testing tools can be especially useful for this type of testing. Acceptance testing - final testing based on specifications of the end-user or customer, or based on use by end-users/customers over some limited period of time. Load testing - testing an application under heavy loads, such as testing of a web site under a range of loads to determine at what point the system's response time degrades or fails. Stress testing - term often used interchangeably with 'load' and 'performance' testing. Also used to describe such tests as system functional testing while under unusually heavy loads, heavy repetition of certain actions or inputs, input of large numerical values, large complex queries to a database system, etc. Performance testing - term often used interchangeably with 'stress' and 'load' testing. Ideally 'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA or Test Plans. Usability testing - testing for 'user-friendliness'. Clearly this is subjective, and will depend on the targeted end-user or customer. User interviews, surveys, video recording of user sessions, and other techniques can be used. Programmers and testers are usually not appropriate as usability testers. Install/uninstall testing - testing of full, partial, or upgrade install/uninstall processes. Recovery testing - testing how well a system recovers from crashes, hardware failures, or other catastrophic problems. failover testing - typically used interchangeably with 'recovery testing' Security testing - testing how well the system protects against unauthorized internal or external access, willful damage, etc; may require sophisticated testing techniques. Compatability testing - testing how well software performs in a particular hardware/software/operating system/network/etc. environment.

 

Exploratory testing - often taken to mean a creative, informal software test that is not based on formal test plans or test cases; testers may be learning the software as they test it. Ad-hoc testing - similar to exploratory testing, but often taken to mean that the testers have significant understanding of the software before testing it.

Context-driven testing - testing driven by an understanding of the environment, culture, and intended use of software. For example, the testing approaches for life. ………………………………………………………………………………………………………………… WEB TESTING How can World Wide Web sites be tested? Web sites are essentially client/server applications - with web servers and 'browser' clients. Consideration should be given to the interactions between html pages, TCP/IP communications, Internet connections, firewalls, applications that run in web pages (such as applets, javascript, plug-in applications), and applications that run on the server side (such as cgi scripts, database interfaces, logging applications, dynamic page generators, asp, etc.). Additionally, there are a wide variety of servers and browsers, various versions of each, small but sometimes significant differences between them, variations in connection speeds, rapidly changing technologies, and multiple standards and protocols. The end result is that testing for web sites can become a major ongoing effort. Other considerations might include:  What are the expected loads on the server (e.g., number of hits per unit time?), and what kind of performance is required under such loads (such as web server response time, database query response times). What kinds of tools will be needed for performance testing (such as web load testing tools, other tools already in house that can be adapted, web robot downloading tools, etc.)? Who is the target audience? What kind of browsers will they be using? What kind of connection speeds will they be using? Are they intra- organization (thus with likely high connection speeds and similar browsers) or Internet-wide (thus with a wide variety of connection speeds and browser types)? What kind of performance is expected on the client side (e.g., how fast should page appear, how fast should animations, applets, etc. load and run)? Will down time for server and content maintenance/upgrades be allowed? how much? What kinds of security (firewalls, encryptions, passwords, etc.) will be required and what is it expected to do? How can it be tested? How reliable are the site's Internet connections required to be? And how does that affect backup system or redundant connection requirements and testing? What processes will be required to manage updates to the web site's content, and what are the requirements for maintaining, tracking, and controlling page content, graphics, links, etc.? Which HTML specification will be adhered to? How strictly? What variations will be allowed for targeted browsers? Will there be any standards or requirements for page appearance and/or graphics throughout a site or parts of a site?? How will internal and external links be validated and updated? How often? Can testing be done on the production system, or will a separate test system be required? How browser caching, variations in browser are option settings, dial-up


        

 

connection variability‘s, and real-world internet 'traffic congestion' problems to be accounted for in testing? How extensive or customized are the server logging and reporting requirements; are they considered an integral part of the system and do they require testing? How are cgi programs, applets, java scripts, ActiveX components, etc. to be maintained, tracked, controlled, and tested?

Some usability guidelines to consider - these are subjective and may or may not apply to a given situation      Pages should be 3-5 screens max unless content is tightly focused on a single topic. If larger, provide internal links within the page. The page layouts and design elements should be consistent throughout a site, so that it's clear to the user that they're still within a site. Pages should be as browser-independent as possible, or pages should be provided or generated based on the browser-type. All pages should have links external to the page; there should be no dead-end pages. The page owner, revision date, and a link to a contact person or organization should be included on each page.

………………………………………………………………………………………………………………… TESTING PLANNING AND DOCUMENTATION USING TEST DIRECTOR STEPS ARE NEEDED TO DEVELOP AND RUN SOFTWARE TESTS The following are some of the steps to consider:                 Obtain requirements, functional design, and internal design specifications and other necessary documents Obtain budget and schedule requirements Determine project-related personnel and their responsibilities, reporting requirements, required standards and processes (such as release processes, change processes, etc.) Determine project context, relative to the existing quality culture of the organization and business, and how it might impact testing scope, approaches, and methods. Identify application's higher-risk aspects, set priorities, and determine scope and limitations of tests Determine test approaches and methods - unit, integration, functional, system, load, usability tests, etc. Determine test environment requirements (hardware, software, communications, etc.) Determine test ware requirements (record/playback tools, coverage analyzers, test tracking, problem/bug tracking, etc.) Determine test input data requirements Identify tasks, those responsible for tasks, and labor requirements Set schedule estimates, timelines, milestones Determine input equivalence classes, boundary value analyses, error classes Prepare test plan document and have needed reviews/approvals Write test cases Have needed reviews/inspections/approvals of test cases Prepare test environment and test ware, obtain needed user manuals/reference documents/configuration guides/installation guides, set up test tracking processes, set up logging and archiving processes, set up or obtain test input data

     

Obtain and install software releases Perform tests Evaluate and report results Track problems/bugs and fixes Retest as needed Maintain and update test plans, test cases, test environment, and test ware through life cycle.

TEST SCENARIO Scenario is a software testing activity. It uses scenario tests, or simply scenarios, which are based on a hypothetical story to help a person think through a complex problem or system. They can be as simple as a diagram for a testing environment or they could be a description written in prose. The ideal scenario has five key characteristics. It is (a) a story that is (b) motivating, (c) credible, (d) complex, and (e) easy to evaluate. These tests are usually different from test cases in that test cases are single steps and scenarios cover a number of steps. Test suites and scenarios can be used in concert for complete system testing. TEST PLAN A test plan is a systematic approach to testing a system such as a machine or software. The plan typically contains a detailed understanding of what the eventual workflow will be. A software project test plan is a document that describes the objectives, scope, approach, and focus of a software testing effort. The process of preparing a test plan is a useful way to think through the efforts needed to validate the acceptability of a software product. The completed document will help people outside the test group understand the 'why' and 'how' of product validation. It should be thorough enough to be useful but not so thorough that no one outside the test group will read it. The following are some of the items that might be included in a test plan, depending on the particular project:                  Title Identification of software including version/release numbers Revision history of document including authors, dates, approvals Table of Contents Purpose of document, intended audience Objective of testing effort Software product overview Relevant related document list, such as requirements, design documents, other test plans, etc. Relevant standards or legal requirements Traceability requirements Relevant naming conventions and identifier conventions Overall software project organization and personnel/contact-info/responsibilities Test organization and personnel/contact-info/responsibilities Assumptions and dependencies Project risk analysis Testing priorities and focus Scope and limitations of testing


Test outline - a decomposition of the test approach by test type, feature, functionality, process, system, module, etc. as applicable Outline of data input equivalence classes, boundary value analysis, error classes Test environment - hardware, operating systems, other required software, data configurations, interfaces to other systems Test environment validity analysis - differences between the test and production systems and their impact on test validity. Test environment setup and configuration issues Software migration processes Software CM processes Test data setup requirements Database setup requirements Outline of system-logging/error-logging/other capabilities, and tools such as screen capture software, that will be used to help describe and report bugs Discussion of any specialized software or hardware tools that will be used by testers to help track the cause or source of bugs Test automation - justification and overview Test tools to be used, including versions, patches, etc. Test script/test code maintenance processes and version control Problem tracking and resolution - tools and processes Project test metrics to be used Reporting requirements and testing deliverables Software entrance and exit criteria Initial sanity testing period and criteria Test suspension and restart criteria Personnel allocation Personnel pre-training needs Test site/location Outside test organizations to be utilized and their purpose, responsibilties, deliverables, contact persons, and coordination issues Relevant proprietary, classified, security, and licensing issues. Open issues Appendix - glossary, acronyms, etc.

Test plans in software development Cem Kaner, co-author of Testing Computer Software (ISBN 0-471-35846-0), has suggested that test plans are written for two very different purposes. Sometimes the test plan is a product; sometimes it's a tool. It's too easy, but also too expensive, to confuse these goals. In software testing, a test plan gives detailed testing information regarding an upcoming testing effort, including     

Scope of testing Schedule
Test Deliverables Release Criteria Risks and Contingencies

Test plan template, IEEE 829 format


Test Plan Identifier (TPI) References Introduction Test Items Software Risk Issues Features to be Tested Features not to be Tested Approach Item Pass/Fail Criteria Entry & Exit Criteria Suspension Criteria and Resumption Requirements Test Deliverables Remaining Test Tasks Environmental Needs Staffing and Training Needs Responsibilities Schedule Planning Risks and Contingencies Approvals Glossary

Test plan identifier Master test plan for the Line of Credit Payment System. TP_PA1.0 References List all documents that support this test plan. Documents that are referenced include:        Project Plan System Requirements specifications. High Level design document. Detail design document. Development and Test process standards. Methodology guidelines and examples. Corporate standards and guidelines.


As this is the "Executive Summary" keep information brief and to the point. Test items (functions) These are things you intend to test within the scope of this test plan. Essentially, something you will test, a list of what is to be tested. This can be developed from the software application inventories as well as other sources of documentation and information. This can be controlled on a local Configuration Management (CM) process if you have one. This information includes version numbers, configuration requirements where needed, (especially if multiple versions of the product are supported). It may also include key delivery schedule issues for critical elements. Remember, what you are testing is what you intend to deliver to the Client. This section can be oriented to the level of the test plan. For higher levels it may be by application or functional area, for lower levels it may be by program, unit, module or build.Software risk issues Identify what software is to be tested and what the critical areas are, such as: 1. 2. 3. 4. 5. 6. Delivery of a third party product. New version of interfacing software. Ability to use and understand a new package/tool, etc. Extremely complex functions. Modifications to components with a past history of failure. Poorly documented modules or change requests.

There are some inherent software risks such as complexity; these need to be identified. 1. 2. 3. 4. Safety. Multiple interfaces. Impacts on Client. Government regulations and rules.


Features to be tested This is a listing of what is to be tested from the user's viewpoint of what the system does. This is not a technical description of the software, but a USERS view of the functions. Set the level of risk for each feature. Use a simple rating scale such as (H, M, L): High, Medium and Low. These types of levels are understandable to a User. You should be prepared to discuss why a particular level was chosen. Sections 4 and 6 are very similar, and the only true difference is the point of view. Section 4 is a technical type description including version numbers and other technical information and Section 6 is from the User‘s viewpoint. Users do not understand technical software terminology; they understand functions and processes as they relate to their jobs. Features not to be tested This is a listing of what is 'not' to be tested from both the user's viewpoint of what the system does and a configuration management/version control view. This is not a technical description of the software, but a user's view of the functions. Identify why the feature is not to be tested, there can be any number of reasons.    Not to be included in this release of the Software. Low risk, has been used before and was considered stable. Will be released but not tested or documented as a functional part of the release of this version of the software.




If this is a master test plan the overall project testing approach and coverage requirements must also be identified. Specify if there are special requirements for the testing.   Only the full component will be tested. A specified segment of grouping of features/components must be tested together.

Other information that may be useful in setting the approach are:   MTBF, Mean Time Between Failures - if this is a valid measurement for the test involved and if the data is available. SRE, Software Reliability Engineering - if this methodology is in use and if the information is available.

How will meetings and other organizational processes be handled? Item pass/fail criteria Show stopper issues. Specify the criteria to be used to determine whether each test item has passed or failed. Entry & exit criteria Specify the criteria to be used to start testing and how you know when to stop the testing process. Suspension criteria & resumption requirements Suspension criteria specify the criteria to be used to suspend all or a portion of the testing activities while resumption criteria specifies when testing can resume after it has been suspended. Test deliverables List documents, reports, charts, that will be presented to stakeholders on a regular basis during testing and when testing has been completed. Remaining test tasks If this is a multi-phase process or if the application is to be released in increments there may be parts of the application that this plan does not address. These areas need to be identified to avoid any confusion should defects be reported back on those future functions. This will also allow the users and testers to avoid incomplete functions and prevent waste of resources chasing nondefects.


Staffing and training needs Training on the application/system. Training for any test tools to be used. The Test Items and Responsibilities sections affect this section. What is to be tested and who is responsible for the testing and training. Responsibilities Who is in charge? This issue includes all areas of the plan. Here are some examples:        Setting risks. Selecting features to be tested and not tested. Setting overall strategy for this level of plan. Ensuring all required elements are in place for testing. Providing for resolution of scheduling conflicts, especially, if testing is done on the production system. Who provides the required training? Who makes the critical go/no go decisions for items not covered in the test plans?

Planning risks and contingencies What are the overall risks to the project with an emphasis on the testing process?  Lack of personnel resources when testing is to begin.

    

Lack of availability of required hardware, software, data or tools. Late delivery of the software, hardware or tools. Delays in training on the application and/or tools. Changes to the original requirements or designs. Complexities involved in testing the applications

Specify what will be done for various events, for example: Requirements definition will be complete by January 1, 20XX, and, if the requirements change after that date, the following actions will be taken:        The test schedule and development schedule will move out an appropriate number of days. This rarely occurs, as most projects tend to have fixed delivery dates. The number of tests performed will be reduced. The number of acceptable defects will be increased. o These two items could lower the overall quality of the delivered product. Resources will be added to the test team. The test team will work overtime (this could affect team morale). The scope of the plan may be changed. There may be some optimization of resources. This should be avoided, if possible, for obvious reasons.



Used to define terms and acronyms used in the document, and testing in general, to eliminate confusion and promote consistent communications. Regional differences There are often localized differences in the use of this term. In some locations, test plan can mean all of the tests that need to be run. Purists would suggest that a collection of tests or test cases is a Test suite. Some locations would consider what is described above as a test strategy. This usage is generally localized to the Indian market. TEST CASES

editTest cases, suites, scripts and scenarios
A test case is a software testing document, which consists of event, action, input, output, expected result and actual result. Clinically defined (IEEE 829-1998) a test case is an input and an expected result. This can be as pragmatic as 'for condition x your derived result is y', whereas other test cases described in more detail the input scenario and what results might be expected. It can occasionally be a series of steps (but often steps are contained in a separate test procedure that can be exercised against multiple test cases, as a matter of economy) but with one expected result or expected outcome. The optional fields are a test case ID, test step or order of execution number, related requirement(s), depth, test category, author, and check boxes for whether the test is automatable and has been automated. Larger test cases may also contain prerequisite states or steps, and descriptions. A test case should also contain a place for the actual result. These steps can be stored in a word processor document, spreadsheet, database or other common repository. In a database system, you may also be able to see past test results and who generated the results and the system configuration used to generate those results. These past results would usually be stored in a separate table. The term test script is the combination of a test case, test procedure and test data. Initially the term was derived from the byproduct of work created by automated regression test tools. Today, test scripts can be manual, automated or a combination of both. The most common term for a collection of test cases is a test suite. The test suite often also contains more detailed instructions or goals for each collection of test cases. It definitely contains a section where the tester identifies the system configuration used during testing. A group of test cases may also contain prerequisite states or steps, and descriptions of the following tests. Collections of test cases are sometimes incorrectly termed a test plan. They might correctly be called a test specification. If sequence is specified, it can be called a test script, scenario or procedure. Summary:




TEST SUITE The introduction to this article provides insufficient context for those unfamiliar with the subject matter. Please help improve the introduction to meet Wikipedia's layout standards . You can discuss the issue on the talk page. The most common term for a collection of test cases is a test suite. The test suite often also contains more detailed instructions or goals for each collection of test cases. It definitely contains a section where the tester identifies the system configuration used during testing. A group of test cases may also contain prerequisite states or steps, and descriptions of the following tests. Collections of test cases are sometimes incorrectly termed a test plan. They may also be called a test script, or even a test scenario. An executable test suite is a test suite that is ready to be executed. This usually means that there exists a test harness that is integrated with the suite and such that the test suite and the test harness together can work on a sufficiently detailed level to correctly communicate with the system under test (SUT). The counterpart of an executable test suite is an abstract test suite. However, often terms test suites and test plans are used, roughly with the same meaning as executable and abstract test suites, respectively. TEST SCRIPT A test script is a short program written in a programming language used to test part of the functionality of a software system. A written set of steps that should be performed automatically can also be called a test script, however this is more correctly called a test case. Any test that is written as a short program is regarded as an automated test. Test scripts written as a short program can either be written using a special manual functional GUI test tool or in a well-known programming language (such as C++, Tcl, Expect, Java, Perl, Python, or more recently, Ruby). Automated test tools can test many areas of system functionality such as the user interface, performance of the system, the system code and the requirements. Automated testing has the advantage over manual testing in that it is easily repeatable, and thus is favoured when doing regression testing. This however is not always the case as automated

tests may be poorly written and can break during playback. Since most systems are designed with human interaction in mind, it is good practice that a human tests the system at some point. For additional information about written test scripts, please see the test case article. TEST EFFORT The test effort shows the expenses for (still to come) tests. There is a relation with test costs and failure costs (direct, indirect, costs for fault correction). Some factors which influence test effort are: maturity of the software development process , quality and testability of the test object, test infrastructure, skills of staff members, quality goals and test strategy. Methods for estimation of the test effort To analyse all factors is difficult, because most of the factors influence each other. Following approaches can be used for the estimation: top-down estimation and bottom-up estimation. The top-down techniques are formula based and they are relative to the expenses for development: Function Point Analysis (FPA) and Test Point Analysis (TPA) amongst others. Bottomup techiques are based on detailed information and involve often experts. The following techniques belong here: Work Breakdown Structure (WBS) and Wide Band Delphi (WBD). Test efforts from literature In literature test efforts relative to total costs are between 20% and 70%. These values are amongst others dependent from the project specific conditions. When looking for the test effort in the single phases of the test process, these are diversely distributed: with about 40% for test specification and test execution each. ………………………………………………………………………………………………………………… TEST-DRIVEN DEVELOPMENT Test-driven development (TDD) is a software development technique that involves repeatedly first writing a test case and then implementing only the code necessary to pass the test. Test-driven development gives rapid feedback. The technique began to receive publicity in the early 2000s as an aspect of Extreme Programming, but more recently is creating more general interest in its own right [1]. Practitioners emphasize that test-driven development is a method of designing software, not merely a method of testing. Along with other techniques, the concept can also be applied to the improvement and removal of software defects from legacy code that was not developed in this way [2]. Requirements

Test-driven development requires that an automated unit test, defining requirements of the code, is written before each aspect of the code itself. These tests contain assertions that are either true or false. Running the tests gives rapid confirmation of correct behaviour as the code evolves and is refactored. Testing frameworks based on the xUnit concept (see the list of unit testing frameworks for an exhaustive list) provide a mechanism for creating and running sets of automated test cases. Test-Driven Development Cycle The following sequence is based on the book Test-Driven Development by Example [3], which many consider to be the original source text on the concept in its modern form. 1. Add a test In test-driven development each new feature begins with writing a test. In order to write a test, the developer must understand the specification and the requirements of the feature clearly. This may be accomplished through use cases and user stories or design documentation may exist to cover the requirements and exception conditions. 2. Run all tests and see the new one fail This validates that the test harness is working correctly and that the new test does not mistakenly pass without requiring any new code. The new test should also fail for the expected reason. This step tests the test itself, in the negative. 3. Write some code The next step is to write some code that will pass the test. The new code written at this stage will not be perfect and may, for example, pass the test in an inelegant way. That is acceptable as later steps will improve and hone it. It is important that the code written is only designed to pass the test; no further (and therefore untested) functionality should be predicted and 'allowed for' at any stage. 4. Run the automated tests and see them succeed If all test cases now pass, the programmer can be confident that the code meets all the tested requirements. This is a good point from which to begin the final step of the cycle. 5. Refactor code Now the code can be cleaned up as necessary. By re-running the test cases the developer can be confident that refactoring is not damaging any existing functionality. The concept of removing duplication is an important aspect of any software design. In this case, however, it also applies to

removing any duplication between the test code and the production code—for example magic numbers or strings that were repeated in both, in order to make the test pass in step 3. The cycle is then repeated, starting with another new test to push forward the functionality. The size of the steps can be as small as the developer likes, or get larger if s/he feels more confident. If the code written to satisfy a test does not fairly quickly do so, then the step-size may have been too big, and maybe the increment should be split into smaller testable steps. When using external libraries (such as Microsoft ADO.NET) it is important not to make increments that are so small as to be effectively testing the library itself. Development style There are various aspects to using test-driven development, for example the principles of "Keep It Simple, Stupid" (KISS) and "You Ain't Gonna Need It" (YAGNI). By focusing on writing only the code necessary to pass tests, designs can be cleaner and clearer than is often achieved by other methods[3]. To achieve some advanced design concept (such as a Design Pattern), tests are written that will generate that design. The code may remain simpler than the target pattern, but still pass all required tests. This can be unsettling at first but it allows the developer to focus only on what is important. Test-driven development requires the programmer to first fail the test cases. The idea is to ensure that the test really works and can catch an error. Once this is shown, the normal cycle will commence. This has been coined the "Test-Driven Development Mantra", known as red/green/refractor where red means fail and green is pass. Test-driven development constantly repeats the steps of adding test cases that fail, passing them, and refactoring. Receiving the expected test results at each stage reinforces the programmer's mental model of the code, boosts confidence and increases productivity. Benefits Programmers using pure TDD on new (" Greenfield") projects report they only rarely feel the need to invoke a debugger. Test-driven development can help to build software better and faster. It offers more than just simple validation of correctness, but can also drive the design of a program. By focusing on the test cases first, one must imagine how the functionality will be used by clients (in this case, the test cases). Therefore, the programmer is only concerned with the interface and not the implementation. This benefit is complementary to Design by Contract as it approaches code through test cases rather than through mathematical assertions or preconceptions. The power test-driven development offers is the ability to take small steps when required. It allows a programmer to focus on the task at hand as the first goal is to make the test pass. Exceptional cases and error handling are not considered initially. Tests to create these

extraneous circumstances are implemented separately. Another advantage is that test-driven development, when used properly, ensures that all written code is covered by a test. This can give the programmer, and subsequent users, a greater level of trust in the code. While it is true that more code is written with TDD than without TDD, total code implementation time is typically shorter. Large numbers of tests help to limit the number of defects in the code. The early and frequent nature of the tests helps to catch defects early in the development cycle, preventing them from becoming endemic and expensive problems. Eliminating defects early in the process usually avoids lengthy and tedious debugging later in the project. Limitations Test-driven development, as a design technique, can produce inappropriate and low-quality code just as rapidly as high-quality code. A team should use other techniques, such as discussion and code reviews for example, to ensure not only that the correct features are implemented, but that the resulting code uses acceptable styles. James Carr has created a catalog of TDD antipatterns [1], as discussed on the TDD mailing list, and others have proposed that Tester Driven Development itself is a design anti-pattern. Test-driven development only proves correctness of design and functionality according to the test cases written. An incorrect test that does not represent the specifications will produce incorrect code. Some of the emphasis on correctness and design shifts to designing test cases since they are the drivers. As a result, a test-driven development is only as good as its tests. Due to the rapid alternation of tasks described above, of course each developer is responsible for their own tests—they should not be written in bulk, in advance, for example. (Contrarily, code that submits to tests through all phases of its growth will achieve many Design For Test goals implicitly. When developers only write code that can indeed be tested, verification with formal unit tests becomes easier.) As a design technique, test-driven development is unlikely to produce new algorithms, although it will verify their behavior. Computer science research should therefore only use it in conjunction with other techniques. Automated testing may not be feasible at the limits of computer science (cryptography, robustness , artificial intelligence, computer security etc). TDD is difficult to use in some situations, such as graphical user interfaces or relational databases, where systems with complex input and output were not designed for isolated unit testing or refactoring. Code visibility Test-suite code clearly has to be able to access the code it is testing. On the other hand normal design criteria such as information hiding, encapsulation and the separation of concerns should not be compromised. Thus, unit test code for TDD is usually written within the same project or module as the code being tested. In object oriented design this still does not

provide access to private data and methods. In some languages these can be made internal to allow test-suite access, but this is still a distortion of the original intent. Where the facility is provided by the language, it may be better to put the test code into a partial class to allow such privileged access to these members during development. In all cases, thought must be given to the question of deployment. When the final library or executable is assembled or built, is it desirable to include in it all the test code that was used in its development? If it is written within the project or module, maybe as partial classes, then it is likely that it would be included by default. This will make the delivered run-time much larger than necessary, and may introduce security and other unexpected problems. It is unlikely that test code compiled into a delivered binary will ever be run; it is mainly useful to a developer with full access to the source code. In many compiled languages, this can largely be solved by the use of compiler directives surrounding all test code to prevent deployment.

#if DEBUG ...//test class #endif

In scripting languages, keeping all test code in separate folders may help remind users not to deploy it, for example to live web servers, especially if the folders have, for example '_DO_NOT_DEPLOY' as part of their name. Fakes, mocks and integration tests If a code project is going to be developed by the creation of several hundred test cases, and the developers are expected to rerun these tests very frequently throughout the development, then it is necessary for these tests to run extremely quickly. For this reason unit tests, as used in TDD, should never cross process boundaries, i.e. they should never lead to any calls to code running in a different process to the one under test. Much more so, they should never cause and wait for any network access, across host boundaries. This appears to lead to a problem when the code under development relies on a database or a web service or any other external process or service. This is not a problem, however, but an opportunity and a driving force to design more modular, more testable and more re-usable code. Two steps are necessary: 1. Whenever external access is going to be needed in the final design, an interface should be defined that prescribes the access that will be available. 2. The interface should be implemented in two ways, one of which really accesses the external process, and the other is a fake or mock object. Fake objects need do little more than log a message such as "Person object saved" to a log file or to the console, if that. Mock objects differ in that they themselves may contain test assertions that will make the test fail, for example, if the person's name and other data are not valid. Fake

and mock object methods that return data, ostensibly from a data store or user, can help the test process by always returning the same, realistic data that the tests can rely upon. The disadvantage of this approach is that the actual database or other external-access code is never tested by the TDD process. This must be avoided, and other tests are needed that instantiate the test-driven code with its 'real' implementations of the interfaces discussed above. Many find it useful if these tests are kept quite separate from the TDD unit tests, and are referred to as integration tests. There will be fewer of them, and they need be run less often than the unit tests. They can nonetheless be implemented using the same, for example xUnit, testing framework. Integration tests that test writing to a database should be careful always to clean their test data out after the test, even if the test fails for some other reason. This can be achieved using some combination of the following techniques where relevant and available to the developer:    the TearDown method integrated into many test frameworks try...catch...finally exception handling structures where these are available database transactions where a transaction atomically includes perhaps a write, a read and a matching delete operation.

………………………………………………………………………………………………………………… DEFECT MANAGEMENT USING TEST DIRECTOR DEFECT TRACKING In engineering, defect tracking is the process of finding defects in a product, (by inspection, testing, or recording feedback from customers), and making new versions of the product that fix the defects. Defect tracking is important in software engineering as complex software systems typically have tens or hundreds of thousands of defects: managing, evaluating and prioritizing these defects is a difficult task: defect tracking systems are computer database systems that store defects and help people to manage them. DEFECT REPORTING TRACKING WORKFLOW BUG TRACKING SYSTEM A bug tracking system is a software application that is designed to help programmers keep track of reported software bugs in their work. It may be regarded as a sort of issue tracking system. Many bug-tracking systems, such as those used by most open source software projects, allow users to enter bug reports directly. Other systems are used only internally in a company or organization doing software development. Typically bug tracking systems are integrated with other software project management applications.

Bug tracking system is essential and used extensively by companies developing software products. A major component of a bug tracking system is a database that records facts about known bugs. Facts may include the time a bug was reported, its severity, the erroneous program behavior, and details on how to reproduce the bug; as well as the identity of the person who reported it and any programmers who may be working on fixing it. Typical Bug Tracking Systems support the concept of the life cycle for a bug which is tracked through status assigned to the bug. A bug tracking system should allow administrators to configure which users can view or edit a bug in a certain status, move the bug to another status or delete the bug. The system should also allow administrators to configure the bug statuses and to what status a bug in a particular status can be moved to. In a corporate environment, a bug-tracking system may be used to generate reports on the productivity of programmers at fixing bugs. However, this may sometimes yield inaccurate results because different bugs may have different levels of severity and complexity. The severity of a bug may not be directly related to the complexity of fixing the bug. There may be different opinions among the managers, architects and the developers about the relative ease of fixing bugs. local bug trackers are usually a Computer Program used by a team of application support professionals (often a help desk) to keep track of issues communicated to software developers. Using an LBT allows support professionals to track bugs in their "own language" and not the "language of the developers." In addition, LBT use allows a team of support professionals to track specific information about users who have called to complain that may not always be needed in the actual development queue. (Thus, there are two tracking systems when an LBT is in place.) ………………………………………………………………………………………………………………… …………… DESIGN PREDICATES Design predicates are a method, invented by Thomas McCabe, to quantify the complexity of the integration of two units of software. Each of the four types of design predicates have an associated integration complexity rating. For pieces of code that apply more than one design predicate, integration complexity ratings can be combined. The sum of the integration complexity for a unit of code, plus one, is the maximum number of test cases necessary to exercise the integration fully. Though a test engineer can typically reduce this by covering as many previously uncovered design predicates as possible with each new test. Also, some combinations of design predicates might be logically impossible. Types of Calls Unconditional Call

Unit A always calls unit B. This has an integration complexity of 0. For example:

unitA::functionA() { unitB->functionB(); }

Conditional Call Unit A may or may not call unit B. This integration has a complexity of 1, and needs two tests: one that calls B, and one that doesn't.

unitA::functionA() { if (condition) unitB->functionB(); }

Mutually Exclusive Conditional Call This is like a programming language's switch statement. Unit A calls exactly one of several possible units. Integration complexity is n - 1, where n is the number of possible units to call.

unitA::functionA() { switch (condition) { case 1: unitB->functionB(); break; case 2: unitC->functionC(); break; ... default: unitN->functionN(); break; } }

Iterative Call In an iterative call, unit A calls unit B at least once, but maybe more. This integration has a complexity of 1. It also requires two tests: one that calls unit B once, and one test that calls it more than once.

unitA::functionA() { do {

unitB->functionB(); } while (condition); }

Combining Calls Any particular integration can combine several types of calls. For example, unit A may or may not call unit B; and if it does, it can call it one or more times. This integration combines a conditional call, with its integration complexity of 1, and an iterative call, with its integration complexity of 1. The combined integration complexity totals 2.

unitA::functionA() { if (condition) { for ( i = 0 ; i < someNumber ; i++ ) { unitB->functionB(); } } }

Since the number of necessary tests is the total integration complexity plus one, this integration would require 3 tests. In one, unit B isn't called. In another, unit B is called once. And in the final, unit B is called more than once. ………………………………………………………………………………………………………………… …………… STATIC CODE ANALYSIS Static code analysis is the analysis of computer software that is performed without actually executing programs built from that software (analysis performed on executing programs is known as dynamic analysis ). In most cases the analysis is performed on some version of the source code and in the other cases some form of the object code. The term is usually applied to the analysis performed by an automated tool, with human analysis being called program understanding or program comprehension. The sophistication of the analysis performed by tools varies from those that only consider the behavior of individual statements and declarations, to those that include the complete source code of a program in their analysis. Uses of the information obtained from the analysis vary from highlighting possible coding errors (e.g., the lint tool) to formal methods that mathematically prove properties about a given program (e.g., its behavior matches that of its specification). Some people consider software metrics and reverse engineering to be forms of static analysis.

A growing commercial use of static analysis is in the verification of properties of software used in safety-critical computer systems and locating potentially vulnerable code. Formal methods Formal methods are the term applied to the analysis of software (and hardware) whose results are obtained purely through the use of rigorous mathematical methods. The mathematical techniques used include denotational semantics , axiomatic semantics , operational semantics, and abstract interpretation. It has been proven that finding possible run-time errors is undecidable: there is no mechanical method that can always answer truthfully whether programs may or not exhibit runtime errors. This result dates from the works of Church, Gödel and Turing in the 1930s (see the halting problem and Rice's theorem). As with most undecidable questions, one can still attempt to give useful approximate solutions. Some of the implementation techniques of formal static analysis include:   

Model checking considers systems that have finite state or may be reduced to finite state by abstraction; Abstract interpretation models the effect that every statement has on the state of an
abstract machine (i.e., it 'executes' the software based on the mathematical properties of each statement and declaration). Use of assertions in program code as first suggested by Hoare logic. There is tool support for some programming languages (e.g., the SPARK programming language (a subset of Ada) and the Java Modeling Language — JML — using ESC/Java and ESC/Java2).

………………………………………………………………………………………………………………… …………… DYNAMIC PROGRAM ANALYSIS Dynamic code analysis is the analysis of computer software that is performed with executing programs built from that software on a real or virtual processor (analysis performed without executing programs is known as static code analysis ). Such tools may require loading of special libraries or even recompilation of program code. Examples  

Valgrind, performances run on a virtual processor, can detect memory errors (e.g. connected with misuse malloc and free) and race conditions in multithread programs. Dmalloc, library for checking memory allocation and leaks. Software must be



What is dynamic analysis and how can it be automated? Dynamic Analysis uses test data sets to execute software in order to observe its behavior and produce test coverage reports. This assessment of source code ensures consistent levels of high quality testing and correct use of capture/playback tools.

………………………………………………………………………………………………………………… …………… ELEPHANT IN CAIRO An elephant in Cairo is a term used in computer programming to describe a piece of test data that is designed to ensure that an algorithm is working. The term derives from a humorous essay circulated on the Internet and published in Byte magazine in September 1989 that described how various professions would go about hunting elephants, with programmers following the algorithm: 1. Go to Africa. 2. Start at the Cape of Good Hope. 3. Work northward in an orderly manner, traversing the continent alternately east and west until you get to Cairo, 4. During each traverse pass, o Catch each animal seen. o Compare each animal caught to a known elephant. o Stop when a match is detected. Experienced programmers modify Algorithm A by placing a known elephant in Cairo to ensure that the algorithm will terminate. Go to Africa. Put an elephant in Cairo. Start at the Cape of Good Hope. Work northward in an orderly manner, traversing the continent alternately east and west until an elephant is found, 5. During each traverse pass, o Catch each animal seen. o Compare each animal caught to the elephant in Cairo. 6. If the elephant is in Cairo, you put it there. 1. 2. 3. 4. ………………………………………………………………………………………………………………… TEST AUTOMATION

editManual vs. Automated
Some writers believe that test automation is so expensive relative to its value that it should be used sparingly. Others, such as advocates of agile development, recommend automating

100% of all tests. A challenge with automation is that automated testing requires automated test oracles (an oracle is a mechanism or principle by which a problem in the software can be recognized). Such tools have value in load testing software (by signing on to an application with hundreds or thousands of instances simultaneously), or in checking for intermittent errors in software. The success of automated software testing depends on complete and comprehensive test planning. Software development strategies such as test-driven development are highly compatible with the idea of devoting a large part of an organization's testing resources to automated testing. Many large software organizations perform automated testing. Some have developed their own automated testing environments specifically for internal development, and not for resale.


Over the past few years , tools that help programmers quickly create applications with graphical user interfaces have dramatically improved programmer productivity. This has increased the pressure on testers, who are often perceived as bottlenecks to the delivery of software products. Testers are being asked to test more and more code in less and less time.
Test automation is one way to do this, as manual testing is time consuming. As and when different versions of software are released, the new features will have to be tested manually time and again. But, now there are tools available that help the testers in the automation of the GUI which reduce the test time as well as the cost, other test automation tools support execution of performance tests. Many test automation tools provide record and playback features that allow users to record interactively user actions and replay it back any number of times, comparing actual results to those expected. However, reliance on these features poses major reliability and maintainability problems. Most successful automators use a software engineering approach, and as such most serious test automation is undertaken by people with development experience. A growing trend in software development is to use testing frameworks such as the xUnit frameworks (for example, JUnit and NUnit) which allow the code to conduct unit tests to determine whether various sections of the code are acting as expected in various circumstances. Test cases describe tests that need to be run on the program to verify that the program runs as expected. All three aspects of testing can be automated. Another important aspect of test automation is the idea of partial test automation, or automating parts but not all of the software testing process. If, for example, an oracle cannot reasonably be created, or if fully automated tests would be too difficult to maintain, then a software tools engineer can instead create testing tools to help human testers perform their jobs more efficiently. Testing tools can help automate tasks such as product installation, test data creation, GUI

interaction, problem detection (consider parsing or polling agents equipped with oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion. Test automation is expensive and it is an addition, not a replacement, to manual testing. It can be made cost-effective in the longer term though, especially in regression testing. One way to generate test cases automatically is model-based testing where a model of the system is used for test case generation, but research continues into a variety of methodologies for doing so. GUI SOFTWARE TESTING In computer science, GUI software testing is the process of testing graphical user interface-based software to ensure it meets its written specifications. This is normally done through the use of a variety of test cases. Test Case Generation To generate a ‗good‘ set of test cases, the test designer must be certain that their suite covers all the functionality of the system and also has to be sure that the suite fully exercises the GUI itself. The difficulty in accomplishing this task is twofold: one has to deal with domain size and then one has to deal with sequences. In addition, the tester faces more difficulty when they have to do regression testing. The size problem can be easily illustrated. Unlike a CLI (command line interface) system, a GUI has many operations that need to be tested. A very small program such as Microsoft WordPad has 325 possible GUI operations[1]. In a large program, the number of operations can easily be an order of magnitude larger. The second problem is the sequencing problem. Some functionality of the system may only be accomplishable by following some complex sequence of GUI events. For example, to open a file a user may have to click on the File Menu and then select the Open operation, and then use a dialog box to specify the file name, and then focus the application on the newly opened window. Obviously, increasing the number of possible operations increases the sequencing problem exponentially. This can become a serious issue when the tester is creating test cases manually.

Regression testing becomes a problem with GUIs as well. This is because the GUI may
change significantly across versions of the application, even though the underlying application may not. A test designed to follow a certain path through the GUI may not be able to follow that path since a button, menu item or dialog may not be where it used to be. These issues have driven the GUI testing problem domain towards automation. Many different techniques have been proposed to automatically generate test suites that are complete and that simulate user behavior. Most of the techniques attempt to build on previously used CLI techniques. However, most have scaling problems when they are applied to GUI‘s. For example, Finite State Machine-based modeling[2][3] — where a system is modeled as a finite state machine and a program is used to

generate test cases that exercise all states — can work well on a system that has a limited number of states but may become overly complex and unwieldy for a GUI. (See also modelbased testing ). Planning and artificial intelligence A novel approach to test suite generation, adapted from a CLI technique involves using a planning system. Planning is a well studied technique from the artificial intelligence (AI) domain that attempts to solve problems that involve 4 parameters: an initial state, a goal state, a set of operators and a set of objects to operate on. Planning systems determine a path from the initial state to the goal state by using the operators. An extremely simple planning problem would be one where you had two words and one operation called ‗change a letter‘ that allowed you to change one letter in a word to another letter – the goal of the problem would be to change one word into another. For GUI testing, the problem is a bit more complex. In [1] the authors used a planner called IPP to demonstrate this technique. The method used is very simple to understand. First, the systems UI are analyzed to determine what operations are possible. These operations become the operators used in the planning problem. Next an initial system state is determined. Next a goal state is determined that the tester feels would allow exercising of the system. Lastly the planning system is used to determine a path from the initial state to the goal state. This path becomes the test plan. Using a planner to generate the test cases has some specific advantages over manual generation. A planning system by its very nature generates solutions to planning problems in a way that is very beneficial to the tester. First, the plans are always valid. What this means is that the output of the system can be one of two things, a valid and correct plan that uses the operators to attain the goal state or no plan at all. This is beneficial because much time can be wasted when manually creating a test suite due to invalid test cases that the tester thought would work but didn‘t. Secondly, a planning system pays attention to order. Often to test a certain function, the test case must be complex and follow a path through the GUI where the operations are performed in a specific order. When done manually, this can lead to errors and also can be quite difficult and time consuming to do. Finally, and most importantly, a planning system is goal oriented. What this means and what makes this fact so important is that the tester is focusing test suite generation on what is most important, testing the functionality of the system. When manually creating a test suite, the tester is more focused on how to test a function (i.e. The specific path through the GUI). By using a planning system, the path is taken care of and the tester can focus on what function to test. An additional benefit of this is that a planning system is not restricted in any way when generating the path and may often find a path that was never anticipated by the tester. This problem is a very important one to combat. Generating test suites using Genetic Algorithms Another interesting method of generating GUI test cases uses the theory that good GUI test coverage can be attained by simulating a novice user. One can speculate that an expert user of a

system will follow a very direct and predictable path through a GUI and a novice user would follow a more random path. The theory therefore is that if we used an expert to test the GUI, many possible system states would never be achieved. A novice user, however, would follow a much more varied, meandering and unexpected path to achieve the same goal so it‘s therefore more desirable to create test suites that simulate novice usage because they will test more. The difficulty lies in generating test suites that simulate ‗novice‘ system usage. Using Genetic algorithms is one proposed way to solve this problem. Novice paths through the system are not random paths. First, a novice user will learn over time and generally won‘t make the same mistakes repeatedly, and, secondly, a novice user is not analogous to a group of monkeys trying to type Hamlet, but someone who is following a plan and probably has some domain or system knowledge. Genetic algorithms work as follows: a set of ‗genes‘ are created randomly and then are subjected to some task. The genes that complete the task best are kept and the ones that don‘t are discarded. The process is again repeated with the surviving genes being replicated and the rest of the set filled in with more random genes. Eventually one gene (or a small set of genes if there is some threshold set) will be the only gene in the set and is naturally the best fit for the given problem. For the purposes of the GUI testing, the method works as follows. Each gene is essentially a list of random integer values of some fixed length. Each of these genes represents a path through the GUI. For example, for a given tree of widgets, the first value in the gene (each value is called an allele) would select the widget to operate on, the following alleles would then fill in input to the widget depending on the number of possible inputs to the widget (for example a pull down list box would have one input…the selected list value). The successes of the genes are scored by a criterion that rewards the best ‗novice‘ behavior. The system to do this testing described in [6] can be extended to any windowing system but is described on the X window system. The X Window system provides functionality (via XServer and the editors' protocol) to dynamically send GUI input to and get GUI output from the program without directly using the GUI. For example, one can call XSendEvent () to simulate a click on a pull-down menu, and so forth. This system allows researchers to automate the gene creation and testing so for any given application under test, a set of novice user test cases can be created. Running the test cases At first the strategies were migrated and adapted from the CLI testing strategies. A popular method used in the CLI environment is capture/playback. Capture playback is a system where the system screen is ―captured‖ as a bitmapped graphic at various times during system testing. This capturing allowed the tester to ―playback‖ the testing process and compares the screens at the output phase of the test with expected screens. This validation could be automated since the screens would be identical if the case passed and different if the case failed. Using capture/playback worked quite well in the CLI world but there are significant problems when one tries to implement it on a GUI based system [7]. The most obvious problem one finds

that the screen in a GUI system may look different while the state of the underlying system is the same, making automated validation extremely difficult. This is because a GUI allows graphical objects to vary in appearance and placement on the screen. Fonts may be different, window colors or sizes may vary but the system output is basically the same. This would be obvious to a user, but not obvious to an automated validation system. To combat this and other problems, testers have gone ‗under the hood‘ and collected GUI interaction data from the underlying windowing system [8]. By capturing the window ‗events‘ into logs the interactions with the system are now in a format that is decoupled from the appearance of the GUI. Now, only the event streams are captured. There is some filtering of the event streams necessary since the streams of events are usually very detailed and most events aren‘t directly relevant to the problem. This approach can be made easier by using an MVC architecture for example and making the view (i.e. the GUI here) as simple as possible while the model and the controller hold all the logic. Another approach is to use an HTML interface or a three-tier architecture which make it also possible to better separate the user interface from the rest of the application. Another way to run tests on a GUI is to build a driver into the GUI so that commands or events can be sent to the software from another program [6]. This method of directly sending events to and receiving events from a system is highly desirable when testing since the input and output testing can be fully automated and user error is eliminated. List of GUI testing tools This is a list of tools for automated functional GUI software testing. Cross-Platform tools   

Eggplant : automated user-interface testing of a device running on any platform that uses the Virtual Network Computing (VNC) protocol. Squish : automated GUI testing tool on Windows, Linux/Unix, Mac OS X, embedded
Linux for Qt, Web/HTML, Java/SWT, Java/Swing, Mac OS X, Tk, 4JS, XView, ... SilkTest is a leading functional GUI testing tool]] for Windows, and several flavors of UNIX.

Windows test tools       

Badboy is an inexpensive automated testing tool targeted specifically at testing of web

KeepLiquid TestAutomation is a keyword and data driven GUI test automation tool. NUnitForms is an open source extension for unit testing. QuickTest Professional (QTP) is a leading functional GUI testing tool. Ranorex is a free GUI test and automation Library for C++, Python and for all .NET
languages. SilkTest is a leading functional GUI testing tool]]. TestDrive-Gold, code-free testing of GUI and browser applications.

  

TestPartner an automated functional and regression testing tool from Compuware

WinRunner is a leading functional GUI testing tool. Wintask is an automation and regression test tool for GUI.

Linux or UNIX test tools 

Dogtail - An open source GUI test framework written in Python

Java test tools   

abbot : an open source Java GUI Testing Framework (Open source) GUIdancer: a tool for creating automated GUI tests for programs written with Java and Swing, without programming. GTT: GTT is a tool for developing and executing test scripts for Java swing-based
applications. Test designers develop test scripts by capturing and replaying user interactions (Open source). jemmy : a tool developed by NetBeans (can be used independently as well) for testing Swing applications. jfcunit : a tool for testing Swing applications using an Extreme programming methodology (Open source). Marathon an open source Java GUI Test Tool focuses on end-user testing (Open source) PGUIRobot : a very easy to use automated tool for functional and GUI testing Java programs, it can also be used for making canned demonstrations by event and screen capture (Open source). UISpec4J : a Java GUI Testing Framework whose APIs are designed to make the test scripts as close as possible to human-readable text (Open source). TestPartner an automated functional and regression testing tool from Compuware Corporation. qftestJUI, now QF-Test : is a tool for the creation, execution and management of automated tests for Java/Swing and Java/SWT applications.

   

  

Web only test tools    

Selenium - Selenium tests run directly in a browser. Runs in IE, Mozilla and Firefox on
Windows, Linux, and Macintosh. Watir - An opensource webtesting framework written in Ruby. Works only under Windows/IE6. WET Web Tester - An opensource webtesting framework inspired by Watir.Includes a script generation utility Sahi - Open source, java based testing tool for automated testing of web/html/ajax applications

Mercury Interactive Tools - Win Runner, QTP Compuware Tools - QA Tester Empirix Tools eTester Related tools

 

AutoHotkey is a free GUI automation/scripting tool for windows applications, source
available under the GPL. Jameleon is an automated testing framework that uses plug-ins to test database, desktop, web, and mainframe software applications. Used mainly for integration, regression, functional, and acceptance testing at the code/object level.


WinRunner, Mercury Interactive’s enterprise functional testing tool. It is used to quickly create and run sophisticated automated tests on your application. Winrunner helps you automate the testing process, from test development to execution. You create adaptable and reusable test scripts that challenge the functionality of your application. Prior to a software release, you can run these tests in a single overnight run- enabling you to detect and ensure superior software quality.
What's New in WinRunner 7.5?

Automatic Recovery

The Recovery Manager provides an easy-to-use wizard that guides you through the process of defining a recovery scenario. You can specify one or more operations that enable the test run to continue after an exception event occurs. This functionality is especially useful during unattended test runs, when errors or crashes could interrupt the testing process until manual intervention occurs.

Silent Installation Now you can install WinRunner in an unattended mode using previously recorded installation preferences. This feature is especially beneficial for those who use enterprise software management products or any automated software distribution mechanisms. Enhanced Integration with TestDirector WinRunner works with both TestDirector 6.0, which is client/server-based, and TestDirector 7.x, which is Web-based. When reporting defects from WinRunner’s test results window, basic information about the test and any checkpoints can be

automatically populated in TestDirector’s defect form. WinRunner now supports version control, which enables updating and revising test scripts while maintaining old versions of each test. Support for Terminal Servers Support for Citrix and Microsoft Terminal Servers makes it possible to open several window clients and run WinRunner on each client as a single user. Also, this can be used with LoadRunner to run multiple WinRunner Vusers. Support for More Environments WinRunner 7.5 includes support for Internet Explorer 6.x and Netscape 6.x, Windows XP and Sybase's PowerBuilder 8, in addition to 30+ environments already supported by WinRunner 7.
WinRunner provides the most powerful, productive and cost-effective solution for verifying enterprise application functionality. For more information on WinRunner, contact a


Interactive local representative for pricing, evaluation, and distribution information.

Win Runner(Features & Benefits) Test functionality using multiple data combinations in a single test WinRunner's DataDriver Wizard eliminates programming to automate testing for large volumes of data. This saves testers significant amounts of time preparing scripts and allows for more thorough testing. Significantly increase power and flexibility of tests without any programming The Function Generator presents a quick and error-free way to design tests and enhance scripts without any programming knowledge. Testers can simply point at a GUI object, and WinRunner will examine it, determine its class and suggest an appropriate function to be used. Use multiple verification types to ensure sound functionality WinRunner provides checkpoints for text, GUI, bitmaps, URL links and the database, allowing testers to compare expected and actual outcomes and identify potential problems with numerous GUI objects and their functionality. Verify data integrity in your back-end database Built-in Database Verification confirms values stored in the database and ensures transaction accuracy and the data integrity of records that have been updated, deleted and added.

View, store and verify at a glance every attribute of tested objects WinRunner’s GUI Spy automatically identifies, records and displays the properties of standard GUI objects, ActiveX controls, as well as Java objects and methods. This ensures that every object in the user interface is recognized by the script and can be tested. Maintain tests and build reusable scripts The GUI map provides a centralized object repository, allowing testers to verify and modify any tested object. These changes are then automatically propagated to all appropriate scripts, eliminating the need to build new scripts each time the application is modified. Test multiple environments with a single application WinRunner supports more than 30 environments, including Web, Java, Visual Basic, etc. In addition, it provides targeted solutions for such leading ERP/CRM applications as SAP, Siebel, PeopleSoft and a number of others.

Using Rapid Test Script wizard


Start->Program Files->Winrunner->winruner Select the Rapid Test Script Wizard (or) create->Rapid Test Script wizard Click Next button of welcome to script wizard Select hand icon and click on Application window and Cilck Next button Select the tests and click Next button Select Navigation controls and Click Next button Set the Learning Flow(Express or Comprehensive) and click Learn button Select start application YES or NO, then click Next button Save the Startup script and GUI map files, click Next button Save the selected tests, click Next button Click Ok button Script will be generated.then run the scripts. Run->Run from top Find results of each script and select tools->text report in Winrunner test results.

Using GUI-Map Configuration Tool:


Open an application.

        

Select Tools-GUI Map Configuration;Windows pops-up. Click ADD button;Click on hand icon. Click on the object, which is to be configured. A user-defined class for that object is added to list. Select User-defined class you added and press ‗Configure‘ button. Mapped to Class;(Select a corresponding stanadard class from the combo box). You can move the properties from available properties to Learned Properties. By selecting Insert button Select the Selector and recording methods. Click Ok button Now, you will observe Winrunner indentifying the configured objects.

Using Record-ContextSensitive mode:

    

Create->Record context Sensitive Select start->program files->Accessories->Calculator Do some action on the application. Stop recording Run from Top; Press ‗OK‘.

Using Record-Analog Mode:

       

Create->Insert Function->from function generator Function name:(select ‗invoke_application‘ from combo box). Click Args button; File: mspaint. Click on ‗paste‘ button; Click on ‗Execute‘ button to open the application; Finally click on ‗Close‘. Create->Record-Analog. Draw some picture in the paintbrush file. Stop Recording Run->Run from Top; Press ‗OK‘.

GUI CHECK POINTS-Single Property Check:

 

Create->Insert function->Function Generator-> (Function name:Invoke_application; File :Flight 1a) Click on‘paste‘ and click on‘execute‘ & close the window.

       

Create->Record Context sensitive. Do some operations & stop recording. Create->GUI Check Point->For single Property. Click on some button whose property to be checked. Click on paste. Now close the Flight1a application; Run->Run from top. Press ‗OK‘ it displays results window. Double click on the result statement. It shows the expected value & actual value window.

GUI CHECK POINTS-For Object/Window Property:

         

Create->Insert function->Function Generator-> (Function name:Invoke_application; File :Flight 1a) Click on‘paste‘ and click on‘execute‘ & close the window. Create->Record Context sensitive. Do some operations & stop recording. Create->GUI Check Point->Object/Window Property. Click on some button whose property to be checked. Click on paste. 40Now close the Flight 1a application; Run->Run from top. Press ‗OK‘ it displays results window. Double click on the result statement. It shows the expected value & actual value window.

GUI CHECK POINTS-For Object/Window Property:

        

Create->Insert function->Function Generator-> (Function name:Invoke_application; File :Flight 1a) Click on‘paste‘ and click on‘execute‘ & close the window. Create->Record Context sensitive. Do some operations & stop recording. Create->GUI Check Point->For Multiple Object. Click on some button whose property to be checked. Click on Add button. Click on few objects & Right click to quit. Select each object & select corresponding properties to be checked for that object: click ‗OK‘.


Run->Run from Top. It displys the results.


For object/window.

           

Create->Insert function->Function Generator-> (Function name:Invoke_application; File :Flight 1a) Click on‘paste‘ and click on‘execute‘ & close the window. Create->Record Context sensitive. Enter the Username, Password & click ‗OK‘ button Open the Order in

Flight Reservation Application

Select File->Fax Order& enter Fax Number, Signature Press ‗Cancel‘ button. Create->Stop Recording. Then open

Fax Order in Flight Reservation Application

Create->Bitmap Check->For obj.window; Run->run from top. The test fails and you can see the difference.

For Screen Area:

     

Open new

Paint Brush file;

Create->Bitmapcheck point->from screen area. Paint file pops up; select an image with cross hair pointer. Do slight modification in the Run->Run from Top. The test fails and you can see the difference of images.

paint file(you can also run on the same paint file);

Using Default check(for MS-Access only)

   

Create->Database Check Point->Default check Select the Specify Click Next button Click Create button

SQL Statement check box

       

Type New DSN name and Click New button Then select a driver for which you want to set up a database & double clcik that driver Then select Browse button and retype same DSN name and Click save button. Click Next button & click Finish button Select database button & set path of the your database name Click ‗OK‘ button & then Click the your DSN window ‗OK‘ button Type the SQL query in SQL box Theb click Finish button Note : same process will be Custom Check Point

Runtime Record Check Point.

 

Repeat above 10 steps. Type query of two related tables in

SQL box Ex: select Orders.Order_Number,

Flights.Flight_Number from Orders, Flights where Flight.Flight_Number=Orders.Flight_Number.

      

Select Finish Button Select hand Icon button& select Order No in your Application Click Next button. Select hand Icon button& select Filght No in your Application Click Next button Select any one of the following check box 1. One match record 2. One or more match records. 3. No match record select Finish button the script will be generated.

Synchronization Point

For Obj/Win Properties:

      

Open start->Programs->Win Runner->Sample applications->Flight1A. Open winrunner window Create->RecordContext Sensitive Insert information for new Order &click on "insert Order" button After inserting click on "delete" button Stop recording& save the file. Run->Run from top: Gives your results.

Without Synchronization:


settings->General Options->Click on "Run" tab. "Timeout for checkpoints& Cs statements‘ value:10000 follow 1 to 7->the test display on "Error disabled.

Message" that "delete" button is

With Synchronization:

    

Keep Timeout value:1000 only Go to the Test Script file, insert pointed after "Insert Order" button, press statement. Create->Synchronization->For Obj/Window Property Click on"Delete Order" button & select enable property; click on "paste". It inserts the Synch statement.

For Obj/Win Bitmap:

      

Create-> Record Context Sensitive. Insert information for new order & click on "Insert order" button Stop recording & save the file. Go to the TSL Script, just before inserting of data into "date of flight" insert pointer. Create->Synchronization->For Obj/Win Bitmap is selected. (Make sure

flight reservation is empty) click on "data of flight" text box

Run->Run from Top; results are displayed. Note:(Keep "Timeout value" :1000)

Get Text: From Screen Area:

(Note: Checking whether Order no is increasing when ever Order is created)

        

Open Flight1A; Analysis->graphs(Keep it open) Create->get text->from screen area Capture the No of tickets sold; right clcik &close the graph Now , insert new order, open the graph(Analysis->graphs) Go to Winrunner window, create->get text->from screen area Capture the No of tickets sold and right click; close the graph Save the script file Add the followinf script; If(text2==text1) tl_step("text comparision",0,"updateed"); else tl_step("text comparision",1,"update property"); Run->Run from top to see the results.

Get Text: For Object/Window:

      

Open a "Calc" application in two windows (Assuming two are two versions) Create->get text->for Obj/Window Click on some button in one window Stop recording Repeat 1 to 4 for Capture the text of same object from another "Calc" application. Add the following TSL(Note:Change "text" to text1 & text2 for each statement) if(text1==text2) report_msg("correct" text1); Else report_msg("incorrect" text2); Run & see the results

Using GUI-Spy:

Using the GUI Spy, you can view and verify the properties of any GUI object on selected application

    

Tools->Gui Spy… Select Spy On ( select Object or Window) Select Hand icon Button Point the Object or window & Press Ctrl_L + F3. You can view and verify the properties.

Using Virtual Object Wizard:


         

tools->Virtual Object Wizard. Click Next Button Select standard class object for the virtual object Ex: class:Push_button Click Next button Click Mark Object button Drag the cursor to mark the area of the virtual object. Click Next button Assign the Logical Name, This name will appear in the test script when you record object. Select Yes or No check box Click Finish button

  

Go to winrunner window & Create->Start Recording. Do some operations Stop Recording

Using Gui Map Editor:

Using the GUI Map Editor, you can view and modify the properties of any GUI object on selected application. To modify an object’s logical name in a GUI map file

     

Tools->GUI Map Editor Select Learn button Select the Application A winrunner message box informs ―do you want to learn all objects within the window‖ & select ‗yes‘‘ button. Select perticular object and select Modify Button Change the Logical Name& click ‗OK‘ Button Save the File

To find an object in a GUI map file:

    

Choose Tools > GUI Map Editor. Choose View > GUI Files. Choose File > Open to load the GUI map file. Click Find. The mouse pointer turns into a pointing hand. Click the object in the application being tested. The object is highlighted in the GUI map file.

To highlight an object in a Application:

    

Choose Tools > GUI Map Editor. Choose View > GUI Files. Choose File > Open to load the GUI map file. Select the object in the GUI map file Click Show. The object is highlighted in the Application.

Data Driver Wizard


Start->Programs->Wirunner->Sample applications->Flight 1A



Flight Reservation Application

Go to Winrunner window Create->Start recording Select file->new order, insert the fields; Click the Insert Order Tools->Data Table; Enter different Customer names in one row and Tickets in another row. Default that two column names are Noname1 and Noname2. Tools->Data Driver Wizard Click Next button &select the data table Select Parameterize the test; select Line by Line check box Click Next Button Parameterize each specific values with column names of tables;Repeat for all Finalli Click finish button. Run->Run from top; View the results.

Merge the GUI Files:

Manual Merge

      

Tools->Merge GUI Map Files A WinRunner message box informs you that all open GUI maps will be closed and all unsaved changes will be discarded & click ‗OK‘ button. Select the Manual Merge. Manual Merge enables you to manually add GUI objects from the source to target files. To specify the Target GUI map file click the browse button& select GUI map file To specify the Source GUI map file. Click the add button& select source GUI map file. Click ‗OK‘ button GUI Map File Manual Merge Tool Opens Select Objects and move Source File to Target File Close the GUI Map File Manual Merge Tool

Auto Merge

  

Tools->Merge GUI Map Files A WinRunner message box informs you that all open GUI maps will be closed and all unsaved changes will be discarded & click ‗OK‘ button. Select the Auto Merge in Merge Type. If you chose Auto Merge and the source GUI map files are merged successfully without conflicts, To specify the Target GUI map file click the browse button& select GUI map file

  

To specify the Source GUI map file. Click the add button& select source GUI map file. Click ‗OK‘ button A message confirms the merge.

Manually Retrive the Records form


     

db_connect(query1,DSN=Flight32); db_execute_query(query1,select * from Orders,rec); db_get_field_value(query1,#0,#0); db_get_headers(query1, field_num,headers); db_get_row(query1,5,row_con); db_write_records(query1,,c:\\str.txt,TRUE,10);


1. web_browser_invoke ( browser, site );

// invokes the browser and opens a specified site. browser The name of browser (IE or NETSCAPE). site The address of the site.

2. web_cursor_to_image ( image, x, y );

// moves the cursor to an image on a page. image The logical name of the image. x,y The x- and ycoordinates of the mouse pointer when moved to an image

3. web_cursor_to_label ( label, x, y );

// moves the cursor to a label on a page. label The name of the label. x,y The x- and y-coordinates of the mouse pointer when moved to a label.

4.web_cursor_to_link ( link, x, y );

// moves the cursor to a link on a page. link The name of the link. x,y The x- and y-coordinates of the mouse pointer when moved to a link.

5.web_cursor_to_obj ( object, x, y );

// moves the cursor to an object on a page. object The name of the object. x,y The x- and ycoordinates of the mouse pointer when moved to an object.

6.web_event ( object, event_name [, x , y ] );

// uns an event on a specified object. object The logical name of the recorded object. event_name The name of an event handler. x,y The x- and y-coordinates of the mouse pointer when moved to an object

7.web_file_browse ( object );

// clicks a browse button. object A file-type object.

8.web_file_set ( object, value );

// sets the text value in a file-type object. object A file-type object. Value A text string.

9. web_find_text ( frame, text_to_find, result_array [, text_before, text_after, index, show ] );

// returns the location of text within a frame.

10. web_frame_get_text ( frame, out_text [, text_before, text_after, index ] );

// retrieves the text content of a frame.

11. web_frame_get_text_count ( frame, regex_text_to_find , count );

// returns the number of occurrences of a regular expression in a frame.

12. web_frame_text_exists ( frame, text_to_find [, text_before, text_after ] );

// returns a text value if it is found in a frame.

13.web_get_run_event_mode ( out_mode );

// returns the current run mode out_mode The run mode in use. If the mode is FALSE, the default parameter, the test runs by mouse operations. If TRUE, is specified, the test runs by events.

14. web_get_timeout ( out_timeout );

// returns the maximum time that WinRunner waits for response from the web. out_timeout The maximum interval in seconds

15.web_image_click ( image, x, y );

// clicks a hypergraphic link or an image. image The logical name of the image. x,y The x- and ycoordinates of the mouse pointer when clicked on a hypergraphic link or an image.

16. web_label_click ( label );

// clicks the specified label. label The name of the label. 17. web_link_click ( link );

// clicks a hypertext link. link The name of link.

18. web_link_valid ( name, valid );

// checks whether a URL name of a link is valid (not broken). name The logical name of a link. valid The status of the link may be valid (TRUE) or invalid (FALSE)

19. web_obj_click ( object, x, y );

object The logical name of an object. x,y The x- and y-coordinates of the mouse pointer when clicked on an object.

20. web_obj_get_child_item ( object, table_row, table_column, object_type, index, out_object );

// returns the description of the children in an object.

21. function returns the count of the children in an object.

web_obj_get_child_item_count ( object, table_row, table_column, object_type, object_count );

22. returns the value of an object property.

web_obj_get_info ( object, property_name, property_value );

23. returns a text string from an object.

web_obj_get_text ( object, table_row, table_column, out_text [, text_before, text_after, index] );

24. returns the number of occurrences of a regular expression in an object.

web_obj_get_text_count ( object, table_row, table_column, regex_text_to_find, count );

25. returns a text value if it is found in an object.


26. web_restore_event_default ( );

//resets all events to their default settings. 27. web_set_event ( class, event_name, event_type, event_status );

// sets the event status.

28. web_set_run_event_mode ( mode );

//sets the event run mode. 29 web_set_timeout ( timeout );

//.sets the maximum time WinRunner waits for a response from the web. 30. web_set_tooltip_color ( fg_color, bg_color );

// sets the colors of the WebTest ToolTip.

31. web_sync ( timeout );

//waits for the navigation of a frame to be completed. 32. web_url_valid ( URL, valid );

Load Runner - Introduction

Load Runner is divided up into 3 smaller applications:

The Virtual User Generator allows us to determine what actions we would like our Vusers, or virtual users, to perform within the application. We create scripts that generate a series of actions, such as logging on, navigating through the application, and exiting the program.

The Controller takes the scripts that we have made and runs them through a schedule that we set up. We tell the Controller how many Vusers to activate, when to activate them, and how to group the Vusers and keep track of them.

The Results and Analysis program gives us all the results of the load test in various forms. It allows us to see summaries of data, as well as the details of the load test for pinpointing problems or bottlenecks.

LoadRunner 7 Features & Benefits
New Tuning Module Add-In The LoadRunner Tuning Module allows customers to isolate and resolve system performance bottlenecks. Once the application has been stress tested using LoadRunner, the Tuning Module provides component test libraries and a knowledgebase that help users isolate and resolve performance bottlenecks.

WAN Emulation Support This powerful feature set enables LoadRunner to quickly point out the effect of the wide area network (WAN) on application reliability, performance, and response time. Provided through technology from Shunra

Software, this WAN emulation capability introduces testing for

bandwidth limits, latency, network errors, and more to LoadRunner.

Sun ONE Studio4 IDE Add-in Mercury Interactive and Sun Microsystems have worked together to integrate LoadRunner with the Sun ONE Studio4 add-in.

JBuilder for Java IDE Add-in LoadRunner now works with Borland's JBuilder integrated development environment (IDE) to create powerful support for J2EE applications. This add-in enables LoadRunner users who create J2EE applications and services with JBuilder to create virtual users based on JBuilder project.

source code within a

Native ICA Support for Citrix MetaFrame LoadRunner now supports Citrix's Independent Computing Architecture (ICA) for the testing of applications being deployed with Citrix MetaFrame. This support is the first native ICA load testing solution of its kind, jointly developed with Citrix.

Web Transaction Breakdown Monitor for Isolating Performance Problems Now you can more efficiently isolate performance problems within your architecture. LoadRunner's Web Transaction Breakdown Monitor splits end-to-end transaction response times for the client, network, and server and provides powerful drill-down capabilities.

Data Wizard for Data-Driven Testing LoadRunner's Data Wizard enables you to quickly create data-driven tests and eliminate manual data manipulation. It connects directly to back-end database servers and imports desired data into test scripts.

Goal-Oriented Testing with AutoLoad The new AutoLoad

technology allows you to pre-define your testing objectives beyond the

number of concurrent users to streamline the testing process.

Enterprise Java Bean Testing By testing EJB components with LoadRunner, you can identify and solve problems during the early stages of application development. As a result, you can optimize performance before clients have been developed and thereby save time and resources.

XML Support

With LoadRunner's XML support, you can quickly and easily view and manipulate XML data within the test scripts.

Hosted Virtual Users
Hosted Virtual Users

How it Work

LoadRunner Hosted Virtual Users complements in-house load testing tools and allows companies to load test their Web-based applications from outside the

firewall using Mercury Interactive's

infrastructure. Customers begin by using LoadRunner Hosted Virtual Users' simple Web interface to schedule tests and reserve machines on

Mercury Interactive's load farm. At the scheduled time,

they select the recorded scripts to be uploaded and start running the tests on the host machines*. These scripts will emulate the behavior of real users on the application and generate load on the

Through LoadRunner Hosted Virtual Users‘ Web interface, testers can view real-time performance metrics, such as hits per second, throughput, transaction response times and hardware resource usage (e.g., CPU and memory levels). They also can view performance metrics gathered by Mercury Interactive‘s

server monitors and correlate this with end-user performance data to diagnose

bottlenecks on the back end.

The interface to LoadRunner Hosted Virtual Users enables test teams to control the load test and view tests in progress, no matter their locations. When the test is complete, testers can analyze results online, as well as download data for further analysis.

*Customers who do not own LoadRunner can download the VUGen component for free to record their scripts. Likewise, the LoadRunner analysis pack can be downloaded for free.

LoadRunner Hosted Virtual Users gives testers complete control of the testing process while providing critical real-time performance information, as well as views of the individual machines generating the load.

Features and Benefits

Provides pre- and post-deployment testing.

At any time in the application lifecycle, organizations can use LoadRunner Hosted Virtual Users to verify performance and fine-tune systems for greater efficiency, scalability and availability. The application under test only needs to be accessible via the Web.

Complements in-house solutions to provide comprehensive load testing.

By combining LoadRunner Hosted Virtual Users with Mercury Interactive's LoadRunner or another in-house load testing tool, operations groups can thoroughly load test their and Internet infrastructures from inside and outside the firewall.

Web applications

Gives customers complete control over all load testing.

Testing groups create the scripts, run the tests and perform their own analyses. They can perform testing at their convenience and easily access all performance data to quickly diagnose performance problems.

Provides access to Mercury Interactive's extensive load testing infrastructure.

With LoadRunner Hosted Virtual Users, organizations do not need to invest in additional hardware,

software or bandwidth to increase their testing coverage. Mercury Interactive‘s load testing
infrastructure is available 24x7 and consists of load farms located worldwide. As a result, organizations can generate real-user loads over the Internet to stress their Web-based applications at any time, from anywhere.

How the Monitors Work To minimize the impact of the monitoring on the system under test, LoadRunner enables IT groups to extract data without having to install intrusive capture agents on the monitored servers. As a result, LoadRunner can be used to monitor the performance of the servers regardless of the hardware and operating system on which they run. Setup and installation of the monitors therefore is trivial. Since all the monitoring information is sampled at a low frequency (typically 1 to 5 seconds) there is only a negligible effect on the servers.

Supported Monitors

Astra LoadTest and LoadRunner support monitors for the following components:

Client-side Monitors

End-to-end transaction monitors - Provide end-user response times, hits per second, transactions per second

Hits per Second and Throughput

Hits per Second


Throughput The Throughput graph shows the amount of throughput on the Web server (y-axis) during each second of the scenario run (x-axis). Throughput is measured in kilobytes and represents the amount of data that the Vusers received from the server at any given second. You can compare this graph to the Transaction Response Time graph to see how the throughput affects transaction performance.
HTTP Responses

HTTP Responses The HTTP Responses per Second graph shows the number of HTTP status codes,

which indicate the status of HTTP requests, for example, the request was successful,the page was not found returned from the Web server during each second of the scenario run (x-axis), grouped by status code. Load Testing Monitors

Pages Downloaded per Second


Pages Downloaded per Second The Pages Downloaded per Second graph shows the number of

Web pages downloaded from the server during each second of the scenario run. This


User-defined Data Point

User Defined Data Points graph allows you to add your own measurements by defining a data point function in your Vuser script. Data point information is gathered each time the script executes the function or step. The User-Defined Data Point graph shows the average value of the data points during the scenario run. The x-axis represents the number of seconds elapsed since the start time of the run. The y-axis displays the average values of the recorded data point statements. Transaction Monitors


Transaction Response Time The Transaction Response time graph shows the response time of transactions in seconds (y-axis) as a function of the elapsed time in the scenario (xaxis). ·


Transaction per Second (Passed) The Transaction per Second (Passed) graph shows the number of successful transactions performed per second (y-axis) as a function of the elapsed time in the scenario (x-axis). ·


Transaction per Second (Failed) The Transaction per Second (Failed) graph shows the number of failed transactions per second (y- axis) as a function of the elapsed time in the scenario (x- axis).

Virtual User Status

The monitor's Runtime graph provides information about the status of the Vusers running in the current scenario on all host machines. The graph shows the number of running Vusers, while the information in the legend indicates the number of Vusers in each state. The Status field of each Vuser displays the current status of the Vuser. The following table describes each Vuser status.


   

Running The total number of Vusers currently running on all load generators. · Ready The number of Vusers that completed the initialization section of the script and are ready to run. · Finished The number of Vusers that have finished running. This includes both Vusers that passed and failed · Error The number of Vusers whose execution generated an error.

Web Transaction Breakdown Graphs



DNS Resolution Displays the amount of time needed to resolve the DNS name to an IP address, using the closest

DNS server. The DNS Lookup measurement is a good

indicator of problems in DNS resolution, or problems with the DNS server. ·


Connection Time Displays the amount of time needed to establish an initial connection with the Web server hosting the specified URL. The connection measurement is a good indicator of problems along the network. It also indicates whether the server is responsive to requests. ·


Time To First Buffer Displays the amount of time that passes from the initial HTTP request (usually GET) until the first buffer is successfully received back from the Web server. The first buffer measurement is a good indicator of Web server delay as well as network latency. ·


Server and Network time The Time to First Buffer Breakdown graph also displays each Web page component's relative server and network time (in seconds) for the period of time until the first buffer is successfully received back from the

Web server. If the download

time for a component is high, you can use this graph to determine whether the problem is server- or network- related. ·


Receive Time Displays the amount of time that passes until the last byte arrives from the server and the downloading is complete. The Receive measurement is a good indicator of network quality (look at the time/size ratio to calculate receive rate). ·

  

Client Time Displays the average amount of time that passes while a request is delayed on the client machine due to browser think time or other client-related delays. · Error Time Displays the average amount of time that passes from the moment an HTTP request is sent until the moment an error message (HTTP errors only) is returned · SSL Handshaking Time Displays the amount of time taken to establish an SSL connection (includes the client hello, server hello, client public key transfer, server certificate transfer, and other stages). The SSL Handshaking measurement is only applicable for HTTPS communications ·


FTP Authentication Displays the time taken to authenticate the client. With FTP, a server must authenticate a client before it starts processing the client's commands. The FTP Authentication measurement is only applicable for FTP protocol communications.

Server Monitors

NT/UNIX/Linux monitors - Provide hardware, network and operating system performance metrics, such as CPU, memory and network throughput. The following list describes the recommended objects to be monitored during a load test:

ASP Server Cache HTTP Content Index Internet Information Service Global Logical Disk

Memory Physical Disk Processor Server
ASP Server


Debugging Requests - Number of debugging document requests. Errors during Script Runtime - Number of requests failed due to runtime errors. Errors from ASP Preprocessor - Number of requests failed due to preprocessor errors. Errors from Script Compilers - Number of requests failed due to script compilation errors. Errors/Sec - The number of errors per second. Memory Allocated - The total amount of memory, in bytes, currently allocated by Active Server Pages Request Bytes In Total - The total size, in bytes, of all requests. Request Bytes Out Total - The total size, in bytes, of responses sent to clients. This does not include standard HTTP response headers. Request Execution Time - The number of milliseconds required to execute the most recent request. Request Wait Time - The number of milliseconds the most recent request was waiting in the queue. Requests Disconnected - The number of requests disconnected due to communication failure. Requests Executing - The number of requests currently executing. Requests Failed Total - The total number of requests failed due to errors, authorization failure and rejections. Requests Not Authorized - The number of requests failed due to insufficient access rights. Requests Succeeded - The number of requests that were executed successfully. Requests Timed Out - The number of requests that timed out.


Async Copy Reads/Sec - The frequency of reads from cache pages that involve a memory copy of the data from the cache to the application's buffer. The application will regain control immediately, even if the disk must be accessed to retrieve the page.


Async Data Maps/Sec - The frequency that an application uses a file system, such as NTFS or HPFS, to map a page of a file into the cache to read because it does not wish to wait for the cache to retrieve the page if it is not in main memory.


Async Fast Reads/Sec - The frequency of reads from cache pages that bypass the installed file system and retrieve the data directly from the cache. Normally, file I/O requests will invoke the appropriate file system to retrieve data from a file. This path, however, permits direct retrieval of cache data without file system involvement, as long as the data is in the cache. Even if the data is not in the cache, one invocation of the file system is avoided. If the data is not in the cache, the request (application program call) will not wait until the data has been retrieved from disk, but will get control immediately.


Fast Reads/Sec - The frequency of reads from cache pages that bypass the installed file system and retrieve the data directly from the cache. Normally, file I/O requests invoke the appropriate file system to retrieve data from a file. This path, however, permits direct retrieval of cache data without file system involvement if the data is in the cache. Even if the data is not in the cache, one invocation of the file system is avoided.


Lazy Write Flushes/Sec - The frequency with which the cache's Lazy Write thread has written to disk. Lazy Writing is the process of updating the disk after the page has been changed in memory. In this way, the application making the change to the file does not have to wait for the disk write to be completed before proceeding. More than one page can be transferred on each write operation.

HTTP Content Index

      

%Cached Hits - Percentage of queries found in the query cache. %Cache Misses - Percentage of queries not found in the query cache. Active Queries - Current number of running queries. Cache Items - Number of completed queries in cache.Current Requests Queued - Current number of query requests queued. Queries per Minute - Number of queries per minute. Total Queries - Total number of queries run since service start. Total Requests Rejected Total number of query requests rejected.

Internet Information Service(IIS) Global


Cache Hits % - The ratio of cache hits to all cache requests.

     

Cache Misses - The total number of times a file open, directory listing or service-specific object request was not found in the cache. Cached Files Handles - The number of open file handles cached by all of the Internet Information Services. Current Blocked Async I/O Requests - Current requests temporarily blocked due to bandwidth throttling settings. Directory Listings - The number of directory listings cached by all of the Internet Information Services. Measured Async I/O Bandwith Usage - Measured bandwidth of asynchronous I/O averaged over a minute. Objects - The number of objects cached by all of the Internet Information Services. The objects include file handle tracking objects, directory listing objects and service specific objects.

 

Total Allowed Async I/O Requests - Total requests allowed by bandwidth throttling settings (counted since service startup). Total Blocked Async I/O Requests - Total requests temporarily blocked due to bandwidth throttling settings (counted since service startup).

Logical Disk

     

% Disk Read Time - The percentage of elapsed time that the selected disk drive was busy servicing read requests. % Disk Time - The percentage of elapsed time that the selected disk drive was busy servicing read or write requests. % Disk Write Time - The percentage of elapsed time that the selected disk drive was busy servicing write requests. % Free Space - The ratio of the free space available on the logical disk unit to the total usable space provided by the selected logical disk drive Avg. Disk Bytes/Read - The average number of bytes transferred from the disk during read operations. Avg. Disk Bytes/Transfer - The average number of bytes transferred to or from the disk during write or read operations.



% Committed Bytes in Use - The ratio of the Committed Bytes to the Commit Limit. This represents the amount of available virtual memory in use. Note that the Commit Limit may change if the

paging file is extended. This is an instantaneous value, not an average.


Available Bytes - Displays the size of the virtual memory currently on the Zeroed, Free and Standby lists. Zeroed and Free memory is ready for use, with Zeroed memory cleared to zeros. Standby memory is memory removed from a process's Working Set but still available. Notice that this is an instantaneous count, not an average over the time interval.


Cache Bytes - Measures the number of bytes currently in use by the system cache. The system cache is used to buffer data retrieved from disk or LAN. In addition, the system cache uses memory not in use by active processes in the computer.


Cache Bytes Peak - Measures the maximum number of bytes used by the

system cache.




Cache Faults/Sec - Cache faults occur whenever the

cache manager does not find a

file's page in the immediate cache and must ask the memory manager to locate the page elsewhere in memory or on the disk, so that it can be loaded into the immediate cache.

Physical Disk

      

% Disk Read Time - The percentage of elapsed time that the selected disk drive is busy servicing read requests. % Disk Time - The percentage of elapsed time that the selected disk drive is busy servicing read or write requests. % Disk Write Time - The percentage of elapsed time that the selected disk drive is busy servicing write requests. Avg. Disk Bytes/Read - The average number of bytes transferred from the disk during read operations. Avg. Disk Bytes/Transfer - The average number of bytes transferred to or from the disk during write or read operations. Avg. Disk Bytes/Write - The average number of bytes transferred to the disk during write operations. Avg. Disk Queue Length - The average number of both read and write requests that were queued for the selected disk during the sample interval.



% DPC Time - The percentage of elapsed time that the Processor spent in Deferred Procedure Calls (DPC). When a hardware device interrupts the Processor, the Interrupt Handler may elect to execute the majority of its work in a DPC. DPCs run at lower priority than Interrupts. This counter can help determine the source of excessive time being spent in Privileged Mode.


% Interrupt Time - The percentage of elapsed time that the Processor spent handling hardware Interrupts. When a hardware device interrupts the Processor, the Interrupt Handler will execute to handle the condition, usually by signaling I/O completion and possibly issuing another pending I/O request. Some of this work may be done in a DPC (see % DPC Time.)


% Privileged Time - The percentage of processor time spent in Privileged Mode in non-idle threads. The

Windows NT service layer, the Executive routines, and the Windows NT

Kernel execute in Privileged Mode. Device drivers for most devices other than graphics adapters and printers also execute in Privileged Mode. Unlike some early operating systems,


% Processor Time - Processor Time is expressed as a percentage of the elapsed time that a processor is busy executing a non-idle thread. It can be viewed as the fraction of the time spent doing useful work. Each processor is assigned an idle thread in the idle process that consumes those unproductive processor cycles not used by any other threads.


% User Time - The percentage of processor time spent in User Mode in non-idle threads. All application code and subsystem code execute in User Mode. The graphics engine, graphics device drivers, printer device drivers and the

window manager also execute in User

Mode. Code executing in User Mode cannot damage the integrity of the Windows NT Executive, Kernel, and device drivers. Unlike some early operating systems, Windows NT uses process boundaries for subsystem protection in addition to the traditional protection of User and Privileged modes.



Blocking Requests Rejected - The number of times the server has rejected blocking Server Message Blocks (SMBs) due to insufficient count of free work items. May indicate whether the maxworkitem or minfreeworkitems server parameters need tuning.

 

Bytes Received/Sec - The number of bytes the server has received from the network. This value indicates how busy the server is. Bytes Total/Sec - The number of bytes the server has sent to and received from the network. This value provides an overall indication of how busy the server is.

  

Bytes Transmitted/Sec - The number of bytes the server has sent on the network. This value indicates how busy the server is. Context Blocks Queued/Sec - - The rate that work context blocks had to be placed on the server's FSP queue to await server action. Errors Access Permissions - The number of times file opens on behalf of clients have failed with STATUS_ACCESS_DENIED. Can indicate whether somebody is randomly attempting to access files in hopes of accessing data that was not properly protected.


1.Creating Script Using Virtual User Generator


Start-> program Files->Load Runner->Virtual User Generator Choose File->New Select type and Click Ok Button Start recording

Dialog Box appears

Besides Program to Record, Click Browser Button and Browse for the Application Choose the Working Dir Let start recording into sections Vuser_Init and click Ok button After the application appears, change sections to Actions. Do some actions on the application Change sections to Vuser_End and close the application Click on stop Recording Icon in the tool bar of Vuser Generator Insert the Start_Transaction and End_Transactions. Insert the Rendezvous Point Choose :Vuser->Run, Verify the status of script at the bottom in Execution Log. Choose:File->Save.(Remember the path of the script).

2.Running the script in the Controller with Wizard

    

Start-> program Files->Load Runner->Controller. Choose: wizard option and click OK. Click Next in the welcome Screen In the host list , click add button and mention the machine name Click Next Button Select the related script you are generated in Vuser Generator(GUI Vuser Script,DB script,RTE script)

    

Select Simulation group list, cilck edit button and change the group name ,No of Vuser. Click Next Button Select Finish Button. Choose: Group->Init or Group->Run or Scenario->Start. Finally Load runner Analysis graph report appears.

3.Running the script in the Controller with out Wizard

         

Start-> program Files->Load Runner->Controller. Choose File->New, Four Windows appears Select Vusers window. Select Group->Add Group Vuser Information Box appears Select Group Name, Vuser Quantity, Host Name. Select Script and select Add button, select path of the script. Click OK button. Choose: Group->Init or Group->Run or Scenario->Start. Select Results->Analyse Results.

4.Creating GUI Vuser Script Using Winrunner(GUI Vuser)

         

Start-> Program Files->Winrunner->Winrunner. Chose: File->New Start recording through Create->Record ContextSensitive Mode Invoke the application Do some actions on the application. Select Stop Recording Create->Stop Record Declare the Transactions and Rendezvous Point at the top of script Declare_transaction(); Declare_rendezvous(); Identify where the transaction points to be inserted. Start_transaction(); End_transaction(); Insert the Rendezvous Point Rendezvous(); Save the script and remember the path of the script.

5.Running the script in the Controller with Wizard



        

Choose: wizard option and click OK. Click Next in the welcome Screen In the host list , click add button and mention the machine name Click Next Button In GUI Vuser Script, click add button and mention the name of Script Click Next button. Select Simulation group list, cilck edit button and change the group name ,No of Vuser. Click Next Button Select Finish Button. Select Vuser window, select tools->option Select winrunner tab,set the path of Winrunner

Path:Program files->winrunner->dat->wrun.ini


Select Host window, select Host->details Select Vuser Limits tab, select GUI-winrunner check box. Select winrunner tab , set path

Path:Program files->winrunner->dat->wrun.ini

 


Test Partner

Batch Testing:-we can create batches/suites by using the keyword ―Run‖. In this tool, calling test is known as Driver script and called test is known as Test script. Syntax: Run ―Test script‖ (or) Run(―Test script‖)

Including an Asset:-To include a VBA or other non-Test Partner asset in a script, add a declaration using the following syntax:

$TPInclude "asset Name"

where asset Name is the name of the asset that you are including. Asset names are unique across all asset types, so you don‘t need to specify what type of asset you are including. Object Mapping:-It can be used to provide simplified, easily understood aliases for the names of Windows objects. Once a window is registered in the Object Map, all references to it in scripts, check definitions, and event definitions are made by its alias, rather than by its actual attach name. The attach name is an important concept when testing applications using Test Partner. Check Points:-A check is a definition of the expected state of some aspect of the target

system at a particular

point. In Test Partner, checks are saved and managed as assets. This means you always have the option to reuse a check in more than one script. The following are various checks available in Test Partner. i) Bitmap Check:– Bitmap checks allow you to verify the appearance of a bitmap image. When you create the check, you capture the image within a rectangular area of the screen. When the check is verified, the same area is captured and compared to the defined image. If the two images match according to the criteria you defined, the check passes. If not, the check fails. These checks are used to check the appearance of toolbars, the ii) Clock Check:– Clock checks measure the time the system takes to perform a process. Clock checks help you determine how the system performs under varying CPU or network loads. When you create the check, you specify an acceptable response time. When the check is verified, the system‘s actual response time is recorded and compared to the specified time. It can

desktop, and other windows that contain non-textual information.

 

carry out performance checks on the target application. determine whether, under controlled conditions, the target application performs tasks within pre-defined response times.


record the effects of varying CPU and network loads on the system.

iii) Content Check:-– Content checks test the contents of tables and list controls in a window or

web page. A content Windows NT Version

check enables you to verify the contents of controls that it supports. Currently, tables and list controls in a Windows-based or Web-based application are supported. The 4 desktops are also list controls. The content check for tables enables you to optionally check the number of rows and columns in the table and the case of the text in each table cell. The content check for list controls enables you to optionally check the number of items, positions of the items, which item(s) are selected, the text of each list item, and the case of the text. iv) Field Check:-– Like text checks, Field checks enable you to verify that required text is present in the target application, but they enable you to verify that text as data, such as numbers or dates. For example, you can see if a value falls between a lower and upper limit, or if a particular area of the screen contains today‘s date. You can create field checks that verify the following data:

    

ASCII values Numeric values Date values (fixed and aged) Time values Patterns

v) Property Check:– Property checks verify the properties of the controls in a dialog or web page. You can check the size and position of each control, their legends and IDs, and whether they are active, disabled, selected, or cleared. You can check a single control, or you can check several controls within an application window. vi) Text Check:– Text checks provide an exact comparison of the text in a window or individual area to defined text. If you check a whole screen, areas that contain legitimately variable data, such as dates and

login IDs, can be ignored. Unlike bitmap checks, which simply compare the appearance of an area of the screen with an expected appearance, text checks actually read the displayed data as strings. This enables more sophisticated checking to be performed. Events:-Events are unscheduled occurrences or conditions to which you want the target application to respond in a specified manner. Test Partner supports two categories of events: Wait and Whenever. A Wait event tells Test Partner to wait for a specified occurrence before proceeding. Wait events are useful in situations where you cannot anticipate the amount of time a response will take. An example of a Wait event would be waiting for a

system login prompt. When your script is running

against a network-based application that requires the user to log in, the amount of time it takes for the login prompt to display may vary. To account for this variance, you can insert a Wait event that instructs your script to wait for the login prompt before proceeding to type a username and password. A Whenever event tells Test Partner to watch for a specific occurrence and, if it occurs, perform a special set of steps. Whenever events are useful for trapping unexpected error conditions during a test run. For example, you can include events in your scripts to recognize when the connection to the server has been interrupted by a communications error or a network system message has been received, so that the script can report or work around the problem. In a script, Test Partner automatically inserts the Script_Whenever function by default to handle the whenever event. If a whenever event is inserted into a module, shared module, or class module, you must customize the whenever event handler code. Test Partner supports the following types of Events: i) Bitmap Event:-- A bitmap event detects the presence, absence, or state of a graphic in a window.

ii) Date/Time Event:-Date/Time events enable you to define a date or time condition. Test Partner recognizes the event by monitoring the internal clock of the running. iii) Key Event:-Watch for the entry of a particular keystroke combination by the user. You can use key events to:

computer on which it and the target application are

   

Build your own ―hotkeys‖ to make Test Partner perform an action whenever the hotkey is used. Interrupt a script to take manual control of the target application. Pause a script until a particular key is used. Prevent certain keys from being entered.

iv) Menu Events:-- Watch for when a particular menu item is selected by the user.

v) Mouse Events:-- Watch for when one of the mouse buttons is clicked or released in a certain window.

vi) Screen Events:-A screen event detects the presence or absence of text in a window. The most common use of screen events is to synchronize with host-based applications and to detect vii) Window Events:-A window event detects an action on a window; for example, its creation, destruction, movement, or its existence. Data Driven Test:-The Test Partner Active Data Test Creation wizard provides a non-programmatic way to create datadriven tests. Such tests are useful for testing form-based applications. Using the wizard, you can record a test, choose the fields you want to include in your data file, then populate the data file itself using a data table. To create a data test

error messages.

1. From the Script menu, choose Create Data Test. The Data Test wizard appears. 2. Read the instructions and click Next. 3. Follow the instructions in the Data Layout Assistant for the three steps necessary to define the scope of the test. 4. Enter a name for the Data Table. 5. Exclude any fields you do not want to include in the test by unchecking the Use checkbox and click Next.

6. Read the instructions and click Finish.

TestPartner shows the Data Table, which includes a column for each field you defined and one row of data. The data table also includes a column labelled Results Script, to specify a script to be run after the data driven test. To modify the data table.

1. If you have just recorded a new data test and the data table is open, proceed to step 3.

2. To modify an existing data test, select Modify Data Table on the Script menu and choose the data table you want to change. 3. To add rows and populate the table with test data, right-click on the empty row labelled END ( or any other row) and select Insert on the context menu. Test Partner inserts a new row above the selected row. 4. Enter test data into the cells of the new row.

5. Alternatively, right-click the empty row and choose Import. You can import data from a tabdelimited text (.TXT) or comma-delimited (.CSV) file to populate the cells of the data table. If you select the END row in the table, TestPartner will add multiple rows if needed to accommodate the data. 6. To delete rows from the table, right-click the row and choose Delete. To delete multiple rows, press the Ctrl or Shift (for contiguous rows) key while selecting rows to delete. 7. To launch another script from the data test, insert the name of the script in the Results Script field in the table at any point(s) at which you want to run the script. 8. When you have finished editing the data table, click Save and Close to exit.

Introduction Purpose

The Client/Server approach to providing business solutions, while at the same time delivering more functional and powerful systems, is also introducing new risks. As well as the functional concerns associated with any computer systems, client/server solutions have a much greater associated risk in terms of the performance of the overall system. Load Runner is a software tool that facilitates in measuring the performance of systems prior to employment. This allows performance issues to be addressed before rollout. This paper is a guide to implementing a Load Runner based load-testing solution. It also provides some theoretical background into the area, defining several key concepts. It is recommended that this paper be used in conjunction with the Load Runner Implementation Guide available from Mercury Interactive. This paper should not be seen to replace Mercury‘s guide, but as a supplement to it.

Load/Stress Testing Load and stress testing are two different, but related forms of testing. Stress testing involves testing a system under extreme cases, which will rarely occur under normal conditions. Such testing can involve such scenarios as running many users concurrently for short times, short transactions repeated quickly, and extraordinarily long transactions. Load testing places a more realistic load on the system. Scenarios are modeled on the demand that the system is expected to face under real conditions. Load testing starts by placing a low demand on the system, and gradually increasing the load. This process allows measuring the performance of the system under different loads, as well as determining the maximum load that under which the system can respond acceptably. Load and stress testing are clearly two interrelated processes, generally performed concurrently. It is important to note what load/stress testing isn‘t. They are not functional tests. In order to perform performance testing, it must be assumed that a reasonably stable and functional system is in use, otherwise results are meaningless. With that said, it is still common to uncover functional problems in a system while performing load testing. However, uncovering excessive functional problems can be a sign of inadequate functional testing. Black Box Testing Strictly speaking, black box testing involves testing a system as a whole, not as a system of connected parts. Realistically, however, black box testing still provides some differentiation between clients, the server, and the network connecting them. This means that when performing black box testing, it is possible to determine whether time is being spent in the client, the network, or the server, and hence determine where bottlenecks lie. The limitation of this approach is that knowing, for example, that the server is the bottleneck of a system is not necessarily sufficient. Is the bottleneck caused by a lack of memory, a lack of CPU performance, poor algorithms, or one of many other causes? To answer these questions, diagnostics need to be performed on the server while generating a load. White Box Testing

White box testing treats the system as a collection of many parts. During white box testing, many diagnostics will be run on the server(s), the network, and even on clients. This allows causes of bottlenecks to be much more ably identified, and therefore addressed. White box testing requires much greater technical knowledge of the system than does black box testing. To perform white box testing, knowledge is needed as to the components that make up the system, how they interact, the algorithms involved with the components, and the configuration of the components. It must also be known how to perform diagnostics against these components, and even what diagnostics are needed and appropriate. This knowledge can generally only be provided by someone intimate with the system. A problem with white box testing is the effect that running diagnostics has on system performance. Ideally, running diagnostics should have no impact on performance. However, with the exception of the network, this is rarely possible, running diagnostics consumes some system resources. When considering the appropriateness of various diagnostics, one must consider the benefit vs. the impact on performance. Black Box vs White Box Testing Black box testing is generally much easier to perform than white box testing, requiring systemspecific expertise. However, while black box testing may identify that problems do exist, white box shows with much more clarity where they exist. Black box simply shows whether or not performance is acceptable. This is still useful, as it can show which transactions are taking the most time, as well as highlight problems with certain numbers of transactions. A common use of black box testing is to identify knees in the response of a system against the number of concurrent transactions. Such a knee indicates a limitation in the system that once passed, provides exponentially decreasing system performance, a bad thing if reached. End to End Testing End to end testing is testing from the point of view of the client. It involves manually or automatically using the actual client software to perform business processes, and measures the response time of the application. End to end testing measures system performance from the end users perspective Load Runner Overview Load Runner is a tool, or more accurately a collection of tools, which facilitates in performing load testing on a system. Load Runner is used to create a load or stress on a system, and can be used to measure the responsiveness of the system to that load. The key concept behind Load Runner is the use of virtual users, or vusers. Virtual Users To perform a load test, a system needs many users to access the system concurrently, often one hundred or more. Performing such a test with people is a logistical nightmare. When considering that the test cycle is generally iterative, this problem is compounded. Load Runner overcomes this problem through the use of vusers. A vuser performs the actions that a real user would perform, except that a vuser is controlled by Load Runner, and the actions are performed automatically.

Vusers are created to perform specific transactions in the system. Once created a vuser can be replicated as many times as required, thus producing an effective load as large as is required. With the exception of GUI vusers, virtual users don‘t use the client to perform their actions; they take the place of the client. This allows in many vusers to be run on one workstation, as the overhead associated with the client is not required. This does mean, however, that non-GUI vusers do not test the performance associated with the client side of a system, only the network and server associated performance. Vusers can be of four main types. DB Virtual Users A DB or database vuser performs operations against a database server. Load Runner supports all the most popular interfaces to database servers including but not limited to ODBC, SQL, and various Oracle interfaces. It is also possible to create Load Runner interfaces that allow recording against unsupported databases. Recording a DB vuser is as simple as performing a business process while having Launched the Load Runner Virtual User Generator. This creates a script which is used by Load Runner to perform the recorded business process. Once recorded, a vuser script is then parameterized. This process involves replacing the hard coded data values in the script with parameters, which refer to tables external to the script. Values can then be placed in these tables allowing vusers to perform a single business process, but with different data each time. Web Virtual Users A web virtual user is one that performs actions against a web based system. A web vuser is essentially identical to a DB vuser, except that where the DB vuser is recorded against a client application that communicates directly with a database of some description, a web vuser is recorded against a web based application. RTE Virtual Users Remote Terminal Emulation virtual users are used to emulate users that would be operating on a lightweight, character based terminal. SAP Virtual Users SAP R/3 virtual users are created somewhat differently than the others, and require the SAP R/3 Load Runner add-on. Essentially, a SAP R/3 virtual user is one that performs actions in a SAP R/3 environment. GUI Virtual Users Unlike the previous types of vusers, GUI virtual users actually perform actions using the client. This means that fewer GUI vusers can be run per workstation than the other types of vusers due to the overhead of the client, and is in fact limited to one GUI vuser per PC workstation. GUI vusers are used not so much to generate load on a system, but to test end-to-end performance associated with various business processes. Although Load Runner can control GUI vusers, it is unable to create them. For this purpose, Win Runner is required. Controller

The Load Runner controller is used to create and run scenarios. Once vuser scripts have been created, they can be added into the controller in varying quantities to produce the demand on the system that is required. When running a scenario, the controller coordinates all the vusers, graphical or otherwise. It starts vusers as required, and can force them to rendezvous with each other, thus producing a maximum peak demand. Agent When running a scenario, every workstation that is to be used to generate load must have the agent software installed. This software allows a remote (or local, but this is not advised) controller to send vusers to the workstation to be run. The agent communicates directly with the controller, reporting performance data of the vuser, as well as ensuring that the vuser runs as specified in the scenario.

Virtual User Generator The virtual user generator (VU Gen) is used to generate scripts for all non-GUI vusers. The VU Gen records the behavior of the client software, and automatically generates a script that can be used by the controller to run vusers. Once a script is created, the VU Gen can then be used to parameterize the script. This involves replacing hard-coded values in the script with references, which refer to external tables. The VU Gen provides a semi-automated procedure for doing this. Load Analyzer The load analyzer utility presents the results of running a scenario in a variety of graphical and tabular forms. These graphs and tables can be used to determine the acceptability of a system, and to determine areas of concern. The data presented can be exported in a variety of popular formats, allowing it to be used outside Load Runner.

Preparation Before load testing can begin, a certain amount of preparation is needed; however the exact time required varies considerably from system to system. A guide to the issues that should be addressed follows. Planning This phase identifies the goals of testing, how the goals may be reached, and a timeline for doing so. A typical goal is to conduct load tests to determine where bottlenecks lie in a system, and to suggest ways the system could be improved to overcome these. Note here that this goal implies some degree of white box testing, as it asks us to determine where the bottlenecks lie inside the system.

In this phase, resources are allocated for the load test. This includes people, hardware and time. Project managers in conjunction with testing consultants generally perform this phase. This stage can generally be performed quickly, in no more than a few days. Analysis/Design This phase involves identifying transactions performed in a system, rates of performing transactions, and scenarios that model expected use of the system. It involves performing a risk assessment of all transactions of a system to determine which the most critical, in terms are of: Frequency of use Demand placed on system Importance to the system Any other criteria deemed appropriate A transaction is a series of steps a real user would perform to achieve a business process in the system. Typical transactions include such things as performing a customer query, entering an order, and invoice printing. The exact transactions and complexity thereof vary considerably between systems. It is preferable to limit the complexity of transactions, especially compound transactions, while ensuring that meaning isn‘t lost. This is for practical reasons; complex transactions are more difficult to model, maintain, and reduce flexibility when creating scenarios. It is important to determine exactly what transactions should be used in load testing; creating them all as vuser scripts is time consuming and inefficient. Experience shows that modeling 80% of a user population provides accurate load tests, and this is typically achieved by about 20% of transactions. This typically amounts to 5-10 transactions per application module, although this figure can vary depending on the modularity of a system. The previously mentioned risk assessment is used to determine which transactions should be used in the load test. When determining the transactions to implement, the tradeoff of accuracy of the load tests against the time involved in getting results needs addressing. The risk assessment is best performed by both project managers and experienced testing consultants, and generally takes two or more days. Once identified, the transactions to be performed during load testing must be detailed in a stepby-step fashion. These can later be passed on to test engineers who can then work unassisted to create Load Runner vusers. When detailing the transactions, it is advised to extract the data that can change between runs of the transaction from the rest of the script. This makes it clear to the test engineer exactly what is to be parameterized. Instead of detailing transactions in the above fashion, it is also possible to make available to test engineers people knowledgeable with the system that can show them how to perform transactions as required. This approach requires less planning, and upfront work, however is more likely to lead to transactions which do not accurately reflect the processes of real uses. With transactions identified, it is also necessary to design the scenarios to be run. The scenarios should be designed with the goals of the load testing in mind, and place demand on the server and network as required meeting the goals of the test. Data Dependencies and Consumption Transactions can be divided into two categories; those that do not have any need of information already in the system, and those that do. The first type is easily implemented, but is in a minority. These transactions can simply be recorded, parameterized as appropriate, and run as many times as required. The second type has a dependency on other data in the system. This dependency can be quite weak, such as simply needing an unused customer ID number, or as strict as requiring a specific customer ID number. There are also transactions which consume data. Consider a delete

customer transaction; it is dependent upon a pre-existing customer ID, but once the transaction is performed with that ID, that ID becomes unavailable, it has been consumed. It is important that in this analysis phase, that data dependencies and data consumption are identified. This allows the system to be pre-populated with sufficient data that scenarios can be run several times before requiring a refresh. Testing Environment The environment used for testing can vary considerably depending on the goals of the load test. If the load test aims to simply determine whether the performance of a server is satisfactory, then the test environment can consist of the server and a high-powered workstation connected by a high-speed network. If, however, the impact of the network is a larger concern, then it is necessary to have a test environment that runs over a network identical to the one used when the system goes live. This is especially a concern when testing a system that will be run over a WAN. Isolated Environment Generally, it is preferable to have a testing environment that is isolated from other users. This allows precise measurement of the impact of different loads on the system, with no glitches caused by external network or server access. However, in systems that is to be run over the same network as other systems, which will have users accessing the server for reasons outside the system under test, this is not always desirable. In these cases, it may be preferable to perform load testing with the test environment connected to other systems, as it would be when live. The danger of this approach is that spikes can enter into performance measures, and that the load test can render other activities unusable, due to network and server congestion. An isolated environment is recommended for testing unless there are very good reasons to the contrary. Realistically sized database In many cases, the responsiveness of a system is related to the amount of data in the system. Therefore, it is meaningless to perform load tests on a system unless it contains a realistic amount of information. In this phase, it is important to identify the expected size of the system when it goes live, so that it can be later populated to reach this size. Identify Load Runner Workstations Depending on the exact goals of the load testing, a number of workstations to run vusers must be identified. If the load test aims to do no more than place a heavy load on the server, then one high powered workstation is all that is required. If the effects of various network connections are to be tested, then workstations will be required at the end of these connections, or least at the end of identically simulated connections. Load Runner Installation At this point, Load Runner will need to be installed on at least one machine for development purposes. This install will include all components of Load Runner. Workstations that have been identified as those to run vusers can be installed with the Load Runner agent at this stage, but this is not necessary until just before execution of the scenarios.

Development Development is one of the more time consuming aspects to load testing. Development can easily take one month or more, but this depends entirely on the number and complexity of the transactions to be modeled, as well as the number of people working in development. A proficient test engineer is generally able to produce about 2 scripts per day, with less capable and experienced people capable of about half this. Load Runner Scripts Load Runner vuser scripts form the backbone of a load test. Every transaction must be recorded, parameterized, and tested prior to executing load scenarios. Recording Vuser scripts Recording vuser scripts is normally as simple as performing actions in the client application while recording with the virtual user generator. The V U Gen creates a script file containing all the data access that occurs between the client and the server. In some cases, this is not sufficient, as the client application may communicate with an application such as Excel, which then communicates with the server. The VU Gen does not record this secondary communication. If this is the case, it is possible that an experienced test engineer can first record against the client, and then against the secondary application, and combine the two scripts. This is a complicated procedure, and should not be attempted by inexperienced test engineers. Parametersing Vuser scripts Once a vuser script is recorded, it must then be parameterized. This involves extracting the hardcoded values from within the script, and placing them in an external data table. This allows a single vuser to be run many times, and to use different data on each run. This provides a more realistic demand on the system, as it prevents the server from caching the results of many transactions. Scripts can be parameterized in a variety of ways. Some parameters will be defined to act as global variables, specifying such things as the database name, while others will take on a new value on every iteration of the vuser. Load Runner is extremely flexible in this respect, and can model most requirements that may be placed on the load test. Testing Scripts Test scripts, like any form of program, need to be tested. Initial testing involves running a test in isolation, ensuring that it performs the transaction as intended, that no unforeseen data is consumed, and that all necessary data is available for it. Once this is achieved, multiple instances of a test are run concurrently, ensuring that they don‘t conflict with each other, and that their parameters are correctly set so as not to cause deadlocks. It is also preferable to run different scripts concurrently, ensuring that one script cannot cause another to deadlock. If the analysis phase was performed with due care, this will not be a problem. Win Runner Scripts Win Runner scripts are not used so much to generate load, but to measure the end to end performance of the system when it is under load. As a result of this, it is common for only a subset of Load Runner implemented transactions to be implemented as Win Runner scripts. This

is a practical consideration; scripting all transactions as both GUI and non-GUI vusers is time consuming. However, it is important that sufficient transactions are scripted using Win Runner so as to provide a clear assessment of the end to end performance of a system, and occasionally this does involve scripting all transactions. The tradeoff here is again the time required vs. benefits gained in terms of the usefulness of the performance results. Scripting transactions as GUI vusers is considerably different to non-GUI vusers, and is explained fully in the associated Win Runner implementation documentation. Data Preparation Preparation of data in the system must be completed before scenario execution can commence. This phase ensures that the system contains enough data that the most demanding load scenarios can be executed at least once, but preferably three or more times, before the system requires refreshing. This phase can be performed in parallel to script development, and can normally be achieved quite quickly if the data demands of the transactions were correctly identified in the planning phase. Scenario Creation With transactions scripted as both GUI and non-GUI vusers, scenarios that place the system under load can be implemented. These scenarios should have been previously identified in the earlier planning phase, and all that is required here is creating them inthe Load Runner controller. As a rule of thumb, most scenarios should run for about one hour. This allows the results of the run to be more statistically reliable than a shorter run, and is not so long as to consume too much time. Shorter scenarios should be looked upon unfavorably, and longer scenarios implemented only if there is good cause to do so. Golden Area Before execution is commenced, it is important to create a golden area, or a place where all scripts, scenarios, and a database baseline can be placed. This golden area should be treated as write-once read-many. This means that scripts placed here must be bug-free, as modifications made to files in the golden area can greatly confuse future testing. The database baseline should contain all the data required for performing the most strenuous of the load tests, and is used to refresh the system as required. Preferably, the database baseline placed here should allow several load tests to be performed before requiring the system to be refreshed. As these refreshes are generally time consuming activities, it is ideal if refreshes can be performed overnight, after every day of load testing. It is important to note that it is not always possible or feasible to create database baselines, due to the complexity of a system and its batch processing. Where this is the case, it is important to pre-determine this, and make allowances in creating the data that load tests can consume. This type of scenario is far from ideal, as it is difficult to compare results between runs, as too many system variables have changed. All that can be done is to raise the issue, and continue as best as possible. It is, however, worth noting that systems in which it is "impossible" to revert to baselines, often through ingenuity techniques can be created allowing just that to be done. Execution When this phase is commenced, the testing environment must be correctly set-up, and the scenarios for testing must be created and prioritized. The exact scenarios to be run vary widely depending on the goals set prior to planning the load test. Some common scenarios that are recommended follow.

Baseline Load The baseline or light load scenario involves running small numbers of users, ensuring that the testing environment behaves as expected. As the scripts have been tested under the development environment, problems here are likely to indicate configuration problems in the testing environment. If these occur, they must be addressed before more demanding testing can begin. A second reason for executing a light load scenario is to use the results as a benchmark. Performance results obtained under the baseline load should be more than satisfactory; otherwise the system has no hope of performing adequately under a higher load. Assuming no performance problems were noticed during this scenario, the results obtained can be used as "best case" results. These results indicate how the system performs when it is not under any real form of stress, allowing conclusions to be drawn about the performance of the system during the higher load tests. Medium Load Depending exactly on the requirements of the load testing project, a medium load may constitute what is considered a real-world worse case load, or as little as 50% of this. Where the load that is applied is less than the expected worse case load, the scenario is, used to identify major failings in a system, but does so in a way that doesn't highlight many of the more minor failings, allowing an easier analysis of results. Where the load is equivalent to the expected real-world worse case load, actual performance of the system can be measured, and associated problems clearly identified. Running a medium load before moving directly to a high load allows the more major problems of a system to be identified separately of the smaller issues, by providing a simpler result analysis. Heavy Load Heavy load scenarios test a system at and beyond the worse expected demand it is likely to face. The majority of major deficiencies in the system will have already been identified in the medium load execution, so this phase deals more with assessing the impact of these issues under a heavy load. However, heavy scenarios also identify many other system bottlenecks not previously noticed, which may in fact be partially responsible for earlier identified problems. Heavy load scenarios are generally designed to be far more than a system can handle. They are used not to identify if a system fails, but where it fails first, by how badly, and why. By answering the why question, it can be determined whether a system is as good as it should be. Heavy scenarios often involve starting a scenario running, and incrementally adding more and more load onto the system, until it either fails altogether, or becomes totally unusable. At this point, the load can be slowly decremented. This allows the effect of different loads to be easily analyzed, and plotted graphically. The analysis performed in this phase can vary depending on the exact goals and objectives of the load testing. Results obtained from within Load Runner are often used in conjunction with results obtained directly from the system, where white-box testing has been employed. The majority of scenario execution time will be spent with heavy load scenarios, as these provide the most information about the performance of a system. Analysis The entire reason of performing load testing is so a better understanding of a system‘s performance can be obtained. The results obtained during Load Runner scenario.

Executions are plentiful, and this is compounded if white-box metrics have also been measured. The analysis phase typically occurs in parallel with the execution phase, as both phases are iterative in nature, although analysis occurs with an offset from the execution, and can continue after execution has completed. White Box vs. Black Box Analysis Where white-box metrics have been recorded, they can provide much more insight into where problems exist in a system, as well as to highlight where improvements are most necessary. Black box analysis can show the effects of running different numbers of simultaneous transactions, as well as showing the effect of different combinations of transactions, but the results. However, black box testing will only highlight performance issues on a per transaction basis. White box analysis can show exactly where in a system time is being spent, as well as show other information not directly related to time, such as the percentage of transactions that result in more than 10 rows being fetched. Clearly, this information requires a lower level of knowledge about a system than Load Runner can gain. Black box analysis provides the big picture of a system‘s performance, which is often all that is required. White box analysis concentrates on looking at the system as a collection of co-operating parts, allowing finer detail to be obtained. Results Many load test projects iterate between the execution and analysis phases many times, with system modifications being made between iterations, thus the load tests are used to improve the system. Other load test projects are simply performed to measure the performance of a system, preparing the systems owners for its performance, and allowing them to determine if changes may be required. Depending on the situation, the writing of results may vary. The results of a load test project should directly relate to the original goals and objectives of the load test. They should focus on what system requirements are met, how well they are met, and which fail. Any issues encountered that do not directly relate to the objectives of the project should be mentioned in full, allowing further action to be taken where this is deemed necessary. A write-up of results should include at least part of the analysis, but this is more often desirable as an appendix. Load testing is often performed as part of user acceptance testing, even if only implicitly, and this should be recognized when load test results are being tabulated. The Benefits of Automated Testing If you have ever tested software manually, you are aware of its drawbacks. Manual testing is time-consuming and tedious, requiring a heavy investment in human resources. Worst of all, time constraints often make it impossible to manually test every feature thoroughly before the software is released. This leaves you wondering whether serious bugs have gone undetected. Automated testing with WinRunner addresses these problems by dramatically speeding up the testing process. You can create test scripts that check all aspects of your application, and then run these tests on each new build. As WinRunner runs tests, it simulates a human user by moving the mouse cursor over the application, clicking Graphical User Interface (GUI) objects, and entering Keyboard input—but WinRunner does this faster than any human user. ………………………………………………………………………………………………………………… BIBLIOGRAPHY

Useful Links 00725

One principle in software testing is summed up by the classical Latin question posed by Juvenal: Quis Custodiet Ipsos Custodes (Who watches the watchmen?), or is alternatively referred informally, as the "Heisenbug" concept (a common misconception that confuses Heisenberg's uncertainty principle with observer effect). The idea is that any form of observation is also an interaction, which the act of testing can also affect that which is being tested. In practical terms the test engineer is testing software (and sometimes hardware or firmware) with other software (and hardware and firmware). The process can fail in ways that are not the result of defects in the target but rather result from defects in (or indeed intended features of) the testing tool. There are metrics being developed to measure the effectiveness of testing. One method is by analyzing code coverage (this is highly controversial) - where every one can agree what areas are not at all being covered and try to improve coverage on these areas. Finally, there is the analysis of historical find-rates. By measuring how many bugs are found and comparing them to predicted numbers (based on past experience with similar projects), certain assumptions regarding the effectiveness of testing can be made. While not an absolute measurement of quality, if a project is halfway complete and there have been no defects found, then changes may be needed to the procedures being employed by QA.

Shared By: