Does 1x1 work? Does the strategy work for all defects for one dimension? Closed Off Outside C >= 60
on
off
C
60
Closure defect Misplaced boundary, shifted left Misplaced boundary, shifted right Missing boundary Extra boundary ????? CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
One case needs a little more thought!
1
Does it work for 2 dimensional domains? (For linear boundaries:) Closure defect Misplaced boundary 1 Misplaced boundary 2 Missing boundary Extra boundary
off
on
Any more? In general, this strategy will detect most poorly located boundaries of multiple variables. But consider: will our 1x1 test cases for Boundary 3 spot the mistaken implementation C + 3E >= 220 ? What kind of boundary defect is it ? If we continue to use a single Off point, how many On points do we need in order to detect such defects ? CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
2
The Nx1 strategy Extending to the general case, it takes N On points to confirm the correctness of an (N-1) dimensional boundary of an Ndimensional domain Can see this, for example, for linear boundaries of 2 dimensional domains. It takes 2 On points to uniquely determine the line that forms the boundary. How should the On points be chosen? Far apart, with the Off point midway between them. This greatly increases the cost of testing, particularly for higher dimensions. Non-linear boundaries are even more expensive - they require substantially more On points.
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
3
Is N+1 worth the price? In software engineering we must consider costs and benefits We can consider what defects may be missed by relying on a 1x1 rather than an Nx1 strategy When will 1x1 fail to detect a defect ? What is the likelihood of:
Or (for the non-linear case):
on
on
off
off expected actual
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
4
1x1 vs. Nx1: Summary 1x1 misses a bad boundary only if that boundary passes between our On and Off points. This is very unlikely if we choose the points to be close together. Remember we aren’t dealing with a malicious implementation, we hope! Typically, natural defects cause big distortions of boundaries and will quite likely be caught by the simpler strategy. If we can’t take the chance of missing some less obvious defect, however (safety critical or mission critical systems) we might apply the more expensive strategy. One possible use for Nx1 in other cases is to help determine the type of defect that is present since it provides much more information than the less expensive 1x1. CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
5
Error Guessing and Fault Models
6
Error Guessing Here we design test cases based on our experience with similar software – could be experience of similar domain, design approach, implementation, etc. Objective is to improve effectiveness and efficiency Testers also use intuition about where defects are likely to be found They guess the sorts of defects that will be present, and design test cases to reveal them Simple examples you’ll recognize close to code: - where there is the possibility of a divide-by-zero, - where pointers are being manipulated in C/C++ - where null parameters or return values are allowed.
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
7
Seems obvious… … but there are a couple of important points At a higher level, we are also doing this when we choose a test technique – really, error guessing is a part of all test techniques. It is closely related to the idea of a fault model.
This need not be an ad hoc approach. If we have defect data recorded for similar code we’ve developed in the past, we can make the process more systematic. Gathering and using data in this way raises our testing process maturity.
It is also a skill that separates great testers from the rest
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
8
Role of the Fault Model It is the fault model that leads us to a test strategy and a practical set of test cases Can think of the fault model as answering the question: why do we think the features targeted by a test technique will be worth testing The model is based on good argument or evidence that a certain kind of probing has a good chance of revealing a defect
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
9
Pesticide paradox This is a danger to be aware of in Error Guessing, especially within an organisation Good developers learn from the defects that are discovered They will search for, and remove similar defects and will take care not to repeat them This means that , over time, your fault model becomes ineffective – the implementation is immune to it The better your tests, the faster they “wear out”
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
10
State Transition Testing
11
Stuff you know
(or will know)
A state machine is a model whose outcome is determined by both current and past input The effect of previous inputs is represented by a state The state machine diagram (UML 2.0) depicts all states that a component can be in, the events that trigger transitions between states, and the activities that result. State machine diagrams have been used for a long time to describe the behaviour of a system. Widely used in object-oriented development to show, for example, the lifetime behaviour of objects.
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
12
Elements of the UML State Machine Diagram Trigger event Guard condition (Boolean)
All optional:
Evaluated once only, when the event occurs
Action taken (brief, atomic) Initial state
State1
Event [Guard] /Activity
d o/ acti vi ty
In addition to “do / activity”, can have: entry / activity exit / activity … and others CSIS0403 Implementation, Testing and Maintenance of Software Systems
St ate2 do/ Other acti vi ty
Final state 2006-2007
13
State machines and testing State tables provide a compact representation of state machines Very easy to convert information in the state table into test cases Can help develop test cases to detect defects that might not be revealed using the input/output conditions yielded by equivalence class partitioning – uses a different fault model which includes constraints from sequential control requirements
Build the table, if necessary. Make sure the state table is correct! Typically they are incomplete. CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
14
Faults – incorrect responses
(Binder)
Missing or incorrect transition – wrong resultant state Missing event – stimulus is ignored Missing or incorrect activity – wrong thing happens Extra, missing or corrupt state – behaviour becomes unpredictable Message erroneously accepted – “sneak path” Illegal message failure – fails on receiving an unexpected message Responds to undefined messages –”trap door”
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
15
Example - Testing GUIs (yet another “toy” example…)
$$
Currency Converter
-
x
HK Dollar Amount Yen Equivalent Convert Clear
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
16
State machine diagram
Note, this is a slight misuse of activities just to make it clear what is going on
Idle Enter Amount / display amount
Convert/display error Clear /clear all
Amount Entered Enter Amount / display amount, clear Yen
Enter Amount / display amount
Missing Amount
Convert/display Yen
Display Yen
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
17
State table with holes All possible Current State
One of many possible formats
Enter amount
All possible Events
Convert
Clear
Idle
Missing Amount
Amount Entered
Amount Entered
Amount Entered
Amount Entered
/display amount
/display amount
/display amount, clear Yen
Missing Amount
Display Yen
/display error
/display Yen
Display Yen
Idle
Idle
Idle
/clear all
/clear all
/clear all
But better to test every event from every state to ensure “no change” assumptions are correct. CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
18
Define all events Clear /clear all
Idle
Enter Amount / display amount
Enter Amount / display amount
Convert/display error Clear /clear all
Amount Entered
Enter Amount / display amount, clear Yen
Enter Amount / display amount
Convert/display Yen
Missing Amount
Convert /display error
Display Yen Convert /display Yen
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
19
State table 2. Holes filled Current state
Enter amount
Event
Convert
Clear
Idle
Missing Amount
Amount Entered
Display Yen
Amount Entered
Amount Entered
Amount Entered
Amount Entered
/display amount
/display amount
/display amount
/display amount, clear Yen
Missing Amount
Missing Amount
Display Yen
Display Yen
/display error
/display error
/display Yen
/display Yen
Idle
Idle
Idle
Idle
/clear all
/clear all
/clear all
/clear all
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
20
Creating the test cases Input
Expected Result
Test case ID
Current state
Event
Display
Next state
ST-001
Idle
Enter Amount
display amount
Amount Entered
ST-002
Idle
Convert
display error
Missing Amount
ST-003
Idle
Clear
clear all
Idle
ST-004
Amount Entered
Enter Amount
display amount
Amount Entered
ST-005
Amount Entered
Convert
display Yen
Display Yen
ST-006
….
….
….
….
A failed test case may indicate: ¾ specified states or transitions are missing from the code; ¾ specified transitions are not implemented correctly; ¾ specified activities are not implemented correctly.
etc.
But how about states or event/transitions not shown in the state diagram but present in the code? Extra states. CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
21
Coverage There are various degrees to which we can test. Is it sufficient, for example, to simply check that each state can be visited? How much testing should we do? Questions like this highlight that there is a coverage hierarchy. Each represents a different test strategy. Notice we have several possible levels of thoroughness to choose between. These represent various adequacy criteria against which we can judge the quality of our test suite
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
22
Coverage hierarchy Create a set of test cases such that: All states are visited at least once Every event occurs at least once All transitions are exercised at least once All events occur for all states, whether or not they are supposed to trigger a transition All possible paths are executed at least once
Weak coverage Acceptable coverage
Good coverage Excellent, but generally infeasible coverage
We’ll see a similar concept of levels of coverage in structural testing
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
23
Where it is useful Can be used to model a wide range of, often complex, systems and situations. Examples: Menu-driven software Behaviour of GUI widgets Life cycle and behaviour of business control objects Protocols Device drivers Software that replaces hardware Parsers Installation software Long-lived database transactions Motion control systems Behaviour in OO systems at any scope: class, cluster, component, subsystem, up to the system itself
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
Provides a compact way to express complex rules and interactions 24
