One of the major components of the testing arsenal is finding combinations of inputs that cause problems in the AUT.
For all but the most trivial applications, the number of possible input combinations is somewhere between daunting and infinite.
To get around this, many testers subscribe to Pairwise testing - the argument being that most troublesome combinations arise from
simple pairs of inputs, rather than deeply nested combinations. This is not to deny the presence of properly obscure bugs, it is merely
a technique to direct testing to where it is most likely to produce results soonest. It is left to your own discretion, experience and
knowledge of the AUT to extend the pairwise test cases with your own exploratory testing.
The creation of pairwise tests is a fascinating mathematical exercise. Luckily a team at Microsoft (again?!) have produced PICT to do the hard maths.
It works very well but some people fear the command line, so I produced an Excel spreadsheet as a front end:
The PICTinput sheet allows you to enter test input names and possible values:
| OS | XP | Vista | W2K3 | W2K8 | W2K3_64 | W2K8_64 | W7RC1 |
| Security | Local | Network | Inadequate | Botched | Swiss Cheese | Microsoft Standard | |
| Install | New | Upgrade | Confused | Impossible | Frustrating | ||
| Killbits Installed? | YES | NO | WTF??!!?!? | NEARLY | |||
| Change Folder? | NO | YES | MAYBE | ||||
| User | Admin | N00b | Super User | Super Duper User | Super Smashing Great |
The Control sheet allows you to set the combination depth (e.g. 2 to test by pairs) and output file name.
Pressing the button invokes the following VBA routine, which passes the PICTinput values to the PICT CLI.
Sub Button1_Click()
'create text version of inputs sheet
Worksheets("PICTinput").Activate
Set fs = CreateObject("Scripting.FileSystemObject")
Set a = fs.CreateTextFile("PICTinputs.txt", True)
Dim newrange As Range
For irow = 1 To ActiveSheet.UsedRange.Rows.Count
pictLine = ""
Set newrange = Intersect(Cells.Rows(irow), ActiveSheet.UsedRange)
For Each cell In newrange
If (cell.Column > 2) And (cell.Text <> "") Then
pictLine = pictLine + ","
End If
pictLine = pictLine + cell.Text
If cell.Column = 1 Then
pictLine = pictLine + ":"
End If
Next cell
a.WriteLine (pictLine)
Next irow
a.Close
Worksheets("Control").Activate
'delete existing output
Dim x As String
Dim OutputFileName As String
OutputFileName = ActiveSheet.Range("OutputFileName").Text
ChDir (ThisWorkbook.Path)
x = Dir(OutputFileName)
If x <> "" Then
If MsgBox("Overwrite existing output file " & OutputFileName & "?", vbYesNo + vbExclamation) = vbNo Then
End
Else
Kill (OutputFileName)
End If
End If
'run PICT to create the test cases
Dim ComboDepth As Integer
ComboDepth = ActiveSheet.Range("Combination_Depth").Text
Call Shell("cmd /C pict PICTinputs.txt > " & Chr(34) & OutputFileName & Chr(34) & " /o:" & ComboDepth)
'check that the output file exists
x = Dir(OutputFileName)
Do While x = ""
Application.Wait (Now + TimeValue("0:00:1"))
x = Dir(OutputFileName)
Loop
'open the output and tidy the formatting
Workbooks.Open (OutputFileName)
Range("1:1").Cells.Font.Bold = True
ActiveSheet.Columns.AutoFit
End Sub
and you end up with a table of test cases like this, covering every pair of input conditions:
| OS | Security | Install | Killbits Installed? | Change Folder? | User | ||
|---|---|---|---|---|---|---|---|
| W2K8_64 | Swiss Cheese | Upgrade | WTF??!!?!? | NO | N00b | ||
| XP | Network | Upgrade | YES | MAYBE | Super Smashing Great | ||
| W2K8_64 | Local | Confused | NEARLY | YES | Super User | ||
| W2K3_64 | Inadequate | New | NO | MAYBE | Admin | ||
| Vista | Botched | Frustrating | NO | NO | Super Duper User | ||
| W7RC1 | Local | Impossible | YES | NO | Super Smashing Great | ||
| XP | Local | Impossible | NO | YES | N00b | ||
| W2K8_64 | Microsoft Standard | Impossible | WTF??!!?!? | MAYBE | Super Duper User | ||
| W7RC1 | Network | Frustrating | WTF??!!?!? | YES | Super User | ||
| W2K3_64 | Botched | Upgrade | YES | YES | Super User | ||
| W2K3 | Botched | Confused | NEARLY | NO | Admin | ||
| W2K8 | Botched | New | NEARLY | MAYBE | Super Smashing Great | ||
| W2K3_64 | Local | Frustrating | NEARLY | MAYBE | Super Duper User | ||
| W2K8_64 | Botched | New | YES | YES | Admin | ||
| Vista | Network | Confused | NEARLY | MAYBE | N00b | ||
| W2K8 | Inadequate | Confused | YES | YES | Super Duper User | ||
| W7RC1 | Botched | New | WTF??!!?!? | MAYBE | N00b | ||
| W7RC1 | Microsoft Standard | Upgrade | NO | NO | Admin | ||
| XP | Swiss Cheese | New | NEARLY | NO | Super User | ||
| W2K8 | Network | Impossible | NO | NO | Admin | ||
| XP | Inadequate | Confused | WTF??!!?!? | NO | Super Smashing Great | ||
| XP | Botched | Upgrade | NEARLY | YES | Super Duper User | ||
| W7RC1 | Swiss Cheese | Frustrating | YES | MAYBE | Super Duper User | ||
| W2K3 | Local | New | NO | YES | Super Smashing Great | ||
| Vista | Microsoft Standard | Impossible | YES | YES | Super User | ||
| W2K8 | Local | Frustrating | WTF??!!?!? | NO | Admin | ||
| XP | Microsoft Standard | Frustrating | NEARLY | MAYBE | N00b | ||
| W2K3_64 | Network | New | WTF??!!?!? | NO | Super Duper User | ||
| Vista | Inadequate | Upgrade | NEARLY | MAYBE | Super User | ||
| W2K8_64 | Network | Frustrating | NO | MAYBE | Super Smashing Great | ||
| Vista | Local | New | WTF??!!?!? | YES | Admin | ||
| W2K3 | Swiss Cheese | Confused | NO | YES | Super User | ||
| W2K3 | Network | Impossible | YES | MAYBE | N00b | ||
| W2K3 | Microsoft Standard | Upgrade | WTF??!!?!? | YES | Super Duper User | ||
| W2K8 | Swiss Cheese | Impossible | NEARLY | NO | Super Smashing Great | ||
| Vista | Swiss Cheese | Frustrating | YES | NO | Super Smashing Great | ||
| W2K8_64 | Inadequate | Frustrating | NO | MAYBE | N00b | ||
| W2K8 | Local | Upgrade | NEARLY | MAYBE | N00b | ||
| W2K3 | Inadequate | Impossible | NEARLY | NO | N00b | ||
| W2K8 | Microsoft Standard | Impossible | WTF??!!?!? | YES | Super User | ||
| W2K3_64 | Microsoft Standard | Impossible | YES | NO | Super Smashing Great | ||
| W2K3_64 | Microsoft Standard | Confused | NEARLY | NO | N00b | ||
| W7RC1 | Inadequate | Confused | NEARLY | YES | Super Smashing Great | ||
| XP | Microsoft Standard | New | WTF??!!?!? | NO | Admin | ||
| W2K3_64 | Swiss Cheese | Frustrating | YES | YES | Admin | ||
| XP | Botched | Impossible | WTF??!!?!? | NO | Super Smashing Great | ||
| W2K3 | Network | Frustrating | YES | MAYBE | N00b | ||