:: Exploratory Testing

Transcription

1 :: Exploratory Testing Jonathan Kohl Portions of this presentation are the intellectual property of James Bach and are used with his permission.

2 :: What is Exploratory Testing? Simultaneous learning, test design and execution The opposite of scripted testing (testing from pre-recorded, procedural test cases) This can also include automated tests, log analyzing, etc.

3 :: What is Exploratory Testing? Is not random testing Is not unstructured testing Is not a technique, but an approach Is teachable, repeatable, and manageable Is the heart of excellent testing

4 :: Who does Exploratory Testing? Just about anyone who does testing. You have done exploratory testing if you have: tested different scenarios around a bug fix tracked down suspicious behaviour tested without a script changed your tests on the fly based on program interaction These are all examples of exploratory testing

5 :: Who does Exploratory Testing? All genuine testers do exploratory testing. Some are really good at it. A few can effectively explain and teach it. More of us can learn to do it well, and to explain and teach it

6 :: Exploratory Testing Since exploratory testing is something all testers do at least some of the time, and It is consistently named as the most effective way of providing rapid feedback and finding important bugs, then Why is it frequently discouraged?

7 :: Exploratory Testing Good exploratory testing requires that we are personally accountable for our testing work, whereas few people enjoy reviewing written test procedures few people know how to evaluate the quality of written test procedures most managers seem to think that written tests are a Good Thing Writing poor test procedures won t get you fired. Some companies will even reward you for the poor quality of your test procedures. That means there is little pressure on you to become an excellent tester

8 :: Exploratory Testing There is sometimes confusion about the term exploratory testing It can mean exploring a new feature or software that you haven t used before It can also mean unscripted testing on software you are familiar with Until we know of all the bugs, all the features and behaviours of the software under any and all conditions, we can benefit from exploratory testing

9 :: Discovery and Skill Exploratory testing can be like a voyage of discovery charting the unknown You use observation and inference to build a model of the software you are testing to gather information It is also often like a scientific experiment where we are exploring events that challenge the status quo You build new models, and use confirmatory testing to confirm or challenge the model

10 :: Discovery and Skill Both scientific theory and well-executed exploratory testing rely on skill You try to sense what matters, asking: What are the important variables? You observe, discover, inquire and build a model of the application you are testing You then switch to confirmatory testing to corroborate or contradict your model Exploratory testing is highly interactive and allows for learning by adapting test designs when new information is discovered

11 :: Discovery and Skill Rapid feedback and the discovery of potential problems are valued by development teams In the scientific community, many groundbreaking discoveries are found by accident In software testing, many show-stopping bugs are found by accident Good Exploratory Testers strive to do testing in a disciplined way to increase their odds of uncovering bugs Thoughtful testing allows us to be in the right place at the right time there really are no accidental discoveries

12 :: A Structured Process Exploratory testing, as skilled practitioners view it, is a structured process conducted by a skilled tester, or by lesser skilled testers or users working under reasonable supervision.

13 :: A Structured Process The structure of exploratory testing comes from: Test design heuristics Chartering Time boxing Perceived product risks The nature of specific tests The structure of the product being tested The process of learning the product Development activities Constraints and resources afforded by the project The skills, talents, and interests of the tester The overall mission of testing

14 :: A Structured Process In other words, exploratory testing is not random, it is systematic and disciplined when done by a skilled practitioner

15 :: Contrasting Approaches In scripted testing, tests are first designed and recorded. Then they may be executed at some later time or by a different tester. Tests In exploratory testing, tests are designed and executed at the same time, and they often are not recorded. Tests Product

16 :: Contrasting Approaches Scripted testing emphasizes predictability, accountability and decidability. Tests Exploratory testing emphasizes adaptability, credibility and learning. Tests Product

17 :: Contrasting Approaches Scripted testing is like making a prepared speech. Tests Exploratory testing is like having a conversation. Tests Product

18 :: Decentralizing Testing Instead of trying to solve testing problems: only before test execution begins by investing in expensive test documentation that tends to reduce the total number of tests that can be created only via a designer who is not necessarily the tester while trying to eliminate the variations among testers completely, and all at once They are solved: over the course of the project by minimizing the need for expensive test documentation so that more tests and more complex tests can be created with the same effort via testers who may also be test designers by taking maximum advantage of variations among testers incrementally and cyclically

19 :: Sound Familiar? Exploratory testing is adaptive rather than predictive Where have we heard this before? Agile projects are meant to be adaptive rather than predictive and encourage learning, iterating, communication and feedback Designing all tests test-first pressures testers to be predictive Front loading test development closes the door on many classes of tests

20 :: Agility and Exploration Why should agile teams do exploratory testing? Because an agile development project can accept new and unanticipated functionality so fast, it is impossible to reason out the consequences of every decision ahead of time. In other words, agile programs are more subject to unintended consequences of choices simply because choices happen so much faster. This is where exploratory testing saves the day. Because the program always runs, it is always ready to be explored. - Ward Cunningham, on exploratory testing

21 :: Agility and Exploration Agile teams value: Rapid feedback Test idea generation Working software that they and the customers have confidence in A balanced, risk-based, diverse testing strategy can address these issues Over-reliance on one testing strategy results in a skewed view of testing, and less coverage

22 :: Agility and Exploration When should Agile Teams do exploratory testing? Whenever we want information on a product or feature under development: during an iteration during story development after story development at the end of an iteration

23 :: Agility and Exploration Like Agile development, exploratory testing involves continuous learning. Good Exploratory Testers have: Curiosity Awareness Inquiry Attention to detail Persistence Comfort challenging and being challenged on their testing ideas Most importantly, they rely on the most powerful tool at our disposal: the human mind.

24 :: Agility and Exploration Software projects benefit from diversity in testing Test-Driven Development, User Acceptance Testing and conventional software testing complement each other Combining Agile testing and Exploratory Testing offers the potential for great testing diversity and thorough test coverage

25 :: Exploratory Testing Resources James Bach: Rapid Software Testing Course: Articles: Capturing Documentation, Jonathan Bach:

26 :: Exploratory Testing Resources Cem Kaner: Cem coined the term Exploratory Testing Articles:

27 :: Exploratory Testing Resources Books: The Sciences of the Artificial, Herbert Simon Conjectures and Refutations, Karl Popper Exploring Requirements, Don Gause & Gerald M. Weinberg Abductive Inference, John R. Josesphson Inference to the Best Explanation, Peter Lipton

28 :: Acknowledgements Thank you to James Bach for help and input into this presentation