软件测试论坛

 找回密码
 软件测试论坛注册页
查看: 2537|回复: 59

[英文资料] Resources on Exploratory Testing, Metrics, and Other Stuff

[复制链接]
发表于 2011-12-26 09:56:12 | 显示全部楼层 |阅读模式
软件测试工程师就业班马上开班
Evolving Understanding of Exploratory TestingMy community defines exploratory testing as
a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to optimize the quality of his or her work by treating test design, test execution, test interpretation, and test-related learning as mutually supportive activities that continue in parallel throughout the project. Yes, that's quite a mouthful. It was synthesized by Cem Kaner in 2006, based upon discussions at the Exploratory Testing Research Summit and the Workshop on Heuristic and Exploratory Techniques. (hover the mouse for participants).
Sometimes, when we want to save time, we refer to exploratory testing more concisely: "parallel test design, test execution, and learning". These definitions are not contradictory. The former is more explicit; the other can be uttered more quickly, and is intended to incorporate the latter. (We used to say "simultaneous", instead of "parallel", but people had trouble, apparently, with the idea that simultaneous activities don't have to be happening at equal intensity at every moment, so parallel seems like a better word these days.)
Exploratory approaches to testing live at one end of a continuum. Scripted testing—writing out a list of step-by-step actions to perform, each step paired with a specific condition to observe—is at the other end of this continuum. Scripted approaches are common in software testing. Typically tests are designed by a senior tester, and given to someone else—typically a more junior tester—to execute.
A number of colleagues and I have serious objections to the scripted approach. It is expensive and time-consuming, and seems likely to lead to inattentional blindness. It also separates test design from test execution and result interpretation, and thereby lengthens and weakens the learning and feedback loops that would otherwise support and strengthen them.
James Bach and I recorded a conversation on this subject, which you can listen to here. At the 2008 Conference for the Association for Software Testing, Cem Kaner presented a talk called The Value of Checklists and the Danger of Scripts: What Legal Training Suggests for Testers. You can read that here.
Some claim that exploratory testing is "unstructured", equating it with "ad hoc testing" or "fooling around with the computer". In our definition of exploratory testing, such claims are false and unsupportable, and we reject them. Some may say that they are doing "exploratory testing" when they are behaving in an unskilled, unprofessional manner, but we reject this characterization as damaging not only to exploratory testing, but to the reputation of testers and testing generally. If you are not using the learning garnered from test design and test execution in a continuous and rapid loop to optimize the quality of the work, you are not doing exploratory testing. If exploratory testing is "fooling around with the computer", then forensic investigation is "poking around inside a dead body".
"Ad hoc" presents an interesting problem, because those who equate "ad hoc" with "exploratory" not only misunderstand the latter, but misrepresent the former as meaning "sloppy" or "slapdash". "Ad hoc" means literally "to this", or "to the purpose". The Rogers Commission on the Challenger explosion was an ad hoc commission, but it wasn't the Rogers Sloppy Commission or the Rogers Slapdash Commission; it was formed for a specific purpose, did its work, and was dissolved when its work was done. In that sense, all testing should be "ad hoc". But alas the expression and its original meaning parted company several years ago. Exploratory testing is certainly not ad hoc in its revised sense.
Exploratory testing is not structured in the sense of following a prescribed, step-by-step list of instructions, since that's not what structure means. Structure, per the Oxford English Dictionary, means "the arrangement of and relations between the parts or elements of something complex". In this definition, there is no reference to sequencing or to lists of instructions to follow. So, just as education, nutrition, driving an automobile, juggling, child-rearing, and scientific revolutions are structured and have structure, explororatory testing is also structured. In fact, there are many structures associated with exploratory testing. What follows is an evolving list of lists of those structures:
  • Evolving Work Products, Skills and Tactics, ET Polarities, and Test Strategy. James Bach, Jon Bach, and I authored the latest version of the Exploratory Skills and Dynamics list. This is a kind of evolving master list of exploratory testing structures. James describes it here.
  • Oracles. The HICCUPPS consistency heuristics, which James Bach initiated and which I wrote about in this article for Better Software in 2005. (Actually, at the time it was only HICCUPP—History, Image, Comparable Products, Claims, User Expectations, Purpose, Product—but since then we've also added S, for Standards and Statutes. Mike Kelly also talks about HICCUPP here.
  • Test Strategy. James Bach's Heuristic Test Strategy Model isn't restricted to exploratory approaches, but certainly helps to guide and structure them.
  • Test Framing. Test framing can be seen as a retrospective view of test strategy. Test framing is a key skill that helps us to compose, edit, narrate, and justify the story of our testing in a logical, coherent, and rapid way. The goal of test framing is to link each testing activity with the testing mission via a direct line of logic that connects the mission to the tests. Test framing is described in detail here.
  • Speculation. While this post refers to the overall development process, it makes some important points that are relevant to the nature of rapid, iterative, and exploratory testing. Jim Highsmith on "Don't Plan, Speculate".
  • Data Type Attacks, Web Tests, Testing Wisdom, Heuristics, and Frameworks. Elisabeth Hendrickson's Test Heuristics Cheat Sheet is a rich set of guideword heuristics and helpful reference information.
  • Context Factors, Information Objectives. Cem Kaner most recently delivered his Tutorial on Exploratory Testing for the QAI Quest Conference in Chicago, 2008. There's a similar, but not identical talk here. James Bach's Context Model is a diagram containing many guidewords to context.
  • Quick Tests. In our Rapid Software Testing course, James Bach and I talk about quick tests. The course notes are available for free. Fire up Acrobat and search for "Quick Tests".
  • Coverage (specific). Michael Hunter's You Are Not Done Yet is a detailed set of coverage ideas to help prompt further exploration when you think you're done.
  • Coverage (general). James Bach wrote this article in 2001, in which he summarizes test coverage ideas under the mnemonic "San Francisco Depot."—Structure, Function, Data, Platform, and Operations. Several years later, I convinced him to add an element to the list, so now it's "San Francisco Depot. The last T is for...
  • Time. I realized a few years ago that some guideword heuristics might help us to pay attention to the ways in which products related to time, and vice versa. That turned into a Better Software article called "Time for New Test Ideas".
  • Tours. Mike Kelly's FCC CUTS VIDS Touring Heuristics (note the date) provides a set of structured approaches for touring the application.
  • The Value of Checklists and the Danger of Scripts.Cem Kaner's keynote presentation from the 2008 Conference for the Association for Software Testing describes experiments in overly patterned learning, and the ways in which the legal profession uses checklists as heuristic tools to guide decsion-making.
  • Stopping Heuristics. There are structures to deciding when to stop a given test, a line of investigation, or a test cycle. I catalogued them here, and Cem Kaner made a vital addition here.
  • Accountability, Reporting Progress. James and Jon Bach's description of Session-Based Test Management is a set of structures for making exploratory testing sessions more accountable.
  • Procedure. The General Functionality and Stability Test Procedure. It was designed for Microsoft in the late 1990s by James Bach, and may be the first documented procedure to guide exploratory test execution and investigation.
  • Regression Testing. Karen Johnston provides a list of heuristic guidewords to lend structure to testing for regression in an exploratory way: RCRCRC, for Recent, Core, Risk, Configuration-sensitive, Repaired, and Chronic.
  • The Role of Repetition. In the Rapid Software Testing class, James Bach and I teach heuristics on why you might choose to repeat a test, instead of running a new test. We note that repeated tests and regression tests may intersect, but they're orthogonal categories.
  • Emotions. I gave a talk on emotions as powerful pointers to test oracles at STAR West in 2007. That helped to inspire some ideas about...
  • Noticing, Observation. At STAR East 2009, I did a keynote talk on noticing, which can be important for exploratory test execution. The talk introduces a number of areas in which we might notice, and some patterns to sharpen noticing.
  • Leadership. For the 2009 QAI Conference in Bangalore, India, I did a plenary talk in which I noted several important structural similarities between exploratory testing and leadership.
This is a blog posting that I wrote in September, 2008, summarizing some important points about exploratory testing.
James Bach was interviewed by Matthew Osborn and Federico Silva Armas for a CodingQA podcast in November of 2009. The recording is here, and text summaries can be found here and here.
Meaningful MetricsThe software development and testing business seems to have a very poor understanding of measurement theory and metrics-related pitfalls, so conversations about metrics are often frustrating for me. People assume that I don't like measurement of any kind. Not true; the issue is that I don't like bogus measurement, and there's an overwhelming amount of it out there.
I've written two articles that explain my position on the subject:
I'll suggest that anyone who wants to have a reasonable discussion with me on metrics should read and reflect deeply upon
Software Engineering Metrics: What Do They Measure and How Do We Know (Kaner and Bond) and then explain how their metrics don't run afoul of the problems very clearly identified in the paper.
Here are some more important references on measurement and the risks of distortion and dysfunction:
Show me metrics that have been thoughtfully conceived, reliably obtained, carefully and critically reviewed, and that avoid the problems identified in these works, and I'll buy into the metrics. Otherwise I'll point out the risks, or recommend that they be trashed. As James Bach says, "Helping to mislead our clients is not a service that we offer."
Investigating Hard-To-Reproduce BugsFinding it hard to reproduce the circumstances in which you noticed a bug?
The Heuristic Test Strategy ModelThis document by James Bach describes the test strategy model that is central to the practice of Rapid Testing.
Context-Driven Testing ExplainedCem Kaner and James Bach collaborated on a detailed description of context-driven testing, explaining it and contrasting it with other approaches.
Unpublished Articles
Test MatricesA test matrix is a handy approach to organizing test ideas, tracking results, and visualizing test coverage. Read more here.
Visual SourceSafe DefectsWhile developing a utility to migrate files from Visual SourceSafe (VSS) to another version control package, I had to test Visual SourceSafe itself. These tests demonstrated to me that VSS's file and database management is so defect-ridden as to present a danger to customers using the product in reasonable scenarios and cirucmstances. Although it's an older article (circa 2002), it did turn out to be an excellent example of rapid and exploratory testing approaches, and an example of the kind of test report that I would issue to a client. Your mileage may vary, but these are my findings.
A Review of Error MessagesCreating a good error message is challenging. On the one hand, it needs to be informative, to assist the user, and to suggest reasonable actions to mitigate the problem. On the other hand, it needs to avoid giving hackers and other disfavoured users the kind of information that they seek to compromise security or robustness. Here are some suggestions.
Pairwise Testing and Orthogonal ArraysPairwise and orthogonal array test techniques may allow us to obtain better test coverage—or maybe not. Over the years, I've changed my views on these techniques. I explain all-pairs and orthogonal arrays here, and I then include some tempering of the basic story—and some counter-tempering too.


该贴已经同步到 seanhe的微博
ISTQB
发表于 2011-12-26 11:02:12 | 显示全部楼层
软件测试工程师就业班马上开班
小心大家盯上你哦  
ISTQB
发表于 2011-12-26 11:02:12 | 显示全部楼层
软件测试工程师就业班马上开班
满城尽待年终奖,弄弄才是喇叭腔  
发表于 2011-12-26 11:12:18 | 显示全部楼层
软件测试工程师就业班马上开班
说的真有道理啊!
发表于 2011-12-26 11:12:18 | 显示全部楼层
软件测试工程师就业班马上开班
哈哈,顶你了哦.  
发表于 2011-12-26 11:12:18 | 显示全部楼层
软件测试工程师就业班马上开班
风物长宜放眼量  
发表于 2011-12-26 11:12:18 | 显示全部楼层
拍脑袋决策,拍胸脯保证,拍屁股走人  
发表于 2011-12-26 12:14:39 | 显示全部楼层
呵呵,明白了  
发表于 2011-12-26 12:14:39 | 显示全部楼层
加油啊!!!!顶哦!!!!!  
发表于 2011-12-26 12:14:39 | 显示全部楼层
似曾相识的感觉  
发表于 2011-12-26 12:14:39 | 显示全部楼层
宁愿选择放弃,不要放弃选择。  
发表于 2011-12-26 12:57:58 | 显示全部楼层
#无语  
发表于 2011-12-26 12:57:58 | 显示全部楼层
有才的不在少数啊  
发表于 2011-12-26 12:57:58 | 显示全部楼层
其实回帖算是一种没德德,所以我快成圣人了  
发表于 2011-12-26 12:57:58 | 显示全部楼层
去市场买东西,杀价先杀四分之三,现在杀一半行不通了。  
发表于 2011-12-26 13:06:21 | 显示全部楼层
感謝樓主  
发表于 2011-12-26 13:06:21 | 显示全部楼层
厉害!强~~~~没的说了!  
发表于 2011-12-26 13:06:21 | 显示全部楼层
看贴回复是好习惯  
发表于 2011-12-26 13:06:50 | 显示全部楼层
非常感谢楼主,楼主万岁万岁万万岁!  
发表于 2011-12-26 13:06:50 | 显示全部楼层
去市场买东西,杀价先杀四分之三,现在杀一半行不通了。  
发表于 2011-12-26 13:06:50 | 显示全部楼层
你你你你 呵呵~~~~别这样哦.....  
发表于 2011-12-26 14:00:46 | 显示全部楼层
自己知道了  
发表于 2011-12-26 14:00:46 | 显示全部楼层
真好。。。。。。。。。  
发表于 2011-12-26 14:00:46 | 显示全部楼层
小生对楼主之仰慕如滔滔江水连绵不绝,海枯石烂,天崩地裂,永不变心.  
发表于 2011-12-26 14:00:46 | 显示全部楼层
一个人最大的破产是绝望,最大的资产是希望。  
发表于 2011-12-26 14:11:55 | 显示全部楼层
嘿嘿  
发表于 2011-12-26 14:11:55 | 显示全部楼层
非常感谢楼主,楼主万岁万岁万万岁!  
发表于 2011-12-26 14:11:55 | 显示全部楼层
这贴?不回都不行啊  
发表于 2011-12-26 14:11:55 | 显示全部楼层
不知道说些什么  
发表于 2011-12-26 14:11:55 | 显示全部楼层
我又回复了  
发表于 2011-12-26 14:20:27 | 显示全部楼层
勤奋真能造就财富吗?  
发表于 2011-12-26 15:22:23 | 显示全部楼层
这样的贴子,不顶说不过去啊  
发表于 2011-12-26 15:22:23 | 显示全部楼层
要不我崇拜你?行吗?  
发表于 2011-12-26 15:22:23 | 显示全部楼层
宁愿选择放弃,不要放弃选择。  
发表于 2011-12-26 15:22:23 | 显示全部楼层
一个有信念者所开发出的力量,大于99个只有兴趣者。  
发表于 2011-12-26 16:05:32 | 显示全部楼层
老大,我好崇拜你哟  
发表于 2011-12-26 16:05:32 | 显示全部楼层
很有吸引力  
发表于 2011-12-26 16:05:32 | 显示全部楼层
爷爷都是从孙子走过来的。  
发表于 2011-12-26 16:05:32 | 显示全部楼层
琴棋书画不会,洗衣做饭嫌累。  
发表于 2011-12-26 16:05:32 | 显示全部楼层
女人啊真是那句话~~~无理占三分~~忘记怎么说了 呵呵 ~~你帮我想想  
发表于 2011-12-26 17:52:49 | 显示全部楼层
顶你一下,好贴要顶!  
发表于 2011-12-26 17:52:49 | 显示全部楼层
楼主,支持!  
发表于 2011-12-26 17:52:49 | 显示全部楼层
任何的限制,都是从自己的内心开始的。  
发表于 2011-12-26 17:52:49 | 显示全部楼层
我假装认真地工作,老板假装认真地付薪水  
发表于 2011-12-26 18:01:06 | 显示全部楼层
慢慢来,呵呵  
发表于 2011-12-26 18:01:06 | 显示全部楼层
哈哈,看的人少,回一下  
发表于 2011-12-26 18:20:13 | 显示全部楼层
回帖是种美德.  
发表于 2011-12-26 18:20:13 | 显示全部楼层
楼主也是培训师吗  
发表于 2011-12-26 18:20:13 | 显示全部楼层
又看了一次  
发表于 2011-12-26 18:20:13 | 显示全部楼层
读10年语文,不如聊半年QQ  

本版积分规则

Archiver|手机版|小黑屋|领测软件测试网 ( 京ICP备10010545号-5 )

GMT+8, 2020-12-6 09:23 , Processed in 0.508486 second(s), 12 queries , Xcache On.

Powered by Discuz! X3

© 2001-2013 Comsenz Inc.

快速回复 返回顶部 返回列表