Error guessing
In software testing, error guessing is a test method in which test cases used to find bugs in programs are established based on experience in prior testing.[1] The scope of test cases usually rely on the software tester involved, who uses experience and intuition to determine what situations commonly cause software failure, or may cause errors to appear.[2] Typical errors include divide by zero, null pointers, or invalid parameters.[3]
Error guessing has no explicit rules for testing; test cases can be designed depending on the situation, either drawing from functional documents or when an unexpected/undocumented error is found while testing operations.[1]
Types of errors
[edit]
There are countless software defects that may be encountered upon using this technique. However, here are the most common ones, grouped into categories[4]:
Implementation-level errors
Implementation errors relate to handing specific data conditions.
- Computation error: When the program runs but does not work as intended.
- Domain error: When the branching logic derails the program entirely.
- Subcase error: When a case that is required is missing. A developer must pay attention to extreme values and limits.
- Forgetting error: When a variable is hard to keep track on what it is supposed to represent.
- Edge case error: Rare cases that could involve initial, final, and empty cases.
- Array bounds error: When there are references to out-of-bounds values in loops.
- Input stream error: When the program reads past the intended boundary of the given data due to being given faulty input. For example, empty fields between adjacent separators in a token-delimited file.
- Loop termination error: When a loop's exit is not carefully attended to, causing potential defects that appear under certain use cases rather than all possible loops.[5]
- UI/UX error: When there are problems with the user interface, experience, and design.[5]
- Performance error: When the program's time to execute results goes poorly, lags a lot, or simply crashes under too much load.[5]
- Security error: When there are vulnerabilities that could lead to unauthorized access or data breaches. [5]
- Compatibility errors: When a program behaves differently across different browsers, devices, operating systems, or environments.[5]
User-oriented errors
The second category of errors relates to a testers' or developers' inability to give attention to all areas of the program at once. These errors may not always break the program, but they may adversely affect the quality of the code and ability to build off of for future development as well.[4]
- Requirements focus error: When there are not enough testing cases that represent all possible test cases of a program.
- Unexpected user action error: When the user does an action the developer did not account for.
- Forgetting error: When a variable in the program is hard to keep track of what it is supposed to represent.
History
[edit]Glenford Myers, an American computer scientist, was the first to document the technique in his 1979 book The Art of Software Testing.[6] He presented the technique as an intuitive method built on the tester's experience.[6]
Myers described the technique with a small list of situations that commonly occur across programs. A common example of this is the value zero. Wherever a program accepts input or produces output, zero should be tested as both an input and an output, since zero is a common cause of errors such as division by zero.[6] The empty case should be handled the same way. A program that averages a list of numbers may work as expected but fail when the list is empty, because the count of the list is used as the divisor.
Use in practice
[edit]Error guessing appears both in the software testing literature, as well as in industrial practice, where it is applied alongside formal techniques like boundary-value analysis and equivalence partitioning. Error guessing is used in conjunction with formal techniques instead of in place of them.[7] How well it works depends heavily on the skill of the tester involved.[7]
A more structured form of the technique is the fault attack. This is where the tester writes out a list of defects and failures that seem likely, then designs a test for each one on the list.[7] This is why published catalogs exist. Testing Computer Software, by Cem Kaner, Jack Falk, and Hung Quoc Nguyen, has an appendix that catalogs common software errors.[8] In 2003, Kaner later went on to argue, with Giri Vijayaraghavan, that a catalog like this one is a good source of ideas to test.[9] The tester picks a failure mode from the list and works out what input would cause it. In the same year, James A. Whittaker published How to Break Software. The book is a numbered series of attacks.[10] The first attack tells the tester to supply inputs that force every error message in the program to appear. Later attacks cover input buffer overflows and invalid file names.
Video game studios also use open betas as a strategy to find errors, where participants explore the game freely and report feedback as they go.[11] In the same survey, Cristiano Politowski, Fabio Petrillo and Yann-Gaël Guéhéneuc found that studios test their games almost entirely by hand, and that the testers are usually people who already know the game well. Very little of the work is automated.[11]
Artificial intelligence and error guessing
[edit]Since error guessing depends on a tester's intuition about where a program is likely to fail, it has drawn interest for partial automation by machine learning and artificial intelligence. Large language models (LLMs), which are trained on large collections of text and code, can read a function and its requirements written in natural language to propose test cases for it, and researchers have studied their use for generating unit tests, test inputs, and test oracles.[12]
The existing literature on applying LLMs to test case generation can be described in four major categories: prompt engineering and refinement, feedback-driven loops that repair or regenerate tests after errors are detected, fine-tuning or pre-training a model on code and its associated tests, and hybrid setups that pair a model with an existing testing tool.[13] In the hybrid approach, the model is used to suggest new inputs when the conventional tool stops finding additional coverage.[13] These AI tools are usually delivered as plugins for integrated development environments or as steps in a continuous integration pipeline.[12]
Reported benefits include faster test production, broader branch coverage, and tests that human reviewers find more readable compared to traditional methods and tools.[12] LLMs have also proven useful for reproducing faults described in bug reports, a task that resembles the intuition a human uses when investigating a failure.[12]
Limitations include hallucinated identifiers, incorrect assertions, and missed edge cases.[12] These mistakes are associated with saturating the effective context window of the LLM, which limits the amount of code the model can consider at once.[12][13] Given the current limitations of these models, researchers generally conclude that they should be used as tools to complement human testing rather than being a replacement.[12]
References
[edit]- 1 2 Bernard Homès, Fundamentals of Software Testing (2013), sec. 4.5.3.
- ↑ R.G. Evans, Supercomputational Science (2012), p. 39.
- ↑ Mosley, Daniel J.; Posey, Bruce A. (2002). Just Enough Software Test Automation. Prentice Hall Professional. ISBN 978-0-13-008468-2.
- 1 2 Howden, William E. (July 2011). "Error-Based Software Testing and Analysis". 2011 IEEE 35th Annual Computer Software and Applications Conference Workshops: 161–167. Bibcode:2011cmps.conf...50H. doi:10.1109/COMPSACW.2011.36. ISBN 978-1-4577-0980-7.
{{cite journal}}: CS1 maint: periodical has ISBN (link) - 1 2 3 4 5 "Bugs in Software Testing". GeeksforGeeks. 2022-03-24. Retrieved 2026-08-09.
- 1 2 3 Myers, Glenford J. (1979). The Art of Software Testing. New York: Wiley. ISBN 0-471-04328-1. LCCN 78012923. OCLC 4194539.
- 1 2 3 Certified Tester Foundation Level Syllabus v4.0.1 (PDF) (Report). International Software Testing Qualifications Board. 2024. pp. 43–44.
- ↑ Kaner, Cem; Falk, Jack; Nguyen, Hung Quoc (1999). Testing Computer Software (2nd ed.). New York: Wiley. ISBN 978-0-471-35846-6.
- ↑ Vijayaraghavan, Giri; Kaner, Cem (2003). Bug taxonomies: Use them to generate better tests (PDF). Software Testing Analysis and Review Conference (STAR East).
- ↑ Whittaker, James A. (2003). How to Break Software: A Practical Guide to Testing. Boston: Addison-Wesley. ISBN 0-201-79619-8.
- 1 2 Politowski, Cristiano; Petrillo, Fabio; Guéhéneuc, Yann-Gaël (May 2021). "A Survey of Video Game Testing". 2021 IEEE/ACM International Conference on Automation of Software Test (AST). pp. 90–99. arXiv:2103.06431. Bibcode:2021ast..conf...18P. doi:10.1109/AST52587.2021.00018. ISBN 978-1-6654-3567-3.
- 1 2 3 4 5 6 7 Tasarsu, Murat; Tokmak, Ahmet Vedat; Catal, Cagatay (2026-03-30). "Test case generation using large language models: a systematic literature review". Cluster Computing. 29 (4): 227. doi:10.1007/s10586-026-06021-z. ISSN 1573-7543.
- 1 2 3 Celik, Arda; Mahmoud, Qusay H. (2025-09-09). "A Review of Large Language Models for Automated Test Case Generation". Machine Learning and Knowledge Extraction. 7 (3): 97. doi:10.3390/make7030097. ISSN 2504-4990.