The University of Western Australia
School of Computer Science and Software Engineering
 
 

School of Computer Science and Software Engineering

CITS5501 Software Testing and Quality Assurance

Useful links




Project Management Proverbs

Nothing is impossible for the person who doesn't have to do it themselves.

The sooner you begin coding the later you finish.

Any project can be estimated accurately (once it's completed).

The most valuable and least used WORD in a project manager's vocabulary is "NO".

The most valuable and least used PHRASE in a project manager's vocabulary is "I don't know".

You can con a sucker into committing to an impossible deadline, but you cannot con him into meeting it.

At the heart of every large project is a small project trying to get out.

Resources

This page lists additional resources and material which may be helpful in understanding the topics covered in CITS5501. The only required materials are the lecture slides, the workshop sheets, and freely available resources linked to from them (or from the unit schedule); but the material here may aid your understanding.

Unit topics

This section gives the sources used in developing the course content for each topic, and useful additional reading.

Testing

Material on the theory of testing is mostly drawn from:

  • Paul Ammann and Jeff Offutt, Introduction to Software Testing, 2nd Ed, Cambridge University Press, 2016 [Description on Amazon]

The first edition should also be fine. However, note the textbook is rather dense and theory-heavy – in the lecture and lab material, we’ll be using examples and case studies to elucidate it.

For material on the Python testing frameworks, see the Python documentation for the unittest, doctest and Hypothesis libraries. More information on them, and on test-driven development in Python are available in the following places:

Although it uses the Java language and Java libraries, a good, practical guide to test-driven development is:

  • Viktor Farcic and Alex Garcia, Test-Driven Java Development, Packt Publishing 2015. [Description on Amazon]

It’s not directly used in the unit, but can be a useful source for seeing how test-driven development is done in practice. Another very useful source on good software practices – which does cover testing, but also many other aspects of software development, is:

  • Andrew Hunt and David Thomas, The Pragmatic Programmer, Pragmatic Bookshelf, 1999. [Description on Wikipedia]

There is no need to buy it or read it, but the unit material will sometimes draw on the authors’ way of explaining things.

Roy Osherove provides a good set of guidelines to bear in mind when reviewing your own (or someone else’s) unit tests.

Code coverage tools

In the lectures and workshops, we examine coverage criteria for tests, but do not focus on particular tools that measure the code coverage of an existing set of tests. Code coverage tools exist for nearly every language, however, and for your own interest, you may wish to experiment with some. For Python, the coverage tool (found here) is commonly used; for Java, one of the best-known is JCov.

Call-graph generation tools

We do not cover the use of particular tools that generate visualizations of call graphs for a body of code, but these exist for many languages. If you are interested, you may wish to investigate java-callgraph, which can generate call graphs for Java programs.

Mutation testing packages

In lectures, we mention two mutation testing packages, PIT for Java, and mutpy for Python. Mutation testing packages for many other languages (together with blog posts and papers on the topic of mutation testing) are listed at https://github.com/theofidry/mutation-testing.

Web testing in Python

The book Test-Driven Development in Python provides an excellent guide to test-driven development of web applications in Python. It is available to read for free online, as well as being available in hard-copy and e-book versions. There is far more material on test-driven web development than we will need for this unit, but it may provide helpful background reading.

In particular, it provides a very useful guide on using the Selenium web-testing framework from within Python, and performing unit-testing in the Django web application framework.

QM, standards, risk management, metrics

For the topics of quality management, standards, risk management, and metrics, the material is largely taken from:

  • Pressman, R. Software Engineering: A Practitioner’s Approach. McGraw-Hill, 2005

Any later edition is also fine.

Formal methods and formal specification

For the topics of formal methods and formal specification: in previous years, the unit covered use of the Z notation, a formal specification language. Although still used, use of Z notation is not especially common nowadays, and the unit will instead briefly cover use of the Alloy specification language and model analysis tool, described in the following book:

  • Daniel Jackson, Software Abstractions: Logic, Language, and Analysis, MIT Press, 2012. [Description on Amazon]

Alloy is well-documented, works well for analysing object-oriented systems, and has a freely available graphical tool for analyzing models (available at http://alloytools.org/download.html). It is possible to get started using the tool quite quickly, even before knowing a great deal about the language. Compared with Z, Alloy is a less powerful language – there are things that can be expressed in Z, but not in Alloy – but Alloy has the advantage that it is simpler to learn and use. In addition to the textbook, there is an online tutorial on using Alloy here: http://alloytools.org/tutorials/online/.

For students who are interested – although this goes beyond what we cover in the unit – the freely available textbooks in the series “Software Foundations” provide an excellent guide to using formal methods to improve the reliability of software systems. However, working through those would require a whole semester, and we only have 2 weeks to spend on the topic. (Some more excellent resources on formal methods are listed at https://avigad.github.io/formal_methods_in_education/.)

Students may also be interested in the Dafny language, which includes a Hoare logic–style verifier as parts of its compiler.

Additional resources

These are not required reading for the unit, but may be useful for your self-study or in completing assessments.

Python resources

For a solid coverage of how classes, objects and methods work in Python, I’d suggest looking at chapters 16 and 17 of the interactive course “How To Think Like a Computer Scientist”:

For students who are already very familiar with how classes, objects and methods work in other languages, the official Python tutorial has a shorter explanation of how they work in Python, here:

Test suite examples

The SQLite database engine has a quite comprehensive set of tests, described here.

Techniques used include:

and many others.

Case studies

The website for the 10th edition of Ian Sommerville’s Software Engineering text contains a number of excellent case studies relevant to testing and quality assurance. They illustrate issues related to:

  • requirements specification
  • validation of critical systems
  • engineering for reliability, dependability and safety

They include a description of the Ariane 5 launch accident (in which an Ariane 5 rocket exploded on its maiden flight), the flight control system for the Airbus 340 jet airliner, and several safety- or privacy-critical systems.

Code review checklists

Erik Dietrich provides guidelines on creating your own code review checklist, including suggestions on what items should be used by a code author versus a code reviewer.

Some specific checklists you may wish to investigate include:

Guido van Rossum, creator of the Python programming language, gave a presentation about Mondrian, a web-based code review system he developed, written in Python, and used within Google.

NASA’s Office of Safety and Mission Assurance produced a Software Formal Inspections Guidebook (1993) which you may find useful.

Coding standards and guidelines

Coding standards and guidelines are often used as the basis for code review checklists.

Some you may wish to investigate include:

Debugging resources

Debugging techniques are covered extensively in Andreas Zeller’s book “Why Programs Fail”. There is now also a free Udacity online course associated with the book.

Software Quality Assurance (SQA) Plans

A number of real-world and sample SQA plans are available on the web. Some of them assume a fair amount of background knowledge, but are still useful references for seeing how such plans are typically structured.

Static analysis tools

Although we do not cover them in detail in this unit, many tools are available for analysing code and computing code metrics. Some examples:

  • PMD is an extensible, cross-language, static code analyzer. It scans source code in Java and other languages and looks for potential problems like:

    • potential bugs - e.g. empty try/catch/finally/switch statements
    • dead code - unused local variables, parameters and private methods
    • suboptimal code - wasteful String/StringBuffer usage
    • overcomplicated expressions - unnecessary if statements, for loops that could be while loops
    Custom detection rules can be written using XML and/or Java. PMD also provides the ability to calculate code metrics.
  • FindBugs is software originally developed by the University of Maryland, which uses static analysis to look for bugs in Java code. Similar to PMD, it allows custom “bug filters” to be written.
  • Coverity comprises a suite of static and dynamic software analysis tools, marketed by Synopsis. In addition to providing software commercially, Synopsis provides, as a free service to open-source projects, free analysis using its “Coverity Scan” static analysis prouct.

You may also find it interesting to read “A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World”, an article by the (academic) authors of Coverity on the process of commercializing their static bug-finding tool from a research project.

Advice on giving a technical presentation