Resources

Lecture Slides

These will be published progressively here during the unit.

Topics:

1. Unit info, testing and QA concepts
   Testing concepts
2. Introduction to testing; unit testing
3. Test automation
4. Model-based testing, Input space partitioning
5. Graph-based testing
6. Logic-based testing
7. Syntax-based testing
8. System and integration testing
9. Risk, standards and metrics
10. Intro to formal methods
11. Specification languages
12. Sample exam questions
    Model solutions

Workshop material

These will be published progressively here during the unit.

1. Workshop 1 -- Testing concepts
   Workshop 1 -- Sample solutions
2. Unit tests
   BlueJ walk-through
3. ISP and graph-based testing
   Workshop 3 -- Sample solutions
4. Logic-based testing
   Notes from workshop
5. Syntax-based testing
   Workshop 6 -- Sample solutions
6. Data-driven testing
   Sample solutions
7. Risk assessment
   Sample workshop solutions
   Exercise solutions
8. Formal methods
   Sample solutions
9. Specification languages
   Sample solutions

---

Reading

Testing

Material on the theory of testing is mostly drawn from:

• Paul Ammann and Jeff Offutt, Introduction to Software Testing, 2nd Ed, Cambridge University Press, 2016 [Description on Amazon]

For material on the Python testing frameworks, see the Python documentation for the unittest, doctest and Hypothesis libraries. More information on them, and on test-driven development in Python are available in the following places:

• Unit Testing in Python
• A tutorial on test-driven development in Python (older, but still useful)
• Improve Your Python: Understanding Unit Testing

Although it uses the Java language and Java libraries, a good, practical guide to test-driven development is:

• Viktor Farcic and Alex Garcia, Test-Driven Java Development, Packt Publishing 2015. [Description on Amazon]

It’s not directly used in the unit, but can be a useful source for seeing how test-driven development is done in practice. Another very useful source on good software practices – which does cover testing, but also many other aspects of software development, is:

• Andrew Hunt and David Thomas, The Pragmatic Programmer, Pragmatic Bookshelf, 1999. [Description on Wikipedia]

There is no need to buy it or read it, but the unit material will sometimes draw on the authors’ way of explaining things.

Roy Osherove provides a good set of guidelines to bear in mind when reviewing your own (or someone else’s) unit tests.

Code coverage tools

In the lectures and workshops, we examine coverage criteria for tests, but do not focus on particular tools that measure the code coverage of an existing set of tests. Code coverage tools exist for nearly every language, however, and for your own interest, you may wish to experiment with some. For Python, the coverage tool (found here) is commonly used; for Java, one of the best-known is JCov.

Call-graph generation tools

We do not cover the use of particular tools that generate visualizations of call graphs for a body of code, but these exist for many languages. If you are interested, you may wish to investigate java-callgraph, which can generate call graphs for Java programs.

Mutation testing packages

In lectures, we mention two mutation testing packages, PIT for Java, and mutpy for Python. Mutation testing packages for many other languages (together with blog posts and papers on the topic of mutation testing) are listed at https://github.com/theofidry/mutation-testing.

Web testing in Python

The book Test-Driven Development in Python provides an excellent guide to test-driven development of web applications in Python. It is available to read for free online, as well as being available in hard-copy and e-book versions. There is far more material on test-driven web development than we will need for this unit, but it may provide helpful background reading.

In particular, it provides a very useful guide on using the Selenium web-testing framework from within Python, and performing unit-testing in the Django web application framework.

QM, standards, risk management, metrics

For the topics of quality management, standards, risk management, and metrics, the material is largely taken from:

• Pressman, R. Software Engineering: A Practitioner’s Approach. McGraw-Hill, 2005

Any later edition is also fine.

Formal methods and formal specification

For the topics of formal methods and formal specification, the unit briefly covers use of the Alloy specification language and model analysis tool, described in the following book:

• Daniel Jackson, Software Abstractions: Logic, Language, and Analysis, MIT Press, 2012. [Description on Amazon]

You do not need to purchase the book; the material on the Alloy website will be sufficient for our purposes.

For students who are interested – although this goes beyond what we cover in the unit – the freely available textbooks in the series “Software Foundations” provide an excellent guide to using formal methods to improve the reliability of software systems. However, working through those would require a whole semester, and we only have 2 weeks to spend on the topic. (Some more excellent resources on formal methods are listed at https://avigad.github.io/formal_methods_in_education/.)

Students may also be interested in the Dafny language, which includes a Hoare logic–style verifier as part of its compiler.

Alloy specification language and analyzer

Alloy is well-documented, works well for analysing object-oriented systems, and has a freely available graphical tool for analysing models (available at https://alloytools.org/download.html). It is possible to get started using the tool quite quickly, even before knowing a great deal about the language. In addition to the textbook, there is an online tutorial on using Alloy here: https://alloytools.org/tutorials/online/. An alternative tutorial is available at https://www.doc.ic.ac.uk/project/examples/2007/271j/suprema_on_alloy/Web/.

To install the Alloy analyzer:

1. Download version 4.2 of the Analyzer from the main Alloy website, at https://alloy.mit.edu, under “download”.

   If you have a Mac, you can use the .dmg installer file; if you use Linux or Windows, you need to download the .jar file.

2. You will need to ensure you have a Java runtime installed. On Windows or Mac, if you are not sure whether you do, you can visit https://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html, and download an appropriate Java SE Runtime Environment 8u211.

   On Linux systems: a Java runtime can typically installed using your distribution’s package manager. For instance, on recent versions of Ubuntu, sudo apt install openjdk-8-jre will install one.

3. On a Mac, double-click the .dmg file and follow the instructions.

   On Linux or Windows: once you have installed a Java runtime, simply doublle-click the .jar file you have downloaded, and the analyzer should launch.

Alloy quick reference sheets

For alloy syntax all on one page, try visiting https://thomasalspaugh.org/pub/fnd/alloy.html (web page), or viewing https://www.monperrus.net/martin/alloy-quick-ref.pdf (PDF).

If you’re unsure how to express something in Alloy, these may suggest some possibilities.

Additional resources

These are not required reading for the unit, but may be useful for your self-study or in completing assessments.

Python resources

For a solid coverage of how classes, objects and methods work in Python, I’d suggest looking at chapters 16 and 17 of the interactive course “How To Think Like a Computer Scientist”:

• https://interactivepython.org/runestone/static/thinkcspy/ClassesBasics/toctree.html
• https://interactivepython.org/runestone/static/thinkcspy/ClassesDiggingDeeper/toctree.html

For students who are already very familiar with how classes, objects and methods work in other languages, the official Python tutorial has a shorter explanation of how they work in Python, here:

• https://docs.python.org/3/tutorial/classes.html

Test suite examples

The SQLite database engine has a quite comprehensive set of tests, described here.

Techniques used include:

• unit testing
• coverage measures – the page reports 100% branch test coverage, and 100% modified condition/decision coverage (MC/DC) coverage
• performance tests – these assess non-functional requirements of the engine (i.e., not what it does, but how it does it)
• fuzz testing, an example of random testing
• boundary testing, an example of “black box” testing which focuses on values which at the extreme of their domain.

and many others.

Case studies

The website for the 10th edition of Ian Sommerville’s Software Engineering text contains a number of excellent case studies relevant to testing and quality assurance. They illustrate issues related to:

• requirements specification
• validation of critical systems
• engineering for reliability, dependability and safety

They include a description of the Ariane 5 launch accident (in which an Ariane 5 rocket exploded on its maiden flight), the flight control system for the Airbus 340 jet airliner, and several safety- or privacy-critical systems.

Code review checklists

Erik Dietrich provides guidelines on creating your own code review checklist, including suggestions on what items should be used by a code author versus a code reviewer.

Some specific checklists you may wish to investigate include:

• A code review checklist by Gareth Wilson, of Fog Creek Software (a New-York based consultancy founded by the creators of Stack Overflow)
• A checklist by Surender Reddy Gutha, of Evoke Technologies

Guido van Rossum, creator of the Python programming language, gave a presentation about Mondrian, a web-based code review system he developed, written in Python, and used within Google.

NASA’s Office of Safety and Mission Assurance produced a Software Formal Inspections Guidebook (1993) which you may find useful.

Coding standards and guidelines

Coding standards and guidelines are often used as the basis for code review checklists.

Some you may wish to investigate include:

• the GNU Coding Standards

• the Linux kernel coding style documentation, which recommends, inter alia, that you print off a copy of the GNU Coding Standards and burn it without reading them.

• US National Weather Service Office of Hydrologic Development, Software Development Standards and Guidelines

• the “MISRA C” guidelines, a set of 127 guidelines for the use of C in safety-critical systems, developed by the UK’s Motor Industry Software Reliability Association.

• Google’s Style Guides, for languages including:

  • C++
  • Java
  • Python
  • JavaScript

Debugging resources

Debugging techniques are covered extensively in Andreas Zeller’s book “Why Programs Fail”. There is now also a free Udacity online course associated with the book.

Software Quality Assurance (SQA) Plans

A number of real-world and sample SQA plans are available on the web. Some of them assume a fair amount of background knowledge, but are still useful references for seeing how such plans are typically structured.

• The case studies developed by student teams, and available on the website for Roger Pressman’s software engineering textbook, provide examples of SQA plans.
• Pressman provides a structural template for an SQA plan.
• The US Department of Energy provides a sample SQA plan based an actual Department of Energy project, and an SQA plan used by the Center for Space Research. The DoE Office of Scientific and Technical Information also provides a sample Software quality assurance handbook.
• EOSDIS is software used by NASA to manage data from sources including satellites, aircraft and field measurements. Documents relating to the maintenance and development of this software – including an SQA plan – are available from the NASA website.

Static analysis tools

Although we do not cover them in detail in this unit, many tools are available for analysing code and computing code metrics. Some examples:

• PMD is an extensible, cross-language, static code analyser. It scans source code in Java and other languages and looks for potential problems like:

  • potential bugs - e.g. empty try/catch/finally/switch statements
  • dead code - unused local variables, parameters and private methods
  • suboptimal code - wasteful String/StringBuffer usage
  • overcomplicated expressions - unnecessary if statements, for loops that could be while loops

  Custom detection rules can be written using XML and/or Java. PMD also provides the ability to calculate code metrics.

• FindBugs is software originally developed by the University of Maryland, which uses static analysis to look for bugs in Java code. Similar to PMD, it allows custom “bug filters” to be written.

• Coverity comprises a suite of static and dynamic software analysis tools, marketed by Synopsis. In addition to providing software commercially, Synopsis provides, as a free service to open-source projects, free analysis using its “Coverity Scan” static analysis product.

You may also find it interesting to read “A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World”, an article by the (academic) authors of Coverity on the process of commercializing their static bug-finding tool from a research project.