Why do we Test?

We gather objective information on the current quality of our deliverables in an efficient and effective way; and transform it into constructive feedback, so that the team can decide if our deliverables meet the project-quality-goals and is ready to be released to our customers, in a transparent way for our stakeholders.

Do we? … see if we can find an answer.

Note the question will only be addressed for a specific context.

Test Approach View

NOTE: This is not finished yet!

This view describes the high level principle that guide the Test Approach for this Test Framework. It shows how the Context of a project influences the Test Approach for a project. The Test Approach shows the (most important) high level principle that define the approach and the entities that support this. As a base, the Test Separation Principle shows that What to test and How to do it should be clearly divided. Do not rush into creating tests or even Test Levels, first get the What clear and then go to the How.

It also shows how we use this Test Principle is controlled by Project Scope and Requirements and/or Specification. The Life Cycle supports the Test Principle, Test Techniques should be used defining the What and How or testing. Being transparent on why to test, what needs to be tested, what is not tested and how the ‘what’ is covered in actual test activities or cases (and during test execution), guards that the correct aspects of the Product/System are tested and the information coming from the test activities has a high value and is a good substitute for the actual quality of the Product/System.

Project Context
Requirements / Specification
Test Separation PrincipleMake sure to divide the WHAT you want to test (to achieve the desired coverage level in the given context) from the HOW you achieve that goal. This helps to keep away from the bias towards what CAN I implement and keeps the focus on the identified risk and what should be done to cover it. It also gives an extra coverage view, how good have we covered the items that were identified as item that should be covered.
For example, if a Risk based Strategy is used, the WHAT is determined based on the questions raised by the Risk Assessment on the product (See also Test Strategy), the HOW on the information that is needed to answer them.
Test Strategy
Life Cycle
Test Techniques
Transparency

Views Overview

The views are still in development, some initial views are already defined to get started with the Framework. Time will tell if they are useful enough to be part of the Framework in the future.

  • The Test Approach View describes the high-level principle that guide the Test Approach for this Test Framework. The ‘highest’ view explains the most important blocks in the test environment for the Approach.
  • The Product Risk View helps to determine parts of the Test Strategy.
  • The Test Life Cycle View helps to group/order the identified activities (and serves as a peg for detailing the activities?).
  • More views will follow.

The Test Framework Structure

The Test Framework will be more like an Atlas, showing different views on testing, from different ‘heights’ and on different aspects. Creating a framework that contains everything might be possible, but is too complicated at this moment, this Framework is a first attempt to map the testing aspects relevant for a Test Engineer in a project.

The ‘Atlas’ can be divided into a few big chapters:

  1. Views: The views serve as overview-pictures at several ‘height’ of the framework. They might give a complete picture of the framework looked at from a certain perspective or a zoomed-in view on a specific aspect of the framework. The views show how everything is linked together and uses models to structure the ‘content’ of the view.
  2. Patterns: Analysis and Design Patterns describe analysis (what) and design (how) solutions for a specific context. The patterns can describe a Practice or Activity with Tasks that must be tailored to the specific context of your project/product/team.
  3. Techniques: Describes/lists techniques that can be used to implement (a set of) the Tasks that need to be done for the Test Approach defined for the project.
  4. Skills: Describes/lists Skills that are helpful or required to execute the Practices, Activities or Tasks defined in your Test Approach.
  5. Tools: Describes/lists Tools that are helpful to execute the Practices, Activities or Tasks defined in your Test Approach.
  6. Organization: Describes how the organization can support to create and execute your Test Approach.

How do we Test?

We formulated an answer to the Why do we Test question, but HOW do we do this? How do know that what we are doing is ‘good enough’. What is ‘good enough’ anyway?

How good, means that we need a way to measure, to measure we need a scale. A single measurement will probably not be sufficient, so let’s see if we can create a generic Test Framework to help us create different views that can measure different aspect on its own scale. This can then be our guide to help us determine (and measure) all the aspects of How to Test.

To discuss, compare and improve the ‘way-of-working’/approach in testing we need a Test Framework describing different aspects of testing so we can compare our own approach with the framework. Mapping it to the framework to view how (good) it covers the testing aspect. It also allows to map test-concepts from test literature or other (online) resources. The map can be used to discuss whether the current approach is good enough or/and find ways to improve current weak-spots in the test (and development) approach.

Let’s start ‘filling’ our Test Framework.

Making it more concrete

We can add some more details to the question to make it more concrete:

  1. We (as a team)
    • Testing is not a single person responsibility, certainly not in an Agile environment. The whole team should be involved, although different roles/skills are needed for different tasks within the test approach.
  2. gather objective information
    • Testing does not make a product better, it gathers information, which might show that the product quality (or any of the intermediate work products) is not good enough. There is different kind of information that can be useful for a project, it all depends on the goal of the project. It is good to document (and agree on) the questions that need to be answered with the information that is gathered by testing. These questions can best be based on perceived risk, which also indicates that the questions can change during the execution of the project. Measuring the quality of the product directly is not possible, so we use ‘proxy-metrics’, which come at a risk.
  3. on the current quality
    • Product quality is not a static property, it changes constantly over time. Features that were stable last week can become instable after a new code change. Information gathered on a previous release/sprint is not necessarily valid at this moment. An important part of quality of a product is the ‘remaining risk’ in the current release or any of the intermediate work products.
  4. of our (internal) deliverables,
    • The deliverables will be more than the software executable, it will most probably also contain some documentation (at least of the known defects), interface descriptions and other artifacts. Make sure the deliverables are known and considered in the chosen test strategy. Also the intermediate work products are assessed, like requirements/architecture/design/code/way-of-working/unit-test(-coverage). The deliverables are not limited to the ones that need to be made, but also to those that are made and e.g. used ‘in the field’.
  5. in an efficient and effective way
    • Resources are scarce so the test related activities must also be done efficient and effective, keeping the effort aligned with the risk the project is willing to take and the project-quality-goals.
  6. and transform it into constructive feedback
    • The raw information is (transparently) transformed to constructive feedback, to support the decision making by the team. It relates to the risk that has been identified during earlier activities and concerns of our stakeholders. The current ‘remaining risk’ is also addressed in the feedback.
  7. so that the (agile) team
    • Quality is the responsibility of the team, the team has to decide if the quality is good enough.
  8. can decide if our deliverables meet the project-quality-goals and is ready to be released( or needs to be fixed)
    • The project-quality-goals define the ‘good enough’  which is a reference that was created before or (adjusted) during the execution of the project. The ‘good enough’ is in partly determined by the needs/expectations of the customers (which might conflict) and partly by the needs of the team/project.
    • The project-quality-goals can tell something on the adherence to requirements or other forms of specification, or on the number of open defects that are allowed.
    • Post release (Field)-data can lead to a need to fix the released product/deliverables.
  9. to our customers,
    • Deliverables (or parts of deliverables) can go to different customers, it is important that we know who they are and what they expect from the deliverables. Part of the ‘good enough’ is determined by the needs/expectations of the customers.
  10. in a transparent way for our stakeholders.
    • It should be clear why decisions have been made (on requirements, architecture, design, test coverage/depth, ‘good enough’, …) and that we followed the steps that we agreed to do (‘process’), since the gathered information only has value if we can trace its origin. It also helps to improve our ‘process’ based on the feedback of our (development) performance and our stakeholders.

Does this get us any further in our investigation of the question: Why do we test?

So: How do we this?

Note the question will only be addressed for a specific context.

Context

On this website I will not try to solve all testing problems that exists, it is even more restricted to the project-structure and work domain(s) that I have spend my testing-years. So what is the context in which I am documenting this testing experience?

The Project

The structure of the project is layered, there are several ‘product’-teams who together create the full system which is delivered to the company’s customers. The teams use an Agile method to develop their product releases, incremental system feature development is aligned using SAFe.

The Domain

The system provides an IoT-solution with end-devices, gateways, front-end, back-end and data-components developed by different teams located on different continents.

Most of mine experience is in the embedded devices (end-devices, gateways) and the system (integration and testing).