What are the approaches of the software test

Approach to test case creation in test automation for acceptance test cases in the automotive industry

Table of Contents

1 Introduction

2. Software test

3. The approach for creating the acceptance test cases

4. Sources

1 Introduction

A complete in-house development of vehicle software by a manufacturer (OEM) is nowadays almost impossible due to the high requirements. Various know-how carriers are required during development, which makes the task of coordinating and integrating the software modules on the control unit in the vehicle more difficult. In addition, the high requirements for the correctness of the software must be observed.

This is confirmed in the study HAWK (Challenge Automobile Value Chain). The study predicts far-reaching changes in the value-added architecture in automotive engineering. Suppliers as know-how carriers are increasing their share. The manufacturer's share is expected to fall by almost a third to 25 percent by 20151.

The dependency of the OEMs on several suppliers increases the test complexity. Before the OEMs integrate the SW modules from various suppliers on software in the loop level (SiL level) or test them on hardware in the loop test benches (HiL test benches), the individual modules must be approved. The OEMs then concentrate on the one hand on the acceptance and system tests of the software modules of various suppliers and their integration and on the other hand on the integration of the delivered control unit in the entire vehicle. Embedded systems are increasingly becoming an integral part of safety-critical applications. You take over the control and regulation of complex internal processes and / or interact with the environment. In addition, these devices take on the task of networking distributed and simultaneously running processes between sender and receiver, which can bring about new types of errors. The proportion of errors in software in the medical sector or in the automotive sector is around 20 percent today, so quality assurance is of great importance. Testing is one of the most important pillars of quality assurance. A test is used to check whether the system has been implemented in accordance with the requirements. Various test strategies and concepts are necessary to achieve acceptable test coverage with regard to customer requirements, standards and legal requirements.

2. Software test

The prerequisite for the term software test is the term “error” in software systems. An error is the failure to meet a specified requirement for the software system. There is a discrepancy between the actual behavior and the target behavior. The actual behavior is determined during the operation of the software system. The target behavior is firmly anchored in the specification of the test items. In contrast to those in physical systems, errors in software systems do not arise from wear and tear or aging, but rather during the development of the software. The errors are retained and come into play when the system is executed. Testing has the task of reducing risks when using the software. This is done by uncovering as many errors as possible during testing.2

The aim of testing is to improve quality by correcting a fault condition. Testing is done by running a program in the control unit. You need a test case and a test plan for this. The task of a test case is to determine which entries are made in the program and what the result should be. You need many test cases to cover the entire program in the ECU.3 Test plan or test concept are usually drawn up according to the IEEE 829 standard.4 The content of a test concept according to IEEE 829 consists of the following points:5

Test concept label

The test concept must be marked to ensure that the document is clearly identifiable from other documents. The naming of the documents is archived in every company and specified in accordance with requirements and rules. The document must be stored in the document management system and contain at least the document name, version and release status.

introduction

In the introduction, the project and background are briefly described. The project participants (customer, tester, test manager and other roles) are shown. References to other documents, norms, standards and company rules must be included.

Test objects

The software components to be tested and parts of the system that are not tested should be briefly presented here.

Features to be tested:

This subchapter of the test concept takes into account functions and properties that are to be tested. The reference to the test definition and the test levels must be included.

Performance features that are not tested:

The aspects in the software that should not be tested must be listed here in order to prevent unjustified expectations.

Test strategy

The test objectives are defined on the basis of a risk analysis. Test methods and the necessary automation of the test tasks are defined, the selected strategies are checked for available resources and identified risks are weighed.

Acceptance criteria

Based on the tests carried out, the development team must decide whether the implemented software should be delivered to the customer or not. A short announcement that the software is error-free is not enough. Criteria such as "number of errors found", "severity of errors found" and "costs incurred" are used here.

Criteria for canceling and continuing the test

It can happen that the software does so badly in the first tests that further tests seem pointless. At this point, it must be considered which criteria are selected in order to abort test sequences.

Test documentation

This is where it is decided when and which test documents and results have to be communicated to whom.

Test tasks

The assignment of responsibilities as well as the planning and implementation of the test activities are determined.

Test infrastructure

Test tools, work equipment and test infrastructure must be listed. If special tools and resources are required for test automation, these must be listed.

Responsibilities / competencies

The organizational involvement of the test staff in the project and the division into different test groups and test levels must be planned.

Personnel, induction, training

The resources are planned. The staffing and qualifications are determined and the administration staff is briefed.

Schedule / work plan

The test activities are planned with milestones and communicated to the project manager.

Approval / release

The organizational units that have to approve the test concept are listed. Approval is given by the signature of the management level of an organizational unit.

glossary

Many different terms are used in the test area. These should be recorded in a glossary. The test terms should always be used in the project in order to avoid different interpretations of a specific term.

The classification of the test activities in software development processes is shown using the federal V-model (see Figure 1). The basic idea of ​​the V-Modell is that test activities and development activities are activities that correspond to one another and have equal rights.6 The left branch illustrates the developmental steps. The requirements are designed, specified and then implemented based on the customer's wishes. The right branch stands for integration and test work. The course describes the integration and the testing of the building blocks successively for the growth of the overall system. The left branch records the verifications after each completed step. At the same time, the tester validates whether a sub-product actually solves a specified task.

Figure not included in this excerpt

Figure 1: Federal V-model and test activities7

The main task of the acceptance test cases for the software modules from different suppliers is to check whether they are mature or not for the next test levels in the left branch of the V-model, i.e. Sil integration and HiL tests. Since SiL integration. HiL tests or even system tests involve a lot of effort and costs, the maturity of the software modules is of great importance. At this point it is necessary to remove the software modules in order to check their suitability for the further test levels in the right branch of the V-model. To counteract this problem, approaches are required that take into account the special requirements of control unit software in the automotive industry and, on the other hand, facilitate the acceptance of the software modules. This article presents an approach to the acceptance of software modules. The approach considers manual test case creation in order to derive acceptance test cases for OEMs from the requirements.

3. The approach for creating the acceptance test cases

The approach is based on the integration of the test case creation method "classification tree method" and the "design by contract" concept (DBC). The DBC is based on the definition of Assumptions and Promises for each software module. This means that each software module concludes a contract with other modules. In this contract, the module accepts something and promises to make something available to other modules if its acceptance is fulfilled8. In contrast, the classification tree method starts with the analysis of the functional specification. The aim is to identify the so-called test-relevant aspects of the requirements. By sensible selection of the values ​​that these aspects can assume, one expects differently relevant test cases. This division into different aspects, which can be refined separately below, considerably reduces the complexity of the original test problem9.

In the approach, DBC is used to select the test-relevant aspects and categories in the classification tree method. In the classification tree, no functional categories are considered, but the contracts defined by the DBC. The division of the classification tree into aspects is replaced by DBC contracts. The contracts are divided into Assumptions and Promises. The test case creator must always think of the contract that a software module can conclude with the other modules in the entire architecture of the ECU software.

This division of the tree into different contracts, which can be refined separately in equivalence classes, considerably reduces the complexity of the original test problem. A contract offers the required test data through the assumptions, the promise on the other hand represents the target behavior. Nevertheless, this way of thinking offers a simplification in comparison to the original approach of the classification tree method, since the method prescribes a division of the tree into aspects without doing so to propose a concrete approach to the selection and derivation of the aspects.

This approach enables the creation of all necessary test cases that are required for the acceptance of a correct, integrable and functioning software module, if all the assumptions and their promises of the software module are considered. It is independent of the test levels used and can be used, for example, for acceptance tests on SiL (software in the loop), HiL (hardware in the loop) or overall system level. The classification tree editor CTE / XL can be used to evaluate the approach10 and used in the context of a project.

The approach presented belongs to the category of function-oriented testing. By using the formal “DBC” approach and the classification tree method, the approach meets the requirements of the quality assurance norms and standards that exist in the context of testing software in safety-critical applications such as the automotive industry. The function-oriented approach used is suitable for creating the acceptance test cases. The approach can be seamlessly integrated into the acceptance phase of software modules in the development process.

The approach not only supports OEM in the acceptance of software modules, but also offers a simple procedure for the tester in his task of creating acceptance test cases from requirement specifications. The advantages that distinguish the approach over other test procedures and approaches prevailing in the industry are as follows:

- Simple method and intuitive test case creation approach
- Concentration on the acceptance test cases
- Early detection of consistent and consistent conditions
- Reduction of the test problem
- Use of formal methods in test case creation
- Simplification of the derivation of test cases from the specification sheet
- Use of test tools already established on the market
- Early detection of architecture and specification errors
- Reuse of created test cases

4. Sources

[1]: http://www.automobil-produktion.de/themen/02430/index.php

[2]: http://de.wikipedia.org/wiki/Design_by_contract

[3]: Schneider, Kurt (2007): Adventure software quality, dpunkt-Verlag, Heidelberg

[4]: http://www.systematic-testing.com

[5]: Spillner, Andreas; Linz, Tilo (2006): Basic knowledge software test, dpunkt.verlag.

[...]



1 See http://www.automobil-produktion.de/themen/02430/index.php

2 See Andreas / Tilo (2006), p. 7.

3 See Kurt (2007), p. 83.

4 See Andreas / Tilo (2006), p. 223.

5 See Andreas / Tilo (2006), p. 223.

6 See Andreas / Tilo (2006), p. 39.

7 See Andreas / Tilo (2006), p. 40.

8 See http://de.wikipedia.org/wiki/Design_by_contract

9 See Schneider, Kurt (2007)

10 See http://www.systematic-testing.com