Automatic Test Generation for Function Block Diagrams

A Controlled Experiment in Testing of Safety-Critical Embedded Software


In engineering of safety critical systems, regulatory standards often put requirements on both traceable specification-based testing, and structural coverage on program units. Automated test input generation techniques can be used to generate test data to cover the structural aspects of a program. However, there is no conclusive evidence on how automated test input generation compares to manual test generation, or how testing based on the program implementation relates to specification-based testing.In this paper, we investigate specification- and implementation-based testing of embedded software written in the IEC 1131-3 language, a programming standard used in many embedded safety critical software systems. Further, we measure the efficiency and effectiveness in terms of fault detection. For this purpose, a controlled experiment was conducted, comparing tests created by a total of twenty-three software engineering master students. The participants worked individually on manually designing and automatically generating tests for two IEC 1131-3 programs. Tests created by the participants in the experiment were collected and analyzed in terms of mutation score, decision coverage, number of tests, and testing duration.We found that, when compared to implementation-based testing, specification-based testing yields significantly more effective tests in terms of the number of faults detected. Specifically, specification-based tests more effectively detect comparison and value replacement type of faults, compared to implementation-based tests. On the other hand, implementation-based automated testing leads to fewer tests (up to 85% improvement) created in shorter time than the ones manually created based on the specification.


Enoiu, E. P., Causevic, A., Sundmark, D., & Pettersson, P. (2016). A Controlled Experiment in Testing of Safety-Critical Embedded Software. In IEEE International Conference on Software Testing, Verification and Validation.


Full-text available at


author = {Eduard Paul Enoiu and Adnan Causevic and Daniel Sundmark and Paul Pettersson},
title = {A Controlled Experiment in Testing of Safety-Critical Embedded Software},
note = {Acceptance rate 35/130=26{\%}, submitted 2015},
month = {April},
year = {2016},
booktitle = {IEEE International Conference on Software Testing, Verification and Validation},
url = {}