Test Case Generation - Model Checker

Introduction

In all software development process the time really matters. This is the aspect which is valued as a high-cost.

When the beginner programmers start to learn programming languages, they try to learn basic syntax, some language related paradigms as well. They are so enthusiastic, doing their best to acquire knowledge and study. Some time passes, and now they can write basic to-do list application, it can be mobile, web-based or desktop. At this moment, they don't care about the tests, since they can't evaluate how important can the test be. This is the first phase of every single programmer. The second phase for them is to find a job or internship in a company, to gain more experience and earn experience and money. So this is the time when they encounter real project structure and processes how the big projects are done. Understanding the different people responsibilities, separation of the business side and development teams. Here they see the values of quality assurance(QA), that QAs are responsible for the quality, testing properly the applications, report bugs etc.

So, to say in different words, the second phase is the time for most of the programmers where they ask the question: Do we really need tests? Why should I spend the time in writing the tests which will test my own code? This question is not answered in a small period of time, the one needs particular knowledge in programming and development process to clearly see how important the tests can be.

The main argument for second phase programmers not to write tests is time consumption. They say they could do more valuable tasks meanwhile, instead of writing the tests. So in that case they are right, writing the tests can be time consuming, which is extra cost for the company. But in reality, testing is essential for achieving high quality and standard software.

This paper is intended to review and analyse how much will we win if we automate tests. The perfect example where the strength of test automation is visible.

The interesting fact here is the creation time, when automating the tests we need more time to create tests and prepare the system for automation, but then we significantly win in execution time.

Not only above mentioned is the intention but also how sufficiently we can generate test cases and reduce the time consumption. One of the most interesting part is code coverage, how well will our generated tests will cover the code. Is every case of your program flow included or not?

So my main purpose is first to gather significant information about test generation and code coverage, and if possible then find some different ways to fill the failures for test case generation.

The main methodology which I will talk about is model checker. Model checker method will work if the software product (code, specification, requirements model)  can be represented as a finite state transition system.

Taking Test Automation Seriously

Software testers sometimes are in a hurry and they do not have enough time to write tests, so they are avoiding the fact that testing should be implemented. Their distress lead them to look for a "silver bullet" that will help them recover a discerning pace to their work. Test automatisation is frequently their choice. The dream is that it will make their development process less complex and less demanding and help them to cope with unreasonable timetables.

Automated tests for a graphical user interface are quite difficult to implement. Since day by day graphical user interface(GUI) is becoming more and more complex, interfaces are changing daily during the development processes. Fortunately, significant work has been done in previous years to build tools which will automate tests, they are creating the GUI tests for us, the perfect examples are:  QA Partner done by Segue Software and also Xrunner or WinRunner created by Mercury Interactive.

All software testers have really good understanding and they know best practices why the testing can be critical for software development process. They know the consequences, what will be outcome of neglecting code reviews and ignoring bugs. They see better the results of coding before designing. So testers are people who clear the sky, but sometimes they fail in one thing. They do not understand that test automation should be activity of development process,

thusly needs to cling to critical development practices. This graph should be shown to everyone who rejects the above mentioned sentence.

What are the major challenges for GUI test automation? To start with there are different kinds of GUI test automation, first automation for web-based, which is mainly working with parsing the HTML. Second is desktop automation, which is a bit complicated, since the structure of interface sometimes is unknown. Another type is mobile testing automation, which is as well more complicated than web-based application testing. To answer the question, main challenges are maintainability and reliability.

Sometimes developers and testers underestimate test automation. When they hear the test tool, they are thinking about something really easy and simple to use.

System Model

As mentioned above, we are going to discuss model checker, how it works and what environment should be created in order to fully perform test case generation.

To start with, the system is a subset of different states. The process when the system changes from one state to another is called transition. If we want to test transition between state A and state B when the condition C is true,  we should formulate a condition which will describe the test case testing the transition - the bunch of inputs must take the model to state A; and in state A, C, out condition, must be true. The last is that next state must be B.

I have already mentioned above that the system must contain the finite number of states. Let's say that the system contains n variables {x1, x2, x3, … xn}, where each xi gets its value from domain Di. All this means that reachable state of the space is the subset of all domains like: D = D1 x D2 x D3 … Dn.

What are the initial values of the variables listed above? The predicate p, which is subset of D, is responsible for defining the initial values for variables.

Transition relation for each variable specifies how the variable can be changed while the systems moves from state A to state B. In other words transition for xi can be thought group of three components

• Set of xi pre-state values
• Set of xi post-state values
• Condition

We now have a rough understanding what is system model. It is a tuple of (D, A, p), where D is the state-space of the system, A - transition relation and p - the system initial state.

Test Case Generation

Model checker methodology results have been really promising, it really helps to find all test cases and some leaks(traps) into the coverage. In this chapter I will talk  about the approach itself.

Our aim to generate test cases involves using the model-checker as the core engine. Model-checker has to verify all properties one by one, these properties are generated by model-checker itself and they are called as trap properties. These properties are intended to fail for the given system specification, this leads the model-checker to generate counter-example. The  counter-example shows the sequence of states(guidelines) which should be followed by all conforming implementation. And in the end all this becomes a full and comprehensive test case. After that our task is to generate the trap properties  in such a way that the group of counter-examples generated will be enough to satisfy the coverage criteria that we are dealing with.

Alternatively, counter-example can be generated using mutation. "A mutation describes a syntactic change that may introduce inconsistency, and can be applied to the model or specification , or properties extracted from the model in order to create test-cases - says Gordon Fraser. Here we should also define new expression "Mutation score". This is the number of mutants  which is distinguished by test-suite from the original.

Conclusion

The paper gave us brief understanding of test automation and test generation. Despite the fact that the algorithm of generation is pretty difficult to understand, even more difficult to implement, we have a rough idea how it works.

The fact that the test automation is really important and reduces the time, thus the cost, is obvious. The examples which were given above assures me in the importance of test automation.

To conclude model-checker seems one of the most well-known methodology to generate test cases. The main idea how the model checker is working is that first it tries to generate trap properties and then tries to find all failures within those properties. The next step is to generate counter-examples from those failures, counter-examples than lead us to test case, while bunch of test cases gives us test suite which is our outcome. But as mentioned above, it is quite impossible to fully cover all your cases, since the number of tests are finite.

We have also identified the different types of code coverages, those coverages can be used in different kinds of applications, depending on the complexity.