Tests as function blocks instead of methods

[Roald87]

I was thinking on how to reduce the amount of duplicate code when writing tests, how to make it easier to make different kinds of tests (see #97, #95 and #54) and how to reduce the amount of boilerplate (#29). Having the tests as methods makes this all very hard as the discussion in #29 shows.

Maybe I'm overlooking something why this wouldn't work or be very cumbersome, but I think the method I explain below can even work together with the current way of initializing unit tests. So no backward compatibility issues!

I was thinking of the following. Note: the code only functions as an example, I didn't try it.

What if all tests are FBs instead of methods? Adding a test will function very much like adding a test suite. So instead of adding methods to your test suite you would have something like this:

FUNCTION BLOCK Sum_Tests EXTENDS FB_TestSuite
VAR
    onePlusOne_ShouldBeTwo : FB_OnePlusOne_ShouldBeTwo;
    twoPlusFour_ShouldBeSix : FB_TwoPlusFour_ShouldBeSix;
END_VAR

And then these test function blocks would look like:

FUNCTION BLOCK FB_OnePlusOne_ShouldBeTwo EXTENDS FB_Tests

// Code to test in the body of the FB or in a method e.g. Run()

FB_Tests will get a FB_init, much like FB_TestSuite.

METHOD FB_init : BOOL
VAR_INPUT
    bInitRetains : BOOL; // if TRUE, the retain variables are initialized (warm start / cold start)
    bInCopyCode : BOOL;  // if TRUE, the instance afterwards gets moved into the copy code (online change)
END_VAR

CurrentTestSuiteBeingCalled^.AddTest(THIS);

Furthermore FB_Test will obtain the name in a very similar manner as FB_TestSuite. Also it would probably beneficial to have an interface I_Test:

{attribute 'call_after_init'}
{attribute 'reflection'}
FUNCTION_BLOCK FB_Test IMPLEMENTS I_Test
VAR
    {attribute 'instance-path'}
    {attribute 'noinit'}
    TestName: Tc2_System.T_MaxString; 
END_VAR

If this all works you could do things like creating a FB_TestWithTimeOut (#54). This test would get an extra argument in the FB_Init to set the time out.

FUNCTION BLOCK FB_TestWithTimeOut EXTENDS FB_Test
...
METHOD FB_init : BOOL
VAR_INPUT
    bInitRetains : BOOL; // if TRUE, the retain variables are initialized (warm start / cold start)
    bInCopyCode : BOOL;  // if TRUE, the instance afterwards gets moved into the copy code (online 
change)
    timeOut : TIME;
END_VAR

SUPER^()
THIS^.timeOut := timeOut;

Or a delayed test (#95) would work in a very similar way using a cyclesToWait argument. In your test suite you can then easily see if you have some special tests.

FUNCTION BLOCK Sum_Tests EXTENDS FB_TestSuite
VAR
    onePlusOne_ShouldBeTwo : FB_OnePlusOne_ShouldBeTwo;
    twoPlusFour_ShouldBeSix : FB_TwoPlusFour_ShouldBeSix(timeOut:=T#1s);
    hundredPlusFive_ShouldBeHundredFive : FB_hundredPlusFive_ShouldBeHundredFive(cyclesToWait:=5);
END_VAR

Furthermore you can probably reduce the code of each individual test, because you often have to initialize the same variables. So you can do things like

FUNCTION BLOCK FB_Sum_BaseTest EXTENDS FB_Test
VAR
    fbSum : FB_Sum;
    expected : LREAL;
END_VAR
...
FUNCTION BLOCK FB_OnePlusOne_ShouldBeTwo EXTENDS FB_Sum_BaseTest 

fbSum(1, 1);
expected := 2;
AssertEquals(expected, fbSum.Ans, 'Sum not correct');

Also you could make a FUNCTION BLOCK FB_RandomizedTestSuite EXTENDS FB_TestSuite which can run tests in a random order.

Finally it would also be good to add a check if a test already has passed/failed so it is not run multiple times.

Would something like this work?

[RogerChristopher]

It would work. I use this pattern in a Unit test library I wrote a while ago (and still use) - I have one test per FB that implements an interface (IUnitTest). The test runner has a list of interfaces and runs the tests by iterating through the list calling Execute on each test. This works well, and is much simpler than having multiple test methods.
Roger

[sagatowski]

Hi!

Thank you very much for these greatly documented ideas!

When creating TcUnit my primary inspiration for this was two test unit frameworks I've been using in the past:

• JUnit (Java) (https://junit.org/junit5/)
• Google Test (C++). (https://github.com/google/googletest)

I tried to keep the architecture regarding the tests as close as possible to these two frameworks, and at the same time keeping the terms from ISTQB (http://www.istqb.org/).
Both of these frameworks use functions to declare the tests. I wanted the framework to "feel" like these two very popular frameworks, so that people coming from the non-automation field doing automation would recognize the basic design architecture, instead of coming up with my own.

I'm not sure giving the developers two (quite different) ways of creating tests is a good idea (https://en.wikipedia.org/wiki/Overchoice). Updating all the documentation on the TcUnit-website with these two different ways of creating tests would probably be confusing.

This issue got me thinking that even more importantly, it would be nice to investigate how a design for test fixtures could be made. The concept of test fixture does not exist in TcUnit as such today. I'm very fond of the test fixture design in google test
https://github.com/google/googletest/blob/master/googletest/docs/primer.md

I think this way of declaring and using test fixtures might work quite well with TcUnit as well.

Now, because for example C++ has macros, it's possible to do much more in the testing framework than with TwinCAT. What would be nice to investigate, is if there is a possibility
to do something similar to the parameters of JUnit. For example the @ignore annotation:
https://www.guru99.com/junit-ignore-test.html

TcUnit follows the GoogleTest approach today (by applying DISABLED_ prior to the test name). But with parameters it would be possible to do much more.
I guess the pragmas are the closest thing to this, although these go straight to the compiler which are fixed and defined.

[Roald87]

Hi Jakob!

Good to see that you put so much time, effort and thinking when you started the project! Thanks for doing this.

so that people coming from the non-automation field doing automation would recognize the basic design architecture

Good point. I'm probably not the right person to judge from this standpoint. I've only used the some Pyhton and C# unit test frameworks and both of them quite recently. So I'm not really used to a certain architecture.

I'm not sure giving the developers two (quite different) ways of creating tests is a good idea

I agree that that wouldn't be a great idea. The best thing would probably be to have it temporarily coexist and build a tool to convert the old tests to the new design. Although I saw you once said somewhere that people use TcUnit in many different ways. So they don't always define one test in a separate method. That would complicate making a conversion tool somewhat.

it would be nice to investigate how a design for test fixtures could be made.

If I understand the fixture concept correctly, then that is exactly what I was trying to do with

FUNCTION BLOCK FB_Sum_BaseTest EXTENDS FB_Test

FUNCTION BLOCK FB_OnePlusOne_ShouldBeTwo EXTENDS FB_Sum_BaseTest 

Do you already have an idea how you would like to implement it?

I'm just thinking it through and maybe the following would work? We make a new function

FUNCTION TEST_F(TestFixtureName, TestName)
VAR_IN_OUT
    fbTest : I_TestFixture;
END_VAR
VAR
    TestName : STRING;
END_VAR

TEST(TestName);
fbTest.SetUp();

And the following interface.

INTERFACE I_TestFixture
METHOD SetUp

An example would go as follows:

FUNCTION BLOCK FB_PrepareStateMachineInErrorState IMPLEMENTS I_TestFixture
VAR
    fbStateMachine : FB_SomeStateMachine;
END_VAR

METHOD SetUp
// Do stuff to get fbStateMachine into state error

FUNCTION BLOCK FB_SomeStateMachine_Tests EXTENDS FB_TestSuite

METHOD InErrorStateTryToTurnOn_ShouldStayInErrorState
VAR
    fbStateErrorMachine : FB_PrepareStateMachineInErrorState;
END_VAR

TEST_F(fbStateErrorMachine, 'InErrorStateTryToTurnOn_ShouldStayInErrorState');
fbStateErrorMachine.fbStateMachine.TurnOn();
AssertEquals_INT(
    Expected:=E_State.Error, 
    Actual:=fbStateErrorMachine.fbStateMachine.State,
    Message:='Machine should not leave state error'
);

I'm not sure how to add @annotations to methods. That would only work with FBs I think.Your example of how to ignore a test by using @ignore/DISABLED_ is not really relevant for TcUnit right? You can just as easily ignore it by commenting it out or nor call the method at all.

[Roald87]

@RogerChristopher you made your own unit test library for TwinCAT?

[RogerChristopher]

Yes. It is a minimal implementation - as simple as possible.

I wanted:

1. A test with minimum ceremony - Create a test that extends a base class, assert something and call TestFinished()
2. The ability to run the tests concurrently or sequentially (useful for integration tests)
3. The ability to re-run the tests without a cold restart
4. The ability to re-run the tests after an online change
5. A GUI - it has a visualisation you can drop into a frame in the client project. This frees up the error list to show errors.

I will open-source it when I have time, but you can have access to the repo if you want to - let me know.

Roger

[RodneyRichardson]

Can I add?
6. The ability to run the tests in a CI pipeline, with results exported as standard XML format.

[Roald87]

That sounds very interesting! Can you send a link?

[Aliazzzz]

Yup, i'd like to see your take on it also if you let me! Will be very interesting.

[RogerChristopher]

Just put it on Github - see SimpleUnitTestLibrary. There is a library and an example project. Max number of tests is set via a Param - default is 200.
Its been a bit rushed - let me know if I missed something
Roger

[Aliazzzz]

Hi Roger,
Thank you very much for sharing this.

[DavidHopkinsFbr]

An advantage of defining an I_Test interface and then implementing tests as FBs is that features like delays before starting or timeouts could be implementable as Decorators which wrap/encapsulate an I_Test and implement I_Test themselves, which then allows you to compose multiple decorators around a single base test (if for example you want a test with both a start delay and a timeout).

In the specific case of start delay and timeout there may be some surprising interactions - e.g. the timeout might start counting before or after the start delay period, depending on implementation details and the order in which the FBs are wrapped, but composability in general is usually a pretty nice feature of a library.

Some thought would be required to ensure that only the outermost test FB in the Matrioshka doll of decorators gets registered with the framework for execution & results gathering.

[RGrabichler]

Hallo,
i love this site, your engagement your dedication!
Im very much a selftaught beginner, trying to implement from the beginning all the awesome ideas!

In trying to create Unit Tests for my first serious approach.
But my first problem is, how to create a unit test, where a fb gets an input signal, and after certain time (e.g. 1000ms) the Output Signal is set.
How to implement in TcUnit 1.0.0.0.
My first Problem is, i cant declare the TON in the Method.
Can somebody post a example, how to solve this?

Thanks alot in advance

[sagatowski]

@RGrabichler I'm happy you find TcUnit useful. If you need a timed test there is an example available here:

https://tcunit.org/frequently-asked-questions/?Display_FAQ=703

[RodneyRichardson]

I think this is a great idea, but is obviously a breaking change.

I think a fundamental issue with the TcUnit structure is that it runs each test suite, which in turn will run individual tests. These tests are not explicitly visible to the framework, only via calls to TEST() and AssertXXX() functions. This has several drawbacks:

• The framework cannot prevent a test from being executed, only a test suite. Although each ASSERT function checks if the current test is being ignored or is finished, it will still execute the test function until all tests within the test suite have finished.
• It's difficult or messy to add new test-specific functionality (such as timeouts for tests, or the new ordered tests).

The frameworks you mention above (as well as all the xUnit-style frameworks) allow the test developer to define tests as functions, but they also have the ability to have a pointer to that function, and call it individually. Unfortunately, PLC code does not allow a pointer to a Method (as far as I can see).

I think implementing tests as Function Blocks (possibly with I_Test interface) is the only way to enable this.

[sagatowski]

Hey community!

The more time I have to melt this, the more I agree this is the direction we should go. Making every test implement an interface makes it much more easy to extend with new functionality than what it is today.

It would be good to draw the various use-cases (on napkins/Visio-level is fine enough), and how they would look like using an alternative architecture for this framework, just to get a "feel" of it, and to see how easy/hard it would be to extend with new functionality in the future.

With this said, this will break backwards compatibility, and any major change like this would have to be incorporated into an API-changing version, i.e. V 2.0 of TcUnit, or if someone wants to, a fork of it or completely separate unit testing framework. I want to take the opportunity to mention that there is another framework that is released as open source right now. Together with Peter K. I've written an article about it: https://alltwincat.com/2021/02/16/unit-testing-in-the-world-of-industrial-automation/

I never thought TcUnit would get this widespread, and all this feedback will drive not only this particular framework forward, but I think we together also spred modern software practices to the industry.

Thanks community for your invaluable input.

[AndyP01]

Hi Jakob, great work with this! I'm just starting to explore it.

If we take a look at nUnit (for example) it is possible to provide a test with multiple sets of data by specifying TestCaseData:

e.g.:

[TestFixture]
public class MyTests
{
    [TestCaseSource(typeof(MyDataClass), nameof(MyDataClass.TestCases))]
    public int DivideTest(int n, int d)
    {
        return n / d;
    }
}

public class MyDataClass
{
    public static IEnumerable TestCases
    {
        get
        {
            yield return new TestCaseData(12, 3).Returns(4);
            yield return new TestCaseData(12, 2).Returns(6);
            yield return new TestCaseData(12, 4).Returns(3);
        }
    }  
}

As far as I can tell at the moment, every test with TcUnit requires one method and also a single set of test values. So, if you want to test a 'AddTwoNumbers' function, you need to create multiple test methods like 'OnePlusOneEqualsTwo' and 'TwoPlusTwoEqualsFour'.

Is it possible to do something like 'AddTwoNumbers_test(testCases)' where testCases is an array of test parameters, enabling the test to be called multiple times but with different values?

[Bulkje]

Just ran into the same problem. I tried this:

    AddTwoNumbers(FirstNr := 5, SecondNr := 6, ExpectedSum := 11);
    AddTwoNumbers(FirstNr := 10, SecondNr := 6, ExpectedSum := 16);

But then I get the error "Test with name 'AddTwoNumbers' already exists in test suite (...)" from TcUnit.

[sagatowski]

@AndyP01 & @Bulkje sorry this went under my radar.
It is possible to have the same name for your methods. You actually don't even need to use private methods at all in the test-suite.

What however needs to be unique is the name defined in TEST(TestName := 'YourName'). This is how TcUnit "knows" that this is a new testcase. The private method in the examples is just one way to structure the code.

[Bulkje]

Wow thanks for the fast reply! Glad to hear that this can be done.