orchestrated tests

We need field tests in the real world. That#s the general picture: An orchestrator peer is fed with a number of scripts which are meant to be be performed by tester peers. Each test cases is described with a JSON file split into two sections: A mandatory section contains the actual scripts. An optional seconds section describes requirements for peer's environment, like operating system and its version.

Orchestrated test process

We need at least three SNM peers. One peer will act as orchestrator. It must provide an open port for its tester peers. Start a tester peer e.g. like this

orchestrate csTestCase.json; openTCP 2222

Tester peers must first connect to an orchestrator and send their intent to act as tester peer, e.g. like this:

connect <orchestrator-ip-address> 2222; scriptRQ

scriptRQ requests a test case script. A message is sent providing operating system and version of test peer's host system. The orchestrator will launch a new test case whenever enough fitting tester peers sent a script request.

csTestCase.json could look e.g. like this

{
"PeerRequirements": 
[ 
"1":
{
"osName": "Win",
"osVersion": "11",
},
"2": {
"osName": "Win",
"osVersion": "11",
}
]
,
"testname": "cs1",
"maxDurationInMS": "120000",

"Scripts": 
[
"1": "connectTCP localhost 4444;release CS;sendMessage HiFromAlice;lsMessages;"
"2": "openTCP 4444;block CS;lsMessages;wait 1000;lsMessages;"
]
}

This test case file describes a basic CS scenario. A peer running in a windows 11 environment is meant to run script one. Script 2 is to be performed by a peer running on ubuntu - no constraints on the actual version. This test is labled cs1 which makes it easier to find related log-files. Its maximum duration is set to 120,000 milliseconds which are 20 minutes.

Sending test scripts

The orchestrator adds a few command to each script to solve a few problems:

1. Tester peer process would not stop in case of a failure. A failed test would leave an orphan process in the machine.
2. Tester peer process should stop immediately after running a test case.
3. All test peer must start (more or less) simultaneously.

A timeBomb command (of appr. 20 minutes - check the actual implementation) is added as the very first command to each script. It prevents a tester peer process to run forever.

An exit command is added at the end of the scripts to ensure that the a tester peer process immediately ends after a successful proformance of a test case. It doesn't hurt if a test case script already ends with an exit. The additional exit will not be performed.

SNM peers can be synchronised with block/release commands. The orchestrator adds the following commands directly before the original test case script: connectTCP <orchestrator-ip-address> <port>; release <definedByOrchestrator_1>; block <definedByOrchestrator_1>;

Running test scripts

A SNM peer becomes a potential tester peer after issuing a scriptRQ. It will read and perform a test script send by an orchestrator. Yes, that sounds like a trojan horse and it is. That's the reason why all those test related CLI commands are not part of the SNM library for building real SNM user interfaces.

Following our example, a tester peer would receive a script like this:

timeBomb 1200000; 
connectTCP <orchestrator-ip-address> <orchestratorPort>; release test#42_B_P1; block test#42_R;
connectTCP localhost 4444;release CS;sendMessage HiFromAlice;lsMessages;
exit;

We can see the actual test script in the middle of the script.

A tester peer starts a new process (not a thread). This process starts a new JVM running the SharkNetMessenger jar file fed with the received script. This process will first set a timeBomb.

It connects to the orchestrator, sends a release and waits to be released from a block. Note, the sent release is peer specific, the block is not.

The orchestrator waits for release messages from each involved peers. At some moment, is will receive the message from the last remaing peer.

Now, it sends a release messages to all. Those messages are sent over an existing TCP connection. We cannot predict the order in which that release reaches each peer, though. That's in the hand of the underlying IP network. That's a final race condition that cannot be solved be the orchestrator. Test case script designers must have this in mind and decide whether this can cause a problem.

Tests peers process will start more or less simultaneously and perform the actual test case script. The tester peer process comes to an end either by reaching exit or by being ended fro the timeBomb.

Collecting logs

[TODO]