Test Plan concept

Plan objects are containers for your test logic. They're implemented as a tree of artefacts which can be of different types.

  • The first type is a control structure (usually just referred to as "control"), which is similar to what you find in any programming language (If, For, etc), except for these can simply be dragged and dropped into a Plan and configured. No code needs to be written. As in any programming environment, control structures are used to influence the execution flow of a program (or here test) and are therefore very important nodes in a Plan.
  • The second type is made up of the Keywords which which have already been registered and configured at the time of Plan design. They contain concrete code (whether Java, .NET, JS, etc) which you want to use as a building block of your test plan. From the Plan view, users not required to know or understand the implementation details of the Keywords they're using in their plan. However, they will likely need to pass input data to them and check their outputs.
  • The third type is another Plan. Plans can basically call eachother, this is more of a link feature allowing for modular test design. A similar type of linkage can also be achieved with Composite Keywords which are a special type of Keyword. Use of Composite Keywords is an advance feature which we'll cover in a dedicated section.

In order to define what, when and how a piece of test logic will be executed, you will primarily want to mix Controls and Keywords in your Plan's tree.

Test plan structure

When initially creating a Plan ("New plan" button in the Plan view), you need to pick a logical controller as the root of your tree. By default, you can pick a Sequence for instance, which only ensures the order of its children matches the order they are displayed in the Test Plan Editor.

CreateTestPlan.png

On the right side of the editor, access is given to the 3 aforementioned types of libraries :

  • the control structure library (see tab "Controls") with the different logical controllers.
  • the Keyword library which displays by default the list of all Keyword found on the instance (see the development section for more information on how to design keywords). Keywords can be filtered and searched by name.
  • the "Other plans" library, which allows users to call another test plan from the one they are currently building.

You can add as many nodes as you want by clicking the + buttons of the items shown in the different library tabs. They will then be automatically added to the currently selected node of the Test Plan Tree.

Then, using drag and drop and the upper arrow and lower arrow buttons, you can reorganize nodes and place them in the order that you want.

TestPlanUpdate.png

Groovy expressions

Most of the artefacts used in a Plan will require input parameters passed via a form. The form values can either be plain Strings entered by the user or dynamic values produced by interpreting groovy expressions at run time. In order to switch between these two "worlds", users can toggle the lightning icon of each input. The lightning icon turns blue when groovy mode is active.

See the icon attached to value field in the image below:

Screen Shot 2019-01-11 at 09.59.14.png

Taking the Set artefact as an example, one might want to store the current date in a Plan variable:

Screen Shot 2019-01-11 at 10.02.12.png

Plan export & import

Plans can be exported and imported via buttons in the web app, which will among other things, allow you to download, import and execute the sample plans we've published in our documentation library and share your own plans with others. Just use the export button in order to export an individual plan:

1519135070368-723.png

You can then import it on a different instance by clicking the following button in the root Plans view:

1519135163290-653.png

Don't forget to reload your Plans page after importing plans.

The other button in the Plans view, next to the import button will cause for all of the Plans present in your database to be exported.

Flow Control library

High-level logic controls

In addition to the logic you've built inside your script (at keyword level), STEP offers a wide range of logical controls which can be incorporated to a test plan, thus decoupling the test or business logic from the atomic actions implemented as keywords.

In order to better understand how each of these controls are supposed to be incorporated in a Plan, we strongly recommend checking out our Plan Library, in which we're planning on continuously adding new downloadable and immediately runnable examples.

Supported logic controllers

You will find below a description of what each controller does and of its primary properties. Advanced usage will be documented later on. We sometimes refer to them as "control flow structures" or just "controls".

ArtefactsPrimary usePrimary properties
Sequencesimply guarantees the ordering of the child nodes, as displayed in the tree

No property

simply wrap the child nodes with a Sequence node

Forcreates a For loop at execution time and iterates through its children

Start: the initial value of the for loop
End: the final value of the for loop (exit condition)

Increment: the increment value at the end of each iteration

Counter variable: the name of the variable containing the value of the iteration counter

Number of threads: the number of threads to use in order to parallelize the loop (i.e the execution of the children)

Example for 5 Iterations with a parallelism of 2:

start=1; end=5; inc=1; counter variable=counter; number of threads=2;

ForEachcreates a ForEach loop based on a collection and iterates through the child nodes

item (key of the data accessor), parallel (true/false), threads (number of threads if //), table (Excel File or SQL Query), folder (JDBC URL)

Example 1 
item=datapool; parallel=true; threads=10; table=D:\ExcelDatapool.xlsx
Example 2 
item=datapool; parallel=true; threads=10; table=select FRUIT_NAME, FRUIT_ID from MYFRUITS; folder=jdbc:mysql://localhost/SID,USER,PASSWORD,com.mysql.jdbc.Driver

(also tested with Oracle ojdbc6.jar) IMPORTANT: You need to copy your driver library (jar file) in the lib folder of the agent to use JDBC functionality.

Sessionguarantees that the same driver session will be used for every child node's execution (very important for stateful keywords which rely on a certain driver state as an initial condition)selectionCriteria (optional, for affinity patterns)
 execute the child nodes in a Thread execution pool, allowing concurrent executions of the child nodes

number of threads, number of iterations per thread, pacing (time to pause between thread next executions)

Example: number of threads=2, number of iterations per thread=10, pacing=50ms
Ifonly executes the child nodes if the condition is met

condition (the expression evaluated to know if the content of the node is executed)

Example: condition=my_variable === true
ReturnIn the context of a Composite Keyword, delivers a value back to the parent nodeJust enter key values as you'd like them to be available in the output of a regular keyword
Checkasserts the input expression, useful for validating the output of the parent node

Expression (the expression to be asserted)

Example: expression: output.getString("output1")=="val1"
Casesame as in any programming language, works in combination with switchValue: the value to be potentially matched
Switchsame as in any programming language, works in combination with caseExpression: the expression to switch on to
Setsets a value to a variable, which can then be accessed throughout the Test Plan and sub Test Plans. Combined with the power of groovy, almost anything can be achieved in a Set node.

Key: the name of the variable where STEP will write to

Expression: the groovy expression to be interpreted, it can be anything

Example 1 - Setting up a MySQL connection in groovy (requires a driver on the controller)
key=my_connection; expression=groovy.sql.Sql.newInstance('jdbc:mysql://localhost/SID', 'USER', 'PASSWORD', 'com.mysql.jdbc.Driver')
Example 2 - Using the connection object in a second set to run a SQL query
key=my_result; expression=my_connection.rows('select fruit_name, fruit_id from MYFRUITS where fruit_id = 1')

​​You can then pass the value of my_result as a keyword input in the "argument" box: { "input1" : "my_result.fruit_name.first()" } The double brackets will evaluate the content as a groovy expression even though the expression is located inside a json string.

SynchronizedGuarantee thread safety within a test block by synchronizing all threads on the entire Test Execution.none
CallKeywordTechnical artefact used as part of keyword invocation. Can be used explicitely in order to template keywor calling, due to exposure of the "name" input.

Key & Value: same as in a regular Keyword call, but a name should be set in order to tell step with Keyword needs to be called. Used in conjunction with groovy, this can provide a powerful mechanism for Keyword Call templating.

RetryIfFailsretry mechanism with grace period

maxRetries: number of retries if step fails

gracePeriod: duration to wait in ms after a failed attempt

TestSetused to group up TestCase's as a single unit and executing them in parallelsimply wrap your TestCase's with a TestSet controller, parallelism is derived from the global variable "tec.execution.threads"
TestCasespecific container for a group of nodes, will activate the top-level panel in the execution view for high-level test case execution monitoringsimply wrap your Test Plans with a test case
DataSetUsed to iterate over rows of data in a table (the table itself can be stored in a database, a file, etc) and can be of different format (json, csv, etc)

Source: the type of data source you're wanting to use to retrieve data out of the table
Iterator handle: the pointer to the current row (or "cursor" ind database terms)

Reset at End: whether or not the dataset should resume from the beginning when reaching the end of the table

Example using a JDBC dataset:
Connection string: jdbc:oracle:thin:@<HOSTNAME>:1521/<ServiceName>
Driver class: oracle.jdbc.driver.OracleDriver
Query: select * from fruits
User: foo
Password: bar
Primary write key (used to identify a row uniquely when writing back to it): mykey

EchoUsed to print data in the report nodes of a plan, mostly for debugging or information purposesText: the data you want to display (can be a groovy expression if Groovy mode is toggled)
CallPlanUsed to invoke a plan from within another planKeys & Values: the plan you want to invoke, as for Keywords and CallKeywords, CallPlan can be used for templating Plan executions
AssertValidates the output of a keyword execution

Output attribute: the key containing the data needed for verification
Operator: the type of verification
Expected value: the value you want to check against

Example:

Output attribute: foo
Operator: beginsWith
Expected value: bar

PlaceHolderUsed to execute plan chunks passed from another plan

In conjunction with a plan call, this will be used to defer and delegate the execution of children nodes to another plan.

The placeholder needs to be placed inside the target sub-plan (as a positional marker).

The delegated nodes should be set as children of the sub-plan's node within the parent plan.

ScriptExecutes any arbitrary groovy code

Script: the groovy code to be executed

Example:

Script: myvariable = new Date()

SleepCauses the thread to sleepDuration: the amount of time to sleep in ms.
TestScenarioUsed to parallelize multiple ThreadGroupsSimply place ThreadGroups as children of the TestScenario.
ThreadGroupStarts multiple threads which will execute the content of that node in parallel

Number of threads: the number of threads to be started
Number of iterations per thread: how many times the content should be executed per thread

Load Distribution Parameters: typical Load variables (rampup, pacing, offset)

Example:

Number of threads: 10

Number of iterations per thread: 100

Load Distribution Parameters:
- rampup (nothing, default = pacing)
- pacing: 50000

- 0

WhileIterates over the node content until the condition is not met

Condition: a groovy condition to be evaluated upon each iteration's begin
Pacing: allows for a pacing duration to be observed
Timeout: Forces the thread to continue execution after a certain period of times (serves as a security)
MaxIterations: Force the thread to continue execution after a certain amount of iterations (serves as a security)

Example:

Condition: var_a == var_b
Pacing:
Timeout:
MaxIterations: 50000

WaitForEventWait for a specific event to be receive by the controller. Event can be identified by an eventId or a couple group/eventName

EventId: the ID of the event. 

Group/Name: The group/name used for identifying the event

Example:

EventId: blank

Group: myGroup

Name: myName

Scheduling Test Plan executions

STEP offers you the possibility to schedule exuections of Test Plans based on JAVA CRON expressions. In order to schedule your Test Plan executions, click on the "Execute" button under the plan tab then on the "Schedule" button.

ScheduleTestPlan.png

You are then invited to enter the CRON expression which will define when the plan will be executed :

CronExpression.png

You can then switch to the "Schedule" tab in order to view, edit and control the scheduling entries you set up :

ScheduleTab.png

Here are some basics examples of CRON expressions. If you are not familiar with, we encourage you to take a look at the CRON official documentation.

ExpressionMeans
0 0 12 * * ?Fire at 12:00 PM (noon) every day
0 15 10 * * ?Fire at 10:15 AM every day
0 * 14 * * ?Fire every minute starting at 2:00 PM and ending at 2:59 PM, every day
0 0/5 14 * * ?Fire every 5 minutes starting at 2:00 PM and ending at 2:55 PM, every day

Keyword routing : execution on specific agents

  1. From STEP's Plan editor

Once an Agent as been "typed" as described at here , it is possible to execute a plan using those agents by setting the "Keyword routing" in the plan editor tab before executing it :

1521213726031-520.png

In the example above, the load will be sent only to agents having as property the "AGENT_TYPE" set to "DEV".

If there is no agents marked with this property, the execution will fail with the following error :

1521213968353-124.png

Note: the same can be done at the Session artefact level (assuming your plan uses that artefact).

2. In a text-formatted Plan (i.e ALM, Jira, PlanEditor or via REST / API)

 You can use the reserved variable "route_to_{key}" in order to set or override the default behavior of the router.

For instance, assuming we've type an agent's token pool with the key-value combination "AGENT_TYPE" : "DEV", we can override this behavior by setting the value of variable route_to_AGENT_TYPE to "PROD" in order to route the subsequent keywords to the corresponding agent group (PROD).

In the event of a temporary/scoped re-route, you can simple use another Set in order to reassign the original value back to the "route_to_{key}" variable at the end of the re-routed scope. For instance:

//Saving default value
Set originalRoute=route_to_AGENT_TYPE

//default behavior is route to X
//MyKeyword goes to agent X
MyKeyword myInput="foo"

// now routing Keywords to agent Y
Set route_to_AGENT_TYPE="Y"
MyKeyword myInput="bar"
Set route_to_AGENT_TYPE=originalRoute

// default routing is now active again

Tags:
Created by David Stephan on 2019/01/16 09:59
     
Copyright © exense GmbH
v1.0