Using Variables for Environment Flexibility

In API Fortress, almost any string can be hardcoded or referenced as a variable. Hardcoding is fine as long as you’re building simple tests, however, when:

  • The number of tests is increasing
  • The complexity of tests is increasing
  • The number of tested environments is increasing

It is advisable to parametrize some items. Most of the parametrization you will likely do relates to the HTTP request itself.

Using the Vault

For example, while this is perfectly legit:

harcodedIt may become extremely painful to update tens if not hundreds of tests if the domain changes.

Using the vault to store domain names is a good way to solve this problem by simply adding a “domain” variable in your vault as in:

domainAnd editing the GET like this:

parametrizedIn this way, editing the vault variable will instantly affect all tests.

Using the Environments

Once the domain is parametrized, no one stops you from overriding the variable, if needed.

By activating a preset in the environments section as in:

env2You will be able to hit a different domain in the current session, without actually changing the test.

The same selection can be performed while creating a schedule, to create specific runs hitting specific environments.

In Request Bodies

Variables are not only bound to URLs. Request bodies can also be handled like “templates” when needed, incorporating variables as in:

bodyAnd Basically Anywhere

Variables can be referenced basically anywhere, let’s take this assertion in consideration:

expYes, we’re using variables as expected values.

However…

The platform not only provides you the flexibility but also the freedom to basically combine everything as you want.
With that said, we would also provide you some hints on how to get the best from your experience.

  1. Fill the vault with data that is actually project-wise. Domains, protocols, API keys. They are all fine. We discourage you to introduce test specific variables in here because it would produce an overhead of information that will be unused most of the time,
  2. Fill the globals/input set with test specific variables. Such as paths, Ids, dates, serial numbers,
  3. Make sure that vault + globals/input set make up a complete variable scope for the test. In other words, the test should be able to run without further information,
  4. Use the environments to change the values of the variable scope generated by the vault+global/input sets,
  5. Don’t overdo. Parametrize stuff that can actually change and leave everything else as static strings. Variables are… well, variable, so an excessive and unnecessary use of variables leads to uncertainty and hard to track behaviors.

Environments, Vault, and Variable Overrides

Overview

With the introduction of the “Environments” features, your ability to mix and match your tests settings increases drastically.

Let’s review what our variable containers are, moving from the lowest priority up.

Vault: each project has a vault where variables and snippets are contained. Each vault is global to the project and its variables are injected during test execution.

Globals / Input set: they belong to each test and contain the variables that are needed by the test to perform its duties. Since these variables can be overridden to hit different environments, we generally suggest considering these variables as containers of the default scenario. If the variable has been previously declared in the vault, globals/input sets will win over the Vault.

Overrides: when an overridden variable is declared (using the API, the Scheduler or the Jenkins plugin) its value will be injected in the test when it’s executed. If the variable has already been declared in the vault or the globals/input set, it will be rewritten with the new value.

Environments and presents: they are collections of overrides. You can save an environment with a name and reuse it when running a test.

SET commands: the last item of the chain, the SET commands within a test will introduce a runtime variable. If a variable with that name has already been declared, it will be overwritten.

Suggested Usage

Tests should be as self-contained as possible and should host as much information as possible to perform its operations, with the help of the Vault. In other words, Vault + Globals / Input set should always generate a complete variable scope. Therefore, running a test without any environment selection should at least lead to no syntax or runtime errors.

Environments and overrides should be used to change some of the values, or introduce contextual values, for:

  • hitting a staging server instead of the production one
  • run the test against a special product ID

Double evaluation

All variable containers have a “double evaluation” capability. Meaning that a variable declaration can actually reference another variable.
By doing so you can decide to concentrate the actual data in the variable container that best suits your approach, and then reference it.

In the following example, we are storing the actual domains in the Vault, deciding a default in the Globals, and overriding in the environment:

VAULT:
production_domain: example.com
staging_domain: staging_example.com

GLOBALS:
domain: ${production_domain}

ENVIRONMENTS Name: staging
domain: ${staging_domain}

If run without environment selection or overrides, the test will hit the production domain. If run with the staging environment, the test will hit the staging domain.
The Environments will not know the actual domain, therefore the actual data management will happen within the Vault.

Known Issue: the double evaluation will not work on query params. Workaround: in the test, before performing the call, introduce a SET component to resolve the variable, as in <set var=”apikey” value=”${apikey}”/>

Environments loading

If you’re using environments heavily, you will soon realize that the integration using the APIF API (and any CI/CD plugin) may become unfriendly, as lots of data will need to be copied around.

For this reason, we introduced a special override variable that will allow you to ask API Fortress to load a specific environment when invoking a test using the API.

the apif_env variable, passed as an override, will cause the system to load a specific environment.

Example:

apif_env: staging

Will load the “staging” environment and all its override variables.

Notes

As you get acquinted to the platform, you may be tempted to use all these features at once. While this makes us proud, we want to warn you from the overall complexity you may achieve, expecially when you’re double-referencing variables.
An extensive use of all these techniques altogether may increase the difficulty to debug a problem, when it arises, so use them with caution!

Input Sets and Global Variables

In the test creation process, the definition of global and local variables allows you to parametrize the test to allow more flexibility.

The global variables are meant to be the ones that are common for the whole test (i.e. api key, domain etc). For adding a global variable click on the +Add Global Param in the Data Set panel and fill the name and value.

variable

globalThe variable will be present in the test scope, and will be constant during the test execution.

The input set, instead, is a group of input variables representing a scenario (i.e product id). To add in input set, click the + button in the Input Set section and add a name for the set. Then click + Add Param and add the variable name and value.

inputSet

The unit will be executed once for each input set you have defined. At each iteration, one input set will enter the scope.

If you define a variable both in the global section and the input set, the value used in the test will be the one defined in the input set.

The variables can be defined also in the Vault (for more info see here) and in the Schedule (for more info see here).

The priority order is:

  • if the same variable is defined in the Vault and in the composer (no matter if it is a global variable or an input set), the one defined in the composer will be used.
  • if the same variable is defined in the composer both as global and input set, the value of the input set will be used
  • if the same variable is defined in the Vault (or in the composer) and in the scheduler, the variable defined in the scheduler will be used for the tests.

Using Test State Variables

API tests should rarely need to know facts about themselves and take decisions off those. In fact a test shouldn’t take decisions based on its state, but on what the tested material looks like.

Unfortunately, the real world is not always this perfect and forces you to build introspective tests. API Fortress provides you two mini hacks to extend the test logic.

  • The ws_failures_count variable will allow you to know how many assertions failed while running the unit against the current input set. You might want to use it to stop the test if a certain number of failures has been reached, or decide whether or not to perform a certain branch.
  • The local_ variables. As you know input sets are isolated. This means that any variable I set in the unit while executing input set A will be lost while switching to input set B. This is extremely important because “variable noise” could lead to major uncertainty. However, if I wanted to have a variable to survive input set changes, all I would have to do is use the local_ prefix. As in local_items, local_count.

Most of the times there’s a more conventional way to do things, however when stuff gets nasty, you now know you can use these two tricks to make your life easier.

Data Manipulation Tricks (Random Numbers & Strings)

We present you some tricks that you can use to solve common problems while writing tests.

Unique Emails

If you want to generate a unique random email you can incorporate the current epoch in it. Epochs progress every millisecond so it’s pretty improbable to have a duplicate! The D language extension (see here for more details about this extension) provides date manipulation utilities, so:

<string>${D.nowMillis()}@gmail.com

if ‘string’ is ‘apifortress’ the result will be something like: apifortress1489051395378@gmail.com

String Manipulation

If a data item is a composite string of values and you need to evaluate portions of it, there are numerous ways you can do it.

Let’s say we have this string “55,5,84,65,26,7,82” and you want to work with each number. The simpler way is transforming the string into an array. You can do using the ‘split’ function:

<variable_name>.split('<separator>')

In the example above, if the string is included in a variable named ‘randomNumbers’ writing randomNumbers.split(‘,’) you produce an array like [55,5,84,65,26,7,82]. At this point you can work with it as you generally do when you have an array.

Another trick you can use to improve your tests is extracting something from a string using the ‘substring’ functionality. If you need to extract the first n characters from a string there is a function that helps you: take(n).

Let’s consider the following string: “whatever” saved in a variable named “substring_variable”, if you:

substring_variable.take(4)

“what” will be returned.

Another function for a string is drop(n) that removes the first n characters and return the resulting String. Considering the same string and variable used as example for take(n), if you:

 substring_variable.drop(4)

“ever” will return.

The last example is extracting a substring using index ranges. If we have the string “wherever” by doing:

wherever[1..4]

we will obtain “here”

Dates Manipulation

Maybe your endpoint requires a date in a specific format as parameter.

${D.format(D.nowMillis(),'MM/dd/YY')}

This expression will generate today date (e.g. 03/09/17)

 

Working With the Header (status code, latency, fetch)

An HTTP response is made of a payload (what you are mainly interested in), but also contains contextual information. Using API Fortress you can also deal with the whole response envelope.

When you’re making an HTTP request in the composer, you’re providing a variable name. That variable will host the entire response payload. So let’s say “payload” is the name of that variable. When the operation completes, another variable called <variable_name>_response is also created. Therefore various pieces of information such as HTTP header and metrics are contained in the variable “payload_response.”

So by referencing the “payload_response.statusCode” expression you can access the status code. For example, if you want to run a branch of code when the status code is 400, here’s how you do it:

statusCode

You can have multiple ‘IF’ conditions for checking all the possible status codes you need to check. Very useful for creating positive and negative tests.

multiStatusCode

Headers are also a big part of the response.

Let’s say you want to check that a resource shouldn’t be cached:

response_headers

Furthermore, you can have the performance of the call verified as well. Here is what it looks like in the CODE view:

<assert-less expression="payload_response.metrics.latency" value="350" type="integer"/>

<assert-less expression="payload_response.metrics.fetch" value="350" type="integer"/>

<assert-less expression="payload_response.metrics.overall" value="550" type="integer"/>

Visual Composer view (use the Assert-Less component):

metric test 1 metric test 2

Please note that latency is defined as time to first byte. Fetch is the total download time of the payload.

Connector – BigPanda.io

We love making life easier for our customers. That often means helping them unify their tools. BigPanda is one of those great tools that can help with that process, so we made sure to create included them as one of our connectors! To see this doc on their site click here.

  1. Login to your BigPanda account, and click on the Integrations tab, and then on New Integration.step 1
  2. Click on Monitoring and then Alerts REST API.
    step 2
  3. Next enter a name and click on Generate App Key.
    step 3
  4. Finally, keep the App Key and Authorization code handy for later.
    step 4
  5. Login to API Fortress and click the gear icon. Then click on Alert Groups. This is where all the connectors are located.
    apif step 1
  6. Type in a label (name).
  7. Click on the Edit Connectors button, and then on +Connector to this Group.
    apif step 3
  8. Scroll down to BigPanda.io.
    apif step 4
  9. Remember that App Key and Authorization? Copy and paste them here!
    apif step 5
  10. Click the checkmarkand you are done! All alerts from scheduled events are now being sent to BigPanda.

The Working Copy / Published Tests

We’ve been asked this question a few times so we wrote a post to make it clear.

When you work on a test you are not editing the actual test, but a working copy (a clone) of the test. The working copy can be edited, reset, and saved to continue the work later, but none of these actions alter the state of the live test.

When you’re done editing you will need to PUBLISH the test so that the working copy replaces the current live test to become official.

See the "publish" button top right?
See the “publish” button top right?

Why

Well, API Fortress is meant to alert people when something goes wrong. If you are editing a test and an incomplete test runs based on the schedule, you will receive a failure notification. A false positive.

By using the working copy to make your edits, the scheduler will not execute a test unless it is finalized and published.

To Summarize

  • You don’t edit the live test. When you edit one, you’re actually editing the working copy.
  • If you want to start over from the live test you can ‘Clear Working Copy’ to create a fresh version of the live test that you can edit.
  • When you are done editing click ‘Publish’ so your working copy goes live and the scheduler will use this new test for its executions.
  • If you create a new test but never publish it, the test will be marked as ‘incomplete.’

Improving Your Metrics

The performance of the API can be mission critical in some cases, and cataloging metrics can be as important as collecting them.

The classic approach of creating big tables of HTTP hits with the actual URL being called and its performance is certainly accurate but far from being easy to review because URLs contain IDs and hardly represent what the action was about.

API Fortress, as a default, works in this “classic” (dull?) way, but gives you to possibility to change the “footprint” of requests, based on your organization needs.

To activate this feature you need to add a “config” component to your I/O operation to reconfigure the footprint.

footprint

In this example, most of the request remains the same, but the ID value is actually replaced with the static string “[id]”, therefore making the calls with this pattern equivalent when browsed in the project dashboard.

metrics

And this is just one of the approaches. You’re virtually free to write the footprints that best fit your needs.

Outline a Plan For Your Test

Before you start writing your test, you should plan what you want to do.

Contrary to most testing frameworks, API Fortress is not endpoint centric, so there’s a number of different paths you can take to make the best of your test.

Single Action Test (active)

You’re willing to write a test that is going to verify how an endpoint action works. We’re also considering this scenario to be active, which means you expect API Fortress to perform a call and test the outcome.
If your action takes multiple parameters, we suggest you to create a number of input sets containing combinations of valid input. The unit will be run against each input set. Having multiple inputs is important because it’s the only way you can tell your API is responding uniformly, no matter what you feed it.inputs
Then you need to make a request and use those variables.

get

As you can see we’re composing the URL using global variables and we reference the variables in the query parameters.

Eventually, you will write your test. This is the simplest API Fortress test, covering the need to verify if an endpoint is up and running and working just fine.

Single Action Test (passive)

A test is passive when API Fortress won’t retrieve the payload on its own but will have it forwarded via the API. In this case, no input sets or I/O operations are necessary. The forwarded payload can be referenced using the “payload” variable. The ideal scenario for this mode is testing live request and responses.

Single Action Test (active, just-in-time parameters)

Also an input set can be overridden using the API. If your process requires an agent to trigger an active test with fresh inputs, create only one input set with valid inputs and have your API call override them. If you run the test by schedule, it test will run against the input you manually inserted. If you run it using the API and override the inputs, your test can run against fresh data all the time.

The ideal scenario for such mode is an application where the API is public but the inputs vary a lot over time and cannot be easily determined.

Single Action Test (mixed mode)

There’s a way to have a test to work both in an active way (when triggered by a schedule or manually launched, for example) and in an active way (invoked via API).
Here’s how you do it:

ifnotpayload

The GET will be performed only if payload does not have a value (yet).

If the test has multiple input sets that you don’t want to run when the payload is being forwarded, invoking the API with the “nosets=true” attribute will prevent it. This is generally a preferred over a plain passive test.

Integration test

In integration tests you chain together multiple calls, from authentication and actions to complete flows.

Possible scenarios vary from catalog browsing…

int1

… to full booking process for multiple flights

bookings