11. Using Variables for Environment Flexibility

In API Fortress, you do not have to be concerned with organizing variables in a hierarchy from global to local… why? 

Almost any string can be hardcoded or referenced as a variable in API Fortress. Hardcoding is fine as long as you’re building simple tests, however, it is advisable to parametrize some items when:

  • The number of tests is increasing
  • The complexity of tests is increasing
  • The number of tested environments is increasing

Most of the parametrization you will likely do relates to the HTTP request itself.

Using the Vault

While the following variable is perfectly valid, it may become extremely painful to update tens or hundreds of tests if the domain changes.

harcodedAlternatively, you may use the API Fortress Vault to store domain names to solve this problem. Simply add a “domain” variable in your vault as follows:

domainAnd then edit the GET like this:

parametrizedIn this way, you can eliminate duplicate tasks by simply editing the vault variable to instantly update all tests.

Using the Environments

Once a domain is parametrized, you may override a variable, if needed.

By activating a preset in the environments section, you will be able to hit a different domain in the current session without actually changing the test as in the following:

env2The same selection can be performed while creating a schedule to create specific runs hitting specific environments.

In Request Bodies

Variables are not only bound to URLs. Request bodies can also be handled like “templates” when needed, incorporating variables as in:

bodyAnd Basically Anywhere

Reference variables almost anywhere that you need. Consider the following example assertion:

expYes, we’re using variables as expected values.

However…

API Fortress provides the flexibility and freedom to combine the use of global, local, and hard coded variables as you want. In addition, API Fortress also provides helpful hints as you work with variables.

Review

  1. Fill the vault with data that is project-centric: Domains, protocols, API keys. They are all fine. We discourage you from introducing test-specific variables because it would produce an overhead of information that would go unused most of the time.
  2. Fill the globals/input set with test-specific variables, such as paths, IDs, dates, and serial numbers, etc.
  3. Make sure that the “vault + globals/input set” add up to a complete variable scope for the test. In other words, the test should be able to run without further information.
  4. Use the environments to change the values of the variable scope generated by the vault+global/input sets.
  5. Don’t overdo things. Parametrize stuff that can actually change, and leave everything else as static strings. Variables are… well, variable, so an excessive and unnecessary use of variables leads to uncertainty and hard-to-track behaviors.

parametrization, pass values, pass variables, header variables, body variables

12. Environments, Vault,
       and Variable Overrides

Overview

Gain a wide range of options to mix and match your test settings with our latest “Environments” features. Explore our variable containers from the lowest to the highest priority:

Vault: Each project has a vault where variables and snippets are contained. Each vault is global to the project and its variables are injected during test execution.

Globals / Input set: They belong to each test and contain the variables that are needed by the test to perform its duties. Since these variables can be overridden to hit different environments, we generally suggest considering these variables as containers of the default scenario. If the variable has been previously declared in the vault, globals/input sets will win over the Vault.

Overrides: When an overridden variable is declared (using the API, the Scheduler or the Jenkins plugin) its value will be injected into the test when it’s executed. If the variable has already been declared in the vault or the globals/input set, it will be rewritten with the new value.

Environments and Pre-sets: They are collections of overrides. You can save an environment with a name and reuse it when running a test.

SET commands: The last item of the chain, the SET commands within a test will introduce a runtime variable. If a variable with that name has already been declared, it will be overwritten.

Suggested Usage

Tests should be as self-contained as possible and should host as much information as possible to perform its operations (with the help of the Vault). In other words, Vault + Globals / Input set should always generate a complete variable scope. Therefore, running a test without any environment selection should at least lead to no syntax or runtime errors.

Environments and overrides should be used to change some of the values or introduce contextual values to hit a staging server instead of the production server, or run the test against a special product ID.

Double evaluation

All variable containers have a “double evaluation” capability, meaning that a variable declaration can actually reference another variable.

By doing so, you can decide to store the actual data in the variable container that best suits your approach, and then reference it.

In the following example, we are storing the actual domains in the Vault, deciding a default in the Globals, and overriding in the environment:

VAULT:
production_domain: example.com
staging_domain: staging_example.com

GLOBALS:
domain: ${production_domain}

ENVIRONMENTS Name: staging
domain: ${staging_domain}

If run without environment selection or overrides, the test will hit the production domain. If run with the staging environment, the test will hit the staging domain.
The Environments will not know the actual domain, therefore the actual data management will happen within the Vault.

Known Issue: The double evaluation will not work on query params. As a workaround, in the test before performing the call, introduce a SET component to resolve the variable as in: <set var=”apikey” value=”${apikey}”/>

Environments loading

If you begin using environments heavily, you may soon realize that the integration using the APIF API (and any CI/CD plugin) may become a bit “unfriendly,” as lots of data will need to be copied around.

For this reason, we have introduced a special override variable that allows you to ask API Fortress to load a specific environment when invoking a test using the API.

The apif_env variable, passed as an override, will cause the system to load a specific environment.

Example:

apif_env: staging

This will load the “staging” environment and all its override variables.

Notes

As you become acquainted with the platform, you may be tempted to use all of these features at once before you’ve achieved sufficient expertise. We should warn you that you may not be prepared for the overall complexity that may occur as a consequence, especially if you double-reference variables.

Input Sets and Global Variables

In the test creation process, the definition of global and local variables allows you to parametrize the test to allow more flexibility.

The global variables are meant to be the ones that are common for the whole test (i.e. api key, domain etc). For adding a global variable click on the +Add Global Param in the Data Set panel and fill the name and value.

variable

globalThe variable will be present in the test scope, and will be constant during the test execution.

The input set, instead, is a group of input variables representing a scenario (i.e product id). To add in input set, click the + button in the Input Set section and add a name for the set. Then click + Add Param and add the variable name and value.

inputSet

The unit will be executed once for each input set you have defined. At each iteration, one input set will enter the scope.

If you define a variable both in the global section and the input set, the value used in the test will be the one defined in the input set.

The variables can be defined also in the Vault (for more info see here) and in the Schedule (for more info see here).

The priority order is:

  • if the same variable is defined in the Vault and in the composer (no matter if it is a global variable or an input set), the one defined in the composer will be used.
  • if the same variable is defined in the composer both as global and input set, the value of the input set will be used
  • if the same variable is defined in the Vault (or in the composer) and in the scheduler, the variable defined in the scheduler will be used for the tests.

Using Test State Variables

API tests should rarely need to know facts about themselves and take decisions off those. In fact a test shouldn’t take decisions based on its state, but on what the tested material looks like.

Unfortunately, the real world is not always this perfect and forces you to build introspective tests. API Fortress provides you two mini hacks to extend the test logic.

  • The ws_failures_count variable will allow you to know how many assertions failed while running the unit against the current input set. You might want to use it to stop the test if a certain number of failures has been reached, or decide whether or not to perform a certain branch.
  • The local_ variables. As you know input sets are isolated. This means that any variable I set in the unit while executing input set A will be lost while switching to input set B. This is extremely important because “variable noise” could lead to major uncertainty. However, if I wanted to have a variable to survive input set changes, all I would have to do is use the local_ prefix. As in local_items, local_count.

Most of the times there’s a more conventional way to do things, however when stuff gets nasty, you now know you can use these two tricks to make your life easier.

Data Manipulation Tricks (Random Numbers & Strings)

We present you some tricks that you can use to solve common problems while writing tests.

Unique Emails

If you want to generate a unique random email you can incorporate the current epoch in it. Epochs progress every millisecond so it’s pretty improbable to have a duplicate! The D language extension (see here for more details about this extension) provides date manipulation utilities, so:

<string>${D.nowMillis()}@gmail.com

if ‘string’ is ‘apifortress’ the result will be something like: apifortress1489051395378@gmail.com

String Manipulation

If a data item is a composite string of values and you need to evaluate portions of it, there are numerous ways you can do it.

Let’s say we have this string “55,5,84,65,26,7,82” and you want to work with each number. The simpler way is transforming the string into an array. You can do using the ‘split’ function:

<variable_name>.split('<separator>')

In the example above, if the string is included in a variable named ‘randomNumbers’ writing randomNumbers.split(‘,’) you produce an array like [55,5,84,65,26,7,82]. At this point you can work with it as you generally do when you have an array.

Another trick you can use to improve your tests is extracting something from a string using the ‘substring’ functionality. If you need to extract the first n characters from a string there is a function that helps you: take(n).

Let’s consider the following string: “whatever” saved in a variable named “substring_variable”, if you:

substring_variable.take(4)

“what” will be returned.

Another function for a string is drop(n) that removes the first n characters and return the resulting String. Considering the same string and variable used as example for take(n), if you:

 substring_variable.drop(4)

“ever” will return.

The last example is extracting a substring using index ranges. If we have the string “wherever” by doing:

wherever[1..4]

we will obtain “here”

Dates Manipulation

Maybe your endpoint requires a date in a specific format as parameter.

${D.format(D.nowMillis(),'MM/dd/YY')}

This expression will generate today date (e.g. 03/09/17)

 

Working With the Header (status code, latency, fetch)

An HTTP response is made of a payload (what you are mainly interested in), but also contains contextual information. Using API Fortress you can also deal with the whole response envelope.

When you’re making an HTTP request in the composer, you’re providing a variable name. That variable will host the entire response payload. So let’s say “payload” is the name of that variable. When the operation completes, another variable called <variable_name>_response is also created. Therefore various pieces of information such as HTTP header and metrics are contained in the variable “payload_response.”

So by referencing the “payload_response.statusCode” expression you can access the status code. For example, if you want to run a branch of code when the status code is 400, here’s how you do it:

statusCode

You can have multiple ‘IF’ conditions for checking all the possible status codes you need to check. Very useful for creating positive and negative tests.

multiStatusCode

Headers are also a big part of the response.

Let’s say you want to check that a resource shouldn’t be cached:

response_headers

Furthermore, you can have the performance of the call verified as well. Here is what it looks like in the CODE view:

<assert-less expression="payload_response.metrics.latency" value="350" type="integer"/>

<assert-less expression="payload_response.metrics.fetch" value="350" type="integer"/>

<assert-less expression="payload_response.metrics.overall" value="550" type="integer"/>

Visual Composer view (use the Assert-Less component):

metric test 1 metric test 2

Please note that latency is defined as time to first byte. Fetch is the total download time of the payload.

Connector – BigPanda.io

We love making life easier for our customers. That often means helping them unify their tools. BigPanda is one of those great tools that can help with that process, so we made sure to create included them as one of our connectors! To see this doc on their site click here.

  1. Login to your BigPanda account, and click on the Integrations tab, and then on New Integration.step 1
  2. Click on Monitoring and then Alerts REST API.
    step 2
  3. Next enter a name and click on Generate App Key.
    step 3
  4. Finally, keep the App Key and Authorization code handy for later.
    step 4
  5. Login to API Fortress and click the gear icon. Then click on Alert Groups. This is where all the connectors are located.
    apif step 1
  6. Type in a label (name).
  7. Click on the Edit Connectors button, and then on +Connector to this Group.
    apif step 3
  8. Scroll down to BigPanda.io.
    apif step 4
  9. Remember that App Key and Authorization? Copy and paste them here!
    apif step 5
  10. Click the checkmarkand you are done! All alerts from scheduled events are now being sent to BigPanda.

The Working Copy / Published Tests

We’ve been asked this question a few times so we wrote a post to make it clear.

When you work on a test you are not editing the actual test, but a working copy (a clone) of the test. The working copy can be edited, reset, and saved to continue the work later, but none of these actions alter the state of the live test.

When you’re done editing you will need to PUBLISH the test so that the working copy replaces the current live test to become official.

testControlPanel

Why

Well, API Fortress is meant to alert people when something goes wrong. If you are editing a test and an incomplete test runs based on the schedule, you will receive a failure notification. A false positive.

By using the working copy to make your edits, the scheduler will not execute a test unless it is finalized and published.

To Summarize

  • You don’t edit the live test. When you edit one, you’re actually editing the working copy.
  • If you want to start over from the live test you can ‘Clear Working Copy’ to create a fresh version of the live test that you can edit.
  • When you are done editing click ‘Publish’ so your working copy goes live and the scheduler will use this new test for its executions.
  • If you create a new test but never publish it, the test will be marked as ‘incomplete.’

Improving Your Metrics

The performance of the API can be mission critical in some cases, and cataloging metrics can be as important as collecting them.

The classic approach of creating big tables of HTTP hits with the actual URL being called and its performance is certainly accurate but far from being easy to review because URLs contain IDs and hardly represent what the action was about.

API Fortress, as a default, works in this “classic” (dull?) way, but gives you to possibility to change the “footprint” of requests, based on your organization needs.

To activate this feature you need to add a “config” component to your I/O operation to reconfigure the footprint.

footprint

In this example, most of the request remains the same, but the ID value is actually replaced with the static string “[id]”, therefore making the calls with this pattern equivalent when browsed in the project dashboard.

metrics

And this is just one of the approaches. You’re virtually free to write the footprints that best fit your needs.

Outline a Plan For Your Test

Before you start writing your test, you should plan what you want to do.

Contrary to most testing frameworks, API Fortress is not endpoint centric, so there’s a number of different paths you can take to make the best of your test.

Single Action Test (active)

You’re willing to write a test that is going to verify how an endpoint action works. We’re also considering this scenario to be active, which means you expect API Fortress to perform a call and test the outcome.
If your action takes multiple parameters, we suggest you to create a number of input sets containing combinations of valid input. The unit will be run against each input set. Having multiple inputs is important because it’s the only way you can tell your API is responding uniformly, no matter what you feed it.inputs
Then you need to make a request and use those variables.

get

As you can see we’re composing the URL using global variables and we reference the variables in the query parameters.

Eventually, you will write your test. This is the simplest API Fortress test, covering the need to verify if an endpoint is up and running and working just fine.

Single Action Test (passive)

A test is passive when API Fortress won’t retrieve the payload on its own but will have it forwarded via the API. In this case, no input sets or I/O operations are necessary. The forwarded payload can be referenced using the “payload” variable. The ideal scenario for this mode is testing live request and responses.

Single Action Test (active, just-in-time parameters)

Also an input set can be overridden using the API. If your process requires an agent to trigger an active test with fresh inputs, create only one input set with valid inputs and have your API call override them. If you run the test by schedule, it test will run against the input you manually inserted. If you run it using the API and override the inputs, your test can run against fresh data all the time.

The ideal scenario for such mode is an application where the API is public but the inputs vary a lot over time and cannot be easily determined.

Single Action Test (mixed mode)

There’s a way to have a test to work both in an active way (when triggered by a schedule or manually launched, for example) and in an active way (invoked via API).
Here’s how you do it:

ifnotpayload

The GET will be performed only if payload does not have a value (yet).

If the test has multiple input sets that you don’t want to run when the payload is being forwarded, invoking the API with the “nosets=true” attribute will prevent it. This is generally a preferred over a plain passive test.

Integration test

In integration tests you chain together multiple calls, from authentication and actions to complete flows.

Possible scenarios vary from catalog browsing…

int1

… to full booking process for multiple flights

bookings