The Variables System in API Fortress

When an API Fortress test is executed, a variable stack is built. Every variable in the scope is accessible through API Fortress expressions and components.

The contribution to the variable stack happens in this order:

  • Global Vault
  • Project Vault
  • Test’s global variables
  • Test’s input set variables
  • Selected environment variables (also referred to as “Overrides”)
  • SET components within the test

The order is important because during the run-down if a variable is redefined, then it gets rewritten.
Say, for example, the the “domain” variable is declared in the Global Vault. Then this variable needs not to be redeclared in the other 5 stages to retain its original value.
Given the nature of these variables stores, a strategy naturally arises.

Global Vault

This is a place to store variables that live across the whole organization. Variables in this area are critical and generally stable.

Example: assume your whole organization has two domains performing two different things, and that’s all the testing you will do with API Fortress. Then two ideal domain_1 and domain_2 variables would be probably be placed here.

Project Vault

In this vault, you will store variables that serve as a common ground for all tests within a project.

Following our previous example, assuming the two services need to use two different authentication tokens, then this is the place where an auth_token variable should be stored.

Test Global

This is where you store variables that are specific to the test but are not related to a specific scenario.

For example, if you’re testing the /api/products API, then you can store the endpoint as a variable in the Test Global.

Test Input Set

This is where you store variables that are specific to a scenario.

For example, if you’re testing the /api/product/:id API, the ID is specific to a scenario, so that should be a variable in the input set. The ability to have multiple input sets in a set allows you to run a test against multiple scenarios.

Environment / Overrides

An environment (or override) is a set of variables that represent a temporary change in the premises of a test.

For example if, as a default, the domain variable reflects a production environment, but you occasionally want to run the test against a staging environment, this is where you redefine the variable. Being “environment” the last step prior to test execution, it will win over the default.

Note:
 as a general rule of thumb, we strongly advocate for not relying on environments for the test to function properly. Tests should be self-sufficient and already be able to run before the environment stage, which should be used to change the value to certain variables, not defining new ones.
 

SET Component

Finally, the SET component allows you to create variables during test execution. This is useful when a certain variable acquires a value based on what happens during the test, such as a key in a payload, or a computed value.

Downloader 101

What is the Downloader:

The API Fortress Downloader is the agent that retrieves the resources (payloads) to be tested. For cloud customers, we have various downloaders already available to you such as US East, US West, and Europe. 

For self hosted (on-premises) customers it is important that you deploy the Downloaders along with the API Fortress Dashboard. Without at least one Downloader, the dashboard doesn’t have the ability to make the API call and get the response. You can deploy multiple Downloaders in various locations, so factors such as latency and download time can be measured more precisely.


Local (Hybrid) Use of Downloader:

The API Fortress Remote Download Agent can also sit inside of your infrastructure to allow the cloud (SaaS) platform to test systems that are not exposed externally. It will listen to an HTTPS port for jobs requested by an API Fortress engine. The agent will perform an HTTP(S) request to an endpoint as described in the job, and once completed will serialize the data back to the engine, adding contextual information such as the metrics. No data is retained in the agent memory after job completion. The agent will use the DNS settings provided by the machine it’s installed on.

The Downloader Is Very Configurable:

You may disable SSL validation:
https://apifortress.com/doc/disable-ssl-validation/

It can be configured to go through a proxy:
https://apifortress.com/doc/proxy-settings-in-downloader/

The Downloader (aka: RemoteDownloadAgent) receives inbound HTTPS connections from the dashboard, encrypting everything with its own certificate. 

However, you can also install an actual certificate to a RemoteDownloadAgent:
https://apifortress.com/doc/keystores-for-downloader/

 

Input Sets and Global Variables

In the test creation process, the definition of global and local variables allows you to parametrize the test to allow more flexibility.

The global variables are meant to be the ones that are common for the whole test (i.e. api key, domain etc). For adding a global variable click on the +Add Global Param in the Data Set panel and fill the name and value.

variable

globalThe variable will be present in the test scope, and will be constant during the test execution.

The input set, instead, is a group of input variables representing a scenario (i.e product id). To add in input set, click the + button in the Input Set section and add a name for the set. Then click + Add Param and add the variable name and value.

inputSet

The unit will be executed once for each input set you have defined. At each iteration, one input set will enter the scope.

If you define a variable both in the global section and the input set, the value used in the test will be the one defined in the input set.

The variables can be defined also in the Vault (for more info see here) and in the Schedule (for more info see here).

The priority order is:

  • if the same variable is defined in the Vault and in the composer (no matter if it is a global variable or an input set), the one defined in the composer will be used.
  • if the same variable is defined in the composer both as global and input set, the value of the input set will be used
  • if the same variable is defined in the Vault (or in the composer) and in the scheduler, the variable defined in the scheduler will be used for the tests.

Using Test State Variables

API tests should rarely need to know facts about themselves and take decisions off those. In fact a test shouldn’t take decisions based on its state, but on what the tested material looks like. Unfortunately, the real world is not always this perfect and forces you to build introspective tests. API Fortress provides you two mini hacks to extend the test logic.
  • The ws_failures_count variable will allow you to know how many assertions failed while running the unit against the current input set. You might want to use it to stop the test if a certain number of failures has been reached, or decide whether or not to perform a certain branch.
  • The local_ variables. As you know input sets are isolated. This means that any variable I set in the unit while executing input set A will be lost while switching to input set B. This is extremely important because “variable noise” could lead to major uncertainty. However, if I wanted to have a variable to survive input set changes, all I would have to do is use the local_ prefix. As in local_items, local_count.
Most of the times there’s a more conventional way to do things, however when stuff gets nasty, you now know you can use these two tricks to make your life easier.

Data Manipulation Tricks (Random Numbers & Strings)

We present you some tricks that you can use to solve common problems while writing tests.

Unique Emails

If you want to generate a unique random email you can incorporate the current epoch in it. Epochs progress every millisecond so it’s pretty improbable to have a duplicate! The D language extension (see here for more details about this extension) provides date manipulation utilities, so:
<string>${D.nowMillis()}@gmail.com
if ‘string’ is ‘apifortress’ the result will be something like: apifortress1489051395378@gmail.com

String Manipulation

If a data item is a composite string of values and you need to evaluate portions of it, there are numerous ways you can do it. Let’s say we have this string “55,5,84,65,26,7,82” and you want to work with each number. The simpler way is transforming the string into an array. You can do using the ‘split’ function:
<variable_name>.split('<separator>')
In the example above, if the string is included in a variable named ‘randomNumbers’ writing randomNumbers.split(‘,’) you produce an array like [55,5,84,65,26,7,82]. At this point you can work with it as you generally do when you have an array. Another trick you can use to improve your tests is extracting something from a string using the ‘substring’ functionality. If you need to extract the first n characters from a string there is a function that helps you: take(n). Let’s consider the following string: “whatever” saved in a variable named “substring_variable”, if you:
substring_variable.take(4)
“what” will be returned. Another function for a string is drop(n) that removes the first n characters and return the resulting String. Considering the same string and variable used as example for take(n), if you:
 substring_variable.drop(4)
“ever” will return. The last example is extracting a substring using index ranges. If we have the string “wherever” by doing:
wherever[1..4]
we will obtain “here”

Dates Manipulation

Maybe your endpoint requires a date in a specific format as parameter.
${D.format(D.nowMillis(),'MM/dd/YY')}
This expression will generate today date (e.g. 03/09/17)  

Working With the Header (status code, latency, fetch)

An HTTP response is made of a payload (what you are mainly interested in), but also contains contextual information. Using API Fortress you can also deal with the whole response envelope. When you’re making an HTTP request in the composer, you’re providing a variable name. That variable will host the entire response payload. So let’s say “payload” is the name of that variable. When the operation completes, another variable called <variable_name>_response is also created. Therefore various pieces of information such as HTTP header and metrics are contained in the variable “payload_response.” So by referencing the “payload_response.statusCode” expression you can access the status code. For example, if you want to run a branch of code when the status code is 400, here’s how you do it: statusCode You can have multiple ‘IF’ conditions for checking all the possible status codes you need to check. Very useful for creating positive and negative tests. multiStatusCode Headers are also a big part of the response. Let’s say you want to check that a resource shouldn’t be cached: response_headers Furthermore, you can have the performance of the call verified as well. Here is what it looks like in the CODE view:
<assert-less expression="payload_response.metrics.latency" value="350" type="integer"/>

<assert-less expression="payload_response.metrics.fetch" value="350" type="integer"/>

<assert-less expression="payload_response.metrics.overall" value="550" type="integer"/>
Visual Composer view (use the Assert-Less component): metric test 1 metric test 2 Please note that latency is defined as time to first byte. Fetch is the total download time of the payload.

The Working Copy / Published Tests

We’ve been asked this question a few times so we wrote a post to make it clear. When you work on a test you are not editing the actual test, but a working copy (a clone) of the test. The working copy can be edited, reset, and saved to continue the work later, but none of these actions alter the state of the live test. When you’re done editing you will need to PUBLISH the test so that the working copy replaces the current live test to become official. testControlPanel

Why

Well, API Fortress is meant to alert people when something goes wrong. If you are editing a test and an incomplete test runs based on the schedule, you will receive a failure notification. A false positive. By using the working copy to make your edits, the scheduler will not execute a test unless it is finalized and published.

To Summarize

  • You don’t edit the live test. When you edit one, you’re actually editing the working copy.
  • If you want to start over from the live test you can ‘Clear Working Copy’ to create a fresh version of the live test that you can edit.
  • When you are done editing click ‘Publish’ so your working copy goes live and the scheduler will use this new test for its executions.
  • If you create a new test but never publish it, the test will be marked as ‘incomplete.’

Improving Your Metrics

The performance of the API can be mission critical in some cases, and cataloging metrics can be as important as collecting them. The classic approach of creating big tables of HTTP hits with the actual URL being called and its performance is certainly accurate but far from being easy to review because URLs contain IDs and hardly represent what the action was about. API Fortress, as a default, works in this “classic” (dull?) way, but gives you to possibility to change the “footprint” of requests, based on your organization needs. To activate this feature you need to add a “config” component to your I/O operation to reconfigure the footprint. footprint In this example, most of the request remains the same, but the ID value is actually replaced with the static string “[id]”, therefore making the calls with this pattern equivalent when browsed in the project dashboard. metrics And this is just one of the approaches. You’re virtually free to write the footprints that best fit your needs.

Outline a Plan For Your Test

Before you start writing your test, you should plan what you want to do. Contrary to most testing frameworks, API Fortress is not endpoint centric, so there’s a number of different paths you can take to make the best of your test.
Single Action Test (active)
You’re willing to write a test that is going to verify how an endpoint action works. We’re also considering this scenario to be active, which means you expect API Fortress to perform a call and test the outcome. If your action takes multiple parameters, we suggest you to create a number of input sets containing combinations of valid input. The unit will be run against each input set. Having multiple inputs is important because it’s the only way you can tell your API is responding uniformly, no matter what you feed it.inputs Then you need to make a request and use those variables. get As you can see we’re composing the URL using global variables and we reference the variables in the query parameters. Eventually, you will write your test. This is the simplest API Fortress test, covering the need to verify if an endpoint is up and running and working just fine.
Single Action Test (passive)
A test is passive when API Fortress won’t retrieve the payload on its own but will have it forwarded via the API. In this case, no input sets or I/O operations are necessary. The forwarded payload can be referenced using the “payload” variable. The ideal scenario for this mode is testing live request and responses.
Single Action Test (active, just-in-time parameters)
Also an input set can be overridden using the API. If your process requires an agent to trigger an active test with fresh inputs, create only one input set with valid inputs and have your API call override them. If you run the test by schedule, it test will run against the input you manually inserted. If you run it using the API and override the inputs, your test can run against fresh data all the time. The ideal scenario for such mode is an application where the API is public but the inputs vary a lot over time and cannot be easily determined.
Single Action Test (mixed mode)
There’s a way to have a test to work both in an active way (when triggered by a schedule or manually launched, for example) and in an active way (invoked via API). Here’s how you do it: ifnotpayload The GET will be performed only if payload does not have a value (yet). If the test has multiple input sets that you don’t want to run when the payload is being forwarded, invoking the API with the “nosets=true” attribute will prevent it. This is generally a preferred over a plain passive test.
Integration test
In integration tests you chain together multiple calls, from authentication and actions to complete flows. Possible scenarios vary from catalog browsing… int1 … to full booking process for multiple flights bookings