Hacks : test state variables

API tests should rarely need to know facts about themselves and take decisions off those. In fact a test shouldn’t take decisions based on its state, but on what the tested material looks like.

Unfortunately the real world is not always this perfect and forces you to build introspective tests. API Fortress provides you two mini hacks to extend the test logic.

  • The ws_failures_count variable will allow you to know how many assertions failed while running the unit against the current input set. You might want to use it to stop the test if a certain number of failures has been reached, or decide whether or not to perform a certain branch.
  • The local_ variables. As you know input sets are isolated. This means that any variable I set in the unit while executing input set A will be lost while switching to input set B. This is extremely important because “variable noise” could lead to major uncertainty. However, if I wanted to have a variable to survive input sets change, all I have to do is using the local_ prefix. As in local_items, local_count.

Most of the times there’s a more conventional way to do things, however when stuff gets nasty, you now know you can use these two tricks to make your life easier.

Integration Testing – Multistep Tests

One of the core features of the platform is the ability to create proper integration tests.

An Integration test is when you test a complete flow of calls, simulating what an API consumer would experience. Exercising singular endpoints leaves a lot of opportunities for missed bugs and vulnerabilities.

Here is a quick guide on how to do it using a token based authentication API.

Even easier: Login to your account and go to the Examples project. There is an example test named “Retail: Integration – Products.” It’s easier to understand by looking at it first.

First, get the token by making your POST call. Name the payload variable “loginPayload.”

login

Under the login procedure add an assertion named “Set (variable).” Set Var as “access_token,” keep Variable Mode as String, and set Value to “${loginPayload.access_token}” You’ll notice that what we are doing here is specifically taking the access_token variable in the loginPayload response, and setting it as “access_token.”
Note: The dollar sign and bracket are important when referencing variables.
tokenaccess_token

Next, make a GET call to your search endpoint. Name this payload as “resultsPayload.” This call requires that access token from the login procedure. So we passed it in the header as required for this specific API. We added Header Name as “Authorization” and Value as “${access_token}.” Again, notice the dollar sign and bracket. That is how you reference variables.

search

Finally, let’s dive into each result from the search payload one-by-one, using the “product id” and the “access token” variables we have set so far.

  1. Add a “for each” assertion and reference the “resultsPayload.products” object.
  2. Add a new “Set (variable)” assertion to set the “id” variable as every single “resultsPayload.product” that is returned. Notice we set the string to “${_1.id}” The system uses _1 automatically when recognizing a subroutine. Makes it easier when things get into many sub levels.
  3. Make a GET to the product details endpoint, using our new “id” variable as the id parameter. Again, you can still reference the original “${access_token}” to make this call. Variables last through the entire test unless overwritten.

each

Again, take a look at the integration test in your Examples project, which comes with every account. Remember how many of us learned HTML first by looking at the source of our favorite websites? This is much easier to understand when you see it.

Data manipulation tricks

We present you some tricks that you can use to solve common problems while writing tests.

Unique emails

If you want to generate a unique random email you can incorporate the current epoch in it. Epochs progress every millisecond so it’s pretty improbable to have a duplicate! The D language extension (see here for more details about this extension) provides date manipulation utilities, so:

<string>${D.nowMillis()}@gmail.com

if ‘string’ is ‘apifortress’ the result will be something like: apifortress1489051395378@gmail.com

String manipulation

If a data item is a composite string of values and you need to evaluate portions of it, there are numerous ways you can do it.

Let’s say we have this string “55,5,84,65,26,7,82” and you want to work with each number. The simpler way is transforming the string into an array. You can do using the ‘split’ function:

<variable_name>.split('<separator>')

In the example above, if the string is included in a variable named ‘randomNumbers’ writing randomNumbers.split(‘,’) you produce an array like [55,5,84,65,26,7,82]. At this point you can work with it as you generally do when you have an array.

Another trick you can use to improve your tests is extracting something from a string using the ‘substring’ functionality. If you need to extract the first n characters from a string there is a function that helps you: take(n).

Let’s consider the following string: “whatever” saved in a variable named “substring_variable”, if you:

substring_variable.take(4)

“what” will be returned.

Another function for a string is drop(n) that removes the first n characters and return the resulting String. Considering the same string and variable used as example for take(n), if you:

 substring_variable.drop(4)

“ever” will return.

The last example is extracting a substring using index ranges. If we have the string “wherever” by doing:

wherever[1..4]

we will obtain “here”

Dates manipulation

Maybe your endpoint requires a date in a specific format as parameter.

${D.format(D.nowMillis(),'MM/dd/YY')}

This expression will generate today date (e.g. 03/09/17)

 

Working with the Response object

An HTTP response is made of a payload -what you are mainly interested in- but also contextual information.

Using API Fortress you generally verify the payload, but you can also deal with the whole response “envelope”.

In the composer, when you’re making an HTTP request, you’re providing a variable name. That variable will host the response payload.

Let’s say “payload” is the name of that variable. When the operation completes, another variable called <variable_name>_response is also created, in this example “payload_response”. That variable will host the response “envelope”.

So by referencing the “payload_response.statusCode” expression you can access the status code. For example, if you want to run a branch of code when the status code is 400, here’s how you do it:

statusCode

You can have multiple ‘IF’ conditions for checking all the possible status codes you need to check.

multiStatusCode

Headers are also a big part of the response.

Let’s say you want to check that a resource shouldn’t be cached:

response_headers

The Working Copy / Published Tests

We’ve been asked this question a few times so we wrote a post to make it clear.

When you work on a test you are not editing the actual test, but a working copy (a clone) of the test. The working copy can be edited, reset, and saved to continue the work later, but none of these actions alter the state of the live test.

When you’re done editing you will need to PUBLISH the test so that the working copy replaces the current live test to become official.

See the "publish" button top right?
See the “publish” button top right?

Why

Well, API Fortress is meant to alert people when something goes wrong. If you are editing a test and an incomplete test runs based on the schedule, you will receive a failure notification. A false positive.

By using the working copy to make your edits, the scheduler will not execute a test unless it is finalized and published.

To summarize

  • You don’t edit the live test. When you edit one, you’re actually editing the working copy.
  • If you want to start over from the live test you can ‘Clear Working Copy’ to create a fresh version of the live test that you can edit.
  • When you are done editing click ‘Publish’ so your working copy goes live and the scheduler will use this new test for its executions.
  • If you create a new test but never publish it, the test will be marked as ‘incomplete.’

Improving Your Metrics

The performance of the API can be mission critical in some cases, and cataloging metrics can be as important as collecting them.

The classic approach of creating big tables of HTTP hits with the actual URL being called and its performance is certainly accurate but far from being easy to review because URLs contain IDs and hardly represent what the action was about.

API Fortress, as a default, works in this “classic” (dull?) way, but gives you to possibility to change the “footprint” of requests, based on your organization needs.

To activate this feature you need to add a “config” component to your I/O operation to reconfigure the footprint.

footprint

In this example, most of the request remains the same, but the ID value is actually replaced with the static string “[id]”, therefore making the calls with this pattern equivalent when browsed in the project dashboard.

metrics

And this is just one of the approaches. You’re virtually free to write the footprints that best fit your needs.

Writing Your First Test

The easiest way to write your first test is using the tool we named Magic. To put it simply, with Magic you can make an API call and then it will automatically create a test based on the payload.

First of all, you need to make an API call with the console. For this example we are using our own test API:
https://mastiff.apifortress.com/app/api/examples/retail/product?id=511

1

Type in the URL and then press the Send button.

Next, tap the magic wand icon at the top right corner of the Console to launch the Magic tool.

1b

The following screen will allow you to choose whether you want to create the input set (based on the data provided in the request) or skip it. Press Continue.

2

The next screen, allows you to choose if you want to create the assertions. You do, so press Continue once again.

3

At the end you will see a report about the applied changes.

4

Here is the result: a fully working draft of an API payload test.

5

Now, you can launch the test and see the report….

6

Tests are meant to validate an endpoint, so it makes a lot of sense to add more input sets and have the unit run against them.

 

7

Magic is able to understand basic things based on a single request and response. Refining it is mostly about transforming vague assertions to be more specific.
For example, we know that payload.content.options.mainColor can be a finite number of values. Therefore, using an “assert exist” assertion is a bit poor…

8

So let’s change it to an “assert in” that verifies the field value to be either “red” or “black_plain.”

9

Done! Your first test is ready to be scheduled now.

Outline a Plan For Your Test

Before you start writing your test, you should plan what you want to do.

Contrary to most testing frameworks, API Fortress is not endpoint centric, so there’s a number of different paths you can take to make the best of your test.

Single Action Test (active)

You’re willing to write a test that is going to verify how an endpoint action works. We’re also considering this scenario to be active, which means you expect API Fortress to perform a call and test the outcome.
If your action takes multiple parameters, we suggest you to create a number of input sets containing combinations of valid input. The unit will be run against each input set. Having multiple inputs is important because it’s the only way you can tell your API is responding uniformly, no matter what you feed it.inputs
Then you need to make a request and use those variables.

get

As you can see we’re composing the URL using global variables and we reference the variables in the query parameters.

Eventually, you will write your test. This is the simplest API Fortress test, covering the need to verify if an endpoint is up and running and working just fine.

Single Action Test (passive)

A test is passive when API Fortress won’t retrieve the payload on its own but will have it forwarded via the API. In this case, no input sets or I/O operations are necessary. The forwarded payload can be referenced using the “payload” variable. The ideal scenario for this mode is testing live request and responses.

Single Action Test (active, just-in-time parameters)

Also an input set can be overridden using the API. If your process requires an agent to trigger an active test with fresh inputs, create only one input set with valid inputs and have your API call override them. If you run the test by schedule, it test will run against the input you manually inserted. If you run it using the API and override the inputs, your test can run against fresh data all the time.

The ideal scenario for such mode is an application where the API is public but the inputs vary a lot over time and cannot be easily determined.

Single Action Test (mixed mode)

There’s a way to have a test to work both in an active way (when triggered by a schedule or manually launched, for example) and in an active way (invoked via API).
Here’s how you do it:

ifnotpayload

The GET will be performed only if payload does not have a value (yet).

If the test has multiple input sets that you don’t want to run when the payload is being forwarded, invoking the API with the “nosets=true” attribute will prevent it. This is generally a preferred over a plain passive test.

Integration test

In integration tests you chain together multiple calls, from authentication and actions to complete flows.

Possible scenarios vary from catalog browsing…

int1

… to full booking process for multiple flights

bookings