Use the Vault

The Vault is a unique part of the API Fortress platform. Not in terms of the idea, but in the flexibility offered by ours. It allows you to save, edit, and reuse almost anything. This can mean variables, code snippets (think reused authentication flows), or even data!

Learn the Basics

Advanced Use Cases

The IF Component

One of the most powerful components in API Fortress (and in programming) is the IF. This gives you the flexibility to create assertions that can be specific to certain conditions. For example, IF statuscode==200 vs statuscode=400. Meaning you can create positive and negative validations in a single test.

Automatically

By API

The platform has extensive APIs specifically created to empower engineers and testers to execute tests however they choose. You can read more about integration into a CI pipeline here. For specific use cases see some below, or search our docs.

By Command-Line Tool

The platform also has command-line tools available to paying customers, or potential customers doing a proof of concept. It allows full execution of test, and is particularly useful for CI pipelines. It can help return a lot of useful test report data right back into your CI platforms, like Jenkins. [Learn More]

Best Practice — Advanced

Variablize for Test Reuse

Domain, endpoints, and data sets are some of the parts of a test that should be easy to change quickly. You should never have to duplicate a test to execute it against a new staging environment. If you are doing that than your system needs to be reconsidered.

As much as possible aim to have only a single test of a certain type. If you built a partner API, then try to have a single detailed test that can be a regression test during a CI pipeline, and then also used to monitor the live environment. [Learn More]

Business Logic Validating

As described in Course 1, it is important to go beyond the technical facts of a payload. You should also have knowledge as to the business objects. This means that if the Uber payload includes a reference to UberSidecar, but Uber hasn’t announced a Sidecar product yet, this is a major issue for them from a technical and business secrets perspective.

Perhaps they have a major advertising initiative to announce the ability to order a ride in the sidecar of a motorcycle, and this would undermine that. It’s important that your tests are knowledgeable about the goal of the APIs technically, as well as from a business perspective.

Using Lots of Data

One of the most common errors we see with testing is using the same small set of data for every test. A CSV with 50 product IDs is a good start, but now that we can automate this work there is no reason to not write a much more comprehensive test that can validate hundreds of product IDs. You must break free from using small CSV files for datasets.

If you are already using databases or APIs that’s great! If not, we suggest you start this process as well. A test is only as good as the amount of information it is testing against.
[Learn More Using Files | Using JDBC]

Integration / End-to-End Test

Most API programs are a collection of APIs that are meant to interact with each other. Therefore, it is important to create tests that do exactly that. A singular test that goes through a series of calls and validates the responses at each stage. This is what is called an integration test, although it is also called an end-to-end test.

This means writing detailed tests that reproduce normal user flows. For example, if we are talking about an ecommerce platform, a normal user flow would be:

Search > View Product Details Page > Add to Cart > Checkout

A functional test for just the Search API is a great start, but the entire flow also needs to be validated. You should have as many integration tests as there are expected user flows. [Learn More]

Confirm Response Matches Request

This is in the same thinking as the above, but with that advent of PSD2 and Open Banking it’s worth mentioning on its own. When you are requesting information, such as your personal banking history, it is important to make sure only your information is returned. It is also important to validate you can do wildcard searches and get the results of other.

That is exactly the sort of functional error that causes a huge vulnerability for large organizations like the United States Post Office.

Integrations Galore

Every company has an existing workflow, and it’s important to choose a platform that helps keep those integrations seamless. In the basics we talked about integrating with your CI pipeline, but it’s also useful to have notifications and data in one place. If your organization has a lot invested in Slack, Kibana, and TestRail (for example) than having API test notifications and results in one place is imperative. [Learn More]

3. Creating an Integration Test / Using an API as a Datasource

Now let’s take our existing functional test above and use it as the first step in an integration test. Not only that, but that first API call actually contains an array of product IDs. What if we use it as a datasource, and then iterate on each of them individually?

First, we have the original test:

Now, to do this work requires the use of two new components. The For Each component helps you iterate through a series of data (product IDs in this case), and the Set which creates a temporary variable to reference. This should seem fairly straight forward if you have taken some introduction computer science and programming classes.

If you look at the above GUI view of this test, you see that we have our original test. It makes the first API call, and then tests each object in that response. Then, and this is where this becomes an integration test that is using an API to be data-driven, we have that For Each component. You’ll notice that it is referencing the variable that we stored the entire response, then the array, and finally the object we want to use – productsPayload.content.products.pick(2). You’ll notice the .pick(2) we added at the end. This is entirely optional, and what it does is randomly select X number of items from the list we are iterating through. This is useful if the dataset is far too large. Ours is small so we would remove that and allow every product ID to be tested. We will leave it for training purposes.

Next we have the Set component, which is creating a new variable called id. The value of that variable depends on the location the For Each is at in the iteration. Here we set the value as ${_1}. This is saying “using the current location in the array”. Traversing an array is a bit of a larger topic, so we’ll get into that later.

Finally, you see the next GET component. You’ll notice we are using the id that we stored from the first GET call to populate into the second GET call. Each iteration of the For Each will call the current id in the subsequent GET call.

That’s it! What we’ve done here is made an API call to the products list API, then tested that API thoroughly, and then used that API as the datasource for the subsequent GET which dives into each of those product IDs individually. You then want to test that entire API response as well. So here is what that entire test looks like complete in the GUI.

To see the code itself, go to the Examples directory and open the test Retail: Integration – Products.

end to end, iterate

2. Building a Strong Functional Test

Note: This example exists in your Example project, and it is named Retail: Integration – Products.

Ok, so let’s build our first functional test! We will start by using this API call.
(https://mastiff.apifortress.com/app/api/examples/retail/products)

By clicking that link, or using the HTTP composer to make the call, you get this response.

As you can see there are 5 objects, so at minimum we want to make sure they ‘exist.’ What API Fortress allows you to do is validate the objects exist and the data is as expected. This is done using our XML markup language, or our GUI composer.

*Side Bar*
API Fortress was specifically built to bridge the gap between testers and engineers, allowing you to write detailed API tests in whatever format you are most comfortable with. Our composer has a drag-and-drop interface that writes the XML code for you, and it also makes it easier to visually understand the nature of the test. With that said, our GUI composer also has a Code View which exposes the underlying XML. This XML can be written or edited in the GUI, or using your own IDE. Meaning that how you want to write and/or edit a test is completely unlocked, so you can work how you are most comfortable.

In the GUI composer view that test looks like this:

If you clicked on Code, or were looking at the test in your own IDE, this is what you’d see:

<unit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="main" xsi:noNamespaceSchemaLocation="http://apifortress.com/app/unit.xsd">
   <requirements />
   <configs />
   <sequence name="main">
      <get url="${protocol}${domain}${productsEndpoint}" params="[:]" var="productsPayload" mode="json" />
      <assert-equals expression="productsPayload.status" value="200" comment="" />
      <assert-is expression="productsPayload.success" type="boolean" comment="" />
      <assert-exists expression="productsPayload.content.act" comment="" />
      <assert-is expression="productsPayload.content.products" type="array" comment="" />
   </sequence>
</unit>

Now if we look at the code itself carefully, it’s pretty clear what we’re seeing.

  • The unit section is referencing our markup Namespace file (useful when using your own IDE).
  • Requirements and Sequence are more advanced feature we can discuss later.
  • GET is where we make the API call. Within this call it contains the necessary information to make the call (can include more if we’re dealing with oAuth for example). Some important things to note is how we have variablized protocol, domain, and then the chosen endpoint. You’ll also notice the Data Sets section on the left contains those same variables and the data associated. This is hugely important to allow the test to be easily run against any environment. We will go into more detail on that later, but think of it as futureproofing the test from location specificity.
    Then there is also var=”productsPayload”. What this does is call an API, gets a response, and then stores that entire payload into a variable called productsPayload. This is then referenced in the next assertions.
  • Note: Performance can be an assertion within the test itself. See this page for more information.
  • Now we get to the assertions! There are over 70 assertions, so we won’t get into too much detail here, but you can see the next four lines are assertions for each object. Looking at the first we see:
    <assert-equals expression=”productsPayload.status” value=”200″ comment=””/>
    This line calls the assert-equals assertion, which validates an object exists and is equal to a chosen amount. Expression references the specific object we are talking about. You’ll notice that it has productsPayload.status. So it’s referencing the status object within the productsPayload variable. Remember, the GET call gets an API response and stores it in that variable. This is useful for when you are dealing with multiple payloads (variables) in an integration test.
    It’s important to note that with the Generate Test feature you can have this entire structure generated for you in seconds. This frees you to focus on the more important and tricky aspects of writing detailed tests.

Scheduled

Just as important as testing an API before and after release, is validating that API is always up and functioning as expected. This is the difference between Uptime (which an APM can measure) and Functional Uptime, which is a powerful and unique aspect of API Fortress.

Ultimately, it means using your existing detailed functional/integration tests, and scheduling them to run against a live environment. This way they are constantly testing for uptime, functionality, and performance. The details on using the scheduler are here, but there are some important notes.

Note 1: A test must be Published before you can schedule it. This allows for there to be a finalized version that doesn’t get affected when you start editing a test. This is because when you edit a test it creates a Working Copy.

Note 2: You should monitor your own APIs, but you can also monitor third party APIs! Have a partner API with an SLA? Make sure that API is working as contractually agreed upon.

1. What is an API?

APIs are the term used to describe web services that connect different platforms or data services together. An example is the Uber application. The way that you search, book, and pay are all done through APIs.

APIs are not complicated, they are just data. When you search for vehicles in the Uber API, it makes this HTTP call:

https://api.uber.com/v1/products?server_token=[token]&latitude=40.6797300818661&longitude=-73.9639477463489

The response is in JSON format and human readable. Here is a snippet:

{
"products": [
{
"capacity": 2,
"product_id": "929fcc19-8cb4-4007-a54f-3ab34473700f",
"Price_details": {
"service_fees": [],
"cost_per_minute": 0.74,
"distance_unit": "mile",
"minimum": 8,
"cost_per_distance": 1.62,
"base": 0,
"cancellation_fee": 5,
"currency_code": "USD"
},
"Image": "https://d1a3f4spazzrp4.cloudfront.net/car-types/mono/mono-uberpool.png",
"cash_enabled": false,
"shared": true,
"short_description": "Pool",
"display_name": "UberPool",
"product_group": "rideshare",
"description": "Shared rides, door to door"
}
]
}

Glossary

HTTP APIs

There are many different types of APIs, the term itself has become vague unfortunately. For our purposes we will always be talking about HTTP APIs. APIs that can be hit with an HTTP call.

There are two types of HTTP APIs, REST and SOAP. That is a huge topic, but from a very high level:

SOAP

This was the most commonly used format, but is now seen as a legacy technology. It’s advantage is that it’s simple and can return a bunch of data. The negative is that it is very poor for more interactive usage, like we see with mobile applications today.

REST

REST is the most common protocol we see today, and more robust overall. It’s specifically made for interactive usage, such as mobile applications.

GraphQL

Raising in popularity and made by Facebook.

api, learn, 101

The response of an API call can come in many different formats. Again, for simplicity we will just focus on the ones that apply to what API Fortress can test.

XML

This is a very straightforward format that can return in virtually any layout. An example:

<?xml version="1.0" encoding="UTF-8"?>
<XML>
   <note>
      <to>Patrick</to>
      <from>Mom</from>
      <subject>Reminder</subject>
      <body>We need milk</body>
   </note>
</XML>

JSON

This is the more commonly seen format in REST APIs today. The Uber example is in JSON. It’s more standardized and therefore cleaned for machines to understand.

Object

This is the item that has data associated with it. If you look at the Uber response again, display_name, description, and Image are examples of objects.

Assertion

A rule or specific test against a single object and/or piece of data. The API Fortress platform is powered by a proprietary XML language with over 70 assertions, that handle just about every scenario in a very quick and easy way to write.

101, introduction

3. What is Continuous Testing?

API testing isn’t easy, and there are many different ways in which a bad API can cause catastrophic harm to an organization. We will dive into best practices and details a little later, but let’s quickly run through what we’re talking about when it comes to continuous testing.

As a reference point, this is what a normal HTTP based REST API, with a JSON output looks like. Each of these types of testing are part of a continuous testing strategy. They cover a good amount, and we do our best to explain them, but keep in mind terminology and definitions are always evolving.

Functional Testing

Looking at the API above, you see a payload with various objects. The first item you should be testing is that the API responds as expected from a functional standpoint.

This means every object exists, and the responses are within expected ranges. For example the quantity object should always be a whole number. There can’t be .5 baseball caps.

Validating Business Logic

Imagine you are an ecommerce company with a product API, similar to the one we shared. First, you should be testing that each product has all the objects it expects.

The next layer is where skill and creativity in test creation are required. Let’s say that you search for ‘red’ and you receive both shoes and pants. Now all products should contain a size object, but they are very different size ranges. Adding the intelligence to a test that says, “If this is a pair of shoes, size should be between 4 and 17.” That is business logic validation. This is where you need a skilled team that knows the expectations of the API, and how to manipulate and validate them.

End-to-End / Integration Testing

APIs are meant to interact with one another, and that flow has to work in its entirety. By creating proper integration tests you can validate flows that resemble actual users behaviors. For example, creating a single test that can
Search > Pick a Size > Add to Cart> Checkout

For many companies that entire flow can be done on the API level, and should be. This allows you to confirm that when you add a product to cart, it actually works.

Monitoring

Monitoring is probably the most valuable, and yet underused, type of testing. APIs issues are often only found in live environments when using live data.

The goal of a proper monitoring strategy is to go beyond measuring uptime, but also measuring functional uptime. An API can return a statuscode of 200 and still be failing. We have even seen APIs built so that even when there is an error it returns as a 200. What’s important to note is that you can use existing functional tests as your monitor. With the right platform choice, you can schedule them to run against live environments in certain intervals. Then use that monitoring to have a real-time status page across departments.

Load Testing

This is fairly easy to understand. Make sure your APIs perform when under the stress of a lot of users. This can help find memory leaks, or under performing databases.

It’s also important, again, to use existing tests as your load tests. You will never have 10,000 concurrent users only performing a single call. They will perform different types of calls, or a series of calls with some randomness. Use existing functional and end-to-end tests.

A Summary

Ultimately, continuous testing means leveraging all of these types of tests, automatically. A constant series of functional and non-functional (performance) tests to constantly validate your APIs.