4. Using The Vault, Variables and Environemnts

Using The Vault The Vault is a unique part of the API Fortress platform. Not in terms of the idea, but in the flexibility offered by ours. It allows you to save, edit, and reuse almost anything. This can mean variables, code snippets (think reused authentication flows), or even data! The Vault has two levels, project and global, the project vault will allow use of stored values across any test within that project. Similarly the global vault will allow use of stored values across any test within any project. Using Variables and Environments If properly setup any API Fortress test can be run against any environment. You’ll notice that is what we did in our functional test by turning the API URL into three separate parts –  protocol, domain, and endpoint. This allows you to set the default location under Data Sets (in the left pane), but also you can override those amounts with the Environments tab (shown below). The Environments panel lets you change anything, not just environments. You can run the test against a certain environment, using a different API key, and datasource. This effectively allows you to have pre-set runtime variable overrides.
  To learn more about The Vault and Environments see below links: Learn the Basics, Advanced Use Cases, Environments Basics, Environments Advanced

Best Practices – Optional

Integrate with your CI/CD

Integrating your API testing into your existing CI/CD flow is key if you are trying to keep all of your deployments quick and frequent. Kicking off a test or suite of tests during the build process allows you to build your new deployment and test it all at once.

Integrate Notification/Data Analysis Systems

Set up notifications to alert you via email, slack, and more when tests fail.

Do you already use a platform to analyze all your data, well you can send all your API Fortress test data out to those platforms as well. Now all of your data can live in one place, making it easy to pull and analyze all project data.

Integrate with Test Case Managers

Integrating with your Test Case Manager and communicating the results of tests is important. Having all of your information in your test case manager including test result data makes it easy to understand the entire project. Connectors can be setup to automatically send test results to your test case manager upon completion of scheduled tests.

API Testing University

API Testing University

Courses

Course 1

API Testing Introduction

Course 1 will give you a high level understanding of what an API is and why testing APIs is important. This course will guide you on all you need to know to get started with continuous API testing.

Start Course

Course 2

Best Practices

Once you have completed Course 1, jump into Course 2 where you will learn some basic and more advances methodologies for creating good API tests.

Start Course

Course 3

Writing Your First Test

Now that you have learned the basics and some helpful tips on how to do API testing, check out Course 3. In this course you will learn how to create your first test. You will learn the different ways to begin writing a test, as well as some more advanced techniques to help you simulate real life scenarios.

Start Course

Course 4

Variables and Environments

In Course 4, you will learn how using variables and environments in your API tests can make the process easier and more effective. In this course you will learn how to use environments to reuse one test against all systems (production, staging, UAT, etc..)

Start Course

Course 5

Advanced Test Creation

Up to this point you have learned how to create tests and learned some more advanced practices (i.e. variables and environments). In Course 5, you will learn even more advanced API testing techniques, like dealing with authentications.

Start Course

Course 6

Executing Tests

Now that you know the basics and some more advanced methods for creating API tests, you will learn how to execute these tests in a couple different ways. In Course 6, you will see automated ways of kicking off and scheduling the tests you have created.

Start Course

These classes include high-level overviews of best practices for API testing that our team has gathered over six years as a leader in the API testing space. We will be constantly updating our best practices lessons over time. If you have any questions or suggestions, please contact us at support@apifortress.com.

The IF Component

One of the most powerful components in API Fortress (and in programming) is the IF. This gives you the flexibility to create assertions that can be specific to certain conditions. For example, IF statuscode==200 vs statuscode=400. Meaning you can create positive and negative validations in a single test.

Automatically

By API

The platform has extensive APIs specifically created to empower engineers and testers to execute tests however they choose. You can read more about integration into a CI pipeline here. For specific use cases see some below, or search our docs.

By Command-Line Tool

The platform also has command-line tools available to paying customers, or potential customers doing a proof of concept. It allows full execution of test, and is particularly useful for CI pipelines. It can help return a lot of useful test report data right back into your CI platforms, like Jenkins. [Learn More]

Best Practice — Advanced

Variablize for Test Reuse

Domain, endpoints, and data sets are some of the parts of a test that should be easy to change quickly. You should never have to duplicate a test to execute it against a new staging environment. If you are doing that than your system needs to be reconsidered.

As much as possible aim to have only a single test of a certain type. If you built a partner API, then try to have a single detailed test that can be a regression test during a CI pipeline, and then also used to monitor the live environment. [Learn More]

Business Logic Validating

As described in Course 1, it is important to go beyond the technical facts of a payload. You should also have knowledge as to the business objects. This means that if the Uber payload includes a reference to UberSidecar, but Uber hasn’t announced a Sidecar product yet, this is a major issue for them from a technical and business secrets perspective.

Perhaps they have a major advertising initiative to announce the ability to order a ride in the sidecar of a motorcycle, and this would undermine that. It’s important that your tests are knowledgeable about the goal of the APIs technically, as well as from a business perspective.

Using Lots of Data

One of the most common errors we see with testing is using the same small set of data for every test. A CSV with 50 product IDs is a good start, but now that we can automate this work there is no reason to not write a much more comprehensive test that can validate hundreds of product IDs. You must break free from using small CSV files for datasets.

If you are already using databases or APIs that’s great! If not, we suggest you start this process as well. A test is only as good as the amount of information it is testing against.
[Learn More Using Files | Using JDBC]

Integration / End-to-End Test

Most API programs are a collection of APIs that are meant to interact with each other. Therefore, it is important to create tests that do exactly that. A singular test that goes through a series of calls and validates the responses at each stage. This is what is called an integration test, although it is also called an end-to-end test.

This means writing detailed tests that reproduce normal user flows. For example, if we are talking about an ecommerce platform, a normal user flow would be:

Search > View Product Details Page > Add to Cart > Checkout

A functional test for just the Search API is a great start, but the entire flow also needs to be validated. You should have as many integration tests as there are expected user flows. [Learn More]

Confirm Response Matches Request

This is in the same thinking as the above, but with that advent of PSD2 and Open Banking it’s worth mentioning on its own. When you are requesting information, such as your personal banking history, it is important to make sure only your information is returned. It is also important to validate you can do wildcard searches and get the results of other.

That is exactly the sort of functional error that causes a huge vulnerability for large organizations like the United States Post Office.

Integrations Galore

Every company has an existing workflow, and it’s important to choose a platform that helps keep those integrations seamless. In the basics we talked about integrating with your CI pipeline, but it’s also useful to have notifications and data in one place. If your organization has a lot invested in Slack, Kibana, and TestRail (for example) than having API test notifications and results in one place is imperative. [Learn More]

3. Creating an Integration Test / Using an API as
     a Datasource

Now let’s take our existing functional test above and use it as the first step in an integration test. Not only that, but that first API call actually contains an array of product IDs. What if we use it as a datasource, and then iterate on each of them individually?

First, we have the original test:

Now, to do this work requires the use of two new components. The For Each component helps you iterate through a series of data (product IDs in this case), and the Set which creates a temporary variable to reference. This should seem fairly straight forward if you have taken some introduction computer science and programming classes.

If you look at the above GUI view of this test, you see that we have our original test. It makes the first API call, and then tests each object in that response. Then, and this is where this becomes an integration test that is using an API to be data-driven, we have that For Each component. You’ll notice that it is referencing the variable that we stored the entire response, then the array, and finally the object we want to use – productsPayload.content.products.pick(2). You’ll notice the .pick(2) we added at the end. This is entirely optional, and what it does is randomly select X number of items from the list we are iterating through. This is useful if the dataset is far too large. Ours is small so we would remove that and allow every product ID to be tested. We will leave it for training purposes.

Next we have the Set component, which is creating a new variable called id. The value of that variable depends on the location the For Each is at in the iteration. Here we set the value as ${_1}. This is saying “using the current location in the array”. Traversing an array is a bit of a larger topic, so we’ll get into that later.

Finally, you see the next GET component. You’ll notice we are using the id that we stored from the first GET call to populate into the second GET call. Each iteration of the For Each will call the current id in the subsequent GET call.

That’s it! What we’ve done here is made an API call to the products list API, then tested that API thoroughly, and then used that API as the datasource for the subsequent GET which dives into each of those product IDs individually. You then want to test that entire API response as well. So here is what that entire test looks like complete in the GUI.

To see the code itself, go to the Examples directory and open the test Retail: Integration – Products.

end to end, iterate

2. Building a Strong Functional Test

Note: This example exists in your Example project, and it is named Retail: Integration – Products.

Ok, so let’s build our first functional test! We will start by using this API call.
(https://mastiff.apifortress.com/app/api/examples/retail/products)

By clicking that link, or using the HTTP composer to make the call, you get this response.

As you can see there are 5 objects, so at minimum we want to make sure they ‘exist.’ What API Fortress allows you to do is validate the objects exist and the data is as expected. This is done using our XML markup language, or our GUI composer.

*Side Bar*
API Fortress was specifically built to bridge the gap between testers and engineers, allowing you to write detailed API tests in whatever format you are most comfortable with. Our composer has a drag-and-drop interface that writes the XML code for you, and it also makes it easier to visually understand the nature of the test. With that said, our GUI composer also has a Code View which exposes the underlying XML. This XML can be written or edited in the GUI, or using your own IDE. Meaning that how you want to write and/or edit a test is completely unlocked, so you can work how you are most comfortable.

In the GUI composer view that test looks like this:

If you clicked on Code, or were looking at the test in your own IDE, this is what you’d see:

<unit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="main" xsi:noNamespaceSchemaLocation="http://apifortress.com/app/unit.xsd">
   <requirements />
   <configs />
   <sequence name="main">
      <get url="${protocol}${domain}${productsEndpoint}" params="[:]" var="productsPayload" mode="json" />
      <assert-equals expression="productsPayload.status" value="200" comment="" />
      <assert-is expression="productsPayload.success" type="boolean" comment="" />
      <assert-exists expression="productsPayload.content.act" comment="" />
      <assert-is expression="productsPayload.content.products" type="array" comment="" />
   </sequence>
</unit>

Now if we look at the code itself carefully, it’s pretty clear what we’re seeing.

  • The unit section is referencing our markup Namespace file (useful when using your own IDE).
  • Requirements and Sequence are more advanced feature we can discuss later.
  • GET is where we make the API call. Within this call it contains the necessary information to make the call (can include more if we’re dealing with oAuth for example). Some important things to note is how we have variablized protocol, domain, and then the chosen endpoint. You’ll also notice the Data Sets section on the left contains those same variables and the data associated. This is hugely important to allow the test to be easily run against any environment. We will go into more detail on that later, but think of it as futureproofing the test from location specificity.
    Then there is also var=”productsPayload”. What this does is call an API, gets a response, and then stores that entire payload into a variable called productsPayload. This is then referenced in the next assertions.
  • Note: Performance can be an assertion within the test itself. See this page for more information.
  • Now we get to the assertions! There are over 70 assertions, so we won’t get into too much detail here, but you can see the next four lines are assertions for each object. Looking at the first we see:
    <assert-equals expression=”productsPayload.status” value=”200″ comment=””/>
    This line calls the assert-equals assertion, which validates an object exists and is equal to a chosen amount. Expression references the specific object we are talking about. You’ll notice that it has productsPayload.status. So it’s referencing the status object within the productsPayload variable. Remember, the GET call gets an API response and stores it in that variable. This is useful for when you are dealing with multiple payloads (variables) in an integration test.
    It’s important to note that with the Generate Test feature you can have this entire structure generated for you in seconds. This frees you to focus on the more important and tricky aspects of writing detailed tests.

1. What is an API?

APIs are the term used to describe web services that connect different platforms or data services together. An example is the Uber application. The way that you search, book, and pay are all done through APIs.

APIs are not complicated, they are just data. When you search for vehicles in the Uber API, it makes this HTTP call:

https://api.uber.com/v1/products?server_token=[token]&latitude=40.6797300818661&longitude=-73.9639477463489

The response is in JSON format and human readable. Here is a snippet:

{
"products": [
{
"capacity": 2,
"product_id": "929fcc19-8cb4-4007-a54f-3ab34473700f",
"Price_details": {
"service_fees": [],
"cost_per_minute": 0.74,
"distance_unit": "mile",
"minimum": 8,
"cost_per_distance": 1.62,
"base": 0,
"cancellation_fee": 5,
"currency_code": "USD"
},
"Image": "https://d1a3f4spazzrp4.cloudfront.net/car-types/mono/mono-uberpool.png",
"cash_enabled": false,
"shared": true,
"short_description": "Pool",
"display_name": "UberPool",
"product_group": "rideshare",
"description": "Shared rides, door to door"
}
]
}

Glossary

HTTP APIs

There are many different types of APIs, the term itself has become vague unfortunately. For our purposes we will always be talking about HTTP APIs. APIs that can be hit with an HTTP call.

There are two types of HTTP APIs, REST and SOAP. That is a huge topic, but from a very high level:

SOAP

This was the most commonly used format, but is now seen as a legacy technology. It’s advantage is that it’s simple and can return a bunch of data. The negative is that it is very poor for more interactive usage, like we see with mobile applications today.

REST

REST is the most common protocol we see today, and more robust overall. It’s specifically made for interactive usage, such as mobile applications.

GraphQL

Raising in popularity and made by Facebook.

api, learn, 101

The response of an API call can come in many different formats. Again, for simplicity we will just focus on the ones that apply to what API Fortress can test.

XML

This is a very straightforward format that can return in virtually any layout. An example:

<?xml version="1.0" encoding="UTF-8"?>
<XML>
   <note>
      <to>Patrick</to>
      <from>Mom</from>
      <subject>Reminder</subject>
      <body>We need milk</body>
   </note>
</XML>

JSON

This is the more commonly seen format in REST APIs today. The Uber example is in JSON. It’s more standardized and therefore cleaned for machines to understand.

Object

This is the item that has data associated with it. If you look at the Uber response again, display_name, description, and Image are examples of objects.

Assertion

A rule or specific test against a single object and/or piece of data. The API Fortress platform is powered by a proprietary XML language with over 70 assertions, that handle just about every scenario in a very quick and easy way to write.

101, introduction