4. Functionally Test the Security of an API

According to Gartner over 95% of security vulnerabilities in APIs are related to “functional” or “human” errors. That is why functional testing of your APIs is so important to general API security. A good API security policy includes testing of API functionality before release, as well as constant monitoring of those APIs using the same detailed functional tests.

Below we list some of the types of tests we suggest that will help you not be in the 95% of API breaches. PLEASE NOTE: This is by no means a comprehensive list, but a series of suggestions that will constantly be evolving.

Fuzz Tests:

  • Invalid Input Test
    The goal of this test is to validate how the API functions when given invalid data. Fundamentally, it’s a data-driven test with a series of invalid inputs that helps to reveal potential exceptions. Some of these exceptions may include crashes, assertion failures, error responses that give out too much information, and potential memory leaks.
  • Wildcard Testing
    This one is pretty straightforward, and yet it is one of the reasons for a major breach at the US Postal Service. The goal here is to see how the API reacts when a wildcard (*) is used in place of an input value. For most APIs, it shouldn’t be allowed as it would return a broad dataset.

Authentication / Access Control:

  • Common Logins
    This test is meant to simply go through the most common username and password combinations to try to see if one of those is valid. This includes logins such as admin/admin. These tests are particularly useful before pushing an API live when you have internal credentials used for debugging.
  • Brute Force
    This test randomly generates login credentials such as usernames and passwords in an attempt to brute force its way in. The goal here is to validate that the API only allows a certain amount of login attempts before forcing a timeout.
  • Your Authentication Flow
    It is important to create tests that validate your authentication flows. This can even include 3-leg OAuth using something like our open source 3loa project.
  • Access Control
    In this test, the goal is to confirm that a user with permissions of X can only achieve X. For example, in API Fortress, a person with “analyst” rights should only be able to view results and tests. They cannot edit tests or change settings.

Sensitive Data

  • Header Validation
    Headers contain critical details of the API transaction, but they can also sometimes contain sensitive information. In this test, we analyze the header and give some examples of potential weaknesses. This can include revealing an Apache server version that has a known weakness, or the authentication type.
  • Sensitive Information
    In this test, the goal is to search for string patterns that should never appear in a payload, such as credit cards, and social security numbers, etc…

4. Using The Vault, Variables and Environments

Using The Vault The Vault is a unique part of the API Fortress platform. Not in terms of the idea, but in the flexibility offered by ours. It allows you to save, edit, and reuse almost anything. This can mean variables, code snippets (think reused authentication flows), or even data! The Vault has two levels, project and global, the project vault will allow use of stored values across any test within that project. Similarly the global vault will allow use of stored values across any test within any project. Using Variables and Environments If properly setup any API Fortress test can be run against any environment. You’ll notice that is what we did in our functional test by turning the API URL into three separate parts –  protocol, domain, and endpoint. This allows you to set the default location under Data Sets (in the left pane), but also you can override those amounts with the Environments tab (shown below). The Environments panel lets you change anything, not just environments. You can run the test against a certain environment, using a different API key, and datasource. This effectively allows you to have pre-set runtime variable overrides.
You can run the test against a certain environment, using a different API key, and datasource. This effectively allows you to have pre-set runtime variable overrides.
  To learn more about The Vault and Environments see below links: Learn the Basics, Advanced Use Cases, Environments Basics, Environments Advanced

3. Best Practices – Optional

Integrate with your CI/CD

Integrating your API testing into your existing CI/CD flow is key if you are trying to keep all of your deployments quick and frequent. Kicking off a test or suite of tests during the build process allows you to build your new deployment and test it all at once.

Integrate Notification/Data Analysis Systems

Set up notifications to alert you via email, slack, and more when tests fail. Do you already use a platform to analyze all your data, well you can send all your API Fortress test data out to those platforms as well. Now all of your data can live in one place, making it easy to pull and analyze all project data.

Integrate with Test Case Managers

Integrating with your Test Case Manager and communicating the results of tests is important. Having all of your information in your test case manager including test result data makes it easy to understand the entire project. Connectors can be setup to automatically send test results to your test case manager upon completion of scheduled tests.

API Testing University

API Testing University

Courses

Course 1

API Testing Introduction

Course 1 will give you a high level understanding of what an API is and why testing APIs is important. This course will guide you on all you need to know to get started with continuous API testing.

Start Course

Course 2

Best Practices

Once you have completed Course 1, jump into Course 2 where you will learn some basic and more advances methodologies for creating good API tests.

Start Course

Course 3

Writing Your First Test

Now that you have learned the basics and some helpful tips on how to do API testing, check out Course 3. In this course you will learn how to create your first test. You will learn the different ways to begin writing a test, as well as some more advanced techniques to help you simulate real life scenarios.

Start Course

Course 4

Advanced Test Creation

Up to this point you have learned how to create tests and learned some more advanced practices (i.e. variables and environments). In Course 5, you will learn even more advanced API testing techniques, like dealing with authentications.

Start Course

Course 5

Executing Tests

Now that you know the basics and some more advanced methods for creating API tests, you will learn how to execute these tests in a couple different ways. In Course 6, you will see automated ways of kicking off and scheduling the tests you have created.

Start Course

These classes include high-level overviews of best practices for API testing that our team has gathered over six years as a leader in the API testing space. We will be constantly updating our best practices lessons over time. If you have any questions or suggestions, please contact us at support@apifortress.com.

2. The IF Component

One of the most powerful components in API Fortress (and in programming) is the IF.

This gives you the flexibility to create assertions that can be specific to certain conditions. For example, IF statuscode==200 vs statuscode=400. Meaning you can create positive and negative validations in a single test.

Another use case is to build resilient tests that can work with asynchronous APIs. For example call the API, IF status is not complete, wait a few seconds and retry the API call.

1. Validate Metrics Within a Test

Monitoring your internal APIs is always a good step for keeping track of its performance. When building tests to monitor the performance of your API, we recommend not only testing for functionality and load but also testing for metrics such as fetch, latency, and overall times.

These performance metrics can be turned into an assertion themselves using the assert-less component.

The platform also has a robust dashboard that allows for detailed notifications in terms of global performance. 

These metrics can also be pulled from our Insights API and put into any analytics platform of your choice that supports JSON inputs. The data can be pulled all at once or in a streaming fashion. Learn more in the API documentation.

2. Automatically

By API

The platform has extensive APIs specifically created to empower engineers and testers to execute tests however they choose. You can read more about integration into a CI pipeline here. For specific use cases see some below, or search our docs.

By Command-Line Tool

The platform also has command-line tools available to paying customers, or potential customers doing a proof of concept. It allows full execution of test, and is particularly useful for CI pipelines. It can help return a lot of useful test report data right back into your CI platforms, like Jenkins. [Learn More]

2. Best Practice – Advanced

Variablize for Test Reuse

Domain, endpoints, and data sets are some of the parts of a test that should be easy to change quickly. You should never have to duplicate a test to execute it against a new staging environment. If you are doing that than your system needs to be reconsidered.

As much as possible aim to have only a single test of a certain type. If you built a partner API, then try to have a single detailed test that can be a regression test during a CI pipeline, and then also used to monitor the live environment. [Learn More]

Business Logic Validating

As described in Course 1, it is important to go beyond the technical facts of a payload. You should also have knowledge as to the business objects. This means that if the Uber payload includes a reference to UberSidecar, but Uber hasn’t announced a Sidecar product yet, this is a major issue for them from a technical and business secrets perspective.

Perhaps they have a major advertising initiative to announce the ability to order a ride in the sidecar of a motorcycle, and this would undermine that. It’s important that your tests are knowledgeable about the goal of the APIs technically, as well as from a business perspective.

Using Lots of Data

One of the most common errors we see with testing is using the same small set of data for every test. A CSV with 50 product IDs is a good start, but now that we can automate this work there is no reason to not write a much more comprehensive test that can validate hundreds of product IDs. You must break free from using small CSV files for datasets.

If you are already using databases or APIs that’s great! If not, we suggest you start this process as well. A test is only as good as the amount of information it is testing against.
[Learn More Using Files | Using JDBC]

Integration / End-to-End Test

Most API programs are a collection of APIs that are meant to interact with each other. Therefore, it is important to create tests that do exactly that. A singular test that goes through a series of calls and validates the responses at each stage. This is what is called an integration test, although it is also called an end-to-end test.

This means writing detailed tests that reproduce normal user flows. For example, if we are talking about an ecommerce platform, a normal user flow would be:

Search > View Product Details Page > Add to Cart > Checkout

A functional test for just the Search API is a great start, but the entire flow also needs to be validated. You should have as many integration tests as there are expected user flows. [Learn More]

Confirm Response Matches Request

This is in the same thinking as the above, but with that advent of PSD2 and Open Banking it’s worth mentioning on its own. When you are requesting information, such as your personal banking history, it is important to make sure only your information is returned. It is also important to validate you can do wildcard searches and get the results of other.

That is exactly the sort of functional error that causes a huge vulnerability for large organizations like the United States Post Office.

Integrations Galore

Every company has an existing workflow, and it’s important to choose a platform that helps keep those integrations seamless. In the basics we talked about integrating with your CI pipeline, but it’s also useful to have notifications and data in one place. If your organization has a lot invested in Slack, Kibana, and TestRail (for example) than having API test notifications and results in one place is imperative. [Learn More]

3. Creating an Integration Test / Using an API as
     a Datasource

Now let’s take our existing functional test above and use it as the first step in an integration test. Not only that, but that first API call actually contains an array of product IDs. What if we use it as a datasource, and then iterate on each of them individually?

First, we have the original test:

Original functional test

Now, to do this work requires the use of two new components. The For Each component helps you iterate through a series of data (product IDs in this case), and the Set which creates a temporary variable to reference. This should seem fairly straight forward if you have taken some introduction computer science and programming classes.

Looping through data from one API response to data drive a second API call.

If you look at the above GUI view of this test, you see that we have our original test. It makes the first API call, and then tests each object in that response. Then, and this is where this becomes an integration test that is using an API to be data-driven, we have that For Each component. You’ll notice that it is referencing the variable that we stored the entire response, then the array, and finally the object we want to use – productsPayload.content.products.pick(2). You’ll notice the .pick(2) we added at the end. This is entirely optional, and what it does is randomly select X number of items from the list we are iterating through. This is useful if the dataset is far too large. Ours is small so we would remove that and allow every product ID to be tested. We will leave it for training purposes.

Next we have the Set component, which is creating a new variable called id. The value of that variable depends on the location the For Each is at in the iteration. Here we set the value as ${_1}. This is saying “using the current location in the array”. Traversing an array is a bit of a larger topic, so we’ll get into that later.

Finally, you see the next GET component. You’ll notice we are using the id that we stored from the first GET call to populate into the second GET call. Each iteration of the For Each will call the current id in the subsequent GET call.

That’s it! What we’ve done here is made an API call to the products list API, then tested that API thoroughly, and then used that API as the datasource for the subsequent GET which dives into each of those product IDs individually. You then want to test that entire API response as well. So here is what that entire test looks like complete in the GUI.

Full integration test that uses an API response to data drive the rest of the flow.

To see the code itself, go to the Examples directory and open the test Retail: Integration – Products.

end to end, iterate

2. Building a Strong Functional Test

Note: This example exists in your Example project, and it is named Retail: Integration – Products.

Ok, so let’s build our first functional test! We will start by using this API call.
(https://mastiff.apifortress.com/app/api/examples/retail/products)

By clicking that link, or using the HTTP composer to make the call, you get this response.

JSON response of demo product API.

As you can see there are 5 objects, so at minimum we want to make sure they ‘exist.’ What API Fortress allows you to do is validate the objects exist and the data is as expected. This is done using our XML markup language, or our GUI composer.

*Side Bar*
API Fortress was specifically built to bridge the gap between testers and engineers, allowing you to write detailed API tests in whatever format you are most comfortable with. Our composer has a drag-and-drop interface that writes the XML code for you, and it also makes it easier to visually understand the nature of the test. With that said, our GUI composer also has a Code View which exposes the underlying XML. This XML can be written or edited in the GUI, or using your own IDE. Meaning that how you want to write and/or edit a test is completely unlocked, so you can work how you are most comfortable.

An API Fortress auto generated functional test.

In the GUI composer view that test looks like this:

If you clicked on Code, or were looking at the test in your own IDE, this is what you’d see:

<unit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="main" xsi:noNamespaceSchemaLocation="http://apifortress.com/app/unit.xsd">
   <requirements />
   <configs />
   <sequence name="main">
      <get url="${protocol}${domain}${productsEndpoint}" params="[:]" var="productsPayload" mode="json" />
      <assert-equals expression="productsPayload.status" value="200" comment="" />
      <assert-is expression="productsPayload.success" type="boolean" comment="" />
      <assert-exists expression="productsPayload.content.act" comment="" />
      <assert-is expression="productsPayload.content.products" type="array" comment="" />
   </sequence>
</unit>

Now if we look at the code itself carefully, it’s pretty clear what we’re seeing.

  • The unit section is referencing our markup Namespace file (useful when using your own IDE).
  • Requirements and Sequence are more advanced feature we can discuss later.
  • GET is where we make the API call. Within this call it contains the necessary information to make the call (can include more if we’re dealing with oAuth for example). Some important things to note is how we have variablized protocol, domain, and then the chosen endpoint. You’ll also notice the Data Sets section on the left contains those same variables and the data associated. This is hugely important to allow the test to be easily run against any environment. We will go into more detail on that later, but think of it as futureproofing the test from location specificity.
    Then there is also var=”productsPayload”. What this does is call an API, gets a response, and then stores that entire payload into a variable called productsPayload. This is then referenced in the next assertions.
  • Note: Performance can be an assertion within the test itself. See this page for more information.
  • Now we get to the assertions! There are over 70 assertions, so we won’t get into too much detail here, but you can see the next four lines are assertions for each object. Looking at the first we see:
    <assert-equals expression=”productsPayload.status” value=”200″ comment=””/>
    This line calls the assert-equals assertion, which validates an object exists and is equal to a chosen amount. Expression references the specific object we are talking about. You’ll notice that it has productsPayload.status. So it’s referencing the status object within the productsPayload variable. Remember, the GET call gets an API response and stores it in that variable. This is useful for when you are dealing with multiple payloads (variables) in an integration test.
    It’s important to note that with the Generate Test feature you can have this entire structure generated for you in seconds. This frees you to focus on the more important and tricky aspects of writing detailed tests.