These classes include high-level overviews of best practices for API testing that our team has gathered over six years as a leader in the API testing space.
Just as important as testing an API before and after release, is validating that API is always up and functioning as expected. This is the difference between Uptime (which an APM can measure) and Functional Uptime, which is a powerful and unique aspect of API Fortress.
Ultimately, it means using your existing detailed functional/integration tests, and scheduling them to run against a live environment. This way they are constantly testing for uptime, functionality, and performance. The details on using the scheduler are here, but there are some important notes.
Note 1: A test must be Published before you can schedule it. This allows for there to be a finalized version that doesn’t get affected when you start editing a test. This is because when you edit a test it creates a Working Copy.
Note 2: You should monitor your own APIs, but you can also monitor third party APIs! Have a partner API with an SLA? Make sure that API is working as contractually agreed upon.
APIs are the term used to describe web services that connect different platforms or data services together. An example is the Uber application. The way that you search, book, and pay are all done through APIs.
APIs are not complicated, they are just data. When you search for vehicles in the Uber API, it makes this HTTP call:
There are many different types of APIs, the term itself has become vague unfortunately. For our purposes we will always be talking about HTTP APIs. APIs that can be hit with an HTTP call.
There are two types of HTTP APIs, REST and SOAP. That is a huge topic, but from a very high level:
SOAP
This was the most commonly used format, but is now seen as a legacy technology. It’s advantage is that it’s simple and can return a bunch of data. The negative is that it is very poor for more interactive usage, like we see with mobile applications today.
REST
REST is the most common protocol we see today, and more robust overall. It’s specifically made for interactive usage, such as mobile applications.
GraphQL
Raising in popularity and made by Facebook.
api, learn, 101
The response of an API call can come in many different formats. Again, for simplicity we will just focus on the ones that apply to what API Fortress can test.
XML
This is a very straightforward format that can return in virtually any layout. An example:
This is the more commonly seen format in REST APIs today. The Uber example is in JSON. It’s more standardized and therefore cleaned for machines to understand.
Object
This is the item that has data associated with it. If you look at the Uber response again, display_name, description, and Image are examples of objects.
Assertion
A rule or specific test against a single object and/or piece of data. The API Fortress platform is powered by a proprietary XML language with over 70 assertions, that handle just about every scenario in a very quick and easy way to write.
API testing isn’t easy, and there are many different ways in which a bad API can cause catastrophic harm to an organization. We will dive into best practices and details a little later, but let’s quickly run through what we’re talking about when it comes to continuous testing.
As a reference point, this is what a normal HTTP based REST API, with a JSON output looks like. Each of these types of testing are part of a continuous testing strategy. They cover a good amount, and we do our best to explain them, but keep in mind terminology and definitions are always evolving.
Functional Testing
Looking at the API above, you see a payload with various objects. The first item you should be testing is that the API responds as expected from a functional standpoint.
This means every object exists, and the responses are within expected ranges. For example the quantity object should always be a whole number. There can’t be .5 baseball caps.
Validating Business Logic
Imagine you are an ecommerce company with a product API, similar to the one we shared. First, you should be testing that each product has all the objects it expects.
The next layer is where skill and creativity in test creation are required. Let’s say that you search for ‘red’ and you receive both shoes and pants. Now all products should contain a size object, but they are very different size ranges. Adding the intelligence to a test that says, “If this is a pair of shoes, size should be between 4 and 17.” That is business logic validation. This is where you need a skilled team that knows the expectations of the API, and how to manipulate and validate them.
End-to-End / Integration Testing
APIs are meant to interact with one another, and that flow has to work in its entirety. By creating proper integration tests you can validate flows that resemble actual users behaviors. For example, creating a single test that can Search > Pick a Size > Add to Cart> Checkout
For many companies that entire flow can be done on the API level, and should be. This allows you to confirm that when you add a product to cart, it actually works.
Monitoring
Monitoring is probably the most valuable, and yet underused, type of testing. APIs issues are often only found in live environments when using live data.
The goal of a proper monitoring strategy is to go beyond measuring uptime, but also measuring functional uptime. An API can return a statuscode of 200 and still be failing. We have even seen APIs built so that even when there is an error it returns as a 200. What’s important to note is that you can use existing functional tests as your monitor. With the right platform choice, you can schedule them to run against live environments in certain intervals. Then use that monitoring to have a real-time status page across departments.
Load Testing
This is fairly easy to understand. Make sure your APIs perform when under the stress of a lot of users. This can help find memory leaks, or under performing databases.
It’s also important, again, to use existing tests as your load tests. You will never have 10,000 concurrent users only performing a single call. They will perform different types of calls, or a series of calls with some randomness. Use existing functional and end-to-end tests.
A Summary
Ultimately, continuous testing means leveraging all of these types of tests, automatically. A constant series of functional and non-functional (performance) tests to constantly validate your APIs.
API failures are very difficult and nuanced, and because of this can last longer and cost you more money. Sometimes an API issue is actually a data issue, and that’s another reason why we discuss dynamic data sources as a best practice later on.
We obviously believe strongly in the value of API testing and automation, but we can also quote other experts. First, a quote from Forrestor:
“Modern applications require a shift of the current 80% UI automation… shifting about 80% of that test automation [to] API test automation.”
Diego Lo Giudice, Forrester Wave 2017
Our next expert is Michael Cohn in his post about the “Test Automation Pyramid.”
“Where many organizations have gone wrong in their test automation efforts over the years has been in ignoring this whole middle layer of service testing. Although automated unit testing is wonderful, it can cover only so much of an application’s testing needs. Without service-level testing to fill the gap between unit and user interface testing, all other testing ends up being performed through the user interface, resulting in tests that are expensive to run, expensive to write, and brittle.”
Basically, APIs are at the core of modern platforms, and they are as susceptible to issues as the websites and mobile applications. In today’s tech environment, everyone is looking to “innovate faster.” Yes, it’s a buzzword, but it’s also a real thing.
Everyday we deal with huge enterprises transforming their legacy internal systems to APIs, and these APIs then power new mobile apps, internal platforms, partnerships, countless potential new revenue streams for any company. The world is built on REST APIs, and they require the same level of effort in terms of testing and monitoring.
Every endpoint is an integral part of an API program, and that means validating each one works as expected. This even means testing endpoints that add items into a database like Create User. A smart platform allows you to create tests that can add users, validate they were properly entered, and if it’s a success then the user is deleted. [Learn More]
Test Everything
API payloads are filled with information. You should not only be testing every object, but the header also contains vital information. Response-Type, Statuscode, Content-Type, every piece of information in an API call is important to validate.
Test Objects Thoroughly
When testing an API you should make sure all the objects exist, but that is only the first step. You should also be validating the objects have data associated that is within an expected range.
For example, in the below Uber payload you’ll see that
image should always be a properly formatted URL
shared should be boolean (true or false)
display_name should be UberPool, UberX, UberSUV, UberEats, and whatever other products they have available in that city.
Test & Project Organization
Before you begin creating tests for your API program, you should first consider how you want to organize them now and in the future. You may choose to create unique projects for different use cases, or tag the tests thoroughly using a system you create to separate tests that work for version 1 but not version 2. Come up with an organization plan that takes into consideration…
Expected Audience
Integration Test vs Simple Endpoint Test
Versioning
Legacy vs New APIs
Monitor Your APIs
Again, monitoring is the most useful and yet underused form of testing. You need to know more about an API than if it’s up or down, you need to validate it’s functioning correctly.
Schedule your existing functional and integration tests against the live environment to truly understand how your APIs are working in the real world. Every API program should have a status page. [Learn More]
Automate!
The key to delivering new and updated APIs quickly, and with confidence of their quality, is with test execution automation. This is particularly important for organization using CI/CD platforms like Jenkins, TravisCI, Bamboo, Azure DevOps, and more. Add your test suite to your pipelines as soon as you can. [Learn More]
Before we dive into this, we suggest watching this quick video on how to login and generate a test in under a minute. Then read the below to get a better understanding of the nuances of the tests themselves.
The above video shows you the method using Generate Test from a payload, and to generate a test from a spec file you can review this page. This includes Postman Collections.
API Fortress can handle just about any sort of authentication scheme your API is using, including simple, oAuth, etc… We have details on that here.
The only authentication that is a problem is one that is specifically built to not allow for automation, though we have taken steps to support those as well. For example, 3-legged OAuth can now be automated using our open-source helper tool called 3loa.
Not only can you build tests for APIs that require different levels of authentication, but we also highly recommend building tests to validate your security flow.
