API Testing Still In Its Infancy

Note: This post was originally a guest post for the readme.io blog. Reprinted here.

Are you testing your APIs?

I ask people this question daily. Some say they do, but it’s not many. The most conscientious boast of their unit tests and an occasional smoke test, which is progress, but when compared to the rigor we regularly apply to our apps and websites it’s fairly weak. Why is that?

APIs are a hot buzzword today, but the market has reached a point nearing maturity. With that, historically, comes the tools and expectations of full and extensive testing. If I can automatically test the desktop web, mobile site, and app of USA Today (for example), shouldn’t I also be able to confirm their API is producing exactly as intended? Surprisingly, no. On a daily basis we hear these responses:
“We have not been testing our API so far, but no one has complained yet, so it can’t be an issue.”
“No, but why would we need to test the API? We know if it’s up or down.”

There is a fundamental flaw associated with API testing expectations. An API being up or down is RARELY the problem. APIs, like most services today, are up 99.99% of the time. The most common errors actually come from countless other types of issues. Here are the three of the most common issues I see on a daily basis.

  1. *Unit tests are not enough.* Imagine an API as a pipe that feeds water to your bathroom faucet. It is a means to transfer data between services — the water source and your hands. So what happens when the pipe is perfect, but the water running through it is contaminated? Data may transfer perfectly, but the data itself might be flawed. Contaminated data is one of the most common errors today, and is one that I am very focused on. A corruption in the database? Data inputted incorrectly? There are a million ways contamination can happen, but unit and smoke tests almost never catch them. An APIs response has to be validated line-by-line to guarantee it is bug free AND consistent. API consumers expect specific results and consistency is fundamental to the value they expect for their apps and platforms. In other words, constantly test the water coming out of the faucet for purity.
  2. *Documentation.* Like all things in tech, an API is constantly evolving. Even a small alteration can completely break a consumer’s platform or service. It may seem like common sense to make sure that all changes to an API are well documented and communicated, but this process fails ALL THE TIME. How? Well usually what we see is a change being pushed live before an APIs consumers have time to prepare. They are stuck trying to patch for a change they didn’t expect. Keeping documentation up to date is key, but making sure users of the API are aware of the changes BEFORE they go live is important as well. With services like ReadMe.io, your consumers should never be hustling to fix something unexpected due to a failure in documentation.
  3. *Reality versus simulations.* Often, a developer is building and testing all from within their workbench. They might do a manual test with fake data, but what happens when they step out of the simulation and use the API in production? Planes engineers spend years simulating airflow and drag as they develop a new plane. Yet, they only know with 100% confidence the plane works when a test pilot takes off. Simulations are important but they aren’t a guarantee of success. Only if one tests the response using real data from live servers can they have confidence their API is working as intended.

Unit and smoke tests are essential parts of API development. Yet they cannot guarantee a successful API program. The end result (API response) has to be tested to make sure it is coming out as expected. Bad data, a malformation in the JSON/XML, an unexpected response because it was added without proper warning — these are just some of the ways that an API can be up but actually failing its users. With tools like API Fortress, testing the API response has never been easier. There is no reason why an API should not be tested by providers or consumers as thoroughly as we test websites and mobile apps today. The time has come for API testing and monitoring to grow up.

Patrick Poulin
Founder of API Fortress – API testing as a service. He can be reached at patrick@apifortress.com