API Tests vs. Schema Validation

One of the most common questions we receive about API testing is:

What’s the deal? Schema validation was not enough?

Some of these inquiries are moved by real curiosity, others are just polemic, but either cases, the question makes a good point if you’ve never actually tried our platform.

Schema validation has two major flaws:

  1. You need to write a schema
  2. It statically validates your API syntax and grammar, not your data and certainly not your darkest secrets.

For what concerns ( 1 ) deal with it.
( 2 ) is a little more intriguing. Let’s work by example.

{
  "group": "food",
  "items": [ 10,15,17,19 ]
  "install": false,
  "installDay": null
  "delivery": true
  "request_date": 1456249628
}

Of course a schema can determine whether these items are the right types, but no schema can tell that:

  1. Food items should have ids lower than 100
  2. Food items should not be installed (weird, right?)
  3. Food items have no installation day, but they’re deliverable
  4. The response shouldn’t be older than 1 day (caching?)

Moreover, API testing allows you to compare things with the request data. You don’t want to get bathrobes listed while you were looking for your birthday cake. Or, more technically, you don’t want the data within a credit card transaction to be cached.

Right?