Cloud vs On-Premises (local)

Note: Did you receive an Invalid Request message? It’s possible that you are trying to reach an environment that is behind a firewall that our public cloud accounts can’t reach. You may need to whitelist our IPs, or install an on-premises trial. Please contact support@apifortress.com to learn more, or use our chat bot.

The API Fortress platform was uniquely developed to function in our cloud, or from your own virtual private cloud or data center. This was to allow for customers that have security requirements that don’t allow for use of the cloud version (like private staging environments). The entire platform can be deployed behind your own firewall using Docker, Kubernetes or OpenShift.

To trial cloud simply register an account. To trial on-premises please fill out this form and contact sales@apifortress.com to learn more.

Why On-Premises?

There are more details on the on-premises option here, but the main reasons for choosing on-premises over cloud are:

  • On-premises can have access to environments behind firewalls, like staging and preprod.
  • On-premises is unlimited in terms of tests and executions.
  • Load Testing and Mocking are currently only available on-premises.
  • More control over scheduling, database choices, etc…

An on-premises deployment is simple to do. Please contact sales@apifortress.com and we’ll help you get setup.

Load Testing

 

Introduction

API Fortress Load Testing provides it’s users with functional load testing. Rather than simply overburdening a server with requests, API Fortress Load Testing directs properly formatted requests to the appropriate endpoints and records failures at an API endpoint level.

Step 1: Access Load Testing Control Panel

To access Load Testing, from the main page, click on the “Tools” button, and then “Load Testing”.

toolstolt

 

Step 2: Create a Task

This is the main Load Testing screen. From here, you can create and run a new task. You can also run, modify or delete a saved task. To create a new task, click “+New Task

createTest

The “Create New Task” screen allows you to set the parameters for the new test.

  • Name – The name that the test will be referred to by.
  • Project – A drop-down menu of all of the projects available to your current company.
  • Test – Allows you to select a test from the selected project.
  • Tags – Allow you to tag the test for later use.
  • Duration – How long, in seconds or minutes, the test will last.
  • Ramp Up – A warm-up period, during which the load test will make requests of the server, but at a much lower rate.
  • Users/Agent – Defines how many users will be simulated per load testing agent.
  • Server – Allows you to select servers from the column on the right to behave as agents for the load test. Click the “Select” button to select a server.

Once you have successfully created a test, it will appear in the column on the left side of the screen.

taskList

Step 3: Run the Task

You can run the task by hovering over it and clicking the “Play” button. The test will now run at the top of the queue in the middle of the screen. Once it is complete, it will display a summary of the test performance.

runTest

Known Issues / Scheduled Updates

Engine

  • [ETA: TBD] Nested variable referencing (Ie. v1=${v2}, v2=value) for variables declared in input sets, vault and overrides, does not work in:
    Query parameters
    Expression field in assertion
    POST/PUT/PATCH/DELETE bodies
    Workaround:overwrite the variable in the test before using it in one of these fields, to resolve it ahead of time. Example:

    <set var="v1" value="${v2}"/>
  • [ETA: TBD] Assert-Matches for US Zip codes does not include district codes

Dashboard

  • [ETA: End of August] In the test list, provide the user a way to override variables when running a test manually

Composer/Embedded Console

  • [ETA: Next release] Import of requests to console does not resolve nested variables (Ie. v1=${v2} v2=value)
  • [ETA: Next release] Import of requests to console does not resolve vault variables
  • [ETA: End of August] provide the user a way to override variables when running a test in development

Downloader

  • [ETA: End of August] Self signed certificates won’t get accepted unless a certificate is installed in the downloader (no bypass option)

Reports

  • [ETA: Next release] Syntax or runtime errors are rarely reported with details
  • [ETA: Next release] HTTP requests should contain all the details of the request itself
  • [ETA: End of Quarter] On complex tests, reports can become humongous and very hard to load

Notifications

  • [ETA: End of Quarter] heavy API failures cause email flooding. Emails should be stateful
  • [ETA: End of Quarter] Small memory leak causes connectors to slowly grow in memory demand
  • [ETA: Next release] On-premises deployments should be able to connect to message queue and create custom notification systems

Misc

  • [ETA: End of Quarter] On-premises: There’s no way to tell when a license is supposed to expire

API Tests vs. Schema Validation

One of the most common questions we receive about API testing is:

What’s the deal? Schema validation was not enough?

Some of these inquiries are moved by real curiosity, others are just polemic, but either cases, the question makes a good point if you’ve never actually tried our platform.

Schema validation has two major flaws:

  1. You need to write a schema
  2. It statically validates your API syntax and grammar, not your data and certainly not your darkest secrets.

For what concerns ( 1 ) deal with it.
( 2 ) is a little more intriguing. Let’s work by example.

{
  "group": "food",
  "items": [ 10,15,17,19 ]
  "install": false,
  "installDay": null
  "delivery": true
  "request_date": 1456249628
}

Of course a schema can determine whether these items are the right types, but no schema can tell that:

  1. Food items should have ids lower than 100
  2. Food items should not be installed (weird, right?)
  3. Food items have no installation day, but they’re deliverable
  4. The response shouldn’t be older than 1 day (caching?)

Moreover, API testing allows you to compare things with the request data. You don’t want to get bathrobes listed while you were looking for your birthday cake. Or, more technically, you don’t want the data within a credit card transaction to be cached.

Right?