qTest Integration via Jenkins

The following procedure will allow you to integrate API Fortress test results into your qTest instance. By passing JUnit test data through Jenkins and into qTest, your API Fortress test results can be visible in the same space as the rest of your tests.

Install Jenkins Plugins (qTest)

Log in to your Jenkins account. In order for the qTest integration to work, we need to extend Jenkins with a couple of plugins. First, click “Manage Jenkins,” then click “Manage Plugins.”

We’re going to need both the qTest plugin,and the HTTP Request plugin. To find the plugins, click the “Available” tab in the Plugins menu and use the filter in the top right corner to search for them. Install both.

Step 3 – Integrate API Fortress into your CI/CD Process

The first step to integrating API Fortress into your CI/CD process is to grab the generated API hook for the project in question. To do so, head to the Settings panel in API Fortress. This view, seen below, can be accessed from anywhere in the application by clicking the Gear icon in the top right corner of the screen. Please note you need Manager access to generate a webhook. From Settings, click the API Hooks section and generate the hook for your project.

The next step depends on what you’re trying to test. The following steps are going to assume that you wish to run all of the tests in a project. You can also run a single test, or a series of tests with a certain tag. If you would like to learn more about that please contact API Fortress.

To import our data into qTest via Jenkins, we’ll export it in JUnit format using a query parameter. Since we already have our API hook, we just need to add the parameter to do so.

As it stands, our API hook is as follows:

https://mastiff.apifortress.com/app/api/rest/v3/86f81b19-2d29-4879-91d9-6dbb2271fec0861

The normal command to run all of the tests in the project, per the API Fortress docs is /tests/run-all, so we append this on to the end of the API call. We also need to request JUnit outputt via query parameters. First, we need to set sync to true, and then we can set format to JUnit. In short, we need to append ?sync=true&format=junit. That gives us the final API call:

https://mastiff.apifortress.com/app/api/rest/v3/86f81b19-2d29-4879-91d9-6dbb2271fec0861/tests/run-all?sync=true&format=junit

Great! If we make this API call via a browser or a tool like Postman, we can see our results in JUnit. We’re almost there.

Step 4 – Add API Fortress Step to qTest

From the Jenkins dashboard, let’s create a New Item. Next, we’re going to name and create a Freestyle Project. Click the OK button to proceed.

Scroll down the page until you see the “Add Build Step” pulldown menu. Select “HTTP Request.” This option will only be available if you installed the HTTP Request plugin in the previous step. We’re going to paste the API call we created above into the URL line.

Next, we’re going to click the “Advanced” button. Scroll to the bottom of the newly opened view and enter a filename of your choice into the “Output Response to File” line.

Now, we’ve enabled Jenkins to execute API Fortress tests and receive the test data in JUnit format. Next, we’re going to allow it to pass this data on to qTest.

Click on the “Add Post-Build Action” pulldown below the build step we just created and select “Submit JUnit Test Results to qTest.”

Now that we’ve told Jenkins to output the JUnit response to a file, we need to make that file accessible to downstream after-build steps. Click “Add post-build action” and select “Publish JUnit Results.”

In the new window, enter the same filename that we saved our JUnit data to in the API call in the previous step.

Enter the URL of your qTest instance and your qTest – Jenkins API Key. Click the “Retrieve Data” button to populate the next few menus. Select a project, release, and environment (optional) where you want to push the test results. Choose the final options with regards to how you wish the data to be stored in qTest, and click save.

From now on, every time you click “Build Now” with this test, the results will be exported to the qTest Project of your choice automatically.

GitHub and large data sets

Set up GitHub

  • Create a new access token associated to your GitHub profile by accessing the page: https://github.com/settings/tokens1When creating the token make sure you’re selecting the public_repo subcategory or the repo category based on which type of repository that will host your input data.2
    Keep in mind that GitHub will show the token once, so make sure you’re copying and pasting it somewhere safe.
  • Create a repository that will host your input data, if you don’t have one already
  • Commit and push a data source file. It can be a CSV, JSON or XML file. We will use CSV file for this example3

    Create a simple test

  • Create a test
  • Introduce the GitHub component and configure it accordingly4
  • The system will retrieve the document, parse it as a CSV file and assign it to the inputData variable
  • (optional) Verify that everything is set up correctly by adding a comment printing the parsed data as in:5
  • (optional) and run it:6
    NOTE: API Fortress will parse a CSV file as an array (rows) of arrays (columns), so access to the data is positional.
  • Now, let’s iterate over a subset of this input set. Introduce a selection strategy if necessary:
    7This will iterate over a subset of 5 randomly selected items. Other strategies are described in Appendix A
  • (optional) within each iteration, we suggest to introduce a comment that will help you identify which item you’re looking at, while debugging a failure8 9
  • Use the data to perform your HTTP call:10
  • Introduce some assertions, you know, that’s what you use API Fortress for…11
  • And run it12

Appendix A: selection strategies

Simple selectors

  • None. If the number of iterations is greater than 100, the system will randomly select 100 elements, unless you override the maximum iterator size.
    • Pick(n). Ask the system to randomly select a n-sized sample of elements.
      Example: inputData.pick(5)

 

  • Slice. If you’re interested in using a specific slice of data from the inputData, you can slice it according to your needs.
    Example: inputData[10..20] (will select items from index 10 to index 20)

 

Advanced Slicing

Assume you have a 1000 lines CSV file and you need to use them all. While this is technically possible (by overriding the maximum number of iterations) the usefulness of the test may vary on:

  1. How long does the HTTP request takes
  2. How complex the test is going to be
  3. The number of errors the test may trigger

Moreover, the readability of the resulting document may degrade when trying to debug an issue.

Here’s a slicing technique we suggest to ease these points.

  • Introduce the following 2 variables in the global parameters:

    13
  • Use the following expression in your each statement:
    inputData[offset.toInteger()..offset.toInteger()+limit.toInteger()]
    Which reads: slice inputData from the offset index to the offset+limit index

    Note: the toInteger() command is required as variables are always strings and we need to play with numbers.

    By doing so we are setting a baseline: as a default test input data from index 0 to index 99.
  • Introduce as many environments as the slices count, overriding the offset variable15
    Now you can run the test on specific 100 elements slices, by selecting the environment.16 17

    18

  • Finally, you can schedule your slices accordingly:19

Following redirects

By default, API Fortress handles redirects as per RFC standard. So if you are performing a GET and the response wants to redirect you somewhere else, you don’t need to do anything since the redirect will be followed automatically.

Now, since the RFC says that a redirected I/O call should perform the same operation as the original call, and given that, say, re-posting, is a security threat, API Fortress will not automatically follow redirects for any other I/O operation.

Boring, you may say, but following the exact RFC specifications is how you make sure that any client will not be caught off guard!

This specific scenario can be handled applying the following steps:

Make the I/O operation (let’s consider a POST as an example)

  1. step1
    The response header contains the ‘Location’ field. Let’s make sure it’s there, by performing the call in our console tool.step2highlighted
  2. Craft a GET to the ‘Location’ retrieved in the response header of the previous call (i.e. ${payload_response.headers[‘location’]})

    Post307

And you’re done.

Assertions for Metrics / Performance

We have a more detailed guide on working with the HTTP response, but wanted to provide specifics on creating assertions against performance metrics.

Here is an example using the CODE VIEW of the code. Please note that overall refers to fetch and latency combined.

<assert-less expression="payload_response.metrics.latency" value="200" type="integer"/>

<assert-less expression="payload_response.metrics.fetch" value="350" type="integer"/>

<assert-less expression="payload_response.metrics.overall" value="450" type="integer"/>

Setup Connectors (Slack)

Here is a quick guide to setting up Slack integration.

  1. FIRST! Get your Slack webhook. This is the link to generate a new one.
  2. In API Fortress go to company settings (top right gear icon)
  3. Click on Alert Groups
  4. Create a new Alert Group (if necessary)
  5. Add recipients to the Alert Group (if necessary)
  6. Click on the Connectors icon
  7. Choose Slack from the dropdown
  8. Add your Slack webhook
    Important! The full URL is https://hooks.slack.com/services/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,
    Then only part that you need to copy in comes after /services/ i.e.
    T08QUN1SR/B08QUGM8T/qVvPDLTa3OtvskrpyKBTfhvI

notifications - alert group and slack

Setup Notifications (Performance Alerts)

Performance is the first thing that people think of when they are concerned about their APIs. By scheduling accuracy tests to run automatically, the data from those executions can then be used for proper insight into how the API is performing globally.

Step 1. Go to the Dashboard

Step 2. Click on Metrics

Step 3. Click on a Footprint

Step 4. Click on the Bell to Add a Monitor

Step 5. Fill in the Information

Enter the name of the alert, select the type of monitor (latency or fetch), and the max value for the trigger. Latency is the time it takes to ping the endpoint and receive a response. Fetch is the time it takes to download the response itself. Large responses will lead to large fetch times, so be aware of this when setting a number. Values are in milliseconds.

Alert Thresholds (Using JSON Connector)

The JSON Alert Connector sends information about a failure in JSON format and posts it to a given WebHook URL.

The JSON connector with the threshold variation does the same thing with a twist – rather than posting every event, it does so when the error rate exceeds a certain threshold. The connector has been designed to be used with services like Zapier or IFTTT. It can also work with other similar services, but those are two of the most popular ones.

Configure the Connector

  1. Access the company dashboard by clicking the gear icon in the top right corner
  2. Select “Alert groups”
  3. Create a new alert group if you don’t have already one for this purpose
  4. Select the socket icon
  5. Select “+ Connector to this group”
  6. Select “JSON Alert /w threshold” from the list
  7. Enter the required settings
    1. URL:  The URL of the WebHook the alert should be sent to.
    2. Counter: The number of errors per specific test in a period before the alert is actually sent out.
    3. TTL: The time frame, expressed in seconds. (Ex. 3 errors over a TTL of 600sec [10 min])
  8. Save
  9. From the main dashboard, edit the “Settings” the project(s) you want to assign the alert group to if the group has not been assigned yet

Behavior

The connector will collect errors for each specific test. When the first error occurs, the timer expressed by the TTL starts.  When the error reaches the counter number, the alert is sent and the timer resets.

The JSON sent will look like the following:

{
    "date": "2017-12-29T14:36:31+0000",
    "eventId": "5a4aa0a1-d071-4a05-981d-ff57e4ff3897",
    "test": {
        "name": "book",
        "id": "123a"
    },
    "value2": "fake project",
    "value1": "book",
    "companyName": "fake company",
    "eventType": "failure",
    "criticalFailures": [{
        "action": "get",
        "expression": "get http://www.example.com",
        "status": "Generic Failure",
        "failureType": "MissingPropertyException",
        "extra": "Error parsing URL. Missing variable yay"
    }],
    "companyId": 1,
    "failuresCount": 1,
    "location": "Ashburn,Virginia",
    "projectName": "fake project",
    "projectId": 1
}

Typical Use Cases

Zapier – webhook to email

Configure a Zap triggered by a WebHook:

zapier_1That URL is the one that needs to be used in the connector configuration screen.

Add a “Send Email” step, and configure it to use the retrieved data:

zapier_2For this example, we used a template like this:

The test "{{30468424__test__name}}" for the project "{{30468424__projectName}}" Hit the maxium error threshold.
See a sample event by accessing:
https://mastiff.apifortress.com/app/web/events/details/{{30468424__eventId}}?cid={{30468424__companyId}}

Where 30468424 is the Zap ID.

IFTTT – webhook to email

ifttt_1Create an Applet that gets triggered by a WebHook and configure it to send an email. This is pretty simple actually…

 

Copy a Test

Here we will show you how to copy and paste a test. Before that, it should be noted that this isn’t often the right way to go. Take a step back and ask yourself if this is actually the perfect solution. We have specifically built our platform to improve work efficiency. Here are some questions that you should ask yourself:

  • Are you looking to do negative testing? You can likely do that in the same test using data sets and the IF component + status codes.
  • Are you looking to run from different environments? No problem with Presets.
  • Do you want to run most but not all of the tests in a specific project? Execute tests using tags. (example, using a unique webhook and the ‘live’ tag.)

If you still wish to copy a test, the procedure is as follows. First, enter the project and test. Then click over to the Published version of the test, then the Copy to Clipboard option becomes available.

copy 1

Click Copy to Clipboard and then browse to the Tests view of the same, or another project. There Paste from Clipboard becomes available.

copy 2

So here is that entire adventure in video form.

Write Tests in Code

API Fortress has three unique advantages in the market – magic, the visual composer, and the built in assertions/operations. With that said, you are not bound to them exclusively. If you are more comfortable using code, that option is available as well.

Code View

First, the whole test can be seen and edited “naked,” without our glamorous UI. Behind the curtains, the test is described using the XML markup language.

To use it, you simply need to look at the top right of the composer. The default is VISUAL COMPOSER, but right next to it is CODE VIEW. Click that.

code view - 1

Now you will see the markup language that is the basis of API Fortress.

code view - 2More experienced testers may find this to be the most efficient manner to use the platform.
Tip: The best way to learn the markup? Build your tests using the visual composer/magic, then switch to code view and have a look!

A Groovier Approach

Whether you are using the code view, or the visual composer, one important aspect to note is that all “evaluated” fields are actually able to execute a subset of Groovy commands.

For example, let’s take this assertion that verifies whether the “items” element is an array.
assert-is
Or in code view:
<assert-is expression="payload.whatever.items" type="array"/>

Now let’s say you know something more about this array, such as it should always contain more than 3 elements:

assert_greater

Or in code view
<assert-greater expression="payload.whatever.items.size()"  value="3" type="integer"/>

Notice how in the expression field we deliberately used the size() command to retrieve the size of the object at its left.

Even More Serious Grooviness

Moreover, Groovy can be put within SET components.

The first scenario is when you want to set a variable that is not a String. The best way to do it is using the Variable Mode “Object.” The value, in this case, will be evaluated as Groovy code.

set_obj
<set var="number" object="otherNumber+50"/>

Here we are assuming that otherObject is a predefined numeric variable. When the SET is executed, the number variable will be an integer.

The second scenario is when you want to write extensive logic.

Choose the SET component, then choose the item “Language” in the type drop-down (when using Visual Composer), or enter lang=”java” when writing it in Code View.

setOr in code view:

<set var="myVar" lang="java">
<![CDATA[
def item = payload.whatever.items.find { it -> it.id==11 }
return item.amount
]]>
</set>

Templating

What about all the fields that are not explicitly evaluated? Like URL, value, or POST Body? Or the content of a comment? It is often extremely useful to evaluate content on those as well. This is possible using the template syntax.
eq

<assert-equals expression="payload.id" value="${req_id}"/>

This assertion, for example, is evaluating the req_id variable right within the value.

A Little Bit of Everything

Let’s join everything we’ve learned into one snippet:

<set var="data" lang="java">
<![CDATA[
  def items = payoad.whatever.items.find{ it-> it.id>100}
  return items.collect{ it -> it.providerId}
]]>
</set>
<each expression="data">
  <post url="http://www.example.com/${counter+1}" params="[:]" var="payload2" mode="json">
   <postBody contentType="application/json">
     <![CDATA[{"providerId":${_1}}]]>
   </postBody>
  </post>
  <set var="counter" object="counter+1"/>
</each>

Want to learn more about Groovy?

Follow this link to the official documentation:
http://groovy-lang.org/documentation.html

Important Note
For security concerns, the cloud version of API Fortress is sandboxed. Meaning many programming language features are disabled. On-premises eliminates that restraint. While on cloud, if you think a specific feature should be safe to be enabled but is disabled, please contact us and we’ll do our best to help you!

Change Environments and Presets

One of the key aspects of creating meaningful tests is being able to run them against different environments, to make sure that the upcoming release will pass the tests designed for the production release.

To do so, API Fortress introduced the “Environments” feature.

An “environment” is a collection of override variables. An override variable is a variable that will replace the value of a variable that has been hardcoded within the Vault or the test itself, during a predefined session.

You can access the feature in the “tests list” by clicking the “hamburger menu” icon:

 

Or in the composer, clicking on “environments”:

ham2

Simply Overriding Variables

In the override panel, you can type in which variables you want to override and activate the preset:

overBy doing so, you will be overriding the variables for the current session, as long as the preset is activated. This override will be performed to your session only.

Saving the Environment

overOnce your preset is ready and you are willing to save it for later use, you can click on the “save” button and give it a name. The Preset will be associated with the current project and will be available to you and the other users.

Activating a Saved Environment

Once your environment is saved, you can activate it by choosing it from the “environments” dropdown that will show up by clicking on the arrow-down, to the right of the environments icon.

In this example, we named the environment “staging”.

To disable the environment and go back to defaults, just select “None”.

Using a Saved Environment via API

When using the API by invoking a test execution, you can have API Fortress load an environment based on its name.

To do so, just provide the special variable “apif_env” in the params section, and provide the name of the environment you want to load, as in:

{
  "Content-Type":"application/json",
  "payload":"{}",
  "params":{"apif_env":"staging"}
}