Bamboo – Integrate API Tests & Results

Passing data from API Fortress to Atlassian Bamboo allows Bamboo users to include API Fortress test results in their CI/CD process.

Step 1: Generating a Webhook

The first step to integrating API Fortress into your CI/CD process is to grab the generated API hook for the project in question. To do so, head to the Settings panel in API Fortress. This view, seen below, can be accessed from anywhere in the application by clicking the Gear icon in the top right corner of the screen. Please note you need Manager access to generate a webhook. From Settings, click the API Hooks section and generate the hook for your project. The process can be seen in detail in the .gif below.

hook

Step 2: Select or Create a Bamboo Project

After we’ve created our webhook, calling it from within Bamboo is a fairly simple process. First, create a new project in Bamboo. You can also add to an existing project from this screen.

project

Step 3: Adding an HTTP Call

Next, we need to add an HTTP Call component and enter the webhook we generated. Depending on what you wish the call to API Fortress to trigger, you may append different routing on to the end of the webhook. The API Fortress API Documentation is located here.

httpcall

Step 4: Parsing Results

After the request is sent to the API Fortress API, we’ll need to save the JUnit data that’s returned. We do so by adding a JUnit Parser step.

junit

Once the above steps are completed and saved, the build sequence will make a call to API Fortress upon execution, receive the results of the tests, and parse the results.

summary

Integrate With Your CI/CD Platform (simple)

API Fortress was specifically built to make integrating with a CI/CD platform simple. Using our webhook you can execute API tests as part of your build and deployment flow, and get the results back in JSON or Junit format.

Step 1 – Install an HTTP Plugin

Depending on your CI/CD platform, you may need to install a plugin that allows for HTTP requests during the build process. API Fortress will need that to execute the tests.

Step 2 – Generate an API Hook

The first step to integrating API Fortress into your CI/CD process is to grab the generated API hook for the project in question. To do so, head to the Settings panel in API Fortress. This view, seen below, can be accessed from anywhere in the application by clicking the Gear icon in the top right corner of the screen. Please note you need Manager access to generate a webhook. From Settings, click the API Hooks section and generate the hook for your project.

The next step depends on what you’re trying to test. The following steps are going to assume that you wish to run all of the tests in a project. You can also run a single test, or a series of tests with a certain tag. If you would like to learn more about that please contact API Fortress.

As it stands, our API hook is as follows:
https://mastiff.apifortress.com/app/api/rest/v3/86f81b19-2d29-4879-91d9-6dbb2271fec0861

The normal command to run all of the tests in the project, per the API Fortress docs is /tests/run-all, so we append this on to the end of the API call. You may need to request JUnit output. To do that simply takes some query parameters. First, we need to set sync to true, and then we can set format to JUnit. In short, we need to append ?sync=true&format=junit to the webhook call. That gives us the final API call:
https://mastiff.apifortress.com/app/api/rest/v3/86f81b19-2d29-4879-91d9-6dbb2271fec0861/tests/run-all?sync=true&format=junit

Great! If we make this API call via a browser or a tool like Postman, we can see our results in JUnit. We’re almost there.

Step 3 – Execute HTTP Call From Your Platform

From your CI/CD platform’s dashboard, you’ll need to paste the webhook call to the flow. We have more specific docs, such as for Jenkins available.

The test results can then be passed along to platforms like qTest or Zephyr in your CI/CD pipeline.

Setup Connectors (DataDog)

Here is a quick guide to setting up a DataDog integration.

  1. First, we need to generate a new API key in DataDog.
    1. Log in to your DataDog account.
    2. Mouse-over Integrations and then click API
    3. Create a new API key at the top of the view (Note: You must have Admin DataDog account access.)

datadog

  1. In API Fortress go to company settings (top right gear icon)
  2. Click on Alert Groups
  3. Create a new Alert Group (if necessary)
  4. Add recipients to the Alert Group (if necessary)
  5. Click on the Connectors icon
  6. Choose one of the DataDog connectors from the dropdown
  7. Add your DataDog API Key created previously and the DataDog host you wish the connector to pass data to.

connector

Once this process is complete, API Fortress will begin passing data to DataDog where it can be charted in any way you like!

Note: This connector shares events with Datadog, which are outages. If you would like to include performance metrics, such as latency and fetch, please let us know and we can help set that up. It requires a small script.

Load Agent Deployment

A Load Agent is a server instance that provides the simulated users in a load test. Load Testing cannot function without at least one Load Agent.

The provided files (contained in core-server.tgz) are all that you need in order to deploy a Load Agent. This tutorial will explain what changed need to be made to the files within in order to properly deploy the Load Agent.

Before starting the process, there is a step that needs to be taken for clients who received their API Fortress containers before the introduction of Load Testing.

Step 0 (Not for all users) – Activate the Node Container

Open the docker-compose.yml in the main API Fortress directory. It can be located at /core/bin/docker-compose.yml

  • Paste the following code snippet in after the #RABBITMQ section and before the #APIFORTRESS DASHBOARD section:
#NODE
apifortress-node:
   image: theirish81/uitools
   hostname: node.apifortress
   networks:
      - apifortress
   domainname: node.apifortress
   labels:
      io.rancher.container.pull_image: always
  • In the links section of the #APIFORTRESS DASHBOARD configuration, add the following line:
- apifortress-node:node.apifortress
  • Save and close the docker-compose.yml.
  • Open the start_all.sh file in a code editor. It is also located in /core/bin.
  • Copy and paste the following and overwrite the entire contents of the file:
#!/bin/bash
sudo docker-compose up -d apifortress-postgres
sleep 5s
sudo docker-compose up -d apifortress-mongo
sleep 5s
sudo docker-compose up -d apifortress-rabbit
sudo docker-compose up -d apifortress-node
sleep 30s
sudo docker-compose up -d apifortress
sleep 1m
sudo docker-compose up -d apifortress-mailer
sudo docker-compose up -d apifortress-scheduler
sudo docker-compose up -d apifortress-connector
  • Your API Fortress instance can now utilize the API Fortress Node Container which powers Load Testing.

Step 1 – Unzip the provided file (core-server.tgz)

First, unzip the provided file.

Screen Shot 2018-06-05 at 11.44.28 AM

Step 2 – Define the maximum users per Load Agent

Users per agent are the maximum number of virtual users that each Load Agent can provide.

It’s important to remember that large numbers of simulated users will require large amounts of hardware resources. Contact your DevOps team to develop a strategy for resource allocation. 

  • Locate and open the file named application.conf. It is located in core-server/etc.
  • Line 14 of this file (fixed-pool-size) should have it’s value adjusted to match the desired number of maximum users per agent.
  • Line 48 of this file (nr-of-instances) should have it’s value adjusted to match the desired number of maximum users per agent. These two values should match.

Step 3 – Configure Config.yaml

  • Locate and open config.yaml. It is located at core-server/etc.
  • First, we have to configure the baseURL
    • baseURL is located on line 3.
    • If the Load Agent and the API Fortress Dashboard are located on the same server, then you can replace the baseURL with the internal address and port of the Dashboard on the server.
    • If the Load Agent and the API Fortress Dashboard are located on different servers, you can replace the baseURL with the actual URL of the Dashboard. That is to say, the URL you would use to access it via web browser.
  • Next, we need to provide the API Key and Secret.
    • Open the main API Fortress dashboard and click the gear icon in the upper right corner to access the settings menu
    • Click the “API Keys” option in the left sidebar.
    • Click “+API Key” 

Create API Key

(Click image for GIF of procedure)

  • Copy the API Key to line 5 of config.yml.
  • Copy the Secret to line 6 of config.yml.

Step 4 – Adding the Engine

  • The next step is to add the new Engine to API Fortress itself.
  • Log into API Fortress as an administrator.
  • Click the user icon in the upper right corner, and then click “Admin Panel”
  • Click “Engines” on the left side of the screen.
  • Click “+Engine”
  • Enter the name and location of the Engine.
  • The CRN value defaults to a random string. You must change it to something human-readable. This is the internal name of the engine.
  • After modifying the CRN, copy the value to line 11 of config.yml
  • Copy the secret to line 12 of config.yml
  • Select the Owning Company of the Engine. An Engine must be owned by a single company. The default value (Public Engine) should not be chosen.
  • Select “Yes” for “Dedicated to Load Testing
  • Click the green check to save the Engine settings.

Add Engine

(Click image for GIF of procedure)

Step 5 – Deploy the Load Agent

At the desired server location, use the “docker-compose up -d” command to deploy the Load Agent container. After the operation is complete, the Load Agent will be visible to your API Fortress Load Tests. 

Jenkins – Zephyr Enterprise Integration

Step 1 – Install the Zephyr Enterprise Jenkins Plugin

The first step to exporting data to Zephyr Enterprise is to download and configure the Zephyr Enterprise plugin.

From the Jenkins main page, click “Configure” and then “Manage Plugins.” From the “Manage Plugins” window, search for and install “Zephyr Enterprise.”

jenkinsAddons

Step 2 – Configure the Zephyr Enterprise Jenkins Plugin

Click the “Configure System” option in the “Manage Jenkins” menu.

JenkConfig

Scroll down to “Zephyr Server Configuration” and enter your domain and login credentials.

Screen Shot 2018-05-29 at 10.30.39 AM

Click “Test Configuration.” If the test is successful, your Jenkins is properly configured to communicate with your Zephyr instance.

Step 3 – Generate an API Hook

Next, we need to create an API Fortress Webhook to export the test data to Jenkins. To do so, head to the Settings panel in API Fortress. This view, seen below, can be accessed from anywhere in the application by clicking the Gear icon in the top right corner of the screen. Note: You need Manager access to generate a Webhook. From Settings, click the API Hooks section and generate the hook for your project.

The next step depends on what you’re trying to test. The following steps are going to assume that you wish to run all of the tests in a project. You can also run a single test, or a series of tests with a certain tag. If you would like to learn more about that please contact API Fortress.

To import our data into Jenkins as JUnit, we’ll export it in JUnit format using a query parameter. Since we already have our API hook, we just need to add the parameter to do so.

As it stands, our API hook is as follows:

https://mastiff.apifortress.com/app/api/rest/v3/86f81b19-2d29-4879-91d9-6dbb2271fec0861

The normal command to run all of the tests in the project, per the API Fortress docs is /tests/run-all, so we append this on to the end of the API call. We also need to request JUnit output via query parameters. First, we need to set sync to true, and then we can set format to JUnit. In short, we need to append ?sync=true&format=junit. That gives us the final API call:

https://mastiff.apifortress.com/app/api/rest/v3/86f81b19-2d29-4879-91d9-6dbb2271fec0861/tests/run-all?sync=true&format=junit

Great! If we make this API call via a browser or a tool like Postman, we can see our results in JUnit.

Step 4 – Execute HTTP Call from Jenkins

From the Jenkins dashboard, let’s create a New Item. Next, we’re going to name and create a Freestyle Project. Click the OK button to proceed.

Scroll down the page until you see the “Add Build Step” pulldown menu. Select “HTTP Request.” This option will only be available if you installed the HTTP Request plugin in the previous step. We’re going to paste the API call we created above into the URL line. If we save this configuration, we can run the build and see Jenkins receive our JUnit test results in real time.

Next, we’re going to click the “Advanced” button. Scroll to the bottom of the newly opened view and enter a filename of your choice into the “Output Response to File” line.

Step 5 – Publish JUnit Test Results in Jenkins

Now that we’re receiving JUnit data from API Fortress in Jenkins, we need to publish the data so that we can use it further downstream. Click “Add Post-Build Action” and then “Publish JUnit Data.”

In the new window, enter the same filename that we saved our JUnit data to in the API call in the previous step.

Now, we’ve enabled Jenkins to execute API Fortress tests and receive the test data in JUnit format. Next, we’re going to allow it to pass this data on to Zephyr.

Step 6 – Exporting Data to Zephyr

Click “Add Post-Build Action” and select “Publish Test Results to Zephyr Enterprise.” Since we configured the Zephyr plugin in step 2, Zephyr information should populate automatically from your Zephyr Enterprise instance. Select the project, release and cycle of your choice and save the build.

Screen Shot 2018-05-29 at 10.30.05 AM

Test data will now export to Zephyr every time this project is built.

Screen Shot 2018-05-29 at 10.31.14 AM

 

Key/Value Store

The Key/Value Store

The Key/Value store allows API Fortress users to create temporary key/value pairs that can be accessed across different tests. The Key/Value store is accessed via the Key/Value Store Component.

Screen Shot 2018-05-24 at 1.22.48 PM

An extremely important point to note is that these key/value pairs are temporary. They expire after 24 hours has elapsed since the last update to the value itself. 

The Key/Value Store Component has 4 methods available for use. They are:

  • Set

    • Set will create a new key/value pair in the Key/Value store. The value is entered in the “Object” field.

Screen Shot 2018-05-24 at 10.50.19 AM

  • Load

    • Load will recall a value from the Key/Value store when provided with a key.

Screen Shot 2018-05-24 at 10.50.36 AM

  • Push

    • Push will add a value to the end of an existent value of the datatype “Array” in the Key/Value store. If no such key exists, it will create a new array containing the passed in value.  The passed in value is entered in the “Object” field.

Screen Shot 2018-05-24 at 10.51.09 AM

  • Pop

    • Pop will remove a value from the end of an existent value of the datatype “Array” in the Key/Value store.

Screen Shot 2018-05-24 at 10.50.52 AM

 

Basic Workflow

Let’s take a look at how this workflow works in a practical setting. The first example will be a simple set and retrieve of a value in the Key/Value Store.

First, we’ll make a GET request to an endpoint.

Screen Shot 2018-05-24 at 1.21.40 PM

Next, we’ll add a K/V Store component.

component

This first K/V Store component (we’re going to incorporate several) is going to set the Key/Value pair in the Store, so we’re going to use “Set.

Screen Shot 2018-05-24 at 1.46.41 PM

In this case we’re setting the Key “prods” equal to “products[0].name”, which in this case evaluates to “Baseball Cap.”

Next, we’re going to retrieve this Key/Value pair from the store with the “Load” method. In the K/V Store “Load” component, we’re going to assign the retrieved value to the variable “kvprods.”

Screen Shot 2018-05-24 at 1.47.22 PM

Finally, we’ll add in a “Comment” component to ensure that the data was recovered successfully.

Screen Shot 2018-05-24 at 1.48.01 PM

When we run the test, we’re presented with the following result:

Screen Shot 2018-05-24 at 1.48.28 PM

Success!

Push/Pop Workflow

Next, we’re going to take a look at how “Push” and “Pop” work. “Push” and “Pop” are both array methods and behave as they normally do outside of this context. “Push” will append a value to the end of an array. “Pop” will remove the last value in an array.

First, we’re going to use “Push.” It should be noted that “Pop” works similarly but with the opposite result. “Popalso assigns the removed value to a variable which can be used in the context of the test, but can no longer be accessed from the Key/Value Store. We’ll discuss this further when we take a look at “Pop.”

First, we’re going to send a GET request and assign a key in the Key/Value Store to a value from the response body. In this case, we’re going to use “Color,” which is an array.

Screen Shot 2018-05-24 at 1.49.16 PM

Next, we’re going to “Load” and “Comment” this key. We’re doing that so we can actually see the change on the test report at the end of this workflow.

The next step is to “Push” the new data on to the end of the existing array.

Screen Shot 2018-05-24 at 2.43.53 PM

In this case, we’re pushing the integer 999 onto the prods array.

Finally, we’re going to “Load” the modified data into the test from the K/V Store.

Screen Shot 2018-05-24 at 1.51.48 PM

When we run the test, we’re presented with the following test report.

Screen Shot 2018-05-24 at 1.51.59 PM

The comments show us clearly that we have pushed the number 999 onto the array stored in the key prods. 

Now, we’ve added something to the array. Let’s remove it with “Pop!”

The first step is to introduce a “Pop” K/V Store component.

Screen Shot 2018-05-24 at 2.31.17 PM

We provide the “Pop” component with the name of the key from the Key/Value Store, and the name of the variable we’d like to assign the popped value to. Remember, “Pop” removes the last value in an array and returns the value itself. In this case, we’re going to assign it to a variable called “Popped.”

Next, we’re going to recall the modified key from the Key/Value Store. Then, we’re going to Comment both the recalled Key/Value Store value AND the previously popped value.

Screen Shot 2018-05-24 at 2.28.58 PM

In the Test Report, we can clearly see the full workflow. First, we assigned an array to the Key/Value Store with “Set.” Then, we added to that array with “Push.” Finally, we removed the added value with “Pop.” Each time we made a change, we used “Load” to retrieve an updated value from the Key/Value Store.

Screen Shot 2018-05-24 at 2.29.09 PM

The last two comments show the final state of the array in the Key/Value Store and the popped value itself. The popped value will only be available within the scope of this test run. The array in the Key/Value Store will remain retrievable and until 24 hours after it’s most recent modification.

Note: “Load” does not reset the timer. Only “Set,” “Push,” and “Pop” reset the timer. 

Footprint

Consider a scenario where a route has a parameter in it. Let’s take a look at an example:

http://www.whereever.com/<id>/details

Each individual rest run for this route will produce a new line in the metrics view:
http://www.whereever.com/1/details
http://www.whereever.com/2/details
http://www.whereever.com/3/details
http://www.whereever.com/4/details

This sort of reporting will quickly turn our dashboard into a disorganized mess. To produce a single endpoint for reporting from each one of these calls, you can use what we call a ‘footprint.’

How is this accomplished? In the test, you need to add a Config component to the I/O component as seen below:

configFootprint

The Config component has two fields:
Name: the name you want to assign. In this case, you MUST to enter ‘footprint’
Value: The value for the configuration component. To set up a footprint, you must enter the same URL that’s in the I/O Component. Any parameterized portion of the URL must be wrapped in square brackets.

Based upon our example, the value in this case would be:

http://www.wherever.com/whatever/[id]/details

In the project dashboard, after every run of the test instead of
http://www.whereever.com/whatever/1/details
http://www.whereever.com/whatever/2/details
http://www.whereever.com/whatever/3/details
http://www.whereever.com/whatever/4/details

you will find only one endpoint, displayed as:
http://www.whereever.com/whatever/[id]/details

For each endpoint you can use more square brackets, one for each variable that could assume multiple values. For example: http://www.whereever.com/[whatever]/[id]/details/[colors]/whatever

When you write the value of the config, for the ‘static’ part of the endpoint you can also call a variable as in any I/O operation.

Example: ${protocol}/${domain}/[whatever]/[id]/details/[colors]/whatever

is valid syntax.

 

 

Dynamic dates

Have you ever needed to pass a future date as part of the request inside of a test? Perhaps as a check-in or check-out date? You could enter it as static value, but that means you would have to periodically update the date as time went on. Creating a dynamic date in API Fortress is a simple solution for this sort of situation.

Here’s the procedure:

  1. First, open the Composer and add a Set (variable)
    setVar
  2. Then, enter the variable name and leave the mode as String
    varComp
  3. Lastly, enter the following string in the Value field:${D.format(D.plusDays(D.nowMillis(),35), ‘yyyy-MM-dd’)}

valueField
Let’s analyse what this string means:

D.nowMillis(): returns the current Unix epoch in milliseconds
D.plusDays(): returns the provided milliseconds, plus the provided number of days (in our example, we have added 35 days to today date)
D.format(): creates a timestamp with the given format, using the current timezone (in our example yyyy-MM-dd)

As result, you will have something like 2018-05-15

You can obtain a past date, starting from todays date with the following string:
${D.format(D.minusDays(D.nowMillis(),35), ‘yyyy-MM-dd’)}

You can also create a date based on a specified timezone:
${D.format(D.plusDays(D.nowMillis(),35), ‘yyyy-MM-dd’,’America/New_York’)}
The above string create the same date as our first example using New York (EST) as the timezone.

For more details about you can check our reference page

 

How to create a dynamic header

Most APIs have only one response format, either JSON or XML. But what should we do in the case of an API endpoint that can return either JSON OR XML? We could write two different tests, with one supporting each response type, but we’d be repeating a good amount of code in both tests. API Fortress allows you to test both, using the same I/O component and assertion components for almost all test cases. In a few cases, we’ll need to add a bit of extra logic to allow the platform to distinguish between the two formats.

Let’s consider the scenario where you need to pass in the Header the Accept value that is ‘application/json’ if you are testing the JSON and ‘application/xml’ if you are testing the XML. Usually in this case, you should make two different calls, as shown in the image below, in order to be able to pass the different values in the header:

Let’s consider an example. The API call in question requires an “Accept” header. This “Accept” header needs a value of “application/json” if you are testing the JSON case and “application/xml” if you are testing the XML case. Below, we can see the solution to this problem that requires setting up two separate calls. It’s not particularly adherent to the DRY rule of programming. (Don’t repeat yourself!)

both_calls
API Fortress allows you to solve this issue by making only one call.

    1. Let’s start adding the different formats as variables, as seen below.inputSets
    2. Now, we can remove one call and add the format variable in the “Mode” input.
      varFormat
    3. The header is still static at this point. We need to turn it into a dynamic one which changes to suit the data type of the API we are testing. We add a variable component above the I/O operation that will return the appropriate value.setDynamVar
    4. if (format == 'xml')
      return "application/xml";
      else return "application/json";
      To explain: the ‘acceptHeader’ variable will have ‘application/xml’ as value if format is ‘xml’ and ‘application/json’ otherwise (since we have only two different formats, it will be ‘application/json’ only for JSON format
    5. Now, we can finally remove the ‘static’ header and add the ‘dynamic’ header by changing the Header value to ‘${acceptHeader}’dynamicHeader
      Now, the test will be executed two times; once for ‘XML’ and once for ‘JSON’, ensuring that the header will have the correct value.

Jenkins – Integrate Tests & Export Test Results to JUnit

Step 1 – Install Jenkins HTTP Plugin

Log in to your Jenkins account. First, click “Manage Jenkins,” then click “Manage Plugins.”

We’re going to need the HTTP Request plugin. To find the plugins, click the “Available” tab in the Plugins menu and use the filter in the top right corner to search for it.

Step 2 – Generate an API Hook

The first step to integrating API Fortress into your CI/CD process is to grab the generated API hook for the project in question. To do so, head to the Settings panel in API Fortress. This view, seen below, can be accessed from anywhere in the application by clicking the Gear icon in the top right corner of the screen. Please note you need Manager access to generate a webhook. From Settings, click the API Hooks section and generate the hook for your project.

The next step depends on what you’re trying to test. The following steps are going to assume that you wish to run all of the tests in a project. You can also run a single test, or a series of tests with a certain tag. If you would like to learn more about that please contact API Fortress.

To import our data into Jenkins as JUnit, we’ll export it in JUnit format using a query parameter. Since we already have our API hook, we just need to add the parameter to do so.

As it stands, our API hook is as follows:

https://mastiff.apifortress.com/app/api/rest/v3/86f81b19-2d29-4879-91d9-6dbb2271fec0861

The normal command to run all of the tests in the project, per the API Fortress docs is /tests/run-all, so we append this on to the end of the API call. We also need to request JUnit output via query parameters. First, we need to set sync to true, and then we can set format to JUnit. In short, we need to append ?sync=true&format=junit. That gives us the final API call:

https://mastiff.apifortress.com/app/api/rest/v3/86f81b19-2d29-4879-91d9-6dbb2271fec0861/tests/run-all?sync=true&format=junit

Great! If we make this API call via a browser or a tool like Postman, we can see our results in JUnit. We’re almost there.

Step 3 – Execute HTTP Call from Jenkins

From the Jenkins dashboard, let’s create a New Item. Next, we’re going to name and create a Freestyle Project. Click the OK button to proceed.

Scroll down the page until you see the “Add Build Step” pulldown menu. Select “HTTP Request.” This option will only be available if you installed the HTTP Request plugin in the previous step. We’re going to paste the API call we created above into the URL line. If we save this configuration, we can run the build and see Jenkins receive our JUnit test results in real time. These test results can then be passed along to platforms like qTest or Zephyr in a CI/CD pipeline.