Connecting to TestRail

testrail + apif

Test case managers are often critical to helping modern teams manage cases, plans, and runs. Communication of the test results is key, and that’s why API Fortress makes it easy to integrate with many leading platforms today. TestRail is one of them.

API Fortress makes it easy to automate the testing of your APIs, and to trigger those tests to run automatically on a schedule, and during a build process (eg: Jenkins). That test result data can be pushed to your TestRail instance automatically.

Here is a quick guide to setup of how to set it up.

First, click the gear icon in the upper right corner of any view in API Fortress, highlighted in the below image.

2

Next, we’re going to click “Alerts Groups” on the left navigation bar, followed by “+ Alert Group” to create a new group, name it, and finally click the connector button. The GIF below demonstrates this procedure.


AlertGroup

Next, we need to add the TestRail connector to the alert group. Click “+ Connector” and select TestRail in the dropdown that appears.

Screen Shot 2018-06-27 at 11.29.36 AM

Next, we need to define the parameters that we’re going to pass to the TestRail connector. Click the pencil icon to edit the parameters, and then fill out the fields in the modal.

 

1

Username: Your TestRail Username.
Password: The password for the TestRail account you’re using.
Project_Id: The ID (an integer) of the TestRail project you’d like the API Fortress results to populate.
Domain: The subdomain of your TestRail instance. It’s the part of the URL that comes between “https://” and “.testrail.io”

Next, we need to add the alert group to the project. Go to the projects view and click the “settings” icon on the project that you’d like to use the connector for.

4a

In the dropdown that appears, if you begin typing the name of the alert group in the bottom field, it will populate automatically. Select it and click the green check to complete the connection process.

Screen Shot 2018-06-27 at 10.57.38 AM

Your project in API Fortress is now connected with TestRail! It’s important to note that only test results that are generated automatically, either through the scheduler or an API call, will trigger the connector. Manually executed tests (Run Test button for example) will not trigger the connector.

As soon as a test is triggered automatically, we will see the pass/fail result appear in the project of our choice in TestRail, with a link to the test report in API Fortress. Everything you need to know about your API test results in your TestRail instance.

5a

Key/Value Store

The Key/Value Store

The Key/Value store allows API Fortress users to create temporary key/value pairs that can be accessed across different tests. The Key/Value store is accessed via the Key/Value Store Component.

Screen Shot 2018-05-24 at 1.22.48 PM

An extremely important point to note is that these key/value pairs are temporary. They expire after 24 hours has elapsed since the last update to the value itself. 

The Key/Value Store Component has 4 methods available for use. They are:

  • Set

    • Set will create a new key/value pair in the Key/Value store. The value is entered in the “Object” field.

Screen Shot 2018-05-24 at 10.50.19 AM

  • Load

    • Load will recall a value from the Key/Value store when provided with a key.

Screen Shot 2018-05-24 at 10.50.36 AM

  • Push

    • Push will add a value to the end of an existent value of the datatype “Array” in the Key/Value store. If no such key exists, it will create a new array containing the passed in value.  The passed in value is entered in the “Object” field.

Screen Shot 2018-05-24 at 10.51.09 AM

  • Pop

    • Pop will remove a value from the end of an existent value of the datatype “Array” in the Key/Value store.

Screen Shot 2018-05-24 at 10.50.52 AM

 

Basic Workflow

Let’s take a look at how this workflow works in a practical setting. The first example will be a simple set and retrieve of a value in the Key/Value Store.

First, we’ll make a GET request to an endpoint.

Screen Shot 2018-05-24 at 1.21.40 PM

Next, we’ll add a K/V Store component.

component

This first K/V Store component (we’re going to incorporate several) is going to set the Key/Value pair in the Store, so we’re going to use “Set.

Screen Shot 2018-05-24 at 1.46.41 PM

In this case we’re setting the Key “prods” equal to “products[0].name”, which in this case evaluates to “Baseball Cap.”

Next, we’re going to retrieve this Key/Value pair from the store with the “Load” method. In the K/V Store “Load” component, we’re going to assign the retrieved value to the variable “kvprods.”

Screen Shot 2018-05-24 at 1.47.22 PM

Finally, we’ll add in a “Comment” component to ensure that the data was recovered successfully.

Screen Shot 2018-05-24 at 1.48.01 PM

When we run the test, we’re presented with the following result:

Screen Shot 2018-05-24 at 1.48.28 PM

Success!

Push/Pop Workflow

Next, we’re going to take a look at how “Push” and “Pop” work. “Push” and “Pop” are both array methods and behave as they normally do outside of this context. “Push” will append a value to the end of an array. “Pop” will remove the last value in an array.

First, we’re going to use “Push.” It should be noted that “Pop” works similarly but with the opposite result. “Popalso assigns the removed value to a variable which can be used in the context of the test, but can no longer be accessed from the Key/Value Store. We’ll discuss this further when we take a look at “Pop.”

First, we’re going to send a GET request and assign a key in the Key/Value Store to a value from the response body. In this case, we’re going to use “Color,” which is an array.

Screen Shot 2018-05-24 at 1.49.16 PM

Next, we’re going to “Load” and “Comment” this key. We’re doing that so we can actually see the change on the test report at the end of this workflow.

The next step is to “Push” the new data on to the end of the existing array.

Screen Shot 2018-05-24 at 2.43.53 PM

In this case, we’re pushing the integer 999 onto the prods array.

Finally, we’re going to “Load” the modified data into the test from the K/V Store.

Screen Shot 2018-05-24 at 1.51.48 PM

When we run the test, we’re presented with the following test report.

Screen Shot 2018-05-24 at 1.51.59 PM

The comments show us clearly that we have pushed the number 999 onto the array stored in the key prods. 

Now, we’ve added something to the array. Let’s remove it with “Pop!”

The first step is to introduce a “Pop” K/V Store component.

Screen Shot 2018-05-24 at 2.31.17 PM

We provide the “Pop” component with the name of the key from the Key/Value Store, and the name of the variable we’d like to assign the popped value to. Remember, “Pop” removes the last value in an array and returns the value itself. In this case, we’re going to assign it to a variable called “Popped.”

Next, we’re going to recall the modified key from the Key/Value Store. Then, we’re going to Comment both the recalled Key/Value Store value AND the previously popped value.

Screen Shot 2018-05-24 at 2.28.58 PM

In the Test Report, we can clearly see the full workflow. First, we assigned an array to the Key/Value Store with “Set.” Then, we added to that array with “Push.” Finally, we removed the added value with “Pop.” Each time we made a change, we used “Load” to retrieve an updated value from the Key/Value Store.

Screen Shot 2018-05-24 at 2.29.09 PM

The last two comments show the final state of the array in the Key/Value Store and the popped value itself. The popped value will only be available within the scope of this test run. The array in the Key/Value Store will remain retrievable and until 24 hours after it’s most recent modification.

Note: “Load” does not reset the timer. Only “Set,” “Push,” and “Pop” reset the timer. 

Scheduler

The scheduler allows a user to schedule when a test should run.
The scheduled tests must be published.

They should retrieve resources on their own using a GET/POST/PUT/DELETE I/O operations.

You can reach the scheduler page from the Test List page:

schedulerFromTestList

or from the Test Control Panel page:

scheduleFromIntersitial

In the Scheduler, select + Create New Run in the left panel.

schedulerTopPage

Name: The name of the run. Makes it easy to recognize it from a list.
Downloader: Choose from which datacenter the resources should be retrieved from. You can select one or more.
Paused: If checked, the run will be paused and won’t trigger executions.
Try a second execution…: If a test execution fails, another execution will be run after 2m 30s.
Dedicated engine (On Prem): If you are using the On Premises version you can select a dedicated engine to run the test from.
Minutes: In which minutes of the hour the test is going to run. Minimum interval is 5 minutes but the interval you can choose from depends on the account type.
Hours: In which hours of the day the test is going to run.
Days: In which days of the month the test is going to run.
Months: In which months of the year the test is going to run.

Note: The scheduler works by intersecting the provided settings.
Example: set minutes: 5, 15 and days: 1, 5. The test will run every hour at minute 5 and 15, only if the day is either the 1st or 5th.

schedulerOverrides

Overrides: This section allows you to override any variable that is defined in the global section or in data set. You can either write the key/value couples or Import values from presets. For example, if you wrote a test against your production environment and want to keep an eye on how the staging environment reacts to the same test, you can override the ‘domain’ variable with the staging domain for a specific scheduled item that will therefore run the test altering the target host.

In the top part of the page:

schedulerGlobal

Test (drop down): The list of all available test for scheduling (all the test that are published). You can switch from one test schedule to another one without exit from the schedule page. As first item in the list there is the Global option, see below for more details about this option.
Pause All/Run All:  The buttons allow you to pause all or run all the scheduled runs with a single click.
Delete Run: Deletes the selected run.
Save Run: Save the run.

Global Scheduler

Selecting the Global option from the Test drop down you can schedule a single run for all or some of the test available in the project.

Unlike the scheduler for a single test, this one has an extra section where you can select the tests you want to schedule together.

globalSection

Note: The key/value couples inserted in the overrides section at the bottom of the page will be used for all the selected tests. If you need to add values for only a specific test from the scheduled ones, you do not have to insert them there but add them for the single test. To do so, you have firstly to save the scheduled run. Once the schedule is saved, nearby each test name will appear an icon for adding overrides values for the specific test.

overrideGlobal

 

Use the Vault (Stored and Reusable Variables / Code Snippets)

The vault allows you to store variables and code snippets that can be used across an entire project.

Explainer Video!

The link to access the Vault is at the top of the window, as shown below.


The first column shows all of the projects of a company and the Global Vault. Code snippets and variables saved in a specific project are only available in that project. They are not available across projects. If a variable and/or code snippet needs to be available in more projects within the company, they must be saved to the Global Vault. The Global Vault has been built to make variables and code snippets available across all of the projects in a company.



In the snippet section, you will find all of the snippets you have created using the composer (see here for more details). Once you have saved the snippet, from the composer, you can choose whether you want to save it and make it available only for the current project, or for all the projects within the company by saving it in the Global Vault. If you already have a snippet saved for the current project but you need to make it available across all projects, you can easily export them from the current project to the Global Vault by using the import/export feature. 

A good use case for the snippets feature is an authentication flow; you don’t need or want to rewrite all of the steps in every test. You just need to call the snippet that contains the authentication snippet. Another good example is integration testing, where you can reuse various tests to create one larger flow.

In the variable section, you can define variables which will be part of the scope of the tests.

 

If a variable with the same name is defined within the test, it will override the one defined in the Vault. For identical variable names in the global vault and in the project vault, the latter will have higher priority.

Defining a variable in the Vault is helpful when you need to use the same variable across multiple tests. This way, you don’t need to rewrite it every time. For example, a password could be saved as a variable and reused in multiple places.

Just like code snippets, if you need a variable available across multiple projects you can save it in the Global Vault or import it directly from another project.

When you open the Vault tab in the Composer, global snippets and variables are highlighted for ease of identification. 


Here is a quick example on how the Vault can be used in a test.

The Authentication Snippet

First, create a new test. Go to the test list, click +New Test, enter the test name and click Compose. Once the composer appears, we need to enter the call. For this example, we will add a GET request that logs in using a Basic authentication:

Consider a scenario where this login will be required for all the endpoints we have to test. It makes sense for this call to be stored in the Vault.

Select the GET, open the Vault panel and click the + button. Enter a name and description.

 

Now you can proceed creating the test. Once done we need to create the other tests for our API. Once again, click +New Test. Once you are in the composer, you can open the Vault panel and select the snippet we saved in the previous step.

 

To use the login call in the new test, we just need to click the down arrow button next to the snippet, and it will be added into the test.

 

Now we can call the endpoint we want to test. Let’s use the search endpoint. We pass the ‘id’ variable as a query parameter. The authorization token that we parameterized after the login call is passed in as well:

 

Now consider the case where we want to use the same ‘id’ in multiple tests. We don’t set the id as a global param or an input set. We add it to the vault instead. Save the test and exit from the composer. Click on Vault in the header and add the variable ‘id’ here:

 

Once done, go back to the test and check that the variable is available in the Vault panel:

 

Now if you launch the test you can see that the ‘id’ will be replaced with the value you have set in the Vault.

Update Input

The update input component allows you to persist a variable defined inside of the test so that the value will be accessible outside the current scope of the test.

Usually, the component is used in conjunction with the set variable component. First, we set a variable. Then, we make it available outside of the current test with the update input component.

We pass the update input component the name of the variable that we need to persist outside of the test. The component will first try to update a variable of the same name in the current input set. If that doesn’t exist, it will search for a global variable of the same name. If there is no global variable of the same name, it will check the vault. If the variable doesn’t exist there, it will create one with the same name.

Important note: the update input component works only outside of the composer. That is to say, it will only function when a test is executed from the Test List, the Scheduler, or via the API.

In the image above, after calling the login endpoint, we have created a variable called access_token with the set var component. Then, we have updated the value with the update input component. In doing so,  the value of the variable will persist throughout and the value can be used in follow-on tests.

 

JDBC

The JDBC component allows a test to query data from a database.
Typical use cases are:

  • to retrieve data items to use as input data
  • to perform data driven testing

The currently supported databases are: MySQL, PostgreSQL, and Microsft SQL Server.

Configuration keys:

  • Url: the JDBC url to the database. Depending on the database type, URLs will look like the following:
    • jdbc:mysql://database.example.com/databaseName
    • jdbc:postgresql://database.example.com/databaseName
    • jdbc:sqlserver://database.example.com;databaseName=databaseName;
  • Driver: the type of driver; you can choose it from the options available in the drop down:
    • org.postgresql.Driver
    • com.microsoft.sqlserver.jdbc.SQLServerDriver
    • com.mysql.jdbc.Driver
  • Username: the username to access the database
  • Password: the password to access the database
  • Content: the SQL query
  • Variable: the name of the variable that will store the results

The result of the query will be represented as an array where each item is a row.
Every row is a key/value map, as in:

[
  {"id",123,"first_name":"John","last_name":"Doe"},
  {"id",456,"first_name":"Annie","last_name":"Doe"}
]

Therefore, you can then iterate over the results to use them according to your needs.

read-file (on-premises only)

In an on-premises deployment, the read-file command allows you to read a text file from the server local storage, in the /data directory.

Parameters:

Name Type/Value Required
path String Yes
var String Yes
mode “json”,”xml2″,”text”,”csv” Yes

path: the path of the file, relative to the /data/ directory

var: the name of the variable that will carry the read values

mode: how the file has to be parsed

If the file is successfully read, the variable declared in the “var” attribute will contain the structured (in case of json, xml2, csv) or unstructured (in case of text) information you can use as any other piece of data.

Composer Snippets

A snippet is a fragment of test, stored in the Vault, that can be reused in multiple tests.

To create a snippet, click on the first component you want to include in the fragment. Then, hold down the [SHIFT] key and click on the last component you want to include. The selection will be highlighted.

Click the + button in the snippets section of the Vault panel, choose if you want to make it available only for the current project or for all the projects in the company and fill the Name and Description fields. That is it! The fragment of the code is created and ready to be use in every test of the project.

For each snippet, two actions are available:
1) Paste Snippet: Allows you to paste the entire component inside the test, allowing you to edit as needed. The pasted components will lose any reference to the original snippet.
2) Invoke Snippet: This creates a Call component that will invoke the snippet. If the snippet changes, all the tests containing the Call component to that snippet will inherit the changes.

Tag

This component is used to dynamically tag the resulting document of a test execution. You can easily find a document by searching for the specific tag, in the same way you can find a test by searching for the tag you assigned to it.

In the composer you will have the ‘tag’ component as option to be added. In this way, you can add different tags based on dynamic events happening during the test execution, such as a certain value retrieved in the payload. You can assign multiple tags to each test by adding more ‘tag’ components to it.

Parameters:

Name

Type/Value

Required

Value String

Yes

Another way to add a tag to your test is in the test details screen during test definition or edit.

tag

Static tags will be displayed in the tests list.

All tags, dynamic and static will mark the test execution documents. In the project dashboard, you have the ability to filter events by tags.

There is also a dedicated API that does the same. For more info please see the documentation here

Company settings

The Company settings page allows the manager(s) of a company to alter global settings such as  the users that can access the company, who will receive the notifications etc.

The page can be reached by clicking on the top right icon (gear).

The page is composed by different sections: Company, Account, Users, Notifications, Alerts Group, API Keys, API Hooks, Big Screen, Shareable. In the Company page, you can set the Company Name, the Description and the Time Zone.

The Time Zone is important because it will be used when you schedule a test: if you schedule a test at 9am and Time Zone is ‘America/New York’, test will run at 9am New York time, that is different from 9am ‘Europe/London’.

At the top of the page, the ‘Save changes‘ button will make your changes effective.

In the Account page, you can see information about the company:

Contract: the contract type of your account.
Monthly Hits: the number of executions so far for the current month.
Monthly origin downloads: how many times tests downloaded a web service
Overall documents: the number of documents generated and still available for consultation
Monthly SMSs sent: the number of SMSes sent as failure alert

The Users page allows you to add and remove users to the company account.

By clicking ‘+ User‘ you will be presented a popup:

e-mail: the email address of the user you want to add.
Full Name: the name of the user
Time Zone: the Time Zone of the user
Level: the permission level of the user: as ‘Manager’ the user will have access to company settings, tests and analytics. As ‘Developer’ the user will have access to tests and analytics. As ‘analyst’ only analytics will be made available.

In the Notification page, you can add and administer who receives alerts and reports. You can add either an user that has access to the company or an user that does not have access to the company.

The Alerts Groups section is the higher granularity notification system that includes integration with other software and platforms

The API Keys page is useful if you want to use the API. You just need to click the + API Key button to generate a key/secret pair.


In the API Hooks section you can create the project hook for the v3 API. To get started, you will first need to create a hook for the project you want to interact with by clicking + API Hook button, choose the project the hook will refer to, choose a privileged user for this hook and copy the hook in your clipboard. The generated URL will give you the ability to query some resources of the connected project without needing any further authentication, so mind who you will share it with. If you feel the url has been improperly shared, you can revoke it from the page you used to create it.

For all the update endpoints, a JWT token is required as part of the authorization process and the authenticating user has to match the user selected while creating the hook.

For details about all the endpoints, please see the documentation here:

http://docs.apifortressv3.apiary.io/

Lastly, two sections allow users to see status page. Big screens provides a URL to a real-time status page specifically designed for large screens that refreshes automatically made for internal users (it requires a valid username and password). The Shareable section allows you to create a URL for any purpose, such as being embedded in other pages.

For more details about how to Generate a Status Page see here