cloud optimized rest api automation framework...cloud optimized rest api automation framework...

CLOUD OPTIMIZED REST API AUTOMATION FRAMEWORK

Shoukathali C KSoftware Quality Engineer, EMC [email protected]

Shelesh ChopraSenior QA Manager, EMC [email protected]

Gururaj KulkarniQA Manager, EMC [email protected]

mailto:[email protected]



2014 EMC Proven Professional Knowledge Sharing 2

Table of Contents Introduction ................................................................................................................................ 4

What Is REST? .......................................................................................................................... 5

REST Specifications .................................................................................................................. 6

HTTP methods ....................................................................................................................... 6

HTTP GET method ............................................................................................................. 6

HTTP POST method ........................................................................................................... 6

HTTP PUT method ............................................................................................................. 7

HTTP DELETE method ....................................................................................................... 8

HTTP OPTIONS method ..................................................................................................... 9

HTTP status codes ............................................................................................................10

What to Test In a REST-Based Application ...............................................................................10

REST Client...........................................................................................................................10

Tenant UI ..............................................................................................................................10

REST Server .........................................................................................................................11

Required Tests..........................................................................................................................11

Unit tests ...............................................................................................................................11

Integration or system tests .....................................................................................................11

Functional tests .....................................................................................................................12

Load or stress tests ...............................................................................................................12

Getting Started With REST API Testing ....................................................................................13

Feasibility of product to use REST API Tool/Framework .......................................................13

Feasibility study of API testing tool ........................................................................................14

Automate the tests/Actual Script Development ......................................................................14

Choose stable APIs ...........................................................................................................14

Data-driven tests ................................................................................................................14


Create the test modules ........................................................................................................14

Execution and Reporting .......................................................................................................14

Tools Evaluated For REST API Testing [Feasibility Study] ........................................................15

Jersey Client API-Based [Java] .................................................................................................15

Sample Example ................................................................................................................15

Output of the Program........................................................................................................18

Python-Based Testing ...............................................................................................................21

Automation Framework Design .................................................................................................25

Test cases in XML .................................................................................................................26

Sample TC XML example ..................................................................................................26

Authentication certificate Header ...........................................................................................28

XML parser ............................................................................................................................28

HTTP CALL Invoker ..............................................................................................................29

HTTP RESPONSE Validator .................................................................................................29

Logging and reporting ............................................................................................................29

Conclusion ................................................................................................................................29

Disclaimer: The views, processes, or methodologies published in this article are those of the

authors. They do not necessarily reflect EMC Corporation’s views, processes, or

methodologies.


Introduction

The world has adopted cloud technologies such as Infrastructure as a Service (IaaS) and

Software as a Service (SaaS) and is consuming many applications deployed in the cloud. Most

applications are developed using Representational State Transfer (REST) protocol. Manual

testing of REST-based applications has become more difficult and less efficient since the quality

cannot be negotiated on other main factors such as application performance and scalability. An

effective generic automation framework which tests the functionality, scalability, and

performance of REST-based applications will definitely improve quality as well as the time

required to complete the test activity. The approach is very different from the normal way of

automating an application.

After evaluating many tools and analyzing the scope of testing for REST-based applications, we

have come up with a generic automation framework. The tool is focused primarily on;

Portability – Generic and can be expanded across different products

Extensibility – Can add layers to support the future request

Modularity – To allow plug-and-play of separate intelligent modules

Intelligence – Leverages cloud-optimized solutions as well as analytics to store and predict

various requirements

Migration – From legacy to REST API-based framework

Abstraction & Simplified – What is to be automated is user’s choice with a lot of abstraction to

how it is automated

We will explain the generic approach of REST-based applications which supports all areas of

REST-based apps—GUI, RESTCLIENT, RESTAPI. The framework is designed such that it can

be used across platform and across organization irrespective of application build based on

REST protocol.


What Is REST?

Before talking about the testing scope of REST, let us focus on understanding what REST is

and its features. REST is a simple stateless architecture that generally runs over Hypertext

Transfer Protocol (HTTP).

REST is often used in mobile applications, social networking Web sites, and automated

business processes. The REST style emphasizes that interactions between clients and services

is enhanced by having a limited number of operations (verbs). Flexibility is provided by

assigning resources (nouns) their own unique universal resource indicators (URIs). Because

each verb has a specific meaning (GET, POST, PUT, and DELETE), REST avoids ambiguity.

REST is an "architectural style" that basically exploits the existing technology and protocols of

the Web, including HTTP and XML. REST is simpler to use than the well-known Simple Object

Access Protocol (SOAP) approach, which requires writing or using a provided server program

(to serve data) and a client program (to request data). REST uses the four HTTP methods GET,

POST, PUT, and DELETE to execute different operations.

A REST API is a set of operations that can be invoked by means of any the four verbs, using

the actual URI as parameters for your operations. For instance, you may have a method to

query all your accounts which can be called from /accounts/all/. This invokes a HTTP GET and

the 'all' parameter tells your application that it shall return all accounts. REST is the model of the

Web. For example, your browser makes a request to a URL and receives a response. That

request can be a GET or POST (or a PUT, DELETE, or HEAD) and the response can be

anything—HTML, an image file, a PDF, XML, and so on.

REST is an alternative to SOAP-based web services. Where SOAP tries to model the exchange

between client and server as calls to objects, REST tries to be faithful to the web domain.

Thus, when calling a web service written in SOAP, you may write

productService.GetProduct("1") (SOAP)

On the other hand, using REST, you may call a URL with

HTTP GET (REST URL)

http://searchwindevelopment.techtarget.com/definition/HTTP

http://searchsoa.techtarget.com/definition/SOAP


REST Specifications

REST is an architectural style and consists of several constraints to address separation of

concerns, visibility, reliability, scalability, performance, and more. This section will focus on how

to apply REST constraints to the API in the context of HTTP.

HTTP methods

A request message from the client includes a HTTP method and a URI. The HTTP method

specifies an operation that the client wishes to apply to the resource identified by the URI. The

REST API SHALL processes the following common HTTP methods as specified in RFC 2616. If

the requested method is not allowed for the resource, the REST API SHALL replies with an

error response with a HTTP 405 (Method Not Allowed) status code.

HTTP GET method

The GET method is used to retrieve the resource identified by the URI.

The following example shows a HTTP GET request.

GET /user/clients HTTP/1.1

Host: localhost

Example: HTTP GET Request

HTTP POST method

The POST method is used to create a new resource via the controller resource identified by the

URI. Note that the URI identifies a controller resource, which creates a new resource; a new

URI will be assigned to the new resource. The POST method, by its nature, is not idempotent.

If a new resource is created, the REST API SHALL give a response of a HTTP 201 (Created)

status code, and the response SHALL contains a new URI of the new resource in the Location

response header.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html


The following example shows a HTTP POST request and a response.

POST /user/clients HTTP/1.1

Host: localhost

Content-Type: application/json;version=1

Content-Length: n

{ Name: “test.emc.com”,Policy: “policy1”, … }

HTTP/1.1 201 Created

Content-Length: 0

Location: http://localhost/user/clients/6b24569b...-00000000

Example: HTTP POST Request & Response

HTTP PUT method

The PUT method is used to update an existing resource, which is identified by the URI, with the

entity enclosed in the request. It SHALL be an idempotent method. If the URI does not point to

an existing resource, the REST API MAY create a new resource only if the REST API agrees

that the new resource would be assigned with the URI specified in the request.

If the existing resource is modified, the REST API SHALL reply a response with either a HTTP

200 (OK) status code or a HTTP 204 (No Content) status code to indicate successful

completion of the request.


The following example shows a HTTP PUT request and a response.

PUT /user/clients/6b24569b...-00000000 HTTP/1.1

Host: localhost

Content-Type: application/json;version=1

Content-Length: n

{ Name: “test.emc.com”, Policy: “policy2”, … }

HTTP/1.1 204 No Content

Example: HTTP PUT Request & Response

HTTP DELETE method

The DELETE method is used to delete the existing resource identified by the URI. It SHALL be

an idempotent method.

If the DELETE request has been carried out successfully, the REST API SHALL reply a success

response with either a HTTP 200 (OK) status code or a HTTP 204 (No Content) status code.


The following example shows a HTTP DELETE request and a response.

DELETE /user/clients/6b24569b...-00000000 HTTP/1.1

Host: localhost

HTTP/1.1 204 No Content

Example: HTTP DELETE Request & Response

HTTP OPTIONS method

The OPTIONS method is used to retrieve the list of methods supported by the resource

identified by the URI. The client may use the OPTIONS method to determine valid methods that

the resource is associated with.

The REST API SHALL implements the OPTIONS method to return the list of supported

methods in the Allow response header.

The following example shows a HTTP OPTIONS request and a response.

OPTIONS /user/clients HTTP/1.1

Host: localhost

HTTP/1.1 200 OK

Allow: POST,GET,OPTIONS,HEAD

Content-Length: 0

Example: HTTP OPTIONS Request & Response


HTTP status codes

The REST API SHALL use HTTP status codes as specified in RFC 2616. When the REST API

returns an error status code, which is in the 4xx and 5xx ranges, the REST API SHALL use a

common error representation to return details of the error information. The common error

representation SHOULD include the following fields: severity, error code, timestamp, message

ID, and message.

What to Test In a REST-Based Application

Testing of REST-based applications can broadly be classified in the following grouping:

REST Client

Tenant UI

Direct Interaction With REST Server

REST Client

Each REST-based application allows users to form the REST clients and to get the response

back for particular operation. For example;

REST-based app will be tested by making HTTP calls such as (GET, POST, PUT, DELETE) to

the respective URL which maps to a particular REST API and the arguments needed for the API

to function.

The tools that can be used to test REST client and the test approach are explained in the

following sections.

(Http call GETS, API Getrecenttimlineupdate, Argument count))

https://api.twitter.com/1.1/statuses/mentions_timeline.json?count=5

Tenant UI

Tenant UI comes to the next scope of testing. Each REST-based application provides a Tenant

UI to the user to interact with the web services.

Users might not be aware of the actual call a REST client makes to the REST server and also

the processing that happens in the presentation layer.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html


GUI testing that would be done using any of the GUI automation tools will be discussed in later

sections.

REST Server

The final scope of testing with REST-based application is a type of unit testing which is

performed by the developer to assess functionality. In this case, users can directly interact or

invoke the REST API—without making HTTP calls—and validate the same against the actual

output.

Required Tests

There are different tests that can be performed on an API, and each type is designed to validate

a different aspect of the API. This section discusses some of the tests that should be considered

part of the test planning process. We are suggesting a testing use case but extend it further and

you can have built-in product intelligence for automating any use case that traverses the same

modeling example—Config Checker, Servicability, Asset management, Monitoring tools, etc.

The following test models can be augmented using REST-Based API framework

Unit tests

A unit test is a small, atomic test that runs whenever a build of the software is performed.

Developers usually create unit tests at the same time as they develop the APIs, although they

can be written beforehand as well, especially in organizations that adopt test-driven

development methodologies.

Integration or system tests

Unit tests generally test a single module, and if that module is dependent on other modules,

they are emulated using mocks. Integration tests, also called system tests, perform tests similar

to unit tests, but instead of working against mock objects and emulated services, they work

against real implementations.

These tests can only be performed on a system where all of the modules have been deployed

and configured correctly.


Functional tests

Functional tests execute scenarios from a user’s perspective by simulating the user’s behavior.

If we take the example of transferring money between accounts, we might want to create a

battery of functional tests to check transferring money between different accounts belonging to

the same customer, and between accounts belonging to different customers. Functional testing

includes negative testing, which ensures the functionality is stable even when invalid values are

supplied, and security testing, which highlights vulnerabilities and ensures that the system

cannot be abused. Functional tests are performed directly against the user interface of the

application being tested, and should be performed on the API layer.

Load or stress tests

Load tests are designed to ensure that the system performs correctly when subjected to

different loads. The system is usually tested by generating a surge or drop in the number of

concurrent users, or by running different scenarios concurrently. These tests are also referred to

as stress tests when abnormally high loads are placed on the system, which is expected to fail

gracefully.


Getting Started With REST API Testing

REST API testing can be achieved with the process shown below. Here are some steps that

can be taken to ease the process.

Feasibility of product to use REST API Tool/Framework

It is evident that we need to be diligent on what works for a specific product. Not every

framework is optimal for every product and it is very important to plan on the pros and cons of

framework adoption. Understand the product implementation and see where it exposes hooks to

make use of REST calls and adoption.

Fasibility of product to use REST API Tool

Framework analysis

Actual Script Development

Create the test modules

Execution and Reporting


Feasibility study of API testing tool

Decide on an API testing tool that will help you create, maintain, and run tests and that support

the technologies you need to test.

Automate the tests/Actual Script Development

Now you have to decide which tests to automate. To get a quick return on your investment, you

should be looking at tests which meet the following criteria:

Choose stable APIs

The first consideration when starting to automate API testing is how much the API is going

to change. If the API changes frequently, you will be spending your time maintaining the

tests, rather than writing new tests. So start with APIs that are not expected to change

much, and write small tests. As you increase the coverage of your stable APIs, you can start

testing the less stable APIs, and get more value out of early testing.

Data-driven tests

We noted above that APIs could have complex, hierarchical data structures. These data

structures can be filled in essentially infinite combinations of values. With a data-driven test,

you can exercise an API with as many combinations of data as you want by running the test

once for each set of data. API testing tools refer to each of these runs as iterations, where

each iteration is a different test case. If you need to test a new situation, you don’t need to

touch the test—all you need to do is add a new row of data to the test’s data source. In our

case, the XML in which TC is written (Explained in section below.)

Create the test modules

A best practice is to create as many hooks/modules as possible to ensure reusability,

repurposing, and refactoring of the code during evolution phases of both product and

framework. Avoid creating hard wired/dependent and cascaded modules which would create

maintenance overhead in case there are changes to be made.

Execution and Reporting

Self-reporting and automation is needed for real time tracking/reporting and monitoring. Once

you’ve written your test, you want it to work for you. The most effective way to do that is to run

them whenever the API or service is built, preferably as part of your CI system if you have one.


However, when running the tests, ensure that the results are published to the relevant

stakeholders’ dashboards so that they get an indication as soon as any of the tests fail.

Tools Evaluated For REST API Testing [Feasibility Study]

Tools are evaluated based on the scope of REST application—UI, REST CLIENT, and REST

SERVER—and different type of testing [unit testing, integration testing, etc.

Jersey Client API-Based [Java]

Jersey 1.0 is an open-source, production-ready reference implementation of JAX-RS,

the Java API for RESTful Web Services (JSR-311). Jersey makes it easy to create

RESTful web services using Java technology also to interact with web services.

We can use the Jersey client library to interact with REST application either directly

[DIRECT REST SERVER INTERACTION] or by creating an REST CLINT application

and use HTTP GET, POST, PUT DELETE methods for requesting REST resource.

Sample Example

Source code given below uses an open source library [JERSEY client] in order to interact with

Google MAP REST API. It gets the response based on the search criteria mentioned in the GET

URL.

package rest_test;

import java.net.URI;

import javax.ws.rs.core.MediaType;

import javax.ws.rs.core.UriBuilder;

import com.sun.jersey.api.client.Client;

import com.sun.jersey.api.client.ClientResponse;

import com.sun.jersey.api.client.WebResource;

import com.sun.jersey.api.client.config.ClientConfig;

import com.sun.jersey.api.client.config.DefaultClientConfig;

https://jersey.dev.java.net/

https://jsr311.dev.java.net/


import com.sun.jersey.core.util.MultivaluedMapImpl;

import javax.ws.rs.core.MultivaluedMap;

/**

*

* @author Shoukathali chandrakandi

*/

public class SampleRest {

public WebResource service;

public Client client;

public ClientConfig config;

public SampleRest()

{

config = new DefaultClientConfig();

client = Client.create(config);

service = client.resource(getBaseURI());

}

public boolean testDeployed () {

MultivaluedMap queryParams = new MultivaluedMapImpl();

queryParams.add("address", "San Francisco");


queryParams.add("sensor", "false");

String s = service.path("/geocode/json").queryParams(queryParams).accept(MediaType.APPLICATION_JSON).get(String.class);

ClientResponse response = service.path("/geocode/json").queryParams(queryParams).get(ClientResponse.class);

System.out.println(response + "\n \n " );

System.out.println(s);

if(s.length() == 0)

{

return false;

}

else

{

return true;

}

}

private static URI getBaseURI()

{

return UriBuilder.fromUri("http://maps.googleapis.com/maps/api").build();

}

}


Output of the Program

Output of the program above is shown below. Returned response code is 200 OK and types of

response returned of JSON type.

GET http://maps.googleapis.com/maps/api/geocode/json?sensor=false&address=San+Francisco returned a response status

of 200 OK

{

"results" : [

{

"address_components" : [

{

"long_name" : "San Francisco",

"short_name" : "SF",

"types" : [ "locality", "political" ]

},

{

"long_name" : "San Francisco County",

"short_name" : "San Francisco County",

"types" : [ "administrative_area_level_2", "political" ]

},

{

"long_name" : "California",

"short_name" : "CA",


"types" : [ "administrative_area_level_1", "political" ]

},

{

"long_name" : "United States",

"short_name" : "US",

"types" : [ "country", "political" ]

}

],

"formatted_address" : "San Francisco, CA, USA",

"geometry" : {

"bounds" : {

"northeast" : {

"lat" : 37.9297707,

"lng" : -122.3279148

},

"southwest" : {

"lat" : 37.6933354,

"lng" : -123.1077733

}

},

"location" : {


"lat" : 37.7749295,

"lng" : -122.4194155

},

"location_type" : "APPROXIMATE",

"viewport" : {

"northeast" : {

"lat" : 37.812,

"lng" : -122.3482

},

"southwest" : {

"lat" : 37.70339999999999,

"lng" : -122.527

}

}

},

"types" : [ "locality", "political" ]

}

],

"status" : "OK"

}


Verifying the added message in the Queue Passed

----------------------------------------

Total Tc Passed :- 1

Total Tc Failed :- 0

----------------------------------------

BUILD SUCCESSFUL (total time: 3 seconds)

Python-Based Testing

Python is most favorable scripting language today. We evaluated use of Python library to

interact with REST web services. They are:

Urlib/Urllib2

Httplib/httplib2

Python-based code to get the response from REST-based web services is shown in the

following.


import httplib2

import time

import datetime

import xml.etree.ElementTree as et

import json

def GET_request(url):

h = httplib2.Http()

resp, content = h.request(url)

print 'Response Code is:- ' + str(resp.status)

print content

print 'Response Content Type is :- ' + str(resp['content-type'])

if (resp.status == 200 and resp['content-type'] == 'text/html' ) :

print "Response Code and Content-Type Verfication - Passed"

else :

print "Response Code and Content-Type Verfication - Failed"

return resp, content

# Send the GET request

url = 'http://bitworking.org'


# Invoke the GET URL Request

GET_request(url)

Of course, the tools or libraries mentioned above can be used to interact with RESP API

directly. It means that this can be used as it is for Unit testing. If END to END testing is desired,

an Automation framework designed for the same is needed. This will be discussed in detail

later. The following table provides a comparison of tools evaluated as part of REST API testing.


urllib/httlib [Python based] JERCY [Java based] SOUP UI

Urllib (particularly urllib2) handles many things by

default or has appropriate libs to do so. For example,

urllib2 will follow redirects automatically and you can

use cookiejar to handle login scripts. These are all

things you'd have to code yourself if you were using

httplib

if you're dealing solely with http/https and need access

to HTTP specific stuff, use httplib.

For all other cases, use urllib2.

Java based Liscence solution to get

the complete feature

Uses Jercy client UI based

Urllib/urllib2 is built on top of httplib. It offers more

features than writing to httplib directly.

Can create REST client as well as

we can directly talk to REST

server

Scenario based can be

dificult

More of Unit testing

However, httplib gives you finer control over the

underlying connections.


Automation Framework Design

We have explained various tools/libraries that help get the response back from REST web

services. Now let’s explore how to design an automation framework which helps us perform unit

testing, integration testing, and scenario-based testing. The diagram below depicts the low-level

architecture of a REST API automation framework which helps achieve efficient testing.

The tools mentioned below can be used only for unit testing. However, integrating the tools

mentioned above tools should enable complete integration testing as well as other types of

testing.

HTTP CALL INVOKER module would be replaced with the unit testing approach explained

above—Jercy or python-based engine. All other modules mentioned in the diagram would be

the part of automation framework which provides the user with the following functionalities.

Test case writing facility with XML.

Data-driven approach of test case writing so that framework gets scenario-based testing

ability.

Logging and reporting section that includes complete details of logs of real time REST

application test scenarios.

HTTP RESPONSE VALIDATOR

Let’s look at the details of automation framework modules, how it works, and its flowchart.

Test cases in XML

As per new generation automation framework design methodology, it is preferred to have the

test data and test cases written in XMLformat. We suggest this approach because parsing logic

and data retrieval is easier with an XML type protocol. The user has to write the TC in XML

format, for example, TC_REST1.XML. This XML contains the test data and the verification steps

required to verify the REST response data against an actual result—mostly a backend db where

the information is stored.

Sample TC XML example

<? Xml version="1.0" encoding="utf-8”?>

<TC1_RESTAPI>



<RESTAPICALL>PUT Login.json</RETAPICALL>

<TestData>

<Username>demo</username>

<password>demo</password>

<expectedresult>200OK</expectedresult>

</TestData>


<RESTAPICALL>GET timelinedata.json</RETAPICALL>

<TestData>

<usernamename>demo</username>

<expectedresult>200OK</expectedresult>


</TestData>


<CORECALL>verifytimelinedata</CORECALL>

<op>variable</op>

<TestData>

<usernamename>demo</username>

<expectedmessage>”Logged in” <expectedmessage>

</TestData>

</TC1_RESTAPI>

Test case which is represented in XML format has three test steps.

1. PUT username and password to Login UR

2. Expect 200 OK response from REST URL

3. Get the timeline data from REST URL and then does the verification against backend

database.

The test case has to follow the tags such as <RESTAPICALL> and <CORECALL> to invoke the

write method. Understand that RESTAPICALL framework is a call which has to be made to the

REST ENGINE (here, REST ENGINE refers to the portion of code which runs the REST CALL).

Additionally, understand that CORECALL framework has to be executed on the verification

engine, which might be a portion of code which get the control of the backend DB.


Authentication certificate Header

This module stores information required for authorization and authentication—such as HTTP

authentication, basic authentication, Oauth, Oauth2—and the related information for the

authentication mechanism such as the details provided in the following.

Note that these data also stored in XML format.

XML parser

The major portion of the automation framework, XML parser reads both testcase.xml and

autehntication.xml and forms the right command that has to be sent to the HTTPCALLinvoker

module. Basically, this is the program which helps to invoke the REST web service and get the

result. We have already discussed various tools or modules that can be used for these

purposes. Users must choose from the list mentioned above and replace it as HTTPinvoker

module. Hence, a plug-and-play option is provided for the user to choose the tool they want to

use.

XML parser will identify the service and the parameter from TC.xml and make the decision of

call based on the XML tag.


HTTP CALL Invoker

Tools mentioned in the feasibility study section must be used to communicate with the REST

API web services. They can be;

Jercy-based module

Python-based (urlib/httplib) module

SOAP UI

Selenium

REST – Assured

HTTP RESPONSE Validator

This module is responsible for validating the Web service content after parsing it from response

type such as xml/JSON, etc. against the backend database.

Logging and reporting

Finally, this module reports test results into HTML or human readable format.

Conclusion

We have evaluated the REST application architecture and developed an understanding of the

areas of testing and types of testing that has to be done with the same. Following a feasibility

study of the existing tools, we came up with a generic portable plug-and-play automation

framework to support REST-based automation effectively.


EMC believes the information in this publication is accurate as of its publication date. The

information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION

MAKES NO RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO

THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED

WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an

applicable software license.

cloud optimized rest api automation framework...cloud optimized rest api automation framework...

Documents