Declarative end to end functional testing (endly)

This library is compatible with Go 1.8+

Please refer to CHANGELOG.md if you encounter breaking changes.

• Motivation
• Installation
• Introduction
• GettingStarted
• Services
• Secrets
• Unit test
• Usage
• Best Practice
• License
• Credits and Acknowledgements

Motivation

This library was developed to enable simple automated declarative end to end functional testing for web application developed in any language.

It addresses all aspect of testing automation namely:

• Local or remote system preparation including all services required by the application.
• Checking out the application code
• Building and deploying the application as a separate process, or in the container.
• Data preparation including RDBMS, or key/value store
• Test use cases with HTTP, REST or selenium runner.
• Verification of responses, data in datastores or log produced.

Installation

Download the latst endly

or build from sources

#optionally set GOPATH directory
export GOPATH=/Projects/go

go get -u github.com/viant/endly

Introduction

Endly as a comprehensive testing framework automate the following step:

1. System preparation
   1. Local or remote on cloud
   2. System services initialization. (RDBM, NoSQL, caching or 3rd party API, dockerized services)
   3. Application container. (Docker, Application server, i,e, tomcat, glassfish)
2. Application build and deployment
   1. Application code checkout.
   2. Application build
   3. Application deployment
3. Testing
   1. Preparing test data
   2. Actual application testing
      1. Http runner
      2. Reset runner
      3. Selenium runner
   3. Application output verification
   4. Application persisted data verification
   5. Application produced log verification
4. Cleanup
   1. Data cleanup
   2. Application shutdown
   3. Application system services shutdown

Endly automate sequence of actions into reusable tasks and workflows. It uses tabular Neatly format to represent a workflow. Neatly is responsible for converting a tabular document (.csv) into workflow object tree as shown below.

See more about workflow and its lifecycle

A workflow actions invoke endly services to accomplish specific job.

Getting Started

Neatly introduction

Endly introduction

To get you familiar with endly workflows, a few examples of fully functioning applications are included. You can build, deploy and test them end to end all with endly.

1. Web Service
   • Reporter - a pivot table report builder.
     • Test with Rest Runner
     • Data Preparation and Validation (mysql)
2. User Interface
   • SSO - user registration and login application.
     • Test with Selenium Runner
     • Data Preparation and Validation (aersopike)
     • Web Content validation
3. Extract, Transform and Load (ETL)
   • Transformer - datastore to datastore transformer (i.e. aerospike to mysql)
     • Test with Rest Runner
     • Data Preparation and Validation (aersopike, mysql)
4. Runtime - simple http request event logger
   • Logger
     • Test with HTTP Runner
     • Log Validation

Endly Services

Endly services implement Service interface. The following diagram shows service with its component.

1. System services
   • SSH Executor Service
   • Storage Service
   • Process Service
   • Daemon Service
   • Network Service
   • Docker Service
2. Cloud services
   • Amazon Elastic Compute Cloud Service
   • Google Compute Engine Service
3. Build and Deployment Services
   • Sdk Service
   • Version Control Service
   • Build Service
   • Deplyment Service
4. Endpoint Services
   • Http Endpoint Service
5. Runner Services
   • Http Runner Service
   • REST Runner Service
   • Selenium Runner Service
6. Testing Services
   • Validator
   • Log Validator Service
   • Datastore Preparation and Validation Service
7. Notification Services
   • SMTP Service
8. Workflow service
   • Workflow Service

To get the latest list of endly supported services run

endly -s='*'

To check all actions supported by given service run endly -s='[service name]'

i.e

endly -s='docker'

To check request/response for service/action combination run endly -s='[service name]' -a=[action name]

i.e

endly -s='docker' -a='run'

Secrets/Credentials

Endly on its core uses SSH or other system/cloud service requiring credentials. To run system workflow the credentials file/s need to be supplied as various request field.

Endly uses Credentail Config

• it can store user and blowfish encrypted password generated by "endly -c option".
  • it can store google cloud compatible secret.json fields
• it can store AWS cloud compatible fields.
• $HOME/.secret/ directory is used to store endly credentials

Endly service was design in a way to hide user secrets, for example, whether sudo access is needed, endly will output sudo in the execution event log and screen rather actual password.

To generate credentials file to enable endly exec service to run on localhost:

Provide a username and password to login to your box.

mkdir $HOME/.secret
ssh-keygen -b 1024 -t rsa -f id_rsa -P "" -f $HOME/.secret/id_rsa
cat $HOME/.secret/id_rsa.pub >  ~/.ssh/authorized_keys 
chmod u+w authorized_keys

endly -c=localhost -k=~/.secret/id_rsa.pub

Verify that secret file were created
```text
cat ~/.secret/localhost.json

Now you can use ${env.HOME}./secret/localhost.json as you localhost credentials.

Usage

In most case scenario you would use endly app supplied with release binary for your platform. Alternatively, you can build the latest version of endly with the following command:

export GOPATH=~/go
go get -u github.com/viant/endly
go get -u github.com/viant/endly/endly

endly will be build in the $GOPATH/bin

Make sure its location is on your PATH

$ endly -h

When specified workflow or request it can be name of endly predefined workflow or request.

For instance the following command will print shared ec2 workflow in JSON format.

endly -p -w ec2

The following command will run predefined ec2 workflow with -w option

endly -w ec2 -t start awsCredential ~/.secret/aws.json ec2InstanceId i-0ef8d9260eaf47fdd

In case there are more parameters that workflow accepts it is easier to create run.json file representing *workflow.RunRequest

Example of RunRequest

{
  "URL": "manager.csv",
  "Name": "manager",
  "PublishParameters":false,
  "EnableLogging":true,
  "LoggingDirectory":"/tmp/myapp/",
  "Tasks":"init,test",
  "Params": {
    "jdkVersion":"1.7",
    "buildGoal": "install",
    "baseSvnUrl":"https://mysvn.com/trunk/ci",
    "buildRoot":"/build",
    "targetHost": "127.0.0.1",
    "targetHostCredential": "${env.HOME}/secret/localhost.json",
    "svnCredential": "${env.HOME}/secret/adelphic_svn.json",
    "configURLCredential":"${env.HOME}/secret/localhost.json",
    "mysqlCredential": "${env.HOME}/secret/mysql.json",
    "catalinaOpts": "-Xms512m -Xmx1g -XX:MaxPermSize=256m",

    "appRootDirectory":"/use/local",
    "tomcatVersion":"7.0.82",
    "appHost":"127.0.0.1:9880",
    "tomcatForceDeploy":true
  },
  "EventFilter":{}
}

In case you have defined you one UDF or have other dependencies you have to build endly binary yourself. The following template can be used to run a workflow from a command line

#endly.go

package main

//import you udf package  or other dependencies here

import "github.com/viant/endly/bootstrap"

func main() {
	bootstrap.Bootstrap()
}

Unit test

Go lang

To integrate endly with unit test, you can use one of the following

Service action

With this method, you can run any endly service action directly (including workflow with *model.ProcessRequest) by providing endly supported request.

This method runs in silent mode.

        manager := endly.New()
        var context = manager.NewContext(toolbox.NewContext())
        var runRequest = &docker.RunRequest{
                                           Target: target,
                                           Image:  "mysql:5.6",
                                           Ports: map[string]string{
                                               "3306": "3306",
                                           },
                                           Env: map[string]string{
                                               "MYSQL_ROOT_PASSWORD": "**mysql**",
                                           },
                                           Mount: map[string]string{
                                               "/tmp/my.cnf": "/etc/my.cnf",
                                           },
                                           Secrets: map[string]string{
                                               "**mysql**": mySQLcredentialFile,
                                           },
                                       }
                                       
                                       
        
        {//execute vi manager
            response, err := manager.Run(nil, runRequest)
            if err != nil {
                log.Fatal(err)
            }
            dockerResponse := response.(*docker.RunResponse)
		}
		
		{//execute vi helper method, 
		    var runResponse = &docker.RunResponse{}
		    err := endly.Run(context, runRequest, runResponse) //(use 'nil' as last parameters to ignore actual response)
		    if err != nil {
               log.Fatal(err)
            }
		}
		
		
		
		

Workfklow

In this method, a workflow runs with command runner similarly to 'endly' command line. RunnerReportingOptions settings control stdout/stdin and other workflow details.

    runner := cli.New()
	cli.OnError = func(code int) {}//to supres os.Exit(1) in case of error
	err := runner.Run(&workflow.RunRequest{
			URL: "action",
			Tasks:       "run",
			Params: map[string]interface{}{
				"service": "logger",
				"action":  "print",
				"request": &endly.PrintRequest{Message: "hello"},
			},
	}, nil)
    if err != nil {
    	log.Fatal(err)
    }

Best Practice

1. Delegate a new workflow request to dedicated req/ folder
2. Variables in Init, Post should only define state, delegate all variables to var/ folder
3. Flag variable as Required or provide a fallback Value
4. Use Tag Iterators to group similar class of the tests
5. Since JSON inside a tabular cell is not too elegant, try to use Virtual object instead.
6. Organize workflows and data by grouping system, datastore, test functionality together.

Here is an example directory layout.

    manager.csv
        |- system / 
        |      | - system.csv
        |      | - var/init.json (workflow init variables)
        |      | - req/         
        | - regression /
        |       | - regression.csv
        |       | - var/init.json (workflow init variables)
        |       | - <use_case_group1> / 1 ... 00X (Tag Iterator)/ <test assets>
        |       | 
        |       | - <use_case_groupN> / 1 ... 00Y (Tag Iterator)/ <test assets>
        | - config /
        |       
        | -  <your app name for related workflow> / //with build, deploy, init, start and stop tasks 
                | <app>.csv
                | var/init.json 
        
        | - datastore /
                 | - datastore.csv
                 | - var/init.json
                 | - dictionary /
                 | - schema.ddl
    

Finally contribute by creating a pull request with a new common workflow so that other can use them.

License

The source code is made available under the terms of the Apache License, Version 2, as stated in the file LICENSE.

Individual files may be made available under their own specific license, all compatible with Apache License, Version 2. Please see individual files for details.

Credits and Acknowledgements

Library Author: Adrian Witas