Quick Start

Quickstart & Try
More
RPM & binary installation packages
Appendix

Quickstart & Try

This chapter shows the instructions of deployment for single-machine, not cluster. Only hustdb will be used for test, other modules of huststore are skipped. See more details on cluster deployment in here.

$ sudo yum groupinstall -y 'Development tools'
$ sudo yum install -y pcre-devel libidn-devel openssl-devel

$ wget https://github.com/Qihoo360/huststore/archive/v1.7.tar.gz -O huststore-1.7.tar.gz
$ tar -zxf huststore-1.7.tar.gz
$ cd huststore-1.7
$ sh prebuild.sh --prefix=/opt/huststore
$ sudo mkdir /opt/huststore
$ sudo chown -R $USER:$USER /opt/huststore
$ sh build.sh --module=3rd,hustdb

WARNING: please make sure there are enough disk spaces in /opt/huststore (at least 10G), otherwise change a different location.

Start the service:

$ cd /opt/huststore/hustdb
$ sh start.sh

NOTE: it will cost about 30 seconds to initialize data of hustdb for the first time, so do not be panic ^_^

Test the service:

curl -i -X GET 'localhost:8085/status.html'

Infomation returned:

HTTP/1.1 200 OK
Content-Length: 3
Content-Type: text/plain

ok

Stop the service:

$ cd /opt/huststore/hustdb
$ sh stop.sh

See more API from here

Back to top

More

This chapter shows the instructions of deployment for cluster, not single-machine, which is preferred in production environment.

Please read the following instructions section by section without any skip.
At least two machines are required to deploy huststore, for example: ["192.168.1.101", "192.168.1.102"].
A standalone build machine is strongly suggested, for example: 192.168.1.100.

Assumption

user: jobs
build machine : 192.168.1.100
deployment machines : ["192.168.1.101", "192.168.1.102"]
hustdb: ["192.168.1.101:8085", "192.168.1.102:8085"]
hustdb ha: ["192.168.1.101:8082", "192.168.1.102:8082"]
hustmq: ["192.168.1.101:8086", "192.168.1.102:8086"]
hustmq ha: ["192.168.1.101:8080", "192.168.1.102:8080"]

Please replace the arguments above (user, build machine, deployment machines) with your real values in production environment.

Tools

NOTE: All of the tools below are under root folder of huststore.

prebuild.sh

prebuild.sh is used to prepare the build environment of huststore, usage:

usage:
    sh prebuild.sh [option]
    
    [option]
        --help                             show the manual
        --prefix=PATH                      set installation prefix of 3rd & huststore
            
sample:
    sh prebuild.sh --help
    sh prebuild.sh --prefix=/opt/huststore
    sh prebuild.sh

NOTE: if --prefix is not set, prebuild.sh will use /opt/huststore as default installation path. Please make sure there are enough disk spaces in this folder. (at least 10G)

Back to top

build.sh

NOTE: build.sh is generated by prebuild.sh, so you need to run prebuild.sh first.

build.sh is used to build and generate installation package for OPS, usage:

usage:
    sh build.sh [option]

    [option]
        --help                             show the manual

        --module=3rd                       build and generate installation package for third-party libs.
                                           It should be installed FIRST to build other modules.

        --module=hustdb                    build and generate installation package for hustdb
        --module=hustdbha                  build and generate installation package for hustdbha
        --module=hustmq                    build and generate installation package for hustmq
        --module=hustmqha                  build and generate installation package for hustmqha
        
        --clean                            clean obj & bin files of latest build
            
sample:
    sh build.sh --help
    sh build.sh --module=hustdb
    sh build.sh --module=3rd,hustdb
    sh build.sh --module=3rd,hustdb,hustdbha
    sh build.sh --module=3rd,hustmq,hustmqha
    
    sh build.sh --clean

    sh build.sh

NOTE: if --module is not set, build.sh will build all modules of huststore, and generate the installation packages as following:

elf_3rd.tar.gz
elf_hustdb.tar.gz
elf_hustdbha.tar.gz
elf_hustmq.tar.gz
elf_hustmqha.tar.gz

These installation packages are placed in the same folder as prebuild.sh, which could be installed by scp & tar to production environment under installation path /opt/huststore.

If --module is set as specified name, for example, hustdb, then only elf_hustdb.tar.gz will be generated.

WARNING: you need to build & install 3rd first before build other modules:

$ sh build.sh --module=3rd

Back to top

start.sh

After build & install, start.sh will be generated to binary folder of service, which is used to start service. Usage:

sh start.sh

For hustdb and hustmq, you can add option check:

sh start.sh check

It will start hustdb process and wait for the initialization of service. Maximize waiting time : 90 seconds.

Back to top

stop.sh

After build & install, stop.sh will be generated to binary folder of service, which is used to stop service. Usage:

sh stop.sh

Back to top

remote_scp.py

remote_scp.py is a wrapper of scp, which supports batch scp.

usage:
    python remote_scp.py [option] [user] [host_file] [remote_folder] [local_file1] [local_file2] ...
    
    [option]
        --silent                          run in silent mode

sample:
    python remote_scp.py jobs host.txt /opt/huststore/hustdbha/conf nginx.conf hustdbtable.json
    python remote_scp.py --silent jobs host.txt /opt/huststore/hustdbha/conf nginx.conf hustdbtable.json

Note : with option --silent, remote_scp.py will sudo to [user] first, then run scp with option StrictHostKeyChecking=no.

For convienience, you can add contents as below to the file /etc/sudoers of remote machines:

# just a example
%jobs ALL=(jobs) ALL
User_Alias USERS = %jobs
USERS ALL= (jobs) NOPASSWD:ALL
USERS ALL= NOPASSWD: /bin/su - jobs

Back to top

remote_ssh.py

remote_ssh.py is a wrapper of ssh, which supports batch ssh.

usage:
    python remote_ssh.py [option] [user] [host_file] [cmds_file]
    
    [option]
        --silent                          run in silent mode
        
sample:
    python remote_ssh.py jobs host.txt cmds.txt
    python remote_ssh.py --silent jobs host.txt cmds.txt

Note : with option --silent, remote_ssh.py will sudo to [user] first, then run ssh with option StrictHostKeyChecking=no.

For convienience, you can add contents as below to the file /etc/sudoers of remote machines:

# just a example
%jobs ALL=(jobs) ALL
User_Alias USERS = %jobs
USERS ALL= (jobs) NOPASSWD:ALL
USERS ALL= NOPASSWD: /bin/su - jobs

Back to top

remote_prefix.py

remote_prefix.py is used to check and set the installation path of remote machines, usage:

usage:
    python remote_prefix.py [user] [host_file] [prefix] [owner]
sample:
    python remote_prefix.py jobs host.txt /opt/huststore jobs

Arguments:

user : username for ssh
host_file : remote host list file
prefix : installation folder
owner : owner of installation folder

For example, if you run this command:

python remote_prefix.py admin host.txt /opt/huststore jobs

Then remote_prefix.py will login by user admin to the machines stored in host.txt one by one, create folder /opt/huststore if it is not exist, then change its owner to user jobs.

For convienience, you can add contents as below to the file /etc/sudoers of remote machines:

admin ALL = (ALL) ALL

%jobs ALL=(jobs) ALL
User_Alias USERS = %jobs
USERS ALL= (jobs) NOPASSWD:ALL
USERS ALL= NOPASSWD: /bin/su - jobs

remote_deploy.py

remote_prefix.py is used to deploy installation package to remote machines, usage:

usage:
    python remote_deploy.py [option] [user] [host_file] [prefix] [tar]
    
    [option]
        --silent                          run in silent mode
    
sample:
    python remote_deploy.py jobs host.txt /opt/huststore elf_hustdb.tar.gz
    python remote_deploy.py --silent jobs host.txt /opt/huststore elf_hustdb.tar.gz

Arguments:

user : user name for ssh & scp command
host_file : remote host list stored in file
prefix : installation folder in remote host
tar : local elf installation package

For example, if you run this command:

python remote_deploy.py --silent jobs host.txt /opt/huststore elf_hustdb.tar.gz

Then remote_deploy.py will login by user jobs to the machines stored in host.txt one by one, copy installation package elf_hustdb.tar.gz to remote machine, and untar it to the folder /opt/huststore.

remote_service.py

remote_service.py is used to control huststore services in remote machines, usage:

usage:
    python remote_service.py [option] [user] [host_file] [bin_folder] [action]
    
    [option]
        --silent                          run in silent mode
    
    [action]
        --start                           start remote service
        --stop                            stop remote service
        
sample:
    python remote_service.py jobs host.txt /opt/huststore/hustdb --start
    python remote_service.py jobs host.txt /opt/huststore/hustdb --stop
    
    python remote_service.py --silent jobs host.txt /opt/huststore/hustdb --start
    python remote_service.py --silent jobs host.txt /opt/huststore/hustdb --stop

Arguments:

user : user name for ssh command
host_file : remote host list stored in file
bin_folder : binary folder of installed huststore's service in remote host
action : start or stop service

Back to top

make_conf.py

make_conf.py is used to generate configuration of hustdb ha or hustmq ha quickly, usage:

usage:
    python make_conf.py [host_file] [module] [HA port] [backend port]
    
    [module]
        hustdbha
        hustmqha

sample:
    python make_conf.py host.txt hustdbha 8082 8085
    python make_conf.py host.txt hustmqha 8080 8086

Arguments:

host_file : remote hustdb | hustmq host list stored in file
module : hustdbha | hustmqha
HA port : listen port of hustdbha | hustmqha
backend port : listen port of hustdb | hustmq

Back to top

Preparation

Login to build machine (192.168.1.100), download huststore src and untar it:

$ ssh 192.168.1.100  # please replace the ip with your build machine
$ wget https://github.com/Qihoo360/huststore/archive/v1.7.tar.gz -O huststore-1.7.tar.gz
$ tar -zxf huststore-1.7.tar.gz
$ cd huststore-1.7

Edit file hosts:

$ vi hosts

Add contents as below and save, please replace to your real deployment machines：

192.168.1.101
192.168.1.102

Run prebuild.sh to prepare the build environment.

$ sh prebuild.sh --prefix=/opt/huststore
$ sudo mkdir /opt/huststore
$ sudo chown -R $USER:$USER /opt/huststore

WARNING: please make sure there are enough disk spaces in /opt/huststore (at least 10G), otherwise change a different location.

Run remote_prefix.py to set the installation path:

$ python remote_prefix.py jobs hosts /opt/huststore jobs

Back to top

Dependency Deployment

Build third-party and generate installation package:

$ sudo yum groupinstall -y 'Development tools'
$ sudo yum install -y pcre-devel libidn-devel openssl-devel
$ sh build.sh --module=3rd

Install third-party (copy installation package to remote machines and untar to /opt/huststore):

$ python remote_deploy.py --silent jobs hosts /opt/huststore elf_3rd.tar.gz

Back to top

Hustdb Cluster Deployment

hustdb

Build and make installation package of hustdb:

$ sh build.sh --module=hustdb

Deploy hustdb (copy installation package to remote machines and untar to /opt/huststore):

$ python remote_deploy.py --silent jobs hosts /opt/huststore elf_hustdb.tar.gz

Start service (ssh to remote mathines one by one, run script /opt/huststore/hustdb/start.sh):

$ python remote_service.py --silent jobs hosts /opt/huststore/hustdb --start

Type in the below command to test:

curl -i -X GET '192.168.1.101:8085/status.html'
curl -i -X GET '192.168.1.102:8085/status.html'

Infomation returned:

HTTP/1.1 200 OK
Content-Length: 3
Content-Type: text/plain

ok

The result shows that the servers work as expected.

Back to top

hustdb ha

Build and make installation package of hustdb ha:

$ sh build.sh --module=hustdbha

Generate configuration of hustdb ha:

$ python make_conf.py hosts hustdbha 8082 8085
$ cp hustdb/ha/nginx/conf/nginx.conf nginx.conf.db
$ cp hustdb/ha/nginx/conf/hustdbtable.json .

Deploy hustdb ha (copy installation package to remote machines and untar to /opt/huststore, then replace the configuration):

$ python remote_deploy.py --silent jobs hosts /opt/huststore elf_hustdbha.tar.gz
$ cp nginx.conf.db nginx.conf
$ python remote_scp.py --silent jobs hosts /opt/huststore/hustdbha/conf nginx.conf hustdbtable.json
$ rm -f nginx.conf

Start service (ssh to remote mathines one by one, run script /opt/huststore/hustdbha/sbin/start.sh and /opt/huststore/hustdbsync/start.sh):

$ python remote_service.py --silent jobs hosts /opt/huststore/hustdbha/sbin --start
$ python remote_service.py --silent jobs hosts /opt/huststore/hustdbsync --start

Type in commands:

curl -i -X GET '192.168.1.101:8082/version'
curl -i -X GET '192.168.1.102:8082/version'

We should be able to see the below infomation:

HTTP/1.1 200 OK
Server: nginx/1.12.0
Date: Tue, 18 Apr 2017 10:56:55 GMT
Content-Type: text/plain
Content-Length: 13
Connection: keep-alive

hustdbha 1.7

The result shows that the servers work as expected.

Back to top

Hustmq Cluster Deployment

hustmq

Build and make installation package of hustmq:

$ sh build.sh --module=hustmq

Deploy hustmq (copy installation package to remote machines and untar to /opt/huststore):

$ python remote_deploy.py --silent jobs hosts /opt/huststore elf_hustmq.tar.gz

Start service (ssh to remote mathines one by one, run script /opt/huststore/hustmq/start.sh):

$ python remote_service.py --silent jobs hosts /opt/huststore/hustmq --start

Type in the below command to test:

curl -i -X GET '192.168.1.101:8086/status.html'
curl -i -X GET '192.168.1.102:8086/status.html'

Infomation returned:

HTTP/1.1 200 OK
Content-Length: 3
Content-Type: text/plain

ok

The result shows that the servers work as expected.

Back to top

hustmq ha

Build and make installation package of hustmq ha:

$ sh build.sh --module=hustmqha

Generate configuration of hustmq ha:

$ python make_conf.py hosts hustdbha 8080 8086
$ cp hustmq/ha/nginx/conf/nginx.conf nginx.conf.mq

Deploy hustmq ha (copy installation package to remote machines and untar to /opt/huststore, then replace the configuration):

$ python remote_deploy.py --silent jobs hosts /opt/huststore elf_hustmqha.tar.gz
$ cp nginx.conf.mq nginx.conf
$ python remote_scp.py --silent jobs hosts /opt/huststore/hustmqha/conf nginx.conf
$ rm -f nginx.conf

Start service (ssh to remote mathines one by one, run script /opt/huststore/hustmqha/sbin/start.sh):

$ python remote_service.py --silent jobs hosts /opt/huststore/hustmqha/sbin --start

Input the following test command:

curl -i -X GET '192.168.1.101:8080/version'
curl -i -X GET '192.168.1.102:8080/version'

Then server will output the following information:

HTTP/1.1 200 OK
Server: nginx/1.12.0
Date: Tue, 18 Apr 2017 10:54:47 GMT
Content-Type: text/plain
Content-Length: 13
Connection: keep-alive

hustmqha 1.7

Server works just fine if the above result is returned.

Back to top

Summary

All together:

# ssh to build machine

wget https://github.com/Qihoo360/huststore/archive/v1.7.tar.gz -O huststore-1.7.tar.gz
tar -zxf huststore-1.7.tar.gz
cd huststore-1.7

# edit hosts
# vi hosts
# 192.168.1.101
# 192.168.1.102

# prebuild
sh prebuild.sh --prefix=/opt/huststore
python remote_prefix.py jobs hosts /opt/huststore jobs

# build & make installation packages
sh build.sh
python make_conf.py hosts hustdbha 8082 8085
cp hustdb/ha/nginx/conf/nginx.conf nginx.conf.db
cp hustdb/ha/nginx/conf/hustdbtable.json .
python make_conf.py hosts hustmqha 8080 8086
cp hustmq/ha/nginx/conf/nginx.conf nginx.conf.mq


# deploy 3rd
python remote_deploy.py --silent jobs hosts /opt/huststore elf_3rd.tar.gz

# deploy huststore
python remote_deploy.py --silent jobs hosts /opt/huststore elf_hustdb.tar.gz
python remote_deploy.py --silent jobs hosts /opt/huststore elf_hustmq.tar.gz

python remote_deploy.py --silent jobs hosts /opt/huststore elf_hustdbha.tar.gz
cp nginx.conf.db nginx.conf
python remote_scp.py --silent jobs hosts /opt/huststore/hustdbha/conf nginx.conf hustdbtable.json
rm -f nginx.conf

python remote_deploy.py --silent jobs hosts /opt/huststore elf_hustmqha.tar.gz
cp nginx.conf.mq nginx.conf
python remote_scp.py --silent jobs hosts /opt/huststore/hustmqha/conf nginx.conf
rm -f nginx.conf

# start huststore service
python remote_service.py --silent jobs hosts /opt/huststore/hustdb --start
python remote_service.py --silent jobs hosts /opt/huststore/hustmq --start
python remote_service.py --silent jobs hosts /opt/huststore/hustdbha/sbin --start
python remote_service.py --silent jobs hosts /opt/huststore/hustdbsync --start
python remote_service.py --silent jobs hosts /opt/huststore/hustmqha/sbin --start

Back to top

RPM & binary installation packages

We assume that you've already read all contents of Quickstart & Try and More before using RPM & binary installation packages.

The releases page of huststore provides RPM & binary installation packages built from CentOS 6.2:

# RPM installation packages
hust3rd-1.7-1.el6.x86_64.rpm
hust3rd-1.7-1.el6.x86_64.rpm.md5sum
hust3rd-1.7-1.el6.x86_64.rpm.sha1sum
hustdb-1.7-1.el6.x86_64.rpm
hustdb-1.7-1.el6.x86_64.rpm.md5sum
hustdb-1.7-1.el6.x86_64.rpm.sha1sum
hustdbha-1.7-1.el6.x86_64.rpm
hustdbha-1.7-1.el6.x86_64.rpm.md5sum
hustdbha-1.7-1.el6.x86_64.rpm.sha1sum
hustmq-1.7-1.el6.x86_64.rpm
hustmq-1.7-1.el6.x86_64.rpm.md5sum
hustmq-1.7-1.el6.x86_64.rpm.sha1sum
hustmqha-1.7-1.el6.x86_64.rpm
hustmqha-1.7-1.el6.x86_64.rpm.md5sum
hustmqha-1.7-1.el6.x86_64.rpm.sha1sum

# binary installation packages
elf_3rd.tar.gz
elf_3rd.tar.gz.md5sum
elf_3rd.tar.gz.sha1sum
elf_hustdbha.tar.gz
elf_hustdbha.tar.gz.md5sum
elf_hustdbha.tar.gz.sha1sum
elf_hustdb.tar.gz
elf_hustdb.tar.gz.md5sum
elf_hustdb.tar.gz.sha1sum
elf_hustmqha.tar.gz
elf_hustmqha.tar.gz.md5sum
elf_hustmqha.tar.gz.sha1sum
elf_hustmq.tar.gz
elf_hustmq.tar.gz.md5sum
elf_hustmq.tar.gz.sha1sum

Note:

Do NOT forget to check the packages by *.md5sum and *.sha1sum.
The RPM & binary installation packages is just for quickstart, so /opt/huststore is used as installation folder and could not be changed any more as it is set in build process. So if you need to customize the installation folder, please compile and install from source as what Quickstart & Try and More shows.
If you use RPM installation packages for test, please do not forget to change the owner of folder /opt/huststore as what Quickstart & Try and More shows. Besides, you need to install hust3rd-1.7-1.el6.x86_64.rpm first before installing any other module.
If you use binary installation packages for test, please do not forget to create the folder /opt/huststore and change its owner as what Quickstart & Try and More shows. Besides, you need to install elf_3rd.tar.gz first before installing any other module.

Back to top

Appendix

hustdb
- hustdb configuration
hustdb ha
hustmq
- hustmq configuration
hustmq ha
- hustmq ha configuration
API

Back to top

Home

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

quickstart.md

quickstart.md

Quick Start

Quickstart & Try

More

Tools

prebuild.sh

build.sh

start.sh

stop.sh

remote_scp.py

remote_ssh.py

remote_prefix.py

remote_deploy.py

remote_service.py

make_conf.py

Preparation

Dependency Deployment

Hustdb Cluster Deployment

hustdb

hustdb ha

Hustmq Cluster Deployment

hustmq

hustmq ha

Summary

RPM & binary installation packages

Appendix

Files

quickstart.md

Latest commit

History

quickstart.md

File metadata and controls

Quick Start

Quickstart & Try

More

Tools

prebuild.sh

build.sh

start.sh

stop.sh

remote_scp.py

remote_ssh.py

remote_prefix.py

remote_deploy.py

remote_service.py

make_conf.py

Preparation

Dependency Deployment

Hustdb Cluster Deployment

hustdb

hustdb ha

Hustmq Cluster Deployment

hustmq

hustmq ha

Summary

RPM & binary installation packages

Appendix