Skip to content

Latest commit

 

History

History
264 lines (195 loc) · 10.9 KB

DEVELOPMENT.md

File metadata and controls

264 lines (195 loc) · 10.9 KB
Raw

#Environments

##Cloudbreak Application

Use these steps to set up your local development environment, IDEA, command line, UI and database development, and builds for the Cloudbreak application.

Local Development Environment Setup

Before setting up your development environment on OSX, you must install Docker and Boot2docker.

The simplest way to prepare your working environment is to start Cloudbreak on your local machine using the Cloudbreak Deployer. The latest version of the Cloudbreak Deployer is available in this github repository.

Step 1: Create a sandbox directory which will store the necessary configuration files and dependencies of the Cloudbreak Deployer. This directory must be created outside of your cloned Cloudbreak git repository. For example:

mkdir cbd-local

Step 2: Start the entire Cloudbreak ecosystem on your machine by executing the following sequence of commands:

cd cbd-local   
curl -L s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/cloudbreak/cloudbreak-deployer_snapshot_$(uname)_x86_64.tgz | tar -xz -C /usr/local/bin
cbd init  
cbd start

Step 3: If Cloudbreak starts correctly, Cloudbreak application will be available at https://192.168.59.103. For more details and config parameters, check the documentation for Cloudbreak Deployer.

Step 4: Note that the deployer has generated a certs directory under cbd-local directory. You will need it later to set up the IDEA.

Step 5: Next, edit the cbd-local/Profile file, adding the CB_SCHEMA_SCRIPTS_LOCATION environment variable which configures the location of SQL scripts that are in the 'core/src/main/resources/schema' directory in the cloned Cloudbreak git repository. Note that the full path needs to be configured and env variables such as $USER cannot be used.

export CB_SCHEMA_SCRIPTS_LOCATION=/Users/myusername/prj/cloudbreak/core/src/main/resources/schema

Step 6: Use the following command to kill a Cloudbreak container running inside the docker machine and to redirect the Cloudbreak-related traffic to the Cloudbreak running in the IDEA:

cbd util local-dev

IDEA Setup

Step 1: Import Cloudbreak into IDEA as a gradle project.

Step 2:: Import the proper code formatter. To do this, use the File -> Import Settings... menu and select the idea_settings.jar located in the config directory in the Cloudbreak github repository.

Step 3: To launch the Cloudbreak application, execute the com.sequenceiq.cloudbreak.CloudbreakApplication class with VM options:

-XX:MaxPermSize=1024m
-Dcb.cert.dir=FULL_PATH_OF_THE_CERTS_DIR_GENERATED_BY_CBD
-Dcb.client.id=cloudbreak
-Dcb.client.secret=cbsecret2015
-Dcb.db.port.5432.tcp.addr=192.168.59.103
-Dcb.db.port.5432.tcp.port=5432
-Dspring.cloud.consul.host=192.168.59.103
-Dcb.identity.server.url=http://192.168.59.103:8089
-Dserver.port=9091

The -Dcb.cert.dir=FULL_PATH_OF_THE_CERTS_DIR_GENERATED_BY_CBD value above needs to be replaced with the full path of certs directory generated by Cloudbreak Deployer. For example: -Dcb.cert.dir=/Users/myusername/prj/cbd-local/certs

Step 4: To launch the Periscope (Autoscaling) application execute the com.sequenceiq.periscope.PeriscopeApplication class with VM options:

-XX:MaxPermSize=1024m
-Dperiscope.client.id=periscope
-Dperiscope.client.secret=cbsecret2015
-Dperiscope.identity.server.url=http://192.168.59.103:8089
-Dperiscope.db.tcp.addr=192.168.59.103
-Dperiscope.db.tcp.port=5432
-Dperiscope.cloudbreak.url=http://192.168.59.103:8080
-Dserver.port=8085
-Dperiscope.cert.dir=FULL_PATH_OF_THE_CERTS_DIR_GENERATED_BY_CBD

Local Development Setup for UI

By default, the Cloudbreak UI is a service called 'Uluwatu' that runs inside a docker container. The UI files (JS and CSS) are installed as part of the docker container. To keep the changes between the UI and backend application in sync, you may want to load the UI files from the local environment. To do this, perform the following steps:

Step 1: Install npm and bower locally. On Mac OSX, you can do this by running:

brew install npm
npm install -g bower

Step 2: Install the dependencies of the UI project locally:

cd $CLOUDBREAK_HOME/cloudbreak/web && npm install
cd $CLOUDBREAK_HOME/cloudbreak/web/app/static && bower install

where $CLOUDBREAK_HOME is the directory where the Cloudbreak project is cloned. This step may take a while depending on your network speed.

Step 3: Add the following line to the Profile file:

export ULUWATU_VOLUME_HOST=$CLOUDBREAK_HOME/cloudbreak/web

Step 4: Restart Cloudbreak and switch to the local environment, as described above

cbd restart
cbd util local-dev

The export line causes the specified path to be loaded as a volume inside the docker container running the web UI process. Refresh the browser cache to make sure the files are reloaded from this location.

Command Line Setup

To run Cloudbreak from command line:

Step 1: Create a property file, for example application.properties, with the content below:

cb.cert.dir=FULL_PATH_OF_THE_CERTS_DIR_GENERATED_BY_CBD
cb.client.id=cloudbreak
cb.client.secret=cbsecret2015
cb.db.port.5432.tcp.addr=192.168.59.103
cb.db.port.5432.tcp.port=5432
spring.cloud.consul.host=192.168.59.103
cb.identity.server.url=http://192.168.59.103:8089
server.port=9091

Step 2: Execute java -jar -XX:MaxPermSize=1024m -Dspring.config.location=file:///path/of/property/application.properties core/build/libs/cloudbreak.jar command at project root.

Note that the path of application.properties must be an absolute path.

Database Development

If schema change is required in Cloudbreak database (cbdb), then the developer needs to write SQL scripts to migrate the database accordingly. In Cloudbreak, the schema migration is managed by MYBATIS Migrations and the cbd tool provides an easy-to-use wrapper for it.

The basic syntax for the migration commands is cbd migrate <database name> <command> [parameters]

For example: cbd migrate migrate status

Step 1: Create a SQL template for schema changes:

cbd migrate cbdb new "CLOUD-123 schema change for new feature"

As as result of the above command, an SQL file template is generated under the path specified in CB_SCHEMA_SCRIPTS_LOCATION environment variable, which is defined in Profile. The structure of the generated SQL template looks like this:

-- // CLOUD-123 schema change for new feature
-- Migration SQL that makes the change goes here.



-- //@UNDO
-- SQL to undo the change goes here.

Step 2: Once you have implemented your SQLs, you can execute them with:

cbd migrate cbdb up

If you would like to rollback the last SQL file, use the down command:

cbd migrate cbdb down

To check the database status, use:

cbd migrate cbdb status

#Every script that has not been executed will be marked as ...pending... in the output of status command:

------------------------------------------------------------------------
-- MyBatis Migrations - status
------------------------------------------------------------------------
ID             Applied At          Description
================================================================================
20150421140021 2015-07-08 10:04:28 create changelog
20150421150000 2015-07-08 10:04:28 CLOUD-607 create baseline schema
20150507121756 2015-07-08 10:04:28 CLOUD-576 change instancegrouptype hostgroup to core
20151008090632    ...pending...    CLOUD-123 schema change for new feature

------------------------------------------------------------------------

Building

Gradle is used for build and dependency management. Since Gradle wrapper is added to the Cloudbreak git repository, building can be done with:

./gradlew clean build

##Cloudbreak Deployer

###Contributing

Development process happens on separate branches. Open a pull-request to contribute your changes.

To build the project:

# make deps needed only once
make deps

make install

To run the unit tests:

make tests

If you want to test the binary CircleCI build from your branch named fix-something, to validate the PR binary cbd tool will be tested. It is built by CircleCI for each branch.

branch=fix-something
circle_url=https://circleci.com/api/v1/project/hortonworks/cloudbreak-deployer
circle_token=[CIRCLE_TOKEN]
latest_build=$(curl -s -G -d circle-token=${circle_token} ${circle_url}/tree/${branch}?filter=completed\&limit=1 | grep -m 1 build_num | sed 's/[^0-9]*//g')
curl -sL $(curl -s -G -d circle-token=${circle_token} ${circle_url}/${latest_build}/artifacts | grep url | grep -i $(uname) | cut -d\" -f 4) | tar -xz -C $(dirname $(which cbd))

Snapshots

We recommend that you always use the latest release, but you might also want to check new features or bugfixes in the master branch. All successful builds from the master branch are uploaded to the public S3 bucket. You can download it using:

curl -L s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/cloudbreak/cloudbreak-deployer_snapshot_$(uname)_x86_64.tgz | tar -xz

Instead of overwriting the released version, download it to a local directory and use it by referring to it as ./cbd

./cbd --version

Testing

Shell scripts shouldn’t raise exceptions when it comes to unit testing. basht is used for testing. See this link to learn why not bats or shunit2.

You must cover your bash functions with unit tests and run the test with:

make tests

Release Process for the Clodbreak Deployer Tool

The master branch is always built on CircleCI.

When you want to create a new release, run:

make release-next-ver

The make release-next-ver performs the following steps:

  • On the master branch:
    • Updates the VERSION file by increasing the patch version number (For example, from 0.5.2 to 0.5.3).
    • Updates CHANGELOG.md with the release date.
    • Creates a new Unreleased section at the top of CHANGELOG.md.
  • Creates a PullRequest for the release branch:
    • Creates a new branch with a name like release-0.5.x.
    • This branch should be the same as origin/master.
    • Creates a pull request on the release branch.

Acceptance

Now you should test this release. You can update to it by running curl -L -s https://github.com/hortonworks/cloudbreak-deployer/archive/release-x.y.z.tar.gz | tar -xz -C $(dirname $(which cbd)). Comment with LGTM (Looking Good To Me).

Once the PR is merged, CircleCI will: