Home
English | δΈζ
Derrick is a tool to help you dockerizing application in seconds. Derrick focus on the developer's workflow in local development environment. Derrick will inspect your workspace and generate Dockerfile, docker-compose.yml, Jenkinsfile, etc. for you. You can simply use Derrick to stream line your DevOps process in container way smoothly.
Linux & Mac
pip install python-derrick
# Download NodeJs demo repo
git clone git@github.com:ringtail/derrick-nodejs-demo.git
# Enter project folder
cd derrick-nodejs-demo
# Run Derrick
derrick init
# Docker Image Build
derrick up
The results of the first execution of derrick init are as follows
8888888b. d8b 888
888 "Y88b Y8P 888
888 888 888
888 888 .d88b. 888d888888d888888 .d8888b888 888
888 888d8P Y8b888P" 888P" 888d88P" 888 .88P
888 88888888888888 888 888888 888888K
888 .d88PY8b. 888 888 888Y88b. 888 "88b
8888888P" "Y8888 888 888 888 "Y8888P888 888
===================================================
Derrick is a scaffold tool to migrate applications
You can use Derrick to migrate your project simply.
===================================================
This is the first time to run Derrick.
Successfully create DERRICK_HOME in /Users/zhongweilzw/.derrick
Derrick detect your platform is NodeJs and compile successfully.
In the root directory of the project, Dockerfile, docker-compose.yml and other files have been generated. Then the results of the execution of derrick build are as follows.
Building web
Step 1/14 : FROM registry.cn-beijing.aliyuncs.com/codepipeline/node:6 AS base
---> c0cea7b613ca
Step 2/14 : WORKDIR /app
---> Using cache
---> c8c07dfd8dc9
Step 3/14 : COPY package.json .
---> Using cache
---> 055581ce2b5c
Step 4/14 : RUN npm set progress=false && npm config set depth 0
---> Using cache
---> 623f6b32520b
Step 5/14 : RUN npm install --only=production --registry=https://registry.npm.taobao.org
---> Using cache
---> f69ea9289d5c
Step 6/14 : RUN cp -R node_modules prod_node_modules
---> Using cache
---> 53ac8e86e88b
Step 7/14 : RUN npm install --registry=https://registry.npm.taobao.org
---> Using cache
---> d75ad55f841b
Step 8/14 : FROM base As test
---> d75ad55f841b
Step 9/14 : COPY . /app
---> Using cache
---> 5eb0247a9f71
Step 10/14 : RUN npm test
---> Using cache
---> 765536e52fc6
Step 11/14 : FROM base AS release
---> d75ad55f841b
Step 12/14 : COPY --from=base /app/prod_node_modules /app/node_modules
---> c1f1db7c2dc9
Step 13/14 : COPY . /app
---> 6c340f7d54ee
Step 14/14 : CMD npm start
---> Running in 3bec7ae892b4
---> 9ab9c384c029
Removing intermediate container 3bec7ae892b4
Successfully built 9ab9c384c029
Successfully tagged registry.cn-beijing.aliyuncs.com/ringtail/nodejs-demo:latest
WARNING: Image for service web was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating nodejsdemo_web_1 ...
Creating nodejsdemo_web_1 ... done
Your application has been up to running! You can run `docker ps` to get exposed ports.
then you can begin local validation and integration of subsequent processes.
pip install --target=~/.derrick/rigging rigging_package_name
# You can also copy custom Rigging to ~/.derrick/rigging
After installing a custom Rigging, Derrick will load custom Rigging and built-in Rigging, if there are more than two Rigging at the same time to meet the detection conditions, then Derrick will prompt a list to choose which one would you like to use to dockerize your application.
(venv)β nodejs-demo git:(master) derrick init
More than one rigging can handle the application.
? Which rigging would you like to choose? (Use arrow keys)
β― [R] NodejsRigging [P] NodeJs
[R] NodejsDevOpsRigging [P] NodeJs
Running follow command
python setup.py develop
Learn about how Derrick converts code into Dockerfile and other files will help you better hack Derrick. The realization of the principle of Derrick is mainly from CloudFoundry buildpack, if you want to further understand the principle of buildpack, developers can view this article .Here, we will briefly introduce the core concepts of buildpack. We can divide every running application into three abstracted necessary factors: code ,application configuration, application dependency (basic environment, operation dependency).The problem that buildpack has to solve is how to change code through some processes into a running application. CloudFoundry buildpack is a collection of scripts, usually a buildpack will contain three scripts, detect, compile and release. when the code is submitted to the CloudFoundry, three scripts will be executed in turn, and then try to ship code into a running service.
The main purpose of detect script is to detect the current buildpack will be able to handle your code, if the detection is successful and compile script will be executed to convert the detection to dockerfile or someting else. Common detection method includes configuration detection, custom configuration detection etc..
For example, a NodeJs application detections will contains package.json detection,language version detection, start command detection and system dependencies detection and so on.
When the compile script has been executed, the release script will be invoked, and the purpose of the script is to describe how to start the current application, similar to the startup script. Through these three commands, CloudFoundry attempts to convert code into running service.
Through the introduction above, it is not difficult to find that buildpack has some limitations, because many characteristic information including configuration, system dependencies and so on can not be obtained by means of detection. Indeed, buildpack can only solve 80% of the scenarios, so CloudFoundry as a PaaS platform will predefine some project templates to guarantee the Success rate. CloudFoundry also support custom buildpack, allowing developers to solve their own characteristic demand through this way.
So how does Derrick draw lessons from the buildpack mechanism? The goal of CloudFoundry is to generate a running service, and the goal of Derrick is to generate a dockerfile of running service, so what Derrick needs is detect and compile script in buildpack but release script is no more needed. Therefore, Derrick will first detect whether the current language or framework can be processed by detect script. If it can be processed, then compile script will be called and Dockerfile will be generated.
There are some core concepts in Derrick and learn about their specific meanings will helps us to further understand the principles of Derrick implementation.
Derrick means "A kind of huge lifting machine in dock", Derrick is used to lift containers on the dock. And the rope binding container is Rigging. Lifting different container requires different Rigging. Similarly Derrick detect, compile different languages and frameworks requires different Rigging. Therefore, the role of converting different languages or platforms in Derrick is Rigging. For example, there is a NodeJs Rigging for NodeJs platform, MavenRigging for Maven platform. Rigging is an implementation of buildpack in Derrick.
The Command is the meaning as it's name in Derrick.There are two built-in commands(init and build). The init command will execute the whole process of a Rigging, the build command will build the docker image according to the generated Dockerfile.
Detector is the meaning of it's name, Rigging has a large part of the function is to detect the source code structure, configuration, and so on. In order to make the detection process can be reused and easy, we put forward the concept of Detector, developers can use the defined Detector or custom Detector to develop their custom Rigging much more simply.
Before developing a custom Rigging, let's look at how the Derrick init command uses the specific Rigging to generate the relevant information. Rigging is divided into two categories, one is built-in Rigging, and the other is custom Rigging. Derrick allows developers to define custom Rigging and install custom Rigging like a plugin. Derrick loads Rigging is conducted through RiggingManager. RiggingManager handles loading, operation and management of all Rigging life cycle.
Derrick init command will get all the registration Rigging in RiggingManager at execution time and calls the Rigging detect method to determine whether the Rigging can handle the current language or platform in turn. if the Rigging can handle ,it will be marked. when the Rigging traversal is completed, if the number of Rigging marked are more than one. Then developers can choose which Rigging to be called for subsequent flow. If there is only one Rigging, then the compile method of current Rigging will be called directly. If no Rigging is marked, then the current language and platform cannot be processed.
Rigging is also a standard Python module that can be installed through pip, so the standard structure of a Rigging is as follows:
βββ nodejs_rigging
βββ __init__.py
βββ nodejs_rigging.py
βββ templates
βββ Dockerfile.j2
βββ .dockerignore
The contents in nodejs_rigging.py are as follows
#! /usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function
import os
from derrick.core.detector_report import DetectorReport
from derrick.core.rigging import Rigging
from derrick.detectors.image.node import NodeVersionDetector
PLATFORM = "NodeJs"
class NodejsRigging(Rigging):
def detect(self, context):
"""
:param context:
:return: handled(bool),platform(string)
"""
workspace = context.get("WORKSPACE")
package_json_file = os.path.join(workspace, "package.json")
if os.path.exists(package_json_file) is True:
return True, PLATFORM
return False, None
def compile(self, context):
dr = DetectorReport()
docker_node = dr.create_node("Dockerfile.j2")
docker_node.register_detector(NodeVersionDetector())
return dr.generate_report()
First of all, Rigging must inherit from derrick.core.rigging.Rigging or its subclasses, each Rigging needs to implement two methods, one is the detect method, and the other is the compile method.
The results returned by the detect method is handled (bool) and platform (string), The handled's value means that the current Rigging is able to detect the language or framework. The platform's value means the content detected by Rigging. And the content will be chosen and showed after the command finished.
The compile method's core goal is to detect and generate the corresponding config files. in other words, the core goal of the compile is to detect the result of system, the corresponding template assembly action. while in Derrick,A Rigging only need to generate a template file and a variables Python dict.Then the final rendering action will be completed by Derrick.the default template engine in Derrick is jinja2, if there is a non jinja2 file, Derrick will just copy it without variable rendering.Here is an example of a NodeJs application's Dockerfile rendered.
Dockerfile.j2 content
# stage1
FROM {{ version }} AS base
WORKDIR /app
COPY package.json .
RUN npm set progress=false && npm config set depth 0
RUN npm install --only=production
RUN cp -R node_modules prod_node_modules
RUN npm install
# stage 2
FROM base As test
COPY . /app
RUN npm test
# stage 3
FROM base AS release
COPY --from=base /app/prod_node_modules /app/node_modules
COPY . /app
CMD ["npm","start"]
For example, in this case, the goal of compile function is to generate a version variable required in Dockerfile.j2.In order to simplify developer development in Derrick, compile function only needs to return the configuration information of the template, and Derrick will do the remaining rendering of the template. For example, in this case, compile function eventually only needs to return a Python dict.
{
"Dockerfile.j2":{
"version":"node:6"
}
}
So,What is the role of DetectorReport above. Why not just use simple Python dict object but rather define DetectorReport to generate Python dict? The purpose of DetectorReport is to simplify the development. Sometimes what we need is not only the image version. so we should have some simple methods to handle complex scenarios such as override the image version.In the actual scene we need to detect much more information relying on a lot of detectors. so how to organize the dict generation become more complicated. In addition, if you want to reuse some Rigging as your base Rigging to implement. We shoud also consider about how to resue the results detected by base Rigging. And DetectorReport is born for them. DetectorReport is actually a tree structure definition for parsing and anti parsing of dict.
dr = DetectorReport()
docker_node = dr.create_node("Dockerfile.j2") docker_node.register_detector(NodeVersionDetector())
return dr.generate_report()
Finally, let's look at how NodeVersionDetector detects the image version.
#! /usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function
import subprocess
from derrick.core.detector import Detector
from derrick.core.logger import Logger
NODEJS_4 = "node:4"
NODEJS_6 = "node:6"
NODEJS_8 = "node:8"
NODEJS_LATEST = "node:latest"
class NodeVersionDetector(Detector):
def execute(self):
output = subprocess.check_output(["node", "--version"], shell=False)
version = NodeVersionDetector.get_most_relative_version(output)
return {"version": version}
@staticmethod
def get_most_relative_version(version):
version_num = str(version)[1:]
version_arr = version_num.split(".")
detect_version = NODEJS_LATEST
try:
base_version = version_arr[0]
if base_version == "4":
detect_version = NODEJS_4
if base_version == "6":
detect_version = NODEJS_6
if base_version == "8":
detect_version = NODEJS_8
except Exception as e:
Logger.debug("system version is %s,error message is %s" % (version, e.message))
return detect_version
NodeVersionDetector implementation of the execute method of Detector and returns a dict. And this dict will eventually automatically extend to the corresponding node when the DetectorReport traversal generates dict.
For example,if we want to develop a nodejs_devops_rigging based on a built-in NodejsRigging, we can simply extend the existing Rigging and you can also extend derrick.core.rigging.Rigging directly. In this case nodejs_devops_rigging is extended from NodejsRigging.
βββ nodejs_devops_rigging
βββ __init__.py
βββ nodejs_devops_rigging.py
βββ templates
βββ Dockerfile.j2
βββ Jenkinsfile
βββ docker-compose.yml.j2
Create a directory structure from above and here is the following content in nodejs_devops_rigging.py.
#! /usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function
from derrick.core.detector import Detector
from derrick.core.detector_report import DetectorReport
from derrick.rigging.nodejs_rigging.nodejs_rigging import NodejsRigging
from whaaaaat import prompt
class NodejsDevOpsRigging(NodejsRigging):
def compile(self, context):
report = super(NodejsDevOpsRigging, self).compile(context)
dr = DetectorReport()
dr.parse_report(report)
dockerfile_node = dr.create_node("docker-compose.yml.j2")
dockerfile_node.register_detector(DockerComposeDetector(), context)
print(dr.generate_report())
return dr.generate_report()
class DockerComposeDetector(Detector):
def execute(self, *args, **kwargs):
questions = [
{
'type': 'input',
'name': 'image',
'message': 'What\'s your image repo ',
},
{
'type': 'input',
'name': 'ports',
'message': 'What\'s your application ports',
}
]
answers = prompt(questions)
return answers
NodejsDevOpsRigging is extended from NodejsRigging and reuse the detect function and override the compile function .
def compile(self, context):
report = super(NodejsDevOpsRigging, self).compile(context)
dr = DetectorReport()
dr.parse_report(report)
dockerfile_node = dr.create_node("docker-compose.yml.j2")
dockerfile_node.register_detector(DockerComposeDetector(), context)
return dr.generate_report()
Here is the content of DockerComposeDetector below.
class DockerComposeDetector(Detector):
def execute(self, *args, **kwargs):
questions = [
{
'type': 'input',
'name': 'image',
'message': 'What\'s your image repo ',
},
{
'type': 'input',
'name': 'ports',
'message': 'What\'s your application ports',
}
]
# ιθΏδΊ€δΊεΌηζΉεΌε―δ»₯θ·εΎζ’ζ΅ηη»ζ
answers = prompt(questions)
return answers
Such a NodejsRigging based on the built-in NodejsRigging is compeled.
Then copy this folder to ~/.derrick/rigging. And run Derrick init
(venv)β nodejs-demo git:(master) derrick init
More than one rigging can handle the application.
? Which rigging would you like to choose? (Use arrow keys)
β― [R] NodejsRigging [P] NodeJs
[R] NodejsDevOpsRigging [P] NodeJs
select option by directional key and have fun.
on going