From a5451810c7cb7d033389a93e178c32f244323eaf Mon Sep 17 00:00:00 2001 From: wuyi05 Date: Thu, 13 Jul 2017 11:18:26 +0800 Subject: [PATCH 1/2] update readme --- README.md | 77 +++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 49 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index 4ef25bf8..73647eb9 100644 --- a/README.md +++ b/README.md @@ -1,64 +1,75 @@ # PaddlePaddle Cloud -## Using Command-line To Submit Cloud Training Jobs +PaddlePaddle Cloud is a Distributed Deep-Learning Cloud Platform for both cloud +providers and enterprises. + +PaddlePaddle Cloud use [kubernetes](https://kubernetes.io) as it's backend job +dispatching and cluster resource management center. And use [PaddlePaddle](https://github.com/PaddlePaddle/Paddle.git) +as the deep-learning frame work. Users can use web pages or command-line tools +to submit their deep-learning training jobs remotely to make use of power of +large scale GPU clusters. -[English tutorials](./doc/usage_en.md) +## Using Command-line To Submit Cloud Training Jobs [δΈ­ζ–‡ζ‰‹ε†Œ](./doc/usage_cn.md) +English tutorials(comming soon...) + ## Deploy PaddlePaddle Cloud ### Pre-Requirements -- PaddlePaddle Cloud needs python to support `OPENSSL 1.2`. To check it out, simply run: - ```python - >>> import ssl - >>> ssl.OPENSSL_VERSION - 'OpenSSL 1.0.2k 26 Jan 2017' - ``` -- Make sure you have `Python > 2.7.10` installed. +- PaddlePaddle Cloud use kubernetes as it's backend core, deploy kubernetes cluster + using [Sextant](https://github.com/k8sp/sextant) or any tool you like. + ### Run on kubernetes - Build Paddle Cloud Docker Image + ```bash # build docker image git clone https://github.com/PaddlePaddle/cloud.git cd cloud/paddlecloud docker build -t [your_docker_registry]/pcloud . + # push to registry so that we can submit paddlecloud to kubernetes docker push [your_docker_registry]/pcloud ``` -- We use [volume](https://kubernetes.io/docs/concepts/storage/volumes/) to mount MySQL data and cert files, such as CephFS, GlusterFS and etc..., the follow is a example using [hostpath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath): + +- We use [volume](https://kubernetes.io/docs/concepts/storage/volumes/) to mount MySQL data, + cert files and settings, in `k8s/` folder we have some samples for how to mount + stand-alone files and settings using [hostpath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath). Here's + a good tutorial of creating kubernetes certs: https://coreos.com/kubernetes/docs/latest/getting-started.html - create data folder on a Kubernetes node, such as: ```bash mkdir -p /home/pcloud/data/mysql mkdir -p /home/pcloud/data/certs ``` - - Copy Kubernetes CA files (ca.pem, ca-key.pem, ca.srl) to `pcloud_data/certs` folder - - Copy Kubernetes admin user key (admin.pem, admin-key.pem) to `pcloud_data/certs` folder - - Copy CephFS Key file(admin.secret) to `pcloud_data/certs` folder - - Copy `/paddlecloud/settings.py` file to `pcloud_data` folder + - Copy Kubernetes CA files (ca.pem, ca-key.pem, ca.srl) to `/home/pcloud/data/certs` folder + - Copy Kubernetes admin user key (admin.pem, admin-key.pem) to `/home/pcloud/data/certs` folder + - Copy CephFS Key file(admin.secret) to `/home/pcloud/data/certs` folder + - Copy `paddlecloud/settings.py` file to `/home/pcloud/data` folder - Configure `cloud_deployment.yaml` - `spec.template.spec.containers[0].volumes` change the `hostPath` which match your data folder. - `spec.template.spec.nodeSelector.`, edit the value `kubernetes.io/hostname` to host which data folder on.You can use `kubectl get nodes` to list all the Kubernetes nodes. - Configure `settings.py` - Add your domain name to `ALLOWED_HOSTS`. - - Configure `DATACENTERS` to your backend storage. -- Configure `cloud_ingress.yaml` - - `spec.rules[0].host` specify your domain name + - Configure `DATACENTERS` to your backend storage, supports CephFS and HostPath currently. + You can use HostPath mode to make use of shared file-systems like "NFS". +- Configure `cloud_ingress.yaml` is your kubernetes cluster is using "ingress" + to proxy HTTP traffics, or you can configure `cloud_service.yaml` to use [NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport) + - if using ingress, configure `spec.rules[0].host` to your domain name - Deploy cloud on Kubernetes - `kubectl create -f k8s/cloud_deployment.yaml` - `kubectl create -f k8s/cloud_service.yaml` - - `kubectl create -f k8s/cloud_ingress.yaml` + - `kubectl create -f k8s/cloud_ingress.yaml`(optianal) -To test or visit the website, find out the kubernetes [ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) IP addresses, and bind it to your `/etc/hosts` file: -``` -# your ingress IP address -192.168.1.100 cloud.paddlepaddle.org -``` +To test or visit the website, find out the kubernetes [ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) IP +addresses, or the NodePort. -Then open your browser and visit http://cloud.paddlepaddle.org. +Then open your browser and visit http://, or +http://: - Prepare public dataset @@ -66,10 +77,20 @@ Then open your browser and visit http://cloud.paddlepaddle.org. ```bash kubectl create -f k8s/prepare_dataset.yaml ``` - -### Run locally -Make sure you are using a virtual environment of some sort (e.g. `virtualenv` or + +### Run locally without docker + +- You still need a kubernetes cluster when try running locally. +- Make sure you have `Python > 2.7.10` installed. +- Python needs to support `OPENSSL 1.2`. To check it out, simply run: + ```python + >>> import ssl + >>> ssl.OPENSSL_VERSION + 'OpenSSL 1.0.2k 26 Jan 2017' + ``` +- Make sure you are using a virtual environment of some sort (e.g. `virtualenv` or `pyenv`). + ``` virtualenv paddlecloudenv # enable the virtualenv @@ -77,6 +98,7 @@ source paddlecloudenv/bin/activate ``` To run for the first time, you need to: + ``` npm install pip install -r requirements.txt @@ -102,4 +124,3 @@ EMAIL_BACKEND = 'django_sendmail_backend.backends.EmailBackend' You may need to use `hostNetwork` for your pod when using mail command. Or you can use django smtp bindings just refer to https://docs.djangoproject.com/en/1.11/topics/email/ - From dfdee848b2a408902f832cb98cdeab69000db747 Mon Sep 17 00:00:00 2001 From: wuyi05 Date: Thu, 13 Jul 2017 14:17:26 +0800 Subject: [PATCH 2/2] update --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 73647eb9..0b7e93f3 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ PaddlePaddle Cloud is a Distributed Deep-Learning Cloud Platform for both cloud providers and enterprises. -PaddlePaddle Cloud use [kubernetes](https://kubernetes.io) as it's backend job +PaddlePaddle Cloud use [Kubernetes](https://kubernetes.io) as it's backend job dispatching and cluster resource management center. And use [PaddlePaddle](https://github.com/PaddlePaddle/Paddle.git) as the deep-learning frame work. Users can use web pages or command-line tools to submit their deep-learning training jobs remotely to make use of power of @@ -46,7 +46,7 @@ English tutorials(comming soon...) ``` - Copy Kubernetes CA files (ca.pem, ca-key.pem, ca.srl) to `/home/pcloud/data/certs` folder - Copy Kubernetes admin user key (admin.pem, admin-key.pem) to `/home/pcloud/data/certs` folder - - Copy CephFS Key file(admin.secret) to `/home/pcloud/data/certs` folder + - Optianal: copy CephFS Key file(admin.secret) to `/home/pcloud/data/certs` folder - Copy `paddlecloud/settings.py` file to `/home/pcloud/data` folder - Configure `cloud_deployment.yaml` @@ -56,7 +56,7 @@ English tutorials(comming soon...) - Add your domain name to `ALLOWED_HOSTS`. - Configure `DATACENTERS` to your backend storage, supports CephFS and HostPath currently. You can use HostPath mode to make use of shared file-systems like "NFS". -- Configure `cloud_ingress.yaml` is your kubernetes cluster is using "ingress" +- Configure `cloud_ingress.yaml` is your kubernetes cluster is using [ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) to proxy HTTP traffics, or you can configure `cloud_service.yaml` to use [NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport) - if using ingress, configure `spec.rules[0].host` to your domain name - Deploy cloud on Kubernetes @@ -65,7 +65,7 @@ English tutorials(comming soon...) - `kubectl create -f k8s/cloud_ingress.yaml`(optianal) -To test or visit the website, find out the kubernetes [ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) IP +To test or visit the website, find out the kubernetes ingress IP addresses, or the NodePort. Then open your browser and visit http://, or