command-line interface redesign #291

Yancey1989 · 2017-08-10T03:31:52Z

Fixed #285

gongweibao · 2017-08-10T03:34:29Z

doc/client_design.md

+`pfs [command] [path1|path2..]`
+
+- `ls`: `pfs ls <path>` will list all folders and files under the path.
+- `put`: `pfs put <src> <desc>` will upload the local folder or file to the cloud path.


不能用put，还是得用copy。src dest 将来可能都是云端路径。

感觉put和get会更方便理解一点。cp可以只用做云端路径之间的拷贝。和ftp的命令类似。

同感，感觉后续可以同时支持cp和put, 用于云端拷贝和上传文件。

typhoonzero · 2017-08-10T04:27:12Z

doc/client_design.md

+`pfs [command] [path1|path2..]`
+
+- `ls`: `pfs ls <path>` will list all folders and files under the path.
+- `put`: `pfs put <src> <desc>` will upload the local folder or file to the cloud path.


感觉put和get会更方便理解一点。cp可以只用做云端路径之间的拷贝。和ftp的命令类似。

typhoonzero · 2017-08-10T04:32:55Z

doc/client_design.md

+  - `paddle submit job -name <job-name> -cpu 1 ...`
+  - `paddle get [job|worker|quota]`
+  - `paddle describe job <job name>`, `paddle describe worker <worker name>`
+  - `paddle delete job <job name>`


Should be paddle delete <job name>

Why use delete instead of kill? It seems there are no other things to delete?

只是觉得Kill一个 Resource，貌似不是很容易被理解，不过确实PaddleCloud上没有其他Resource是需要被kill的，先改回kill了。

typhoonzero · 2017-08-10T04:38:13Z

doc/client_design.md

+  - `name`: You can specify a name for the given resource
+  - `flags`: You can specify any flags override default values for `command` or `resource`
+
+- Command


Command => Sub-Commond

wanghaoshuang

来自忠实用户的建议。

wanghaoshuang · 2017-08-10T04:34:35Z

doc/client_design.md

+  - `submit`: submit the specify resource, we only support submit a PaddleJob for now
+  - `get`: get all instances for the specified resource
+  - `describe`: output the description for the specify resource
+  - `delete`: delete the specific resource, we only support delete a PaddleJob for now


原来的kill是不是更简短好用？

Done. delete => kill

wanghaoshuang · 2017-08-10T04:35:32Z

doc/client_design.md

+- Command
+  - `submit`: submit the specify resource, we only support submit a PaddleJob for now
+  - `get`: get all instances for the specified resource
+  - `describe`: output the description for the specify resource


感觉describe 有点长...用info合适么？

可是info是一个名词的缩写，貌似不太符合我们设想的语法规范。

wanghaoshuang · 2017-08-10T04:38:15Z

doc/client_design.md

+
+- Command
+  - `submit`: submit the specify resource, we only support submit a PaddleJob for now
+  - `get`: get all instances for the specified resource


get可不可以有默认选项，比如paddlecloud get == paddlecloud get jobs ?

可是怎么确定哪个Resource是默认选项呢？worker还是job

wanghaoshuang · 2017-08-10T04:39:33Z

doc/client_design.md

-## Reference
+## PaddleCloud command-line design
+You can use the following syntax to run `paddlecloud` commands from your terminal window:
+`paddlecloud [command] [resource] [name] [flags]`


我一般会把paddlecloud alias 成pcloud...

paddlecloud 确实比较长，我同意改成pcloud. +1

@Yancey1989 是不是需要改一下paddlecloud代码目录和改成pcloud，以及确认下发布的binary以后都是pcloud了。
Edit：哦，这个是design doc，可以等merge了之后再改。

wanghaoshuang · 2017-08-10T04:48:29Z

doc/client_design.md

+  - `quota`, The quota for the current user
+
+- Example
+  - `paddle submit job -name <job-name> -cpu 1 ...`


paddle->paddlecloud?

typhoonzero · 2017-08-11T07:08:48Z

doc/client_design.md

+- `ls`: will list all folders and files under the path.
+- `put`: will upload the local folder or file to the cloud path.
+- `get`: will download the foler or file to the local path.
+- `rm`: will remove the specify folder or file.


need cp command

typhoonzero · 2017-08-11T07:09:54Z

doc/client_design.md

+
+- Example
+  - `pcloud submit job -name <job-name> -cpu 1 ...`
+  - `pcloud get [job|worker|quota]`


I remember as discussed yesterday, pcloud list is more simple than pcloud get jobs, and pcloud trainers will get pods running trainers, pcloud pservers will get pserver pods.

Just to discuss, which will be better for users?

And more, can we change the name "workers" to "trainers", and add "pservers" commnads?

If we want to separate worker into trainer and pserver, we also need master.But i think using workers to express pserver, trainer and master is good for simplified users' operation, users can use a single command to view all the workers state:

pcloud workers ...

rather than three commands:

pcloud trainers ... pcloud servers ... pcloud master ...

typhoonzero · 2017-08-11T07:19:23Z

doc/client_design.md

+  - `pcloud get [job|worker|quota]`
+  - `pcloud describe job <job name>`, `paddle describe worker <worker name>`
+  - `pcloud kill <job name>`
+  - `pcloud logs job <job name>`, `paddle logs worker <worker name>`


For simplicity, pcloud logs <trainer or pserver name>, logs command will only used for getting logs for workers since logs are stored for each "pod".

If we want to aggregate logs, add a new command like pcloud logaggr <job name>

Agree with @typhoonzero 's comment!

helinwang · 2017-08-11T18:35:47Z

doc/client_design.md

+- `ls`: will list all folders and files under the path.
+- `put`: will upload the local folder or file to the cloud path.
+- `get`: will download the foler or file to the local path.
+- `rm`: will remove the specify folder or file.


Need mv command.

helinwang · 2017-08-11T18:36:38Z

doc/client_design.md

+  - `quota`, The quota for the current user
+
+- Example
+  - `pcloud submit job -name <job-name> -cpu 1 ...`


我觉得pcloud最核心的是两件事：
job和pfs。
pfs我觉得挺好的（这些命令经常敲，写成pcloud pfs xx可能太长）：

pcloud ls pcloud put pcloud get pcloud rm pcloud cp pcloud mv

job相关的是否分得太开了：

pcloud get job pcloud describe job pcloud submit job pcloud kill pcloud logs

能不能统一到一个subcommand里面：

pcloud job help pcloud job list pcloud job submit pcloud job kill pcloud job log pcloud job info

剩下的还有：

pcloud resource worker # 这个需要吗，用户会不会在意每个worker的属性（比如IP是多少） pcloud resource quota # 另一种方式是pcloud quota pcloud describe worker # 这个是否跟pcloud resource worker重复 pcloud describe quota # 是否跟pcloud resource quota重复 pcloud get quota # 是否跟pcloud resource quota重复 pcloud get worker # 用户是否在意？ pcloud logs worker # 用户是否在意？跟job log有重复的地方

pcloud get worker # 用户是否在意？

用户可以用这个命令看到trainer/pserer/master进程的状态，应该也是需要的。

pcloud logs worker # 用户是否在意？跟job log有重复的地方

现在pcloud logs job确实有些不合适的地方：

worker很多的时候日志聚合性能很差（需要在Server端对每个Pod调用一次API）

多个worker日志聚合一起看体验也不是很有好, 可以通过pcloud worker list得到worker的列表，根据需要查看具体worker的日志pcloud logs <worker-name>

赞思考！

job相关的是否分得太开了：
...
能不能统一到一个subcommand里面：
pcloud job help
pcloud job list
pcloud job submit
pcloud job kill
pcloud job log
pcloud job info

刚和 @typhoonzero 也线下讨论了一波，感觉是不是这样也会太长了，毕竟resource比较少，有些像kill/submit等也只会针对job，是不是可以改成:

Job Operation

pcloud jobs # list job pcloud kill <job-name> # kill job pcloud jobs -v <job-name> # job info

Pod Operation (获取直接用Kubernetes中的Pod来表示正在运行实例会比较方便，不需要引入Paddle的Worker概念了，用Pod的名字来区分是trainer/pserver/master进程)

pcloud pods # list all pod, for the current user pcloud logs <pod-name># logs for pod pcloud pods -v <pod-name> # info for pod

PFS的部分的确简化一些会比较方便，我也赞同简短一些，多谢 @helinwang .

@Yancey1989 赞！LGTM！能否更新下这个PR，然大家都reivew一下？

当然，我马上更新下，多谢 @helinwang

gongweibao · 2017-08-17T12:14:27Z

doc/client_design.md

+- `rm <dest>`: remove the specify folder or file on cloud.
+- `mv <src> <dest>`: move a folder or a file from `<src>` 
+to `<dest>`, `<src>` and `<dest>` only support cloud path.
+- `cp <src> <dest`: copy a folder or a file from `<src>` to `<dest>`, `<src>`


cp这个命令有点复杂，对于比较大的文件copy的过程估计会很长，超时和用户的干预(如ctrl + c)是一个问题。
我的基本的想法是cp变成一个job提交的到k8s，然后提示用户这个cp的任务的名字，让用户当做job来管理。

另一个方法是不支持cp，只支持mv？从ceph mv到同一个ceph文件系统应该很快？

mv是元数据的处理应该很快，需要测试一下。
前两天星海还给我们提需求，支持server端的copy。没有这个可能会让用户感到非常的意外。
我觉得还是需要支持的。

也有同感可能cp没有必要：如果是需要对这个数据预处理，生成一份新的数据，则需要数据预处理的job管理的功能；如果是移动文件位置的确mv足够了。

cp一份数据，这样存在多份相同的数据的需求的原始原因是什么呢？

我的基本的想法是cp变成一个job提交的到k8s，然后提示用户这个cp的任务的名字，让用户当做job来管理。

启动一个Job来copy文件会不会过于复杂了，可不可以这样：

Server端收到copy的请求后创建一个GoRoutine来copy文件，同时在目录下touch一个文件并记录copy的状态（RUNNING,DONE,FAILED)

客户端调用API轮训的去check文件的状态

也有同感可能cp没有必要：如果是需要对这个数据预处理，生成一份新的数据，则需要数据预处理的job管理的功能；如果是移动文件位置的确mv足够了。
cp一份数据，这样存在多份相同的数据的需求的原始原因是什么呢？

机房之间的数据拷贝。我们以前使用的是HDFS有一个RemoteCopy的命令，把机房1的数据拷贝到机房2上。同机房的用的很少。
这个的实现看上去不紧急。

@typhoonzero 多个人同时使用一份数据做实验，如果不做拷贝的话，是件挺危险的事情，其中一个人对数据做了更改，将无意中影响其他人，虽说场景有限，但并非无用；如果权限管理完善，限制只读操作，倒是一定程度上避免这种问题

@pkuyym 明白了。我理解是否有一个chmod命令可以更改成只读权限能否满足呢？

感谢@pkuyym提供的使用场景。
我们现在是有一个Home folder，一个公共folder，看来是会需要一个team folder。能否只有上传人有写的权限，team其他人只有读的权限。

我先移除了CP命令，我们需要增加一个team folder, 相关的讨论见： #325 , #328 。

typhoonzero · 2017-08-22T03:03:19Z

doc/client_design.md

+`pcloud pfs [ls|put|get|rm...] [path1|path2...]`
+
+- `ls <dest>`: list all folders and files under the destination path.
+- `put <src> <dest>`: upload the local folder or file to the cloud path.


Should have multiple <src>, for put get mv

helinwang · 2017-08-22T05:00:34Z

Looking good!

typhoonzero

LGTM

command-line interface redesign

223836c

Yancey1989 requested review from helinwang, gongweibao and typhoonzero August 10, 2017 03:31

gongweibao reviewed Aug 10, 2017

View reviewed changes

typhoonzero reviewed Aug 10, 2017

View reviewed changes

wanghaoshuang reviewed Aug 10, 2017

View reviewed changes

Yancey1989 added 3 commits August 10, 2017 14:24

update

19404b4

update

cf9ae71

update

c7eb469

typhoonzero reviewed Aug 11, 2017

View reviewed changes

helinwang reviewed Aug 11, 2017

View reviewed changes

helinwang mentioned this pull request Aug 15, 2017

Command-line parameters should be improved #285

Open

Yancey1989 mentioned this pull request Aug 16, 2017

Curious what is the meaning of the following paddlecloud logs output? #317

Open

update

6d782b6

gongweibao reviewed Aug 17, 2017

View reviewed changes

gongweibao requested a review from xinghai-sun August 18, 2017 02:00

Yancey1989 mentioned this pull request Aug 18, 2017

Does COPY command is necessary? #325

Open

update

10d4422

typhoonzero reviewed Aug 22, 2017

View reviewed changes

update

c1cb5e8

typhoonzero approved these changes Aug 23, 2017

View reviewed changes

Yancey1989 merged commit 66d64b0 into PaddlePaddle:develop Aug 24, 2017

Yancey1989 deleted the commandline_design_doc branch August 24, 2017 02:02

command-line interface redesign #291

command-line interface redesign #291

Conversation

Yancey1989 commented Aug 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wanghaoshuang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yancey1989 Aug 10, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helinwang Aug 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yancey1989 Aug 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helinwang Aug 11, 2017 • edited Loading

Choose a reason for hiding this comment

Yancey1989 Aug 16, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helinwang Aug 16, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gongweibao Aug 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gongweibao Aug 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gongweibao Aug 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helinwang Aug 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helinwang commented Aug 22, 2017

typhoonzero left a comment

Choose a reason for hiding this comment

Yancey1989 Aug 10, 2017 •

edited

Loading

helinwang Aug 11, 2017 •

edited

Loading

Yancey1989 Aug 11, 2017 •

edited

Loading

helinwang Aug 11, 2017 •

edited

Loading

Yancey1989 Aug 16, 2017 •

edited

Loading

helinwang Aug 16, 2017 •

edited

Loading

gongweibao Aug 17, 2017 •

edited

Loading

gongweibao Aug 18, 2017 •

edited

Loading

gongweibao Aug 18, 2017 •

edited

Loading

helinwang Aug 18, 2017 •

edited

Loading