E2E for TensorFlow Integration #381

thandayuthapani · 2019-07-23T15:55:33Z

This PR involves E2E for TensorFlow Integration with volcano

TravisBuddy · 2019-07-23T16:20:34Z

Hey @thandayuthapani,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: cab9b0e0-ad65-11e9-aa77-bf7f12915990

TravisBuddy · 2019-07-23T16:59:40Z

Hey @thandayuthapani,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: 41c164d0-ad6b-11e9-aa77-bf7f12915990

TommyLike · 2019-07-24T01:39:09Z

@thandayuthapani Download every images and run different samples for every application running on volcano would slow down our developing process. Maybe we should setup up a cronjob for this kind of e2e tests?

cc @k82cn @hzxuzhonghu

hzxuzhonghu · 2019-07-24T02:17:49Z

test/e2e/util.go

@@ -645,7 +646,7 @@ func waitJobStateAborted(ctx *context, job *vkv1.Job) error {

 func waitJobPhaseExpect(ctx *context, job *vkv1.Job, state vkv1.JobPhase) error {
 	var additionalError error
-	err := wait.Poll(100*time.Millisecond, oneMinute, func() (bool, error) {
+	err := wait.Poll(100*time.Millisecond, twoMinute, func() (bool, error) {


this may increase total test time, I would like to add a parameter to waitJobPhaseExpect

hzxuzhonghu · 2019-07-24T02:22:22Z

I think this is good, we should have a tf job case. Cronjob does not fit very much, it can not prevent a pr from breaking cases like tf job.

For the speed issue, the docker images should be reusable.

thandayuthapani · 2019-07-24T06:14:56Z

@thandayuthapani Download every images and run different samples for every application running on volcano would slow down our developing process. Maybe we should setup up a cronjob for this kind of e2e tests?

cc @k82cn @hzxuzhonghu

@TommyLike To my Knowledge downloading Images will not affect test time much, since they provide high speed internet, which is around 35MB/s. So downloading images might add 10 more secs in E2E

TravisBuddy · 2019-07-24T06:53:42Z

Hey @thandayuthapani,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: c4fb3320-addf-11e9-aa77-bf7f12915990

TommyLike · 2019-07-24T07:28:02Z

@thandayuthapani Download every images and run different samples for every application running on volcano would slow down our developing process. Maybe we should setup up a cronjob for this kind of e2e tests?
cc @k82cn @hzxuzhonghu

@TommyLike To my Knowledge downloading Images will not affect test time much, since they provide high speed internet, which is around 35MB/s. So downloading images might add 10 more secs in E2E

@thandayuthapani We need to consider how much time it takes to download the image and how much time it takes to load the images into kind and to complete the e2e tests.

⚡ root@husheng-test  ~  docker image ls | grep ed417b28e09c
thanda/tf-operator-example    1.0                                        ed417b28e09c        42 hours ago        1.25GB

 ⚡ root@husheng-test  ~  kind load docker-image thanda/tf-operator-example:1.0  --name integration
Execution time: 0h:02m:57s sec

thandayuthapani · 2019-07-24T09:43:13Z

We need to consider how much time it takes to download the image and how much time it takes to load the images into kind and to complete the e2e tests.
⚡ root@husheng-test  ~  docker image ls | grep ed417b28e09c
thanda/tf-operator-example    1.0                                        ed417b28e09c        42 hours ago        1.25GB
 ⚡ root@husheng-test  ~  kind load docker-image thanda/tf-operator-example:1.0  --name integration
Execution time: 0h:02m:57s sec

1.25GB Image which has been posted is old one, Have updated that image with lighter one.

docker images | grep thanda/tf-operator-example
thanda/tf-operator-example    1.0                 1bdc930499c4        4 hours ago         461MB

And time to load docker-image in my local setup is

time kind load docker-image thanda/tf-operator-example:1.0 --name integration

real	0m22.613s

For E2E test it will take around 3 mins to complete but that is the simplest tensorflow training example I could find.

TommyLike · 2019-07-24T09:49:18Z

@thandayuthapani thanks for your update, and that's why I am thinking if we need considering move this into another job

thandayuthapani · 2019-07-24T11:15:29Z

@TommyLike Have reduced number of train steps, now the test runs in CI for 78 Secs, if that is okay we can maintain that, else we can reduce steps further also if necessary.

TommyLike · 2019-07-24T13:05:42Z

/lgtm

hzxuzhonghu · 2019-07-25T02:55:54Z

@thandayuthapani One point, the image used should be in volcanosh instead of your own repo.

thandayuthapani · 2019-07-25T03:37:16Z

@thandayuthapani One point, the image used should be in volcanosh instead of your own repo.

Sure will move that to volcanosh repo

thandayuthapani · 2019-07-25T10:37:17Z

@thandayuthapani One point, the image used should be in volcanosh instead of your own repo.

Have updated the same.

TommyLike · 2019-07-25T10:54:40Z

example/integrations/tensorflow/dist-mnist/tf-dist-mnist-example.yaml

+                  WORKER_HOST=`cat /etc/volcano/worker.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`;
+                  export TF_CONFIG={\"cluster\":{\"ps\":[${PS_HOST}],\"worker\":[${WORKER_HOST}]},\"task\":{\"type\":\"ps\",\"index\":${VK_TASK_INDEX}},\"environment\":\"cloud\"};
+                  python /var/tf_dist_mnist/dist_mnist.py
+              image: thanda/tf-operator-example:1.0


volcanosh/dist-mnist-tf-example:0.0.1

TommyLike · 2019-07-25T10:54:47Z

example/integrations/tensorflow/dist-mnist/tf-dist-mnist-example.yaml

+                  WORKER_HOST=`cat /etc/volcano/worker.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`;
+                  export TF_CONFIG={\"cluster\":{\"ps\":[${PS_HOST}],\"worker\":[${WORKER_HOST}]},\"task\":{\"type\":\"worker\",\"index\":${VK_TASK_INDEX}},\"environment\":\"cloud\"};
+                  python /var/tf_dist_mnist/dist_mnist.py
+              image: thanda/tf-operator-example:1.0


TravisBuddy · 2019-07-25T12:13:37Z

Hey @thandayuthapani,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: a05e1c20-aed5-11e9-a97b-d704c485e7e4

thandayuthapani · 2019-07-25T12:36:45Z

@k82cn Can you please retrigger this build.

TommyLike · 2019-07-26T13:34:46Z

/lgtm

TravisBuddy · 2019-07-26T13:35:01Z

Hey @thandayuthapani,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: 29ceca70-afaa-11e9-83c1-65c58ed3b4ac

TravisBuddy · 2019-07-27T08:46:27Z

Hey @thandayuthapani,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: 04909210-b04b-11e9-83c1-65c58ed3b4ac

thandayuthapani · 2019-07-30T10:11:38Z

@k82cn @TommyLike Please have a look

TommyLike · 2019-07-30T11:09:55Z

/lgtm

TommyLike · 2019-08-02T01:38:03Z

/approve

TommyLike · 2019-08-02T01:49:13Z

/assign @kevin-wangzefeng

k82cn · 2019-08-20T07:16:35Z

/lgtm
/approve

volcano-sh-bot · 2019-08-20T07:16:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: k82cn, thandayuthapani, TommyLike

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [k82cn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

volcano-sh-bot requested review from kevin-wangzefeng and quinton-hoole July 23, 2019 15:55

volcano-sh-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 23, 2019

thandayuthapani force-pushed the e2e branch from da6ee01 to f3cdd86 Compare July 23, 2019 16:35

hzxuzhonghu reviewed Jul 24, 2019

View reviewed changes

thandayuthapani force-pushed the e2e branch from f3cdd86 to 017fe97 Compare July 24, 2019 06:12

thandayuthapani force-pushed the e2e branch from 017fe97 to 1b3ce8e Compare July 24, 2019 10:51

volcano-sh-bot assigned TommyLike Jul 24, 2019

volcano-sh-bot added the lgtm Indicates that a PR is ready to be merged. label Jul 24, 2019

thandayuthapani force-pushed the e2e branch from 1b3ce8e to 96a4a82 Compare July 25, 2019 09:38

volcano-sh-bot removed the lgtm Indicates that a PR is ready to be merged. label Jul 25, 2019

thandayuthapani force-pushed the e2e branch from 96a4a82 to ae0c6b8 Compare July 25, 2019 09:40

TommyLike reviewed Jul 25, 2019

View reviewed changes

thandayuthapani force-pushed the e2e branch 2 times, most recently from 15aa674 to f4e98ba Compare July 25, 2019 11:32

volcano-sh-bot added the lgtm Indicates that a PR is ready to be merged. label Jul 25, 2019

volcano-sh-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 26, 2019

thandayuthapani force-pushed the e2e branch from f4e98ba to 475f054 Compare July 26, 2019 12:41

volcano-sh-bot removed lgtm Indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jul 26, 2019

thandayuthapani force-pushed the e2e branch from 475f054 to a282cbd Compare July 26, 2019 12:53

volcano-sh-bot added the lgtm Indicates that a PR is ready to be merged. label Jul 26, 2019

volcano-sh-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 27, 2019

thandayuthapani added 2 commits July 27, 2019 13:31

Add simple example for tensorflow integration

f02f83f

Add E2E for TensorFlow integration with Volcano

8d44999

thandayuthapani force-pushed the e2e branch from a282cbd to 8d44999 Compare July 27, 2019 08:04

volcano-sh-bot removed lgtm Indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jul 27, 2019

volcano-sh-bot assigned kevin-wangzefeng Aug 2, 2019

volcano-sh-bot assigned k82cn Aug 20, 2019

volcano-sh-bot added the lgtm Indicates that a PR is ready to be merged. label Aug 20, 2019

volcano-sh-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 20, 2019

volcano-sh-bot merged commit e2faed8 into volcano-sh:master Aug 20, 2019

bibibox mentioned this pull request Nov 11, 2024

add events and actions for task reschedule volcano-sh/apis#140

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E for TensorFlow Integration #381

E2E for TensorFlow Integration #381

thandayuthapani commented Jul 23, 2019

TravisBuddy commented Jul 23, 2019

TravisBuddy commented Jul 23, 2019

TommyLike commented Jul 24, 2019

hzxuzhonghu Jul 24, 2019

thandayuthapani Jul 24, 2019

hzxuzhonghu commented Jul 24, 2019

thandayuthapani commented Jul 24, 2019

TravisBuddy commented Jul 24, 2019

TommyLike commented Jul 24, 2019 •

edited

Loading

thandayuthapani commented Jul 24, 2019

TommyLike commented Jul 24, 2019

thandayuthapani commented Jul 24, 2019

TommyLike commented Jul 24, 2019

hzxuzhonghu commented Jul 25, 2019

thandayuthapani commented Jul 25, 2019

thandayuthapani commented Jul 25, 2019

TommyLike Jul 25, 2019

thandayuthapani Jul 25, 2019

TommyLike Jul 25, 2019

thandayuthapani Jul 25, 2019

TravisBuddy commented Jul 25, 2019

thandayuthapani commented Jul 25, 2019

TommyLike commented Jul 26, 2019

TravisBuddy commented Jul 26, 2019

TravisBuddy commented Jul 27, 2019

thandayuthapani commented Jul 30, 2019

TommyLike commented Jul 30, 2019

TommyLike commented Aug 2, 2019

TommyLike commented Aug 2, 2019

k82cn commented Aug 20, 2019

volcano-sh-bot commented Aug 20, 2019

E2E for TensorFlow Integration #381

E2E for TensorFlow Integration #381

Conversation

thandayuthapani commented Jul 23, 2019

TravisBuddy commented Jul 23, 2019

TravisBuddy Request Identifier: cab9b0e0-ad65-11e9-aa77-bf7f12915990

TravisBuddy commented Jul 23, 2019

TravisBuddy Request Identifier: 41c164d0-ad6b-11e9-aa77-bf7f12915990

TommyLike commented Jul 24, 2019

hzxuzhonghu Jul 24, 2019

Choose a reason for hiding this comment

thandayuthapani Jul 24, 2019

Choose a reason for hiding this comment

hzxuzhonghu commented Jul 24, 2019

thandayuthapani commented Jul 24, 2019

TravisBuddy commented Jul 24, 2019

TravisBuddy Request Identifier: c4fb3320-addf-11e9-aa77-bf7f12915990

TommyLike commented Jul 24, 2019 • edited Loading

thandayuthapani commented Jul 24, 2019

TommyLike commented Jul 24, 2019

thandayuthapani commented Jul 24, 2019

TommyLike commented Jul 24, 2019

hzxuzhonghu commented Jul 25, 2019

thandayuthapani commented Jul 25, 2019

thandayuthapani commented Jul 25, 2019

TommyLike Jul 25, 2019

Choose a reason for hiding this comment

thandayuthapani Jul 25, 2019

Choose a reason for hiding this comment

TommyLike Jul 25, 2019

Choose a reason for hiding this comment

thandayuthapani Jul 25, 2019

Choose a reason for hiding this comment

TravisBuddy commented Jul 25, 2019

TravisBuddy Request Identifier: a05e1c20-aed5-11e9-a97b-d704c485e7e4

thandayuthapani commented Jul 25, 2019

TommyLike commented Jul 26, 2019

TravisBuddy commented Jul 26, 2019

TravisBuddy Request Identifier: 29ceca70-afaa-11e9-83c1-65c58ed3b4ac

TravisBuddy commented Jul 27, 2019

TravisBuddy Request Identifier: 04909210-b04b-11e9-83c1-65c58ed3b4ac

thandayuthapani commented Jul 30, 2019

TommyLike commented Jul 30, 2019

TommyLike commented Aug 2, 2019

TommyLike commented Aug 2, 2019

k82cn commented Aug 20, 2019

volcano-sh-bot commented Aug 20, 2019

TommyLike commented Jul 24, 2019 •

edited

Loading