Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v1alpha2 does not show any data immediately after starting a katib experiment in the HP monitor UI. #704

Closed
asispatra opened this issue Jul 30, 2019 · 8 comments

Comments

@asispatra
Copy link

v1alpha2 does not show any data immediately after starting a katib experiment. We can only see the the experiment entry in the HP monitor. If I click on experiment, It's empty. should I wait till the experiment done?

But v1alpha1 shows the all the trials lists immediately in the UI, once you start a studyjob.

  • Kubeflow version: 0.6.1
  • Kubernetes version: (use kubectl version): v1.15.1
  • OS (e.g. from /etc/os-release): Ubuntu 18.04.2 LTS

Thanks.

@asispatra asispatra changed the title v1alpha2 does not show any data immediately after starting a katib experiment. v1alpha2 does not show any data immediately after starting a katib experiment in the HP monitor UI. Jul 30, 2019
@jlewi
Copy link
Contributor

jlewi commented Oct 2, 2019

I'm not seeing any data in the Katib Monitoring UI even though I have experiments running.

kubectl get experiments
NAME                        STATUS    AGE
tune-191002-081855   Running   7m

katib

I'm using Kubeflow 0.6..2

I've been working on adding HPTuning to the xgboost_synthetic example and you should be able to use that to repo this.
jlewi/examples@621f525

@johnugeorge
Copy link
Member

Currently, UI doesn't show trials till metrics are reported for at least one trial. Till then, UI doesn't show any data.

@gaocegege
Copy link
Member

@jlewi Do you see the experiment in the UI?

Maybe like this.

Screenshot from 2019-10-03 10-22-05

@jlewi
Copy link
Contributor

jlewi commented Oct 5, 2019

@gaocegege no I didn't see any experiment in the UI.

Could this be an issue with multi-user and missing service accounts? I was running my experiments in a namespace other than the Kubeflow namespace.

As part of that I ran into problems with service accounts missing (e.g #843) so one possibility is that the katib backend either isn't listing experiments in other namespace or else is having problems because of missing RBAC permissions/service accounts.

@johnugeorge
Copy link
Member

@jlewi The support for non kubeflow namespace in UI was added recently in v1alpha3. It is not supported in v1alpha2. In v1alpha2, experiments are listed only from Kubeflow namespace

Issue: #774

@gaocegege
Copy link
Member

As @johnugeorge said, we recently support multi-namespace. I think you should be able to get experiments on UI in v1alpha3.

Ref #774

@gaocegege
Copy link
Member

/close

@k8s-ci-robot
Copy link

@gaocegege: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants