Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runner error logs, destructure id of undefined #990

Open
dacbd opened this issue Apr 27, 2022 · 4 comments · Fixed by #1021
Open

runner error logs, destructure id of undefined #990

dacbd opened this issue Apr 27, 2022 · 4 comments · Fixed by #1021
Labels
ci-github cml-runner Subcommand flaky Heisenbugs p2-nice-to-have Low priority

Comments

@dacbd
Copy link
Contributor

dacbd commented Apr 27, 2022

Chasing a different error I have finally uncovered the origin of: Cannot destructure property 'id' of '(intermediate value)' as it is undefined.

of more commonly seen as:

{"level":"error","message":"\tFailed: Cannot destructure property 'id' of '(intermediate value)' as it is undefined."}

When the cml runner flag --single is used, the self-hosted runner is started with ephemeral and automatically de-registers itself when its one job is complete so when cml goes through the shutdown process to also tries and de-register itself the GitHub API returns no results because it's already gone....

@dacbd dacbd added cml-runner Subcommand ci-github flaky Heisenbugs p2-nice-to-have Low priority labels Apr 27, 2022
@dacbd
Copy link
Contributor Author

dacbd commented Apr 27, 2022

This error doesn't break any functionality as far as I can tell.

@0x2b3bfa0
Copy link
Member

Surprised of not seeing the stack trace; it should be part of all the JSON logs. 🤔

cml/bin/cml.js

Line 18 in 352fba4

winston.format.errors({ stack: true }),

This was referenced May 23, 2022
@dacbd dacbd reopened this Aug 22, 2022
@dacbd
Copy link
Contributor Author

dacbd commented Aug 22, 2022

this error is being caught the introduction of #1030? and now seems to be killing the runner process of legitimate jobs:

ubuntu@ip-172-31-19-239:~$ journalctl -n all -u cml.service 
-- Logs begin at Thu 2022-07-21 01:23:30 UTC, end at Mon 2022-08-22 15:11:42 UTC. --
Aug 22 14:49:43 ip-172-31-19-239 systemd[1]: Started cml.service.
Aug 22 14:49:43 ip-172-31-19-239 cml.sh[1911]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Aug 22 14:49:43 ip-172-31-19-239 cml.sh[1911]:                                  Dload  Upload   Total   Spent    Left  Speed
Aug 22 14:49:44 ip-172-31-19-239 cml.sh[1911]: [158B blob data]
Aug 22 14:49:46 ip-172-31-19-239 cml.sh[1911]: {"level":"warn","message":"Github Actions timeout has been updated from 72h to 35 days. Update your workflow accordingly to be able to restart it automatically."}
Aug 22 14:49:46 ip-172-31-19-239 cml.sh[1911]: {"level":"info","message":"Preparing workdir /tmp/tmp.aKGqCxcScZ/.cml/cml-jl6pimbqyz..."}
Aug 22 14:49:46 ip-172-31-19-239 cml.sh[1911]: {"level":"info","message":"Launching github runner"}
Aug 22 14:49:50 ip-172-31-19-239 cml.sh[1911]: {"level":"info","message":"Terraform 1.2.5"}
Aug 22 14:49:50 ip-172-31-19-239 cml.sh[1911]: {"level":"info","message":"Plan: 0 to add, 0 to change, 0 to destroy."}
Aug 22 14:49:50 ip-172-31-19-239 cml.sh[1911]: {"level":"info","message":"Apply complete! Resources: 0 added, 0 changed, 0 destroyed."}
Aug 22 14:49:50 ip-172-31-19-239 cml.sh[1911]: {"level":"info","message":"Outputs: 0"}
Aug 22 14:49:50 ip-172-31-19-239 cml.sh[1911]: {"level":"info","message":"Connected to acpid service."}
Aug 22 14:50:08 ip-172-31-19-239 cml.sh[1911]: {"date":"2022-08-22T14:50:08.441Z","level":"info","message":"runner status","repo":"https://github.com/iterative/cml-playground","status":"ready"}
Aug 22 14:50:20 ip-172-31-19-239 cml.sh[1911]: {"date":"Mon Aug 22 2022 14:50:20 GMT+0000 (Coordinated Universal Time)","error":{},"exception":true,"level":"error","message":"unhandledRejection: Cannot destructure property '
Aug 22 14:50:20 ip-172-31-19-239 cml.sh[1911]: {"level":"error","message":"TypeError: Cannot destructure property 'id' of '(intermediate value)' as it is undefined.","stack":"Error: TypeError: Cannot destructure property 'id
Aug 22 14:50:20 ip-172-31-19-239 cml.sh[1911]: {"level":"info","message":"Unregistering runner cml-jl6pimbqyz..."}
Aug 22 14:50:20 ip-172-31-19-239 cml.sh[1911]: {"level":"error","message":"\tFailed: Bad request - Runner \"cml-jl6pimbqyz\" is still running a job\""}
Aug 22 14:50:20 ip-172-31-19-239 cml.sh[1911]: {"level":"info","message":"Waiting 10 seconds to destroy"}
Aug 22 14:50:23 ip-172-31-19-239 systemd[1]: cml.service: Main process exited, code=exited, status=1/FAILURE
Aug 22 14:50:24 ip-172-31-19-239 systemd[1]: cml.service: Failed with result 'exit-code'.
lines 1-21/21 (END)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-github cml-runner Subcommand flaky Heisenbugs p2-nice-to-have Low priority
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants