executor/ssh: introduce TIUP_SSH_PATH,TIUP_SCP_PATH

[9547]

What problem does this PR solve?

#711

What is changed and how it works?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Code changes

• Has exported function/method change
• Has exported variable/fields change
• Has interface methods change
• Has persistent data change

Side effects

• Possible performance regression
• Increased code complexity
• Breaking backward compatibility

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release notes:

NONE

[CLAassistant]

All committers have signed the CLA.

[9547]

@lucklove PTAL

[lucklove]

BTW, the integrate tests failed, seems caused by this PR, all tests with --native-ssh flag failed

[9547]

BTW, the integrate tests failed, seems caused by this PR, all tests with --native-ssh flag failed

Seems all failed to command-line line 0: missing argument. Could you give me some points, where I'm missing?

[lucklove]

IMO, the biggest challenge for you is that you can't check the error log.

There always been an error message such as Verbose debug logs has been written to /tiup-cluster/tests/tiup-dm/logs/tiup-cluster-debug-2020-09-01-18-15-18.log. but you just can't get /tiup-cluster/tests/tiup-dm/logs/tiup-cluster-debug-2020-09-01-18-15-18.log because it's in github CI.

So you can try to reproduce that in your local host:

# the first step is to build the integrate env
cd tiup/docker
./up.sh --daemon --dev --compose ./docker-compose.dm.yml

# then get into the control node
docker exec -it tiup-cluster-control /bin/sh

# and replay the failed case, eg: dm --native-ssh --do-cases test_cmd
bash /tiup-cluster/tests/tiup-dm/run.sh --native-ssh --do-cases test_cmd

Then you can get the error log (if the bug can be reproduced):

I found that the last error log is:

2020-09-02T03:33:07.686Z        INFO    Execute command finished        {"code": 1, "error": "task.env_init.failed: Failed to initialize TiDB environment on remote host '172.19.0.101', cause: module.user.user_add_failed: Failed to create new system user 'tidb' on remote host, cause: executor.ssh.execute_failed: Failed to execute command over SSH for 'root@172.19.0.101:22' {ssh_stderr: command-line line 0: missing argument.\r\n, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin sudo -H bash -c \"id -u tidb > /dev/null 2>&1 || (/usr/sbin/groupadd -f tidb && /usr/sbin/useradd -m -s /bin/bash -g tidb tidb) && echo 'tidb ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/tidb\"}, cause: exit status 255", "errorVerbose": "task.env_init.failed: Failed to initialize TiDB environment on remote host '172.19.0.101', cause: module.user.user_add_failed: Failed to create new system user 'tidb' on remote host, cause: executor.ssh.execute_failed: Failed to execute command over SSH for 'root@172.19.0.101:22' {ssh_stderr: command-line line 0: missing argument.\r\n, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin sudo -H bash -c \"id -u tidb > /dev/null 2>&1 || (/usr/sbin/groupadd -f tidb && /usr/sbin/useradd -m -s /bin/bash -g tidb tidb) && echo 'tidb ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/tidb\"}, cause: exit status 255\n at github.com/pingcap/tiup/pkg/cluster/executor.(*NativeSSHExecutor).Execute()\n\t/Users/joshua/Desktop/Code/tiup/pkg/cluster/executor/ssh.go:337\n at github.com/pingcap/tiup/pkg/cluster/module.(*UserModule).Execute()\n\t/Users/joshua/Desktop/Code/tiup/pkg/cluster/module/user.go:126\n at github.com/pingcap/tiup/pkg/cluster/task.(*EnvInit).execute()\n\t/Users/joshua/Desktop/Code/tiup/pkg/cluster/task/env_init.go:67\n at github.com/pingcap/tiup/pkg/cluster/task.(*EnvInit).Execute()\n\t/Users/joshua/Desktop/Code/tiup/pkg/cluster/task/env_init.go:46\n at github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute()\n\t/Users/joshua/Desktop/Code/tiup/pkg/cluster/task/task.go:189\n at github.com/pingcap/tiup/pkg/cluster/task.(*StepDisplay).Execute()\n\t/Users/joshua/Desktop/Code/tiup/pkg/cluster/task/step.go:85\n at github.com/pingcap/tiup/pkg/cluster/task.(*Parallel).Execute.func1()\n\t/Users/joshua/Desktop/Code/tiup/pkg/cluster/task/task.go:242\n at runtime.goexit()\n\t/Users/joshua/Desktop/Code/go-darwin-amd64-bootstrap/src/runtime/asm_amd64.s:1374"}

However, I have no ideal why it report this now, because I can execute

export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin sudo -H bash -c \"id -u tidb > /dev/null 2>&1 || (/usr/sbin/groupadd -f tidb && /usr/sbin/useradd -m -s /bin/bash -g tidb tidb) && echo 'tidb ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/tidb

on 172.19.0.101 successfully

[9547]

@lucklove thank you for the way to create the integrate env, I can reproduce it locally, and I'll
fix ASAP

[codecov-commenter]

Codecov Report

Merging #734 into master will decrease coverage by 0.04%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #734      +/-   ##
==========================================
- Coverage   58.07%   58.03%   -0.05%     
==========================================
  Files         255      255              
  Lines       18904    18919      +15     
==========================================
+ Hits        10978    10979       +1     
- Misses       6473     6482       +9     
- Partials     1453     1458       +5     

Flag	Coverage Δ
#coverage	58.03% <ø> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ithub.com/pingcap/tiup/pkg/cluster/executor/ssh.go	63.05% <0.00%> (-2.24%)	⬇️
...c/github.com/pingcap/tiup/pkg/cluster/api/pdapi.go	53.65% <0.00%> (-1.83%)	⬇️
go/src/github.com/pingcap/tiup/pkg/utils/ioutil.go	49.60% <0.00%> (+0.42%)	⬆️
...c/github.com/pingcap/tiup/pkg/cluster/api/dmapi.go	62.77% <0.00%> (+1.45%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 760091f...f89b8db. Read the comment docs.

[lonng]

Rest LGTM

[lonng]

LGTM