topgrade can't find pipx installed from scoop

[soredake]

Erroneous Behavior

Topgrade cannot find pipx from scoop:

❯ topgrade --cleanup --only 'pipx'

── 19:42:36 - pipx ─────────────────────────────────────────────────────────────
pipx failed:
   0: Failed to execute `pipx --version`
   1: program not found

Location:
   src\steps\generic.rs:383
Retry? (y)es/(N)o/(s)hell/(q)uit

── 19:42:38 - Summary ──────────────────────────────────────────────────────────
pipx: FAILED

Expected Behavior

It should be able to use it as i can use it from shell just fine:

Steps to reproduce

1. Install pipx and topgrade from scoop on Windows.
2. Do topgrade --cleanup --only 'pipx'
3. Error.

Possible Cause (Optional)

scoop using shims, maybe this somehow prevents topgrade from using pipx.

Problem persists without calling from topgrade

• Yes
• No

Did you run topgrade through Remote Execution

• Yes
• No

If yes, does the issue still occur when you run topgrade directlly in your
remote host

• Yes
• No

Configuration file (Optional)

Additional Details

• Operation System/Version

Windows 11 Pro 23H2

• Installation

From scoop.

• Topgrade version (topgrade -V)
  Topgrade 14.0.1

Verbose Output (topgrade -v)

❯ topgrade --cleanup --only 'pipx' -v
DEBUG Configuration at C:\Users\user\AppData\Roaming\topgrade.toml
DEBUG Loaded configuration: ConfigFile { include: None, misc: Some(Misc { pre_sudo: None, sudo_command: None, disable: Some([Chocolatey, Scoop, Wsl, WslUpdate, Pnpm]), ignore_failures: None, remote_topgrades: None, remote_topgrade_path: None, ssh_arguments: None, tmux_arguments: None, set_title: None, display_time: None, assume_yes: None, no_retry: None, run_in_tmux: None, cleanup: None, notify_each_step: None, skip_notify: None, bashit_branch: None, only: None, no_self_update: None, log_filters: None }), pre_commands: Some({}), post_commands: None, commands: Some({"cust_pip3": "python -m pip install --upgrade pip"}), python: Some(Python { enable_pip_review: None, enable_pip_review_local: None, enable_pipupgrade: None, pipupgrade_arguments: None }), composer: Some(Composer { self_update: None }), brew: Some(Brew { greedy_cask: None, autoremove: None }), linux: Some(Linux { yay_arguments: None, aura_aur_arguments: None, aura_pacman_arguments: None, arch_package_manager: None, show_arch_news: None, garuda_update_arguments: None, trizen_arguments: None, pikaur_arguments: None, pamac_arguments: None, dnf_arguments: None, nix_arguments: None, nix_env_arguments: None, apt_arguments: None, enable_tlmgr: None, redhat_distro_sync: None, suse_dup: None, rpm_ostree: None, emerge_sync_flags: None, emerge_update_flags: None, home_manager_arguments: None }), git: Some(Git { max_concurrency: None, arguments: None, repos: None, pull_predefined: None }), containers: None, windows: Some(Windows { accept_all_updates: None, self_rename: None, open_remotes_in_new_terminal: None, enable_winget: None, wsl_update_pre_release: None, wsl_update_use_web_download: None }), npm: Some(NPM { use_sudo: None }), yarn: None, vim: None, firmware: Some(Firmware { upgrade: None }), vagrant: None, flatpak: Some(Flatpak { use_sudo: None }), distrobox: Some(Distrobox { use_root: None, containers: None }) }
DEBUG Version: 14.0.1
DEBUG OS: x86_64-pc-windows-msvc
DEBUG Args { inner: ["C:\\Users\\user\\scoop\\apps\\topgrade\\current\\topgrade.exe", "--cleanup", "--only", "pipx", "-v"] }
DEBUG Binary path: Ok("C:\\Users\\user\\scoop\\apps\\topgrade\\current\\topgrade.exe")
DEBUG self-update Feature Enabled: true
DEBUG Configuration: Config { opt: CommandLineArgs { edit_config: false, show_config_reference: false, run_in_tmux: false, cleanup: true, dry_run: false, no_retry: false, disable: [], only: [Pipx], custom_commands: [], env: [], verbose: true, keep_at_end: false, skip_notify: false, yes: None, disable_predefined_git_repos: false, config: None, remote_host_limit: None, show_skipped: false, log_filter: "warn", gen_completion: None, gen_manpage: false, no_self_update: false }, config_file: ConfigFile { include: None, misc: Some(Misc { pre_sudo: None, sudo_command: None, disable: Some([Chocolatey, Scoop, Wsl, WslUpdate, Pnpm]), ignore_failures: None, remote_topgrades: None, remote_topgrade_path: None, ssh_arguments: None, tmux_arguments: None, set_title: None, display_time: None, assume_yes: None, no_retry: None, run_in_tmux: None, cleanup: None, notify_each_step: None, skip_notify: None, bashit_branch: None, only: None, no_self_update: None, log_filters: None }), pre_commands: Some({}), post_commands: None, commands: Some({"cust_pip3": "python -m pip install --upgrade pip"}), python: Some(Python { enable_pip_review: None, enable_pip_review_local: None, enable_pipupgrade: None, pipupgrade_arguments: None }), composer: Some(Composer { self_update: None }), brew: Some(Brew { greedy_cask: None, autoremove: None }), linux: Some(Linux { yay_arguments: None, aura_aur_arguments: None, aura_pacman_arguments: None, arch_package_manager: None, show_arch_news: None, garuda_update_arguments: None, trizen_arguments: None, pikaur_arguments: None, pamac_arguments: None, dnf_arguments: None, nix_arguments: None, nix_env_arguments: None, apt_arguments: None, enable_tlmgr: None, redhat_distro_sync: None, suse_dup: None, rpm_ostree: None, emerge_sync_flags: None, emerge_update_flags: None, home_manager_arguments: None }), git: Some(Git { max_concurrency: None, arguments: None, repos: None, pull_predefined: None }), containers: None, windows: Some(Windows { accept_all_updates: None, self_rename: None, open_remotes_in_new_terminal: None, enable_winget: None, wsl_update_pre_release: None, wsl_update_use_web_download: None }), npm: Some(NPM { use_sudo: None }), yarn: None, vim: None, firmware: Some(Firmware { upgrade: None }), vagrant: None, flatpak: Some(Flatpak { use_sudo: None }), distrobox: Some(Distrobox { use_root: None, containers: None }) }, allowed_steps: [Pipx] }
DEBUG Detected "C:\\Program Files\\Git\\cmd\\git.EXE" as "git"
DEBUG Detected "C:\\Program Files\\WindowsApps\\Microsoft.PowerShell_7.4.1.0_x64__8wekyb3d8bbwe\\pwsh.EXE" as "pwsh"
DEBUG Executing command `C:\Program Files\WindowsApps\Microsoft.PowerShell_7.4.1.0_x64__8wekyb3d8bbwe\pwsh.EXE -NoProfile -Command 'Split-Path $profile'`
DEBUG Path "C:\\Users\\user\\OneDrive\\Documents\\PowerShell" exists
DEBUG Path "C:\\Users\\user\\AppData\\Roaming\\.emacs.d" doesn't exist
DEBUG Cannot find "doas"
DEBUG Detected "C:\\Users\\user\\scoop\\shims\\sudo.EXE" as "sudo"
DEBUG Step "pipx"
DEBUG Detected "C:\\Users\\user\\scoop\\shims\\pipx.CMD" as "pipx"
── 19:47:31 - pipx ─────────────────────────────────────────────────────────────

DEBUG Executing command pipx --version

DEBUG Step "pipx" failed:

0: Failed to execute pipx --version

1: program not found
Location:

src\steps\generic.rs:383

pipx failed:

0: Failed to execute pipx --version

1: program not found
Location:

src\steps\generic.rs:383

Retry? (y)es/(N)o/(s)hell/(q)uit

DEBUG C:\Users\user.ideavimrc does not exist

DEBUG C:\Users\user.intellimacs does not exist

DEBUG Checking if \?\C:\Users\user\AppData\Local\Packages\Microsoft.WindowsTerminal_8wekyb3d8bbwe\LocalState is a git repository

DEBUG Transformed path to C:\Users\user\AppData\Local\Packages\Microsoft.WindowsTerminal_8wekyb3d8bbwe\LocalState

DEBUG Executing command C:\Program Files\Git\cmd\git.EXE rev-parse --show-toplevel

DEBUG Command failed: Err(

0: Command failed: C:\Program Files\Git\cmd\git.EXE rev-parse --show-toplevel
  Stderr:
  fatal: not a git repository (or any of the parent directories): .git

1: C:\Program Files\Git\cmd\git.EXE failed: exit code: 128
Location:

src\steps\git.rs:170)

DEBUG Startup link: Lnk { path: Some("C:\Users\user\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\64Gram.lnk"), header: ShellLinkHeader { header_size: 76, link_clsid: 93045959704944114645041356371858166785, link_flags: HAS_LINK_TARGET_ID_LIST | HAS_LINK_INFO | HAS_NAME | HAS_RELATIVE_PATH | HAS_WORKING_DIR | HAS_ARGUMENTS | IS_UNICODE, file_attributes: (empty), creation_time: 133308061026854264, access_time: 133309393587768394, write_time: 133308061243582882, file_size: 133812224, icon_index: 0, show_command: SW_SHOWNORMAL, hot_key: HotKeyFlags { low_byte: 0, high_byte: 0 }, reserved1: 0, reserved2: 0, reserved3: 0, created_on: Some(2023-06-09T17:41:42.685Z), modified_on: Some(2023-06-09T17:42:04.358Z), accessed_on: Some(2023-06-11T06:42:38.776Z) }, string_data: StringData { name_string: Some("Telegram autorun link.\nYou can disable autorun in Telegram settings."), relative_path: Some("..\..\..\..\..\64Gram Desktop\Telegram.exe"), working_dir: Some("C:\Users\user\AppData\Roaming\64Gram Desktop"), command_line_arguments: Some("-autostart"), icon_location: None }, link_target_id_list: LinkTargetIdList, link_info: LinkInfo { link_info_size: 114, link_info_header_size: 28, link_info_flags: Some(VOLUME_ID_AND_LOCAL_BASE_PATH), volume_id_offset: 28, local_base_path_offset: 55, common_network_relative_link_offset: 0, common_path_suffix_offset: 113, local_base_path_offset_unicode: 27, common_path_suffix_offset_unicode: 3, volume_id: None, local_base_path: Some("C:\Users\user\AppData\Roaming\64Gram Desktop\Telegram.exe"), common_network_relative_link: None, common_path_suffix: None, local_base_path_unicode: None, common_path_suffix_unicode: None }, extra_data: ExtraData { darwin_props: None, special_folder_props: None, console_props: None, console_fe_props: None, environment_props: None, icon_environment_props: None, known_folder_props: None, property_store_props: Some(PropertyStoreDataBlock { block_size: 166, block_signature: 2684354569, property_store: [97, 0, 0, 0, 49, 83, 80, 83, 85, 40, 76, 159, 121, 159, 57, 75, 168, 208, 225, 212, 45, 225, 213, 243, 69, 0, 0, 0, 5, 0, 0, 0, 0, 31, 0, 0, 0, 25, 0, 0, 0, 84, 0, 101, 0, 108, 0, 101, 0, 103, 0, 114, 0, 97, 0, 109, 0, 46, 0, 84, 0, 101, 0, 108, 0, 101, 0, 103, 0, 114, 0, 97, 0, 109, 0, 68, 0, 101, 0, 115, 0, 107, 0, 116, 0, 111, 0, 112, 0, 0, 0, 0, 0, 0, 0, 0, 0, 57, 0, 0, 0, 49, 83, 80, 83, 177, 22, 109, 68, 173, 141, 112, 72, 167, 72, 64, 46, 164, 61, 120, 140, 29, 0, 0, 0, 104, 0, 0, 0, 0, 72, 0, 0, 0, 86, 38, 228, 225, 215, 95, 117, 65, 188, 160, 253, 171, 218, 9, 131, 241, 0, 0, 0, 0, 0, 0, 0, 0] }), shim_props: None, tracker_props: Some(TrackerDataBlock { block_size: 96, block_signature: 2684354563, length: 88, version: 0, machine_id: 547369977551526558338383880546838903, droid: [246641088347836073403379160464379613196, 25846492941788644465674475228989225756], droid_birth: [246641088347836073403379160464379613196, 25846492941788644465674475228989225756] }), vista_and_above_idlist_props: None } }

DEBUG \?\C:\Users\user\AppData\Roaming\64Gram Desktop\Telegram.exe is a file. Checking \?\C:\Users\user\AppData\Roaming\64Gram Desktop

DEBUG Checking if \?\C:\Users\user\AppData\Roaming\64Gram Desktop is a git repository

DEBUG Transformed path to C:\Users\user\AppData\Roaming\64Gram Desktop

DEBUG Executing command C:\Program Files\Git\cmd\git.EXE rev-parse --show-toplevel

DEBUG Command failed: Err(

0: Command failed: C:\Program Files\Git\cmd\git.EXE rev-parse --show-toplevel
  Stderr:
  fatal: not a git repository (or any of the parent directories): .git

1: C:\Program Files\Git\cmd\git.EXE failed: exit code: 128
Location:

src\steps\git.rs:170)

DEBUG Startup link: Lnk { path: Some("C:\Users\user\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\Emby Server.lnk"), header: ShellLinkHeader { header_size: 76, link_clsid: 93045959704944114645041356371858166785, link_flags: HAS_LINK_TARGET_ID_LIST | HAS_LINK_INFO | HAS_NAME | HAS_RELATIVE_PATH | HAS_WORKING_DIR | IS_UNICODE, file_attributes: (empty), creation_time: 133518928781802649, access_time: 133525806377972037, write_time: 133518928781842696, file_size: 464816, icon_index: 0, show_command: SW_SHOWNORMAL, hot_key: HotKeyFlags { low_byte: 0, high_byte: 0 }, reserved1: 0, reserved2: 0, reserved3: 0, created_on: Some(2024-02-08T19:07:58.180Z), modified_on: Some(2024-02-08T19:07:58.184Z), accessed_on: Some(2024-02-16T18:10:37.797Z) }, string_data: StringData { name_string: Some("Emby Server"), relative_path: Some("..\..\..\..\..\Emby-Server\system\EmbyServer.exe"), working_dir: Some("C:\Users\user\AppData\Roaming\Emby-Server\system"), command_line_arguments: None, icon_location: None }, link_target_id_list: LinkTargetIdList, link_info: LinkInfo { link_info_size: 120, link_info_header_size: 28, link_info_flags: Some(VOLUME_ID_AND_LOCAL_BASE_PATH), volume_id_offset: 28, local_base_path_offset: 55, common_network_relative_link_offset: 0, common_path_suffix_offset: 119, local_base_path_offset_unicode: 27, common_path_suffix_offset_unicode: 3, volume_id: None, local_base_path: Some("C:\Users\user\AppData\Roaming\Emby-Server\system\EmbyServer.exe"), common_network_relative_link: None, common_path_suffix: None, local_base_path_unicode: None, common_path_suffix_unicode: None }, extra_data: ExtraData { darwin_props: None, special_folder_props: None, console_props: None, console_fe_props: None, environment_props: None, icon_environment_props: None, known_folder_props: None, property_store_props: Some(PropertyStoreDataBlock { block_size: 69, block_signature: 2684354569, property_store: [57, 0, 0, 0, 49, 83, 80, 83, 177, 22, 109, 68, 173, 141, 112, 72, 167, 72, 64, 46, 164, 61, 120, 140, 29, 0, 0, 0, 104, 0, 0, 0, 0, 72, 0, 0, 0, 86, 38, 228, 225, 215, 95, 117, 65, 188, 160, 253, 171, 218, 9, 131, 241, 0, 0, 0, 0, 0, 0, 0, 0] }), shim_props: None, tracker_props: Some(TrackerDataBlock { block_size: 96, block_signature: 2684354563, length: 88, version: 0, machine_id: 609576601641048345536073861946306420, droid: [246641088347836073403379160464379613196, 25846492941789518103474016908676458855], droid_birth: [246641088347836073403379160464379613196, 25846492941789518103474016908676458855] }), vista_and_above_idlist_props: None } }

DEBUG \?\C:\Users\user\AppData\Roaming\Emby-Server\system\EmbyServer.exe is a file. Checking \?\C:\Users\user\AppData\Roaming\Emby-Server\system

DEBUG Checking if \?\C:\Users\user\AppData\Roaming\Emby-Server\system is a git repository

DEBUG Transformed path to C:\Users\user\AppData\Roaming\Emby-Server\system

DEBUG Executing command C:\Program Files\Git\cmd\git.EXE rev-parse --show-toplevel

DEBUG Command failed: Err(

0: Command failed: C:\Program Files\Git\cmd\git.EXE rev-parse --show-toplevel
  Stderr:
  fatal: not a git repository (or any of the parent directories): .git

1: C:\Program Files\Git\cmd\git.EXE failed: exit code: 128
Location:

src\steps\git.rs:170)

DEBUG Checking if \?\C:\Users\user\OneDrive\Documents\PowerShell is a git repository

DEBUG Transformed path to C:\Users\user\OneDrive\Documents\PowerShell

DEBUG Executing command C:\Program Files\Git\cmd\git.EXE rev-parse --show-toplevel

DEBUG Command failed: Err(

0: Command failed: C:\Program Files\Git\cmd\git.EXE rev-parse --show-toplevel
  Stderr:
  fatal: not a git repository (or any of the parent directories): .git

1: C:\Program Files\Git\cmd\git.EXE failed: exit code: 128
Location:

src\steps\git.rs:170)
── 19:47:34 - Summary ──────────────────────────────────────────────────────────

pipx: FAILED

DEBUG Desktop notification: Topgrade finished with errors

[SteveLauC]

This is so similar to #720

scoop using shims, maybe this somehow prevents topgrade from using pipx.

Is there any way to detect a shim and automatically "dereference" it?

[soredake]

Is there any way to detect a shim

Yes, the binary lives in shims folder:

[SteveLauC]

Except for identifying it from paths, is there any general method so that perhaps we can apply it to the which crate?

[soredake]

I don't think there is, I don't understand why topgrade even have problem with running pipx.cmd.

[SteveLauC]

I don't think there is

This is unfortunate, I am curious about the usage of these shim files, and how PowerShell identifies them and automatically dereferences them?

DEBUG Detected "C:\Users\user\scoop\shims\pipx.CMD" as "pipx"

I don't understand why topgrade even have problem with running pipx.cmd.

From the log, Topgrade identifies this pipx.CMD as the pipx binary, is this also a shim? A google search only tells me that .cmd is a legacy script format coming from the NT era.

[soredake]

In pipx case, pipx.cmd is just a script with this contents:

@rem C:\Users\user\scoop\apps\pipx\current\pipx.bat
@"C:\Users\user\scoop\apps\pipx\current\pipx.bat"  %*

[soredake]

and how PowerShell identifies them and automatically dereferences them?

Shims are just executables, there are no need to explicitly support them in any software, they just work.

[SteveLauC]

I am not familiar with Windows, based on my UNIX experience, it is kinda similar to symlinks, right? Most UNIX syscalls will dereference symlinks when encountered, including the syscall used to replace a process execxxx():

$ cat main.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(void) {
  char *args[] = {"ls", "-l"};

  int res = execve("link_to_ls", args, NULL);

  if (res == -1) {
    perror("execve");
  }

  exit(EXIT_SUCCESS);
}

$ gccs main.c
$ ./a.out
-rwxr-xr-x. 1 steve steve 16784 Mar  4 07:42 a.out
lrwxrwxrwx. 1 steve steve    11 Mar  4 07:40 link_to_ls -> /usr/bin/ls
-rw-r--r--. 1 steve steve   226 Mar  4 07:40 main.c

And it seems that the corresponding Windows syscall won't do this?

[soredake]

As I said above, pipx installed from scoop is just a script(pipx.cmd) with these contents, not a shim executable like i thougt initially:

@rem C:\Users\user\scoop\apps\pipx\current\pipx.bat
@"C:\Users\user\scoop\apps\pipx\current\pipx.bat"  %*

pipx.bat:

@python "%~dp0pipx.pyz" %*

[soredake]

The strange thing is that topgrade can execute scoop fine, and scoop on Windows is .cmd script too.

[SteveLauC]

Some updates on this issue:

From what I have found, this issue is different from #720, the root cause of #720 is a bug of Rust std::process::Command. This issue is still in my work queue, but I am unable to do it this week, will try to fix on the next weekend.

[SteveLauC]

Well, this is actually same with #720, and it has been fixed in the latest main branch: