Occasional Timeout waiting for SSH #79

sanieldalib · 2023-06-29T20:56:14Z

Hey there! Thanks for all your great work on this!

I sometimes get the following error when building an image from IPSW. My ssh_timeout is set to 180s, and I'm killing the tart process before beginning the build to ensure its not a result of too many VMs running (>2).

==> tart-cli.base: Waiting for SSH to become available...
==> tart-cli.base: Timeout waiting for SSH.

Here is what my source block looks like, its essentially identical to the templates provided in https://github.com/cirruslabs/macos-image-templates/blob/master/templates/vanilla-ventura.pkr.hcl

source "tart-cli" "base" {
  # You can find macOS IPSW URLs on various websites like https://ipsw.me/
  # and https://www.theiphonewiki.com/wiki/Beta_Firmware/Mac/13.x
  from_ipsw    = "${var.home}/macOS/UniversalMac_13.3_22E252_Restore.ipsw"
  vm_name      = var.vm_name
  cpu_count    = 4
  memory_gb    = 8
  disk_size_gb = 60
  ssh_username = "admin"
  ssh_password = "admin"
  ssh_timeout  = "180s"
  boot_command = [
    # hello, hola, bonjour, etc.
    "<wait60s><spacebar>",
    # Language
    "<wait30s>english<enter>",
    # Select Your Country and Region
    "<wait30s>united states<leftShiftOn><tab><leftShiftOff><spacebar>",
    # Written and Spoken Languages
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Accessibility
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Data & Privacy
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Migration Assistant
    "<wait10s><tab><tab><tab><spacebar>",
    # Sign In with Your Apple ID
    "<wait10s><leftShiftOn><tab><leftShiftOff><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Are you sure you want to skip signing in with an Apple ID?
    "<wait10s><tab><spacebar>",
    # Terms and Conditions
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # I have read and agree to the macOS Software License Agreement
    "<wait10s><tab><spacebar>",
    # Create a Computer Account
    "<wait10s>admin<tab><tab>admin<tab>admin<tab><tab><tab><spacebar>",
    # Enable Location Services
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Are you sure you don't want to use Location Services?
    "<wait10s><tab><spacebar>",
    # Select Your Time Zone
    "<wait10s><tab>UTC<enter><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Analytics
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Screen Time
    "<wait10s><tab><spacebar>",
    # Siri
    "<wait10s><tab><spacebar><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Choose Your Look
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Enable Voice Over
    "<wait10s><leftAltOn><f5><leftAltOff><wait5s>v",
    # Now that the installation is done, open "System Settings"
    "<wait10s><leftAltOn><spacebar><leftAltOff>System Settings<enter>",
    # Navigate to "Sharing"
    "<wait10s><leftAltOn>f<leftAltOff>sharing<enter>",
    # Navigate to "Screen Sharing" and enable it
    "<wait10s><tab><down><spacebar>",
    # Navigate to "Remote Login" and enable it
    "<wait10s><tab><tab><tab><tab><tab><tab><spacebar>",
    # Open "Remote Login" details
    "<wait10s><tab><spacebar>",
    # Enable "Full Disk Access"
    "<wait10s><tab><spacebar>",
    # Click "Done"
    "<wait10s><leftShiftOn><tab><leftShiftOff><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Disable Voice Over
    "<leftAltOn><f5><leftAltOff>",
  ]

  // A (hopefully) temporary workaround for Virtualization.Framework's
  // installation process not fully finishing in a timely manner
  create_grace_time = "30s"

The text was updated successfully, but these errors were encountered:

fkorotkov · 2023-06-30T15:57:44Z

Do you observe any failures during the installation process of the macOS? This might be related to boot_command not successfully doing the installation so your VM ends up without a user configured to be SSHed in via.

sridhar-vadlamani-ck · 2024-07-01T16:41:34Z

My vanilla script seems to be timing out as well.

    sonoma-vanilla.tart-cli.tart: 85%
    sonoma-vanilla.tart-cli.tart: 86%
    sonoma-vanilla.tart-cli.tart: 87%
    sonoma-vanilla.tart-cli.tart: 90%
    sonoma-vanilla.tart-cli.tart: 100%
==> sonoma-vanilla.tart-cli.tart: Waiting 30s to let the Virtualization.Framework's installation process to finish correctly...
==> sonoma-vanilla.tart-cli.tart: Updating virtual machine resources...
==> sonoma-vanilla.tart-cli.tart: Inspecting machine disk image...
==> sonoma-vanilla.tart-cli.tart: Getting partition table...
==> sonoma-vanilla.tart-cli.tart: Found a recovery partition at GPT entry 3, let's remove it to save space and allow for resizing the main partition...
==> sonoma-vanilla.tart-cli.tart: Successfully updated partitions!
==> sonoma-vanilla.tart-cli.tart: Starting the virtual machine...
==> sonoma-vanilla.tart-cli.tart: Waiting for the VNC server credentials from Tart...
==> sonoma-vanilla.tart-cli.tart: Retrieved VNC credentials, connecting...
    sonoma-vanilla.tart-cli.tart: If you want to view the screen of the VM, connect via VNC with the password "echo-family-amount-copy" to
    sonoma-vanilla.tart-cli.tart: vnc://127.0.0.1:62266
==> sonoma-vanilla.tart-cli.tart: Connected to the VNC!
==> sonoma-vanilla.tart-cli.tart: Typing the commands over VNC...
==> sonoma-vanilla.tart-cli.tart: Successfully started the virtual machine...
==> sonoma-vanilla.tart-cli.tart: Using SSH communicator to connect: 192.168.64.2
==> sonoma-vanilla.tart-cli.tart: Waiting for SSH to become available...
==> sonoma-vanilla.tart-cli.tart: Timeout waiting for SSH.
==> sonoma-vanilla.tart-cli.tart: Waiting for the tart process to exit..

No errors that I can see.

edigaryev · 2024-07-01T16:44:30Z

@sridhar-vadlamani-ck how are you invoking the Packer, from CI, or from a user-interactive Terminal?

Also this might be related to the DHCP shortage, please check out https://tart.run/faq/#changing-the-default-dhcp-lease-time.

sridhar-vadlamani-ck · 2024-07-01T18:58:40Z

It's being invoked in CI, from a non-interactive terminal. $TERM is set to xterm-256color but I haven't passed in any additional flags to the packer script. Are there any additional settings that would be needed to streamline building vanilla vms in CI?

This is my source:


source "tart-cli" "tart" {
  from_ipsw    = "${var.ipsw}"
  vm_name      = "vanilla"
  cpu_count    = 4
  memory_gb    = 8
  disk_size_gb = 40
  ssh_password = "admin"
  ssh_username = "admin"
  ssh_timeout  = "120s"
  boot_command = [
    # hello, hola, bonjour, etc.
    "<wait60s><spacebar>",
    # Language: most of the times we have a list of "English"[1], "English (UK)", etc. with
    # "English" language already selected. If we type "english", it'll cause us to switch
    # to the "English (UK)", which is not what we want. To solve this, we switch to some other
    # language first, e.g. "Italiano" and then switch back to "English". We'll then jump to the
    # first entry in a list of "english"-prefixed items, which will be "English".
    #
    # [1]: should be named "English (US)", but oh well 🤷
    "<wait30s>italiano<esc>english<enter>",
    # Select Your Country and Region
    "<wait30s>united states<leftShiftOn><tab><leftShiftOff><spacebar>",
    # Written and Spoken Languages
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Accessibility
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Data & Privacy
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Migration Assistant
    "<wait10s><tab><tab><tab><spacebar>",
    # Sign In with Your Apple ID
    "<wait10s><leftShiftOn><tab><leftShiftOff><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Are you sure you want to skip signing in with an Apple ID?
    "<wait10s><tab><spacebar>",
    # Terms and Conditions
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # I have read and agree to the macOS Software License Agreement
    "<wait10s><tab><spacebar>",
    # Create a Computer Account
    "<wait10s>admin<tab><tab>admin<tab>admin<tab><tab><tab><spacebar>",
    # Enable Location Services
    "<wait30s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Are you sure you don't want to use Location Services?
    "<wait10s><tab><spacebar>",
    # Select Your Time Zone
    "<wait10s><tab>UTC<enter><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Analytics
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Screen Time
    "<wait10s><tab><spacebar>",
    # Siri
    "<wait10s><tab><spacebar><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Choose Your Look
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Enable Voice Over
    "<wait10s><leftAltOn><f5><leftAltOff><wait5s>v",
    # Now that the installation is done, open "System Settings"
    "<wait10s><leftAltOn><spacebar><leftAltOff>System Settings<enter>",
    # Navigate to "Sharing"
    "<wait10s><leftAltOn>f<leftAltOff>sharing<enter>",
    # Navigate to "Screen Sharing" and enable it
    "<wait10s><tab><tab><tab><tab><tab><spacebar>",
    # Navigate to "Remote Login" and enable it
    "<wait10s><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><spacebar>",
    # Disable Voice Over
    "<leftAltOn><f5><leftAltOff>",
  ]

  // A (hopefully) temporary workaround for Virtualization.Framework's
  // installation process not fully finishing in a timely manner
  create_grace_time = "30s"
}

Also, it's not the DHCP shortage. Timeout is set to 10mins and the vanilla vm build was basically the first one on these machines. I'm still building it out.

fkorotkov · 2024-07-01T19:37:44Z

@sridhar-vadlamani-ck are you trying macOS Sequia Beta 2 host by any chance? We've see this issue on Sequia hosts.

sridhar-vadlamani-ck · 2024-07-01T19:59:16Z

@fkorotkov I'm building from a sonoma ipsw

fkorotkov · 2024-07-01T20:03:36Z

But what about the host? Just checking this. But please also check @edigaryev's tip about DHCP.

sridhar-vadlamani-ck · 2024-07-01T20:43:29Z

The host is also a sonoma instance.

% sw_vers
ProductName:            macOS
ProductVersion:         14.5
BuildVersion:           23F79

These are freshly launched instances with the DHCP lease time set to 10mins per the FAQ already.

sridhar-vadlamani-ck · 2024-07-01T20:44:42Z

I re-ran it a bunch of times, and of about 12 runs, one has succeeded. Everything else seems to fail with the SSH connection issue.

sridhar-vadlamani-ck · 2024-07-03T08:11:43Z

Is there anything else you'd like me to rule out?

edigaryev · 2024-07-03T08:20:47Z

Is there anything else you'd like me to rule out?

Can you connect to the VM using the ssh command after Packer emits the Using SSH communicator to connect: <IP> message?

sridhar-vadlamani-ck · 2024-07-04T08:26:52Z

% ssh -v admin@192.168.64.12
OpenSSH_9.6p1, LibreSSL 3.3.6
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to 192.168.64.12 [192.168.64.12] port 22.
debug1: connect to address 192.168.64.12 port 22: Connection refused
ssh: connect to host 192.168.64.12 port 22: Connection refused

==> sonoma-vanilla.tart-cli.ipsw: Connected to the VNC!
==> sonoma-vanilla.tart-cli.ipsw: Typing the commands over VNC...
==> sonoma-vanilla.tart-cli.ipsw: Successfully started the virtual machine...
==> sonoma-vanilla.tart-cli.ipsw: Using SSH communicator to connect: 192.168.64.12
==> sonoma-vanilla.tart-cli.ipsw: Waiting for SSH to become available...

sridhar-vadlamani-ck · 2024-07-05T18:36:25Z

The SSH issue is only happening when I build the vanilla image. If I build from another image directly, there are no issues.

edigaryev · 2024-07-07T08:38:47Z

The SSH issue is only happening when I build the vanilla image. If I build from another image directly, there are no issues.

This most likely has to do with Remote Login not being enabled in System Settings → Sharing from some reason in the guest.

Make sure that your boot_command sequence is correct for the IPSW you're using by observing how it interacts with the installer/macOS UI.

gitperr · 2024-11-14T08:23:01Z

This started happening to me just now too, but on Sequioa. It happens in the CI environment, but it does not happen when I SSH into the host and run the same commands myself on the terminal.

gitperr · 2024-11-14T14:17:44Z

Can you connect to the VM using the ssh command after Packer emits the Using SSH communicator to connect: <IP> message?

I tried this, and I was able to connect right then. But, the CI environment does not connect.

CodeStage · 2024-11-21T11:16:36Z

I just got a new M4 MacBook Pro and it will hang on "Waiting for SSH to become available..." every time while the VM is actually booted and I can do stuff with it. It still works 100% on my old M1 MacBook. Will investigate a bit more, maybe it's more because it's a fresh installation than because of the new hardware? 🤔 Will investigate a bit more...

ssh -T -v admin@192.168.64.4
OpenSSH_9.8p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/chris/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/100-macos.conf
debug1: /etc/ssh/ssh_config.d/100-macos.conf line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/crypto.conf
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to 192.168.64.4 [192.168.64.4] port 22.
debug1: connect to address 192.168.64.4 port 22: No route to host
ssh: connect to host 192.168.64.4 port 22: No route to host

edigaryev · 2024-11-22T10:19:17Z

@CodeStage are you invoking Packer from the CI?

This is likely related to the newly introduced "Local Network" permission in macOS Sequoia and the fact that Packer binary has no LC_UUID identifier, which is critical for the local network privacy mechanism.

One way to work around this is to either invoke Packer from the Terminal app, or re-built Packer with LC_UUID support:

git clone -b v1.11.2 https://github.com/hashicorp/packer.git
cd packer/
go build -ldflags="-linkmode=external" -o ~/packer .

Afterwards, invoke the resulting ~/packer binary from your CI, and it should result in a "Local Network" permission pop-up

CodeStage · 2024-11-22T11:49:12Z

Thanks for that hint! I was actually running it via Warp which is terminal app. Running it via Apple's terminal app works fine.

Update: After giving Warp the networking permission it worked too 🙈

edigaryev · 2024-11-22T15:25:22Z

hashicorp/packer#13214 should fix this in the next Packer release (assuming it'll be merged).

edigaryev · 2024-11-25T08:59:43Z

The fix is now merged into the Packer, however, it will only appear with the next Packer release (1.12.0 or so), scheduled for the January.

Meanwhile, you can either rebuild Packer to have LC_UUID identifier or use a nightly version which already has it.

edigaryev mentioned this issue Nov 22, 2024

packer 1.9.4 Homebrew/homebrew-core#198616

Closed

6 tasks

edigaryev pinned this issue Nov 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Occasional Timeout waiting for SSH #79

Occasional Timeout waiting for SSH #79

sanieldalib commented Jun 29, 2023 •

edited

Loading

fkorotkov commented Jun 30, 2023

sridhar-vadlamani-ck commented Jul 1, 2024

edigaryev commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024 •

edited

Loading

fkorotkov commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024 •

edited

Loading

fkorotkov commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 3, 2024

edigaryev commented Jul 3, 2024

sridhar-vadlamani-ck commented Jul 4, 2024

sridhar-vadlamani-ck commented Jul 5, 2024

edigaryev commented Jul 7, 2024 •

edited

Loading

gitperr commented Nov 14, 2024

gitperr commented Nov 14, 2024

CodeStage commented Nov 21, 2024 •

edited

Loading

edigaryev commented Nov 22, 2024 •

edited

Loading

CodeStage commented Nov 22, 2024 •

edited

Loading

edigaryev commented Nov 22, 2024

edigaryev commented Nov 25, 2024

Occasional Timeout waiting for SSH #79

Occasional Timeout waiting for SSH #79

Comments

sanieldalib commented Jun 29, 2023 • edited Loading

fkorotkov commented Jun 30, 2023

sridhar-vadlamani-ck commented Jul 1, 2024

edigaryev commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024 • edited Loading

fkorotkov commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024 • edited Loading

fkorotkov commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 3, 2024

edigaryev commented Jul 3, 2024

sridhar-vadlamani-ck commented Jul 4, 2024

sridhar-vadlamani-ck commented Jul 5, 2024

edigaryev commented Jul 7, 2024 • edited Loading

gitperr commented Nov 14, 2024

gitperr commented Nov 14, 2024

CodeStage commented Nov 21, 2024 • edited Loading

edigaryev commented Nov 22, 2024 • edited Loading

CodeStage commented Nov 22, 2024 • edited Loading

edigaryev commented Nov 22, 2024

edigaryev commented Nov 25, 2024

sanieldalib commented Jun 29, 2023 •

edited

Loading

sridhar-vadlamani-ck commented Jul 1, 2024 •

edited

Loading

sridhar-vadlamani-ck commented Jul 1, 2024 •

edited

Loading

edigaryev commented Jul 7, 2024 •

edited

Loading

CodeStage commented Nov 21, 2024 •

edited

Loading

edigaryev commented Nov 22, 2024 •

edited

Loading

CodeStage commented Nov 22, 2024 •

edited

Loading