terraspace build failing - `spawn': No such file or directory - file (Errno::ENOENT) #163

exoaturner · 2021-12-15T18:26:03Z

terraspace build failing - `spawn': No such file or directory - file (Errno::ENOENT)

Checklist

Upgrade Terraspace: Are you using the latest version of Terraspace? This allows Terraspace to fix issues fast. There's an Upgrading Guide: https://terraspace.cloud/docs/misc/upgrading/
Reproducibility: Are you reporting a bug others will be able to reproduce and not asking a question. If you're unsure or want to ask a question, do so on https://community.boltops.com
Code sample: Have you put together a code sample to reproduce the issue and make it available? Code samples help speed up fixes dramatically. If it's an easily reproducible issue, then code samples are not needed. If you're unsure, please include a code sample.

My Environment

Software	Version
Operating System	Ubuntu 20.04
Terraform	1.1.0
Terraspace	0.6.20
Ruby	2.7.0p0

Can recreate environment exactly using docker container ubuntu:20.04 and running the below commands.

Commands:

TF_VERSION="1.1.0"
TS_VERSION="0.6.20"

apt update -y && apt install -y jq curl unzip wget gnupg git ruby-dev build-essential

# Install AWS CLI
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
    && unzip awscliv2.zip \
    && ./aws/install \
    && rm -rf awscliv2.zip aws

# Install Terraform
wget https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip \
    && unzip terraform_${TF_VERSION}_linux_amd64.zip \
    && chmod +x terraform \
    && mv terraform /usr/bin/terraform \
    && rm terraform_${TF_VERSION}_linux_amd64.zip

gem install terraspace:${TS_VERSION}

Expected Behaviour

terraspace build demo results in terraspace cache being built.

Current Behavior

terraspace build demo results in the below error.

Traceback (most recent call last):
        22: from /usr/local/bin/terraspace:23:in `<main>'
        21: from /usr/local/bin/terraspace:23:in `load'
        20: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/exe/terraspace:14:in `<top (required)>'
        19: from /var/lib/gems/2.7.0/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'
        18: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/lib/terraspace/command.rb:61:in `dispatch'
        17: from /var/lib/gems/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
        16: from /var/lib/gems/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
        15: from /var/lib/gems/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
        14: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/lib/terraspace/cli.rb:56:in `build'
        13: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/lib/terraspace/builder.rb:13:in `run'
        12: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/lib/terraspace/builder.rb:77:in `clean'
        11: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/lib/terraspace/compiler/cleaner.rb:9:in `clean'
        10: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/lib/terraspace/compiler/cleaner.rb:16:in `backend_change_purge'
         9: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/lib/terraspace/compiler/cleaner/backend_change.rb:10:in `purge'
         8: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/lib/terraspace/compiler/cleaner/backend_change.rb:21:in `purge?'
         7: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/lib/terraspace/compiler/cleaner/backend_change.rb:40:in `fresh_backend'
         6: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/lib/terraspace/compiler/strategy/mod.rb:6:in `run'
         5: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/lib/terraspace/compiler/strategy/mod.rb:14:in `strategy_class'
         4: from /var/lib/gems/2.7.0/gems/terraspace-0.6.20/lib/terraspace/compiler/strategy/mod.rb:21:in `text_file?'
         3: from /usr/lib/ruby/2.7.0/open3.rb:390:in `capture2e'
         2: from /usr/lib/ruby/2.7.0/open3.rb:208:in `popen2e'
         1: from /usr/lib/ruby/2.7.0/open3.rb:213:in `popen_run'
/usr/lib/ruby/2.7.0/open3.rb:213:in `spawn': No such file or directory - file (Errno::ENOENT)

Step-by-step reproduction instructions

Setup docker container on ubuntu:20.04 installed software as explained in the "My Environment" section. Then follow the below:

git clone https://github.com/boltops-tools/terraspace-demo-terrafile
cd terraspace-demo-terrafile
terraspace bundle
terraspace build demo

Code Sample

Happens with your own demo repo...

Demo Repo: https://github.com/boltops-tools/terraspace-demo-terrafile

Solution Suggestion

Not really a suggestion or recommendation but installing nodejs seems to resolve the problem I am not sure why terraspace would require nodejs. If it does, it definitely not documented anywhere that it does.

curl -sL https://deb.nodesource.com/setup_16.x | bash \
    && apt update -y \
    && apt install -y nodejs

Also from a security perspective having lots of dependencies isn't ideal (larger attack surface).

The text was updated successfully, but these errors were encountered:

exoemay · 2021-12-16T09:06:26Z

In addition to this issue: We are seeing constant issues with randomly updating dependencies. It really seems ruby gems are very often updated without any concern of backwards compatibility. Is it possible to fix all dependencies to specific versions for each version of terraspace? We already needed to manually downgrade various gems to get terraspace to work. This is really not production ready at all, right now.

The above nodejs dependency is just the tip of the iceberg. Earlier we had issues with the 'faraday' gem and now with 'activesupport-7.0.0' as well.

What would help with this and make it possible to use in a real production environment is fixing the versions of all dependencies and providing clear guidelines on other external dependencies, like "build-essentials" (or c-compilers in general) and "nodejs".

General cleanup of dependencies would also be great, as 650MB of dependencies seems quite overkill for an application like terraspace.

tongueroo · 2021-12-16T15:27:56Z

@exoaturner Thanks for the detailed report. 👍 It was just enough info to figure this out.

Here's also a Dockerfile for posterity in case it's helpful in the future:

FROM ubuntu
ENV TF_VERSION="1.1.0"
RUN apt update -y && apt install -y jq curl unzip wget gnupg git ruby-dev build-essential

# Install AWS CLI
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
    && unzip awscliv2.zip \
    && ./aws/install \
    && rm -rf awscliv2.zip aws

# Install Terraform
RUN wget https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip \
    && unzip terraform_${TF_VERSION}_linux_amd64.zip \
    && chmod +x terraform \
    && mv terraform /usr/bin/terraform \
    && rm terraform_${TF_VERSION}_linux_amd64.zip

RUN gem install terraspace --no-document

The issue stems from

terraspace/lib/terraspace/compiler/strategy/mod.rb

Line 21 in 41bb84a

file_type, status = Open3.capture2e("file", filename)

Some folks have binary files in their terraform source code. IE: prepackaged zipped lambda functions. Ran into a while back and added a text or binary file check as part of the Terraspace compile processing.

This check uses the file command. Decided to take this route so didn't have to add another dependency actually: ruby-filemagic, which requires some development header libraries. IE: apt-get install libmagic-dev -y. Related: https://stackoverflow.com/questions/2355866/ruby-how-to-determine-if-file-being-read-is-binary-or-text

Ubuntu is a bare enough Docker image, so it doesn't even have the file command install. Bummer.

Terraspace is not dependent on nodejs. When node was installed it installed the file command as a dependency. So it was a side-effect of installing nodejs that fixed the issue.

In debugging this, installed vim and which also automatically installs file. So the issue seemly disappeared randomly. 🤦🏻‍♂️ Was a tricky one to track down since couldn't even use vim to edit files in the docker container. 🤣

Details of the fix are in: #165

tongueroo · 2021-12-16T17:23:39Z

@exoemay Yes. There are ways to lock down specific versions. Added docs to help: https://terraspace.cloud/docs/install/gems-locking/

Also, note, this comes from a good place. Am hoping there's little more positive energy here. @exoaturner provided a very detailed bug report and is using his own time, free of charge, to help. There will always be ways to improve software and Terraspace. Constructiveness helps motivate everyone to help. Additionally, it fosters a healthy community. Again, this comes from a good place. Hope the docs help and will consider PRs to improve them.

One thing that may help in the future with issues is providing a step-by-step reproduction of the issues. Example repos also help. They can save a tremendous amount of time in helping to fix issues. Even consider creating a PR. Also, added docs showing how to run a custom version of Terraspace to help https://terraspace.cloud/docs/install/gem/custom-version/ Of course, no sweat either way.

exoaturner added the bug Something isn't working label Dec 15, 2021

tongueroo added the help wanted Extra attention is needed label Dec 15, 2021

This was referenced Dec 16, 2021

Use Activesupport All #164

Merged

check if file command is installed #165

Merged

add bundler as a dependency #166

Merged

tongueroo closed this as completed Dec 16, 2021

tongueroo mentioned this issue Dec 27, 2021

Lighter terraspace core: remove terraspace_plugin_* gem dependencies out of core #173

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

terraspace build failing - `spawn': No such file or directory - file (Errno::ENOENT) #163

terraspace build failing - `spawn': No such file or directory - file (Errno::ENOENT) #163

exoaturner commented Dec 15, 2021 •

edited

Loading

exoemay commented Dec 16, 2021

tongueroo commented Dec 16, 2021 •

edited

Loading

tongueroo commented Dec 16, 2021 •

edited

Loading

terraspace build failing - `spawn': No such file or directory - file (Errno::ENOENT) #163

terraspace build failing - `spawn': No such file or directory - file (Errno::ENOENT) #163

Comments

exoaturner commented Dec 15, 2021 • edited Loading

terraspace build failing - `spawn': No such file or directory - file (Errno::ENOENT)

Checklist

My Environment

Expected Behaviour

Current Behavior

Step-by-step reproduction instructions

Code Sample

Solution Suggestion

exoemay commented Dec 16, 2021

tongueroo commented Dec 16, 2021 • edited Loading

tongueroo commented Dec 16, 2021 • edited Loading

exoaturner commented Dec 15, 2021 •

edited

Loading

tongueroo commented Dec 16, 2021 •

edited

Loading

tongueroo commented Dec 16, 2021 •

edited

Loading