Facter still tries to access EC2 metadata service with EC2 in the blocklist

[robert-cam]

We're not running in a cloud environment and don't want or need facter to hit metadata services. The docs say adding EC2 will prevent the facts being resolved but facter still tries to access the metadata services.

Testing with facter 4.2.8 on CentOS 7.9 and AlmaLinux 8.9.

[joshcooper]

@robert-cam can you try with facter 4.5.2 and include the output of facter --debug (and sanitize any sensitive data)

[robert-cam]

facter-debug.txt

[joshcooper]

[2024-01-24 11:22:32.497992 ] DEBUG Facter::Core::Execution::Posix - Executing command: virt-what
[2024-01-24 11:22:32.499457 ] DEBUG Facter::Core::Execution::Posix - Failed while executing 'virt-what': No such file or directory - virt-what
[2024-01-24 11:22:32.499784 ] DEBUG Facter::Core::Execution::Posix - Executing command: lspci
[2024-01-24 11:22:32.517611 ] DEBUG Facts::Linux::Hypervisors::Vmware - No Vmware hypervisor detected.
...
[2024-01-24 11:22:32.518215 ] DEBUG Facter::Core::Execution::Posix - Executing command: vmware -v
[2024-01-24 11:22:32.519675 ] DEBUG Facter::Core::Execution::Posix - Failed while executing 'vmware -v': No such file or directory - vmware
[2024-01-24 11:22:32.519825 ] DEBUG Facter::Core::Execution::Posix - Executing command: virt-what
[2024-01-24 11:22:32.521189 ] DEBUG Facter::Core::Execution::Posix - Failed while executing 'virt-what': No such file or directory - virt-what
[2024-01-24 11:22:32.521600 ] DEBUG Facter::Util::FileHelper - File at: /proc/xen/capabilities is not accessible.
[2024-01-24 11:22:32.521709 ] DEBUG Facter::Core::Execution::Posix - Executing command: ["/usr/sbin/xl", "/usr/sbin/xm"] list
[2024-01-24 11:22:32.523998 ] DEBUG Facter::Resolvers::Xen - Command ["/usr/sbin/xl", "/usr/sbin/xm"] list completed with the following stderr message: sh: [/usr/sbin/xl,: No such file or directory
[2024-01-24 11:22:32.524088 ] DEBUG Facts::Linux::Hypervisors::Xen - No Xen hypervisor detected.
...
[2024-01-24 11:22:32.593064 ] DEBUG Facts::Linux::Virtual - Linux Virtual Resolver
[2024-01-24 11:22:32.593103 ] DEBUG Facts::Linux::Virtual - Fact value is: kvm
...
[2024-01-24 11:22:33.068586 ] DEBUG Facter::Resolvers::Ec2 - Querying Ec2 metadata
[2024-01-24 11:22:33.688348 ] DEBUG Facter::Util::Resolvers::Http - Trying to connect to http://169.254.169.254/latest/api/token but got: execution expired

[lemkep]

Hi. I'm on Debian bookworm with facter 4.5.2 and blocklist does not work for me, too:

# facter --version
4.5.2
# grep mountpoints facter.stderr | cut -c-140
[2024-02-05 16:22:32.529530 ] DEBUG Facter - blocking collection of mountpoints facts 
[2024-02-05 16:22:39.751844 ] DEBUG Facter::FactManager - fact "mountpoints" has resolved to: {"/dev"=>{"device"=>"udev", "filesystem"=

[github-actions]

Migrated issue to FACT-3454

[joshcooper]

@robert-cam @lemkep Could you try installing earlier puppet-agent/facter versions to see if this is a regression?

[robert-cam]

This happens with facter 4.0.52 from the puppet-agent 7.5.0 package:

# facter  --debug 2>&1 |egrep '(EC2|http)'
[2024-02-09 10:07:09.140796 ] DEBUG Facter - blocking collection of EC2 az_metadata facts 
[2024-02-09 10:07:09.507029 ] DEBUG Facter::Util::Resolvers::Http - Request to http://169.254.169.254/metadata/instance?api-version=2020-09-01 failed with error code 403 
# facter -v
4.0.52

Doesn't happen with facter 4.0.51 from the puppet-agent 7.4.1 package:

# facter  --debug 2>&1 |egrep '(EC2|http)'
[2024-02-09 10:09:51.123153 ] DEBUG Facter - blocking collection of EC2 az_metadata facts 
# facter -v
4.0.51

[lemkep]

Seems to be a different issue for me. It is just the case when using puppet libs with '-p', also with an older versions (tested back until 4.2.10).

Without blocklist:

# facter --debug |& egrep "version:|mountpoints" | cut -c -120
[2024-02-09 14:15:30.142853 ] DEBUG Facter - Facter version: 4.4.1
[2024-02-09 14:15:30.646989 ] DEBUG Facter::FactManager - fact "mountpoints" has resolved to: {"/dev"=>{"device"=>"
mountpoints => {
#

With blocklist (blocks):

# facter --debug -c facter.conf |& egrep "version:|mountpoints" | cut -c -120
[2024-02-09 14:15:48.510386 ] DEBUG Facter - Facter version: 4.4.1 
[2024-02-09 14:15:48.510416 ] DEBUG Facter - blocking collection of mountpoints facts
#

With blocklist and '-p':

# facter --debug -p -c facter.conf |& egrep "version:|mountpoints" | cut -c -120
[2024-02-09 14:13:14.748915 ] DEBUG Facter - Facter version: 4.4.1 
[2024-02-09 14:13:14.748946 ] DEBUG Facter - blocking collection of mountpoints facts
[2024-02-09 14:13:21.819326 ] DEBUG Facter::FactManager - fact "mountpoints" has resolved to: {"/dev"=>{"device"=>"
mountpoints => {
#

[tvpartytonight]

@robert-cam could your provide what your facter config file looks like? thanks!

[robert-cam]

I just have a basic config for testing this:

facts : {
    blocklist : [ "EC2" ],
}

[tvpartytonight]

@robert-cam in your example of the issue happening in the 7.5.0 puppet-agent package, that http request is actually coming from the azure resolver, not the EC2 resolver. It looks like you were already trying to block az_metadata in your examples earlier in this thread. Based upon the fragments of this thread, maybe blocking azure_metadata is not working? I haven't been able to reproduce any failures to block the EC2 metadata, but I haven't been looking into blocking azure data.

[robert-cam]

Apologies, I was testing on 2 different VMs and one has this config:

facts : {
    blocklist : [ "EC2", "az_metadata" ],
}

I've updated and they both have the same facter.conf now.

I've checked and the VM I was using to test all the different facter versions was configured to block az_metadata.

[tvpartytonight]

@robert-cam I am going to make the assumption that there is no breakage like you described above between facter versions 4.0.51 and 4.0.52, based upon our az_metadata conversation. However, I am still investigating the EC2 issue you are seeing, but have had no luck reproducing it. I put up a branch on my fork of facter here: https://github.com/tvpartytonight/facter/tree/add_block_filtering_debug_comments

Do you think that you could run facter from that branch in your VM and share the debug output? You may have to do some installation of packages to allow for bundle install to work correctly; I outlined some steps here which should be what is neccesary to run facter from git in your VM.

[robert-cam]

@tvpartytonight here's the debug output with your fork:

# bundle exec facter --debug -p 2>&1 | egrep -i "(EC2|aws|http)"
[2024-03-11 11:00:06.884102 ] DEBUG Facter - blocking collection of EC2 az_metadata facts 
[2024-03-11 11:00:06.936254 ] DEBUG Facter::Resolvers::Ec2 - Querying Ec2 metadata 
[2024-03-11 11:00:16.975009 ] DEBUG Facter::Util::Resolvers::Http - Trying to connect to http://169.254.169.254/latest/api/token but got: Net::ReadTimeout with #<Socket:(closed)> 
[2024-03-11 11:00:26.993533 ] DEBUG Facter::Util::Resolvers::Http - Trying to connect to http://169.254.169.254/latest/meta-data/ but got: Net::ReadTimeout with #<Socket:(closed)> 
[2024-03-11 11:00:37.016814 ] DEBUG Facter::Util::Resolvers::Http - Trying to connect to http://169.254.169.254/latest/user-data/ but got: Net::ReadTimeout with #<Socket:(closed)> 
# bundle exec facter --version
4.6.1
# facter virtual
kvm

[tvpartytonight]

Ahh, sorry @robert-cam I accidentally left my changes on my local fork; can you try again? We are looking for the debug statements that are added here: main...tvpartytonight:facter:add_block_filtering_debug_comments

[robert-cam]

@tvpartytonight ok, debug output attached.
debug.txt

[tvpartytonight]

@robert-cam I realize I made a mistake in that branch which I have now fixed; would you mind fetching the latest for that branch and retrying?

[robert-cam]

@tvpartytonight sure, see attached.
debug.txt

[tvpartytonight]

Ok @robert-cam AFAICT there's nothing wrong with the blocking of facts happening; I added some more debug about the resolution of searched_facts in the branch; can you pull in the changes and share the debug output again?

[robert-cam]

@tvpartytonight done, see attached.
debug.txt
Thanks for looking, it's much appreciated!

[tvpartytonight]

@robert-cam can you run a test for me on the main branch of puppetlabs/facter? I'd like you to add cloud.provider to the list of facts to block, and see if that eliminates the calls to ec2 during facter execution.

[tvpartytonight]

@robert-cam I opened up #2690 to track what I think is really the issue here; I'm going to close this issue out as I think that your issue can be resolved by further blocking the cloud.provider fact. Please let me know if you have any further questions.

[robert-cam]

Confirmed, blocking cloud.provider works for me. Thanks!