updated dd-hander chef-handler to use the web_proxy from agent config

[datwiz]

Updated dd-handler chef-handler to use web_proxy attributes, if set.
Enables the datadog handler to post via the configured proxy without requiring http_proxy env var's to be set prior to the chef-client run.

[miketheman]

Awesome stuff! Thanks for the contribution.

Do you know if using this pattern, the environment variable persists for other applications/services that might look for this variable during a Chef run? Env cars can be considered 'global' state.

[datwiz]

I suspect it does as I had to place the web_proxy settings after the chef_gem section. Otherwise the gem install also tries to use the http_proxy setting and fails to connect to our local gem server.

[miketheman]

Ok, so I can confirm that setting the envvar during a Chef run will persist only for the duration of the Chef run - not beyond, so any application running outside of Chef's environment won't have this setting.

However, this has the unintended behavior of then using the same environment variable for resources like remote_file, one resource that uses HTTP_PROXY (even though it recommends against it).

$ chef-shell
...
 Chef Version: 11.16.4
...
chef > recipe_mode
chef:recipe > ENV['HTTP_PROXY'] = 'https://foobar.com'
 => "https://foobar.com"
chef:recipe > remote_file '/tmp/.rspec' do
chef:recipe >   source 'https://raw.githubusercontent.com/DataDog/chef-datadog/master/.rspec'
chef:recipe ?> end
 => <remote_file[/tmp/.rspec] @name: "/tmp/.rspec" @noop: nil @before: nil @params: {} @provider: Chef::Provider::RemoteFile @allowed_actions: [:nothing, :create, :delete, :touch, :create_if_missing] @action: "create" @updated: false @updated_by_last_action: false @supports: {} @ignore_failure: false @retries: 0 @retry_delay: 2 @source_line: "(irb#1):2:in `irb_binding'" @guard_interpreter: :default @elapsed_time: 0 @sensitive: false @resource_name: :remote_file @path: "/tmp/.rspec" @backup: 5 @atomic_update: true @force_unlink: false @manage_symlink_source: nil @diff: nil @source: ["https://raw.githubusercontent.com/DataDog/chef-datadog/master/.rspec"] @use_etag: true @use_last_modified: true @ftp_active_mode: false @headers: {} @cookbook_name: nil @recipe_name: nil>
chef:recipe > run_chef
[2015-05-29T13:32:05+00:00] INFO: Processing remote_file[/tmp/.rspec] action create ((irb#1) line 2)
[2015-05-29T13:32:05+00:00] DEBUG: touching /tmp/.rspec to create it
[2015-05-29T13:32:05+00:00] INFO: remote_file[/tmp/.rspec] created file /tmp/.rspec
[2015-05-29T13:32:05+00:00] DEBUG: remote_file[/tmp/.rspec] checking for changes
[2015-05-29T13:32:05+00:00] INFO: Unable to access cache at /var/chef. Switching cache to /home/michael/.chef
[2015-05-29T13:32:05+00:00] DEBUG: Cache control headers: {}
[2015-05-29T13:32:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_request
[2015-05-29T13:32:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_request
[2015-05-29T13:32:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_request
[2015-05-29T13:32:05+00:00] DEBUG: Initiating GET to https://raw.githubusercontent.com/DataDog/chef-datadog/master/.rspec
[2015-05-29T13:32:05+00:00] DEBUG: ---- HTTP Request Header Data: ----
[2015-05-29T13:32:05+00:00] DEBUG: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
[2015-05-29T13:32:05+00:00] DEBUG: ---- End HTTP Request Header Data ----
The environment variable HTTP_PROXY is discouraged.  Use http_proxy.
[2015-05-29T13:32:05+00:00] INFO: Unable to access cache at /var/chef. Switching cache to /home/michael/.chef
[2015-05-29T13:32:05+00:00] INFO: Unable to access cache at /var/chef. Switching cache to /home/michael/.chef
[2015-05-29T13:32:05+00:00] INFO: Unable to access cache at /var/chef. Switching cache to /home/michael/.chef
[2015-05-29T13:32:05+00:00] INFO: Unable to access cache at /var/chef. Switching cache to /home/michael/.chef

================================================================================
Error executing action `create` on resource 'remote_file[/tmp/.rspec]'
================================================================================

EOFError
--------
end of file reached

Resource Declaration:
---------------------


Compiled Resource:
------------------
# Declared in (irb#1):2:in `irb_binding'

remote_file("/tmp/.rspec") do
  provider Chef::Provider::RemoteFile
  action "create"
  retries 0
  retry_delay 2
  guard_interpreter :default
  path "/tmp/.rspec"
  backup 5
  atomic_update true
  source ["https://raw.githubusercontent.com/DataDog/chef-datadog/master/.rspec"]
  use_etag true
  use_last_modified true
end

[2015-05-29T13:32:05+00:00] INFO: Running queued delayed notifications before re-raising exception
EOFError: remote_file[/tmp/.rspec] ((irb#1) line 2) had an error: EOFError: end of file reached
    from /usr/lib/ruby/2.1.0/openssl/buffering.rb:182:in `read_nonblock'
    from /usr/lib/ruby/2.1.0/openssl/buffering.rb:182:in `sysread_nonblock'
    from /usr/lib/ruby/2.1.0/openssl/buffering.rb:182:in `read_nonblock'
    from /usr/lib/ruby/2.1.0/net/protocol.rb:153:in `rbuf_fill'
    from /usr/lib/ruby/2.1.0/net/protocol.rb:134:in `readuntil'
    from /usr/lib/ruby/2.1.0/net/protocol.rb:144:in `readline'
    from /usr/lib/ruby/2.1.0/net/http/response.rb:39:in `read_status_line'
    from /usr/lib/ruby/2.1.0/net/http/response.rb:28:in `read_new'
    from /usr/lib/ruby/2.1.0/net/http.rb:915:in `connect'
    from /usr/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
    from /usr/lib/ruby/2.1.0/net/http.rb:852:in `start'
    from /usr/lib/ruby/2.1.0/net/http.rb:1375:in `request'
    from /var/lib/gems/2.1.0/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in `request'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/http/basic_client.rb:65:in `request'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/http.rb:260:in `block in send_http_request'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/http.rb:294:in `block in retrying_http_errors'
... 12 levels...
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/provider.rb:121:in `run_action'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/resource.rb:648:in `run_action'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/runner.rb:49:in `run_action'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/runner.rb:81:in `block (2 levels) in converge'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/runner.rb:81:in `each'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/runner.rb:81:in `block in converge'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/resource_collection.rb:98:in `block in execute_each_resource'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/resource_collection.rb:96:in `execute_each_resource'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/runner.rb:80:in `converge'
    from /var/lib/gems/2.1.0/gems/chef-11.16.4/lib/chef/shell/ext.rb:236:in `run_chef'
    from (irb#1):5chef:recipe >
 => :recipe

Now, in this test, there's no actual proxy server, so real behavior might differ. But the point remains that the use of the environment variable will likely affect other resources.

Can you think of another way to implement this change with something that doesn't affect global state of a Chef run

[datwiz]

I did consider a safer alternate method, which is setting a new env var specifically for the DD web proxy, something like DD_WEB_PROXY.
I had thought the alternate method would require a mod to the dogapi-rb connection code, but digging a bit more, it may be possible make a mod in chef-handler-datadog to wrap the emit calls with a set/revert of an HTTP_PROXY env var that gets passed through into the dogapi methods.
I should probably also confirm the precedence of the ruby Net::HTTP/URI::Generic find_proxy method to make the first var gets picked. HTTPS_PROXY may get picked first.
I'll try out the chef-handler-datadog approach.

[datwiz]

Change has been modified to pass the datadog proxy information to the report handler as a new ENV variable 'DATADOG_PROXY'. A corresponding pull request has also been added to the chef-handler-datadog project to use the new proxy variable on chef handler report runs. The second change also preserves the pre-call values of the environment proxy settings.
Refer to issue DataDog/chef-handler-datadog#64.

[miketheman]

Thanks!