Cannot pull password-protected HTTP URLs after merge of netrc

[frozenfoxx]

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: 6.19.1
• Ruby: 2.5.8p224
• Distribution: Ubuntu 18.04
• Module version: 4.6.0

How to reproduce (e.g Puppet code you use)

  # Pull archive for configuration and extract if newer
  archive { "${facts['networking']['hostname']}.tar.gz":
    ensure          => present,
    path            => "${datasourcedir}/${facts['networking']['hostname']}.tar.gz",
    source          => $archiveurl,
    username        => $archiveuser,
    password        => $archivepassword,
    extract         => true,
    extract_path    => "${datasourcedir}/tool",
    extract_command => 'tar xfz %s --strip-components=1',
    creates         => "${datasourcedir}/file_that_exists_after_extract",
    notify          => Docker::Run['tool']
  }

What are you seeing

Fails to download due to a 401 Unauthorized.

What behaviour did you expect instead

Download of the archive and extraction.

Output log

Error: Execution of '/usr/bin/curl https://company.jfrog.io/artifactory/project/team/tool/archives/fqdn.tar.gz -o /tmp/fqdn.tar.gz_20210116-9592-1yt2mha -fsSLg --max-redirs 5 --netrc-file /tmp/.pupp
et_archive_curl20210116-9592-4uamxq' returned 22: curl: (22) The requested URL returned error: 401
Error: /Stage[main]/Tool::Install/Archive[fqdn.tar.gz]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/curl https://company.jfrog.io/artifactory/project/team/tool/archives/fqdn.tar.gz -o /tmp/fqdn.tar.gz_20210116-9592-1yt2mha -fsSLg --max-redirs 5 --netrc-file /tmp/.puppet_archive_curl20210116-9592-4uamxq' returned 22: curl: (22) The requested URL returned error: 401

Any additional information you'd like to impart

This works perfect with 4.5.0. Looks like it's having some sort of an issue with the netrc file, but since it destroys the file after it fails I can't see what's going in it. I've had to rollback. Seems to be related to this change: #399.

[alexjfisher]

Thanks for the bug report. Are you able to help debug the issue? My first suggestion would be to temporarily disable the netrc file deletion to see if its content looks sane. eg. turn delete_netrcfile into a noop by removing the if from this line.

puppet-archive/lib/puppet/provider/archive/curl.rb

Line 29 in ac5a8a4

return if @netrc_file.nil?

I've not been able to reproduce the issue on ubuntu 20.04 with the following manifest and puppet apply

archive { '/tmp/test':
  source   => 'http://httpbin.org/basic-auth/user/passwd',
  username => 'user',
  password => 'passwd',
}

[frozenfoxx]

If I get a chance I can see if I can get this tested but I'm under a lot of pressure right now to keep these production systems online.

Don't suppose it'd be possible to have a debug flag on the module that leaves the netrc file?

[alexjfisher]

You should be able to reproduce your error using puppet apply and a simple manifest instead of a full agent run.

Something like (untested)
puppet module install --modulepath=/tmp/modules puppet/archive.

Make any change you want to the copy of the module in /tmp/modules and then
puppet apply -t --modulepath=/tmp/modules test.pp where test.pp is a file with just a single archive resource in it.
This should work fine as a non-root user.

[frozenfoxx]

I've run a fresh test, looks like the file isn't being created I think?

Output:

root@fqdn.company.net:/tmp# puppet apply -t --modulepath=/tmp/modules test.pp
Info: Loading facts
Info: Loading facts
Notice: Compiled catalog for fqdn.company.net in environment production in 0.07 seconds
Info: Applying configuration version '1611095716'
Error: Execution of '/usr/bin/curl https://company.jfrog.io/artifactory/project/team/tool/archives/fqdn.tar.gz -o /tmp/fqdn.tar.gz_20210119-22096-rofq3l -fsSLg --max-redirs 5 --netrc-file /tmp/.puppet_archive_curl20210119-22096-uazm9t' returned 22: curl: (22) The requested URL returned error: 401
Error: /Stage[main]/Main/Archive[fqdn.tar.gz]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/curl https://company.jfrog.io/artifactory/project/team/tool/archives/fqdn.tar.gz -o /tmp/fqdn.tar.gz_20210119-22096-rofq3l -fsSLg --max-redirs 5 --netrc-file /tmp/.puppet_archive_curl20210119-22096-uazm9t' returned 22: curl: (22) The requested URL returned error: 401

Code

/tmp/modules/archive/lib/puppet/provider/archive/curl.rb

[...]
  def delete_netrcfile
    #return if @netrc_file.nil?
    return

    #@netrc_file.unlink
    #@netrc_file = nil
  end
[...]

/tmp/test.pp

# Pull archive for configuration and extract if newer
archive { "${facts['networking']['hostname']}.tar.gz":
  ensure          => present,
  path            => "/tmp/${facts['networking']['hostname']}.tar.gz",
  source          => 'https://company.jfrog.io/artifactory/project/team/tool/archives/fqdn.tar.gz',
  username        => 'serviceaccountusername',
  password        => 'serviceaccountpassword',
  extract         => true,
  extract_path    => "/tmp/tool",
  extract_command => 'tar xfz %s --strip-components=1',
  creates         => "/tmp/tool/services-config.json"
}

[alexjfisher]

Ah. My bad. Since it was created with Tempfile it'll still be automatically deleted when the ruby process exits, (which will happen with puppet apply).

Maybe in delete_netrcfile copy it somewhere??

[alexjfisher]

Does your password contain spaces?? I'm guessing it's going to be something like that.

[alexjfisher]

If that is the issue, I can probably fix it. Worse case, we just disable the use of netrc for username/password with spaces in. (There is no way of escaping them it would seem).

Something like...

--- a/lib/puppet/provider/archive/curl.rb
+++ b/lib/puppet/provider/archive/curl.rb
@@ -7,8 +7,14 @@ Puppet::Type.type(:archive).provide(:curl, parent: :ruby) do
 
   def curl_params(params)
     if resource[:username]
-      create_netrcfile
-      params += ['--netrc-file', @netrc_file.path]
+      if resource[:username] =~ %r{\s} || resource[:password] =~ %r{\s}
+        Puppet.warning('Username or password contains a space.  Unable to use netrc file to hide credentials')
+        account = [resource[:username], resource[:password]].compact.join(':')
+        params += optional_switch(account, ['--user', '%s'])
+      else
+        create_netrcfile
+        params += ['--netrc-file', @netrc_file.path]
+      end
     end
     params += optional_switch(resource[:proxy_server], ['--proxy', '%s'])
     params += ['--insecure'] if resource[:allow_insecure]

[frozenfoxx]

Negative, password has no spaces. It's a single HTTP API token.

[cdenneen]

Having same issue... but wondering if this is a windows issue in our case or not. I know Powershell requires the creds to be added as a -Credential parameter to Invoke-WebRequest... so not sure.

Basically we are trying to forcibly override the Archive used in @pcfens https://github.com/pcfens/puppet-filebeat with something like:

Archive <| title == "C:/Windows/Temp/filebeat-6.8.17-windows-x86_64.zip" |>{
  username => 'puppet-read',
  password => $token,
}

[cdenneen]

Think this is the underlying issue... somewhere its getting 2 auth mechanisms that are being passed to JFrog which is passing to S3 and failing:

Unexpected response code 400: <?xml version="1.0" enco
ding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; onl
y the X-Amz-Algorithm query parameter, Signature query string parameter or the A
uthorization header should be specified</Message><ArgumentName>Authorization</Ar
gumentName><ArgumentValue>Basic XXXX</ArgumentValue><RequestId>YYYY<
/RequestId><HostId>ZZZZZ</HostId></Error>