CLI ignoring certificate-related flags

[joeshaw]

Using Vault v0.7.3 ('0b20ae0b9b7a748d607082b1add3663a28e31b68') on macOS 10.12.5 installed from Homebrew.

I am really stumped by the fact that the CLI seems to be ignoring certificate related flags, including -ca-cert, -client-cert, -client-key, and -tls-skip-verify.

I am following the directions from https://www.vaultproject.io/docs/auth/cert.html#authentication.
The instructions for curl work fine, but for the vault CLI I always get:

Put https://127.0.0.1:8200/v1/auth/cert/login: x509: certificate signed by unknown authority

The command-line I'm using is an exact copy from the docs above, except with different paths for the various files.

Passing in -tls-skip-verify has no effect, but setting VAULT_SKIP_VERIFY=1 does, and results in:

Error making API request.

URL: PUT https://127.0.0.1:8200/v1/auth/cert/login
Code: 400. Errors:

* client certificate must be supplied

So it seems like the CLI is not even seeing that I have set those flags. I ran dtruss on the command, and the files I am passing in for CA cert, client cert and client key are neither stat64()ed nor open()ed.

I've tried both vault auth and vault list commands without success.

Things work fine if I set the VAULT_CACERT, VAULT_CLIENT_CERT and VAULT_CLIENT_KEY environment variables.

[jefferai]

Do you see the same issues with the official binary from https://releases.hashicorp.com/vault/0.7.3/vault_0.7.3_darwin_amd64.zip ?

[joeshaw]

@jefferai yes, I do.

[chrishoffman]

I have been trying to recreate this issue but have been unable to do so. I used the vault auth command and curl commands and both successfully logged in and returned a client token. Is there any way you can provide a script that recreates the issue and possibly include sample certificates?

[inkblot]

I have been trying to authenticate using client certificates as well and running into a similar (possibly the same) issue with vault 0.8.3.

Server setup:

• TLS enabled
• CA certificates for the server and clients (these are separate CAs) are in /etc/ssl/certs/ca-certificates.crt
• Policy dorothy exists
• TLS auth enabled at path puppet
• One client certificate added: vault write auth/puppet/certs/dorothy name=dorothy policies=dorothy [email protected] ttl=900

Client setup:

• CA certificates for the server and client are in /etc/ssl/certs/ca-certificates.crt
• VAULT_ADDR uses https and vault commands work using other auth methods

I get an error about the client certificate when I attempt to authenticate, and no attempt is made to connect to the vault server:

inkblot@dorothy:~/certs]$ pwd
/home/inkblot/certs
[inkblot@dorothy:~/certs]$ ls -l
total 8
-r-------- 1 inkblot inkblot 3243 Nov  5 08:33 dorothy.key
-r-------- 1 inkblot inkblot 2057 Nov  5 08:33 dorothy.pem
[inkblot@dorothy:~/certs]$ head -1 dorothy.*
==> dorothy.key <==
-----BEGIN RSA PRIVATE KEY-----

==> dorothy.pem <==
-----BEGIN CERTIFICATE-----
[inkblot@dorothy:~/certs]$ vault auth -method=cert -path=puppet -client-cert=/home/inkblot/certs/dorothy.pem -client-key=/home/inkblot/certs/dorothy.key name=dorothy
Error making API request.

URL: PUT https://vault.movealong.internal:8200/v1/auth/puppet/login
Code: 400. Errors:

* client certificate must be supplied

Strangely, I get a different error if I reference the original paths for the certificate and key (note, this is a puppet agent certificate):

[inkblot@dorothy:~/certs]$ sudo vault auth -address=https://vault.movealong.internal:8200 -method=cert -path=puppet -client-cert=/etc/puppetlabs/puppet/ssl/certs/dorothy.movealong.org.pem -client-key=/etc/puppetlabs/puppet/ssl/private_keys/dorothy.movealong.org.pem name=dorothy
Error making API request.

URL: PUT https://vault.movealong.internal:8200/v1/auth/puppet/login
Code: 400. Errors:

* invalid certificate or no client certificate supplied

In this case, there is connection made to the server but nothing is logged there.

[jefferai]

@inkblot in the first example, are those symlinks? It's possible we don't follow symlinks for those flags; if so, I'm not sure if there is a reason for that.

Are you using HA? If so is the second request against a standby or active node?

[inkblot]

@jefferai They are copies, not symlinks. Ownership and permissions have been altered, but otherwise they're the same.

HA: There are two instances, but I'm not sure I'm doing it right. Wouldn't that just produce hit or miss results if that were the cause? I'm getting consistent errors.

[inkblot]

More to the point on HA...

I'm using a consul backend. I have a consul-template configured nginx reverse proxy that is forwarding tcp streams to vault. The rendered nginx configuration has an upstream for vault that points to exactly one of the two actual vault instances. I can confirm by querying consul that this node is marked active and the other standby. All of my vault client activity is through the nginx proxy to the active vault node. The thing that makes me suspect I'm not doing it right is that there are a large number of errors logged in the standby node that I haven't bothered to look into yet.

[jefferai]

What are those errors?

[inkblot]

The errors on the standby are irrelevant. I have shut the standby down and I still get the same client error when I try to use TLS auth.

[jefferai]

I can't reproduce on master:

$ vault write auth/cert/certs/web display_name=web certificate=@/tmp/vault-test-cluster-150430614/ca_cert.pem
Success! Data written to: auth/cert/certs/web

# tls-skip-verify works:
$ vault auth -method=cert
Put https://127.0.0.1:34081/v1/auth/cert/login: x509: certificate signed by unknown authority
$ vault auth -method=cert -tls-skip-verify
Error making API request.

URL: PUT https://127.0.0.1:34081/v1/auth/cert/login
Code: 400. Errors:

* client certificate must be supplied

# Authing works:
 vault auth -method=cert -tls-skip-verify -client-cert=/tmp/vault-test-cluster-150430614/node2_port_37457_cert.pem -client-key=/tmp/vault-test-cluster-150430614/node2_port_37457_key.pem
Successfully authenticated! You are now logged in.
The token below is already saved in the session. You do not
need to "vault auth" again with the token.
token: ecb96469-d6a3-9395-705b-0411cc9fca30
token_duration: 2764800
token_policies: [default]

# Nothing in env vars, it all came from the CLI:
$ env | grep VAULT
VAULT_ADDR=https://127.0.0.1:34081

# Works from a standby too:
$ VAULT_ADDR=https://127.0.0.1:44061 vault auth -method=cert -tls-skip-verify -client-cert=/tmp/vault-test-cluster-150430614/node2_port_37457_cert.pem -client-key=/tmp/vault-test-cluster-150430614/node2_port_37457_key.pem
Successfully authenticated! You are now logged in.
The token below is already saved in the session. You do not
need to "vault auth" again with the token.
token: 2e68c6e5-21e8-a817-2d17-3e4b7389b08b
token_duration: 2764800
token_policies: [default]

[inkblot]

It looks like you're supplying a CA cert as the client certificate. I assume that means it's self-signed. Does vault require that the client certificate is self-signed? What about key usage?

[jefferai]

I am not using a CA cert as the client cert, I'm using one with server/client usage.

[inkblot]

Pardon, I misread the args. Is the CA certificate for that client cert known to vault?

[jefferai]

Yes, the same CA issued both the client and server certs.

[inkblot]

Is that a requirement?

[jefferai]

No, it is not.