≡ Menu

How to Perform Chef Knife SSL Check and Fetch to Verify Certificate

Chef LogoStarting from Chef server version 12, SSL verification is enabled by default for all the communication that are initiated from any of the chef utilities (knife, chef-client).

For this, Chef server will generate a self-signed SSL certificate during the installation process.

From your Chef workstation when you execute knife command, you should download the SSL certificate before you can get started. If you don’t do this, knife command will not connect to the Chef server.

Why Chef SSL Certificate?

Starting from Chef server version 12, when you perform a simple command like knife client list, it will display the SSL validation failure error message, as the workstation don’t have the server’s SSL certificate for verification.

# knife client list
ERROR: SSL Validation failure connecting to host: datadb - SSL_connect returned=1 errno=0 state=error: certificate verify failed
ERROR: Could not establish a secure connection to the server.
Use `knife ssl check` to troubleshoot your SSL configuration.
If your Chef Server uses a self-signed certificate, you can use
`knife ssl fetch` to make knife trust the server's certificates.
Original Exception: OpenSSL::SSL::SSLError: SSL Error connecting to https://datadb/organizations/thegeekstuff/clients - SSL_connect returned=1 errno=0 state=error: certificate verify failed

Even the chef-client that runs on the individual nodes will need the SSL certificate.

To avoid this SSL error, we should download the self-signed SSL certificate that is generated by the Chef server and store it in the trusted store on the chef client machines including the workstations and individual nodes.

On the Chef workstation when the SSL certificate is installed properly, it will not display the SSL error message when executing knife command.

# knife client list
devdb
proddb
devapp
prodapp

On a high-level, you have to do the following. All of these are explained in this tutorial along with possible error messages that you might get and how to resolve those:

  • On your chef workstation, download the chef server’s SSL certificate
  • On all your individual nodes, make sure a valid chef server’s certificate is installed as part of the bootstrap.
  • Also, verify the check-sum of the downloaded chef server’s SSL certificate.

Note: If you are new to knife command, this might help: 12 Chef Knife Cookbook Command Examples

Knife SSL Check – Error 1 (Invalid chef-repo Directory)

You can use the knife ssl check command to verify whether the Chef SSL certificate is installed properly on your local machine.

You should execute ssl check command inside the chef repository directory. If not, you’ll get the “No knife configuration file found” error message as shown below.

At this stage, this has nothing to do with unable to verify SSL certificate. The following error message is because you are inside a wrong directory when executing the knife ssl check command.

# knife ssl check
WARNING: No knife configuration file found
Connecting to host localhost:443
ERROR: The SSL cert is signed by a trusted authority but is not valid for the given hostname
ERROR: You are attempting to connect to:   'localhost'
ERROR: The server's certificate belongs to 'datadb'

TO FIX THIS ERROR:

The solution for this issue depends on your networking configuration. If you
are able to connect to this server using the hostname datadb
instead of localhost, then you can resolve this issue by updating chef_server_url
in your configuration file.

If you are not able to connect to the server using the hostname datadb
you will have to update the certificate on the server to use the correct hostname.

So, to fix this particular error message, simply cd to chef-repo and execute the knife ssl check command as shown below.

cd ~/chef-repo
knife ssl check

Knife SSL Check – Error 2 (No SSL Certificate Found)

In the following knife ssl check command example, the output indicates that on this Chef workstation, the Chef server’s SSL certificate is not installed.

# cd ~/chef-repo

# knife ssl check
Connecting to host datadb:443
ERROR: The SSL certificate of datadb could not be verified
Certificate issuer data: /C=US/O=YouCorp/OU=Operations/CN=datadb

Configuration Info:

OpenSSL Configuration:
* Version: OpenSSL 1.0.1s  1 Mar 2016
* Certificate file: /opt/chefdk/embedded/ssl/cert.pem
* Certificate directory: /opt/chefdk/embedded/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/root/chef-repo/.chef/trusted_certs"

TO FIX THIS ERROR:

If the server you are connecting to uses a self-signed certificate, you must
configure chef to trust that server's certificate.

By default, the certificate is stored in the following location on the host
where your chef-server runs:

  /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt

Copy that file to your trusted_certs_dir (currently: /root/chef-repo/.chef/trusted_certs)
using SSH/SCP or some other secure method, then re-run this command to confirm
that the server's certificate is now trusted.

When you execute the ssl check command, it will look for the certificate under .chef/trusted_certs directory. If it finds it there, it will verify to make sure it is a valid X.590 SSL certificate. If it can’t find a valid certificate, it will throw the above error message.

Knife SSL Fetch – Error 1 (Network Error – Connection Refused)

Knife ssl fetch command will get the certificate from the Chef server and install it on the individual workstation or node where you are executing the fetch command.

During knife ssl fetch, you might get the following Network error connection refused message as shown below.

# knife ssl fetch
WARNING: Certificates from datadb will be fetched and placed in your trusted_cert
directory (/root/chef-repo/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

ERROR: Network Error: Connection refused - connect(2) for "datadb" port 443
Check your knife configuration and network settings

The same network error connection refused error message for “knife client list” also. So, this error has to do with some unable to resolve/reach using proper name “datadb”.

When you get the above connection refused error message, the same thing will happen for other knife commands also as shown below.

# knife client list
ERROR: Connection refused connecting to https://datadb/organizations/thegeekstuff/clients, retry 1/5
ERROR: Connection refused connecting to https://datadb/organizations/thegeekstuff/clients, retry 2/5

This means that your Chef workstation or individual node is unable to initiate network communication to the Chef server.

To fix this error, you have to make sure your server is using a fully qualified domain name. In the following example, we don’t see the FQDN setup properly.

# hostname -f
datadb

Also, to fix this error message, in my example, my /etc/hosts file looked like the following. Basically you have to make sure both your hostname and FQDN properly setup.

In the following example, the ip-address in the last line the ip-address of the datadb server.

# cat /etc/hosts
127.0.0.1 datadb.thegeekstuff.com datadb
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.2 datadb.thegeekstuff.com datadb

Now, when I execute the hostname -f command, I’ll get the FQDN as shown below.

# hostname -f
datadb.thegeekstuff.com

After the above changes to the /etc/hosts file, now you shouldn’t get the Network Error connection refused message anymore from the knife command.

Knife Fetch SSL certificate and Store it

Finally, use the knife ssl fetch command as shown below to retrieve the SSL certificate from the Chef server and store it to the .chef/trusted_certs directory on the workstation or the individual nodes by either the knife or chef-client command.

The location of where the certificate will be stored when you execute the knife ssl fetch command will be determined by the trusted_certs_dir parameter in the /etc/chef/client.rb file

The following is the output of the knife ssl fetch command when everything is properly working. As shown in the following output, it will download the certificate and store it in the trusted_certs directory.

# knife ssl fetch
WARNING: Certificates from datadb will be fetched and placed in your trusted_cert
directory (/root/chef-repo/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

Adding certificate for datadb in /root/chef-repo/.chef/trusted_certs/datadb.crt

As we see here, on the workstation, the certificate is now downloaded and stored. This certificate will be used by knife command to communicate to the Chef server on SSL.

# ls -l .chef/trusted_certs/
-rw-r--r--. 1 root root 1330 Jun 24 09:44 datadb.crt

As you see, this is a standard SSL certificate. This is a self-signed certificated generated by Chef server.

# cat .chef/trusted_certs/datadb.crt 
-----BEGIN CERTIFICATE-----
ANBgkqhkiG9w0BAQsFADBFMQswCQYDVQQGEwJVUzEQ
ETMBEGA1UECwwKT3BlcmF0aW9uczEPMA0GA1UEAwwG
..
..
IwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAN3l
WO7bpxbQ1qqeuq9aLhNTIM1alwHfvlckmzSJsHRI+V
TaWuk7xncRs4VCUw==
-----END CERTIFICATE-----

Final Knife SSL Check Verification

After installing the SSL certificate properly as shown above, now when you execute the knife ssl check, it will not report any error. The following verifies that the Chef Server’s self-signed SSL certificate is properly installed on this machine and knife or chef-client command can use it without any issue.

# knife ssl check
Connecting to host datadb:443
Successfully verified certificates from `datadb'

Knife SSL Check for a different Chef Server URL

By default, when you execute knife ssl check (or knife ssl fetch), how does it know which Chef server to connect to?

For this, it will use the chef_server_url from the knife.rb file as shown below.

# cat .chef/knife.rb 
current_dir = File.dirname(__FILE__)
log_level                :info
log_location             STDOUT
node_name                "ramesh"
client_key               "#{current_dir}/ramesh.pem"
validation_client_name   "thegeekstuff-validator"
validation_key           "#{current_dir}/thegeekstuff-validator.pem"
chef_server_url          "https://datadb/organizations/thegeekstuff"
cookbook_path            ["#{current_dir}/../cookbooks"]

But, using knife ssl check, you can verify the SSL certificate for a different Chef server by passing the chef server url as a parameter. In the following example, it will connect to datadb-prod Chef server to verify the SSL certificate.

# knife ssl check https://datadb-prod/organizations/thegeekstuff
Connecting to host datadb-prod:443
Successfully verified certificates from `datadb-prod'

Knife SSL Check on Individual Node (On Chef Client)

If you want to verify whether your client machine where the chef-client command is execute has the properly valid Chef server’s SSL certificate, then execute the knife ssl check command and pass the –config parameter as shown below.

# knife ssl check --config /etc/chef/client.rb
Connecting to host datadb:443
Successfully verified certificates from `datadb'

In the above example, the knife ssl check command will use the chef_server_url parameter’s value from the following client.rb file as the Chef Server to connect and verify the SSL certificate.

# cat /etc/chef/client.rb 
log_location     STDOUT
chef_server_url  "https://datadb/organizations/thegeekstuff"
validation_client_name "thegeekstuff-validator"
# Using default node name (fqdn)
trusted_certs_dir "/etc/chef/trusted_certs"

Knife SSL Fetch from a different Chef Server URL

You can also get the SSL certificate from a different chef server (than the default one from the knife.rb) by specifying the chef server’s URL as a parameter to the knife ssl fetch command as shown below.

# knife ssl fetch https://datadb-prod/organizations/thegeekstuff
WARNING: Certificates from datadb-prod will be fetched and placed in your trusted_cert
directory (/root/chef-repo/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

Adding certificate for datadb-prod in /root/chef-repo/.chef/trusted_certs/datadb-prod.crt

Validate the Downloaded Chef Server Certificate

When you have the chef server and the chef workstation running on different machine and when you are doing the knife ssl fetch, it is a good idea to verify the check sum to make sure you have downloaded the correct certificate.

On the Chef Server, the SSL certificate is located under /var/opt/opscode/nginx/ca directory. Use the sha256sum command to get the check-sum as shown below.

# sha256sum /var/opt/opscode/nginx/ca/datadb.crt
bd32b7f8ddc2e4e405f990dd6  /var/opt/opscode/nginx/ca/datadb.crt

On your Chef workstation, where you did “knife ssl fetch” command to download the certificate, execute the following sha256sum command and make sure the following check-sum value matches the above value.

# sha256sum .chef/trusted_certs/datadb.crt 
bd32b7f8ddc2e4e405f990dd6  .chef/trusted_certs/datadb.crt
Add your comment

If you enjoyed this article, you might also like..

  1. 50 Linux Sysadmin Tutorials
  2. 50 Most Frequently Used Linux Commands (With Examples)
  3. Top 25 Best Linux Performance Monitoring and Debugging Tools
  4. Mommy, I found it! – 15 Practical Linux Find Command Examples
  5. Linux 101 Hacks 2nd Edition eBook Linux 101 Hacks Book

Bash 101 Hacks Book Sed and Awk 101 Hacks Book Nagios Core 3 Book Vim 101 Hacks Book