16 KiB
SSL Configuration
Available SSL Configuration Tasks
Omnibus-GitLab supports several common use cases for SSL configuration.
- Allow
https
connections to GitLab instance services - Configure public certificate bundles for external resource connections
Host Services
Administrators can enable secure http using any method supported by a GitLab service.
Service | Manual SSL | Let's Encrypt |
---|---|---|
Primary GitLab Instance Domain | Yes | Yes |
Container Registry | Yes | Yes |
Mattermost | Yes | Yes |
GitLab Pages | Yes | No |
Let's Encrypt Integration
GitLab can be integrated with Let's Encrypt.
Primary GitLab Instance
- Introduced in GitLab 10.5 and disabled by default.
- Enabled by default in GitLab 10.7 and later if
external_url
is set with the https protocol and no certificates are configured.
NOTE: Note: In order for Let's Encrypt verification to work correctly, ports 80 and 443 will need to be accessible to the Let's Encrypt servers that run the validation. Also note that the validation currently does not work with non-standard ports.
CAUTION: Caution:
Administrators installing or upgrading to GitLab 10.7 or later and do not plan on using
Let's Encrypt should set letsencrypt['enable'] = false
in /etc/gitlab/gitlab.rb
to disable.
Add the following entries to /etc/gitlab/gitlab.rb
to enable Let's Encrypt
support for the primary domain:
letsencrypt['enable'] = true # GitLab 10.5 and 10.6 require this option
external_url "https://gitlab.example.com" # Must use https protocol
letsencrypt['contact_emails'] = ['foo@email.com'] # Optional
TIP: Maintenance Tip
Certificates issued by Let's Encrypt expire every ninety days. The optional contact_emails
setting causes an expiration alert to be sent to the configured address when that expiration date approaches.
GitLab Components
Introduced in GitLab 11.0.
Follow the steps to enable basic Let's Encrypt integration and
modify /etc/gitlab/gitlab.rb
with any of the following that apply:
registry_external_url "https://registry.example.com" # container registry, must use https protocol
mattermost_external_url "https://mattermost.example.com" # mattermost, must use https protocol
#registry_nginx['ssl_certificate'] = "path/to/cert" # Must be absent or commented out
NOTE: Under the Hood
The Let's Encrypt certificate is created with the GitLab primary
instance as the primary name on the certificate. Additional services
such as the registry are added as alternate names to the same
certificate. Note in the example above, the primary domain is gitlab.example.com
and
the registry domain is registry.example.com
. Administrators do not need
to worry about setting up wildcard certificates.
Automatic Let's Encrypt Renewal
Introduced in GitLab 10.7.
CAUTION: Caution:
Administrators installing or upgrading to GitLab 12.1 or later and plan on using
their own Let's Encrypt certificate should set letsencrypt['enable'] = false
in /etc/gitlab/gitlab.rb
to
disable automatic renewal. Otherwise, a gitlab-ctl reconfigure
may attempt to renew the
certificates, and thus overwrite them.
Default installations schedule renewals after midnight on every 4th day. The minute is determined by the value in external_url
to help distribute the load
on the upstream Let's Encrypt
servers.
Explicitly set renewal times by adding the following to /etc/gitlab/gitlab.rb
:
# This example renews every 7th day at 12:30
letsencrypt['auto_renew_hour'] = "12"
letsencrypt['auto_renew_minute'] = "30"
letsencrypt['auto_renew_day_of_month'] = "*/7"
Disable automatic renewal with the following in /etc/gitlab/gitlab.rb
:
letsencrypt['auto_renew'] = false
Manual Let's Encrypt Renewal
Renew Let's Encrypt certificates manually using one of the following commands:
sudo gitlab-ctl reconfigure
sudo gitlab-ctl renew-le-certs
CAUTION: Caution:
GitLab 12.1 or later will attempt to renew any Let's Encrypt certificate.
If you plan to use your own Let's Encrypt certificate you must set letsencrypt['enable'] = false
in /etc/gitlab/gitlab.rb
to disable integration. Otherwise the certificate
could be overwritten due to the renewal.
TIP: Tip The above commands require root privileges and only generate a renewal if the certificate is close to expiration. Consider the upstream rate limits if encountering an error during renewal.
Connecting to External Resources
Some environments connect to external resources for various tasks. Omnibus-GitLab
allows these connections to use secure http (https
).
Default Configuration
Omnibus-GitLab ships with the official CAcert.org collection of trusted root certification authorities which are used to verify certificate authenticity.
Other Certificate Authorities
Omnibus GitLab supports connections to external services with self-signed certificates.
NOTE: Compatibility Note Custom certificates were introduced in GitLab 8.9.
TIP: Further Reading For installations that use self-signed certificates, Omnibus-GitLab provides a way to manage these certificates. For more technical details how this works, see the details at the bottom of this page.
Install Custom Public Certificates
NOTE: Note:
A perl interpreter is required for c_rehash
dependency to properly symlink the certificates.
Perl is currently not bundled in Omnibus GitLab.
- Generate the PEM or DER encoded public certificate from your private key certificate.
- Copy the public certificate file only into the
/etc/gitlab/trusted-certs
directory. - Run
gitlab-ctl reconfigure
.
CAUTION: Caution:
If using a custom certificate chain, the root and/or intermediate certificates must be put into separate files in /etc/gitlab/trusted-certs
due to c_rehash
creating a hash for the first certificate only.
Troubleshooting
Git-LFS and other embedded services written in golang report custom certificate signed by unknown authority
NOTE: Note: In GitLab 11.5, the following workaround is no longer necessary, embedded golang apps now use the standard GitLab certificate directory automatically.
The gitlab-workhorse
and other services written in golang use the crypto/tls library from golang
instead of OpenSSL.
Add the following entry in /etc/gitlab/gitlab.rb
to work around the
issue as reported:
gitlab_workhorse['env'] = {
'SSL_CERT_DIR' => '/opt/gitlab/embedded/ssl/certs/'
}
NOTE: Note:
If you have installed GitLab to a path other than /opt/gitlab/
then modify the entry above
with the correct path in your operating environment.
Reconfigure Fails Due to Certificates
ERROR: Not a certificate: /opt/gitlab/embedded/ssl/certs/FILE. Move it from /opt/gitlab/embedded/ssl/certs to a different location and reconfigure again.
Check /opt/gitlab/embedded/ssl/certs
and remove any files other than README.md
that aren't valid x509 certificates.
NOTE: Under the Hood
Running gitlab-ctl reconfigure
constructs symlinks named from the subject hashes
of your custom public certificates and places them in /opt/gitlab/embedded/ssl/certs/
.
Broken symlinks in /opt/gitlab/embedded/ssl/certs/
will be automatically removed.
Files other than cacert.pem
and README.md
stored in
/opt/gitlab/embedded/ssl/certs/
will be moved into the /etc/gitlab/trusted-certs/
.
Custom Certificates Missing or Skipped
GitLab versions 8.9.0, 8.9.1, and 8.9.2 all mistakenly used the
/etc/gitlab/ssl/trusted-certs/
directory. This directory is safe to remove if it
is empty. If it still contains custom certificates then move them to /etc/gitlab/trusted-certs/
and run gitlab-ctl reconfigure
.
If no symlinks are created in /opt/gitlab/embedded/ssl/certs/
and you see
the message "Skipping cert.pem
" after running gitlab-ctl reconfigure
, that
means there may be one of four issues:
- The file in
/etc/gitlab/trusted-certs/
is a symlink - The file is not a valid PEM or DER-encoded certificate
- Perl is not installed on the operating system which is needed for c_rehash to properly symlink certificates.
- The certificate contains the string
TRUSTED
Test the certificate's validity using the commands below:
/opt/gitlab/embedded/bin/openssl x509 -in /etc/gitlab/trusted-certs/example.pem -text -noout
/opt/gitlab/embedded/bin/openssl x509 -inform DER -in /etc/gitlab/trusted-certs/example.der -text -noout
Invalid certificate files produce the following output:
unable to load certificate
140663131141784:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE
To test if c_rehash
is not symlinking the certificate due to a missing perl interpreter:
$ /opt/gitlab/embedded/bin/c_rehash /etc/gitlab/trusted-certs
bash: /opt/gitlab/embedded/bin/c_rehash: /usr/bin/perl: bad interpreter: No such file or directory
If you see this message, you will need to install perl with your distribution's package manager.
If you inspect the certificate itself, then look for the string TRUSTED
:
-----BEGIN TRUSTED CERTIFICATE-----
...
-----END TRUSTED CERTIFICATE-----
If it does, like the example above, then try removing the string TRUSTED
and running gitlab-ctl reconfigure
again.
Custom certificates not detected
If after running gitlab-ctl reconfigure
:
- no symlinks are created in
/opt/gitlab/embedded/ssl/certs/
; - you have placed custom certificates in
/etc/gitlab/trusted-certs/
; and - you do not see any skipped or symlinked custom certificate messages
You may be encountering an issue where Omnibus GitLab thinks that the custom certificates have already been added.
To resolve, delete the trusted certificates directory hash:
rm /var/opt/gitlab/trusted-certs-directory-hash
Then run gitlab-ctl reconfigure
again. The reconfigure should now detect and symlink
your custom certificates.
Let's Encrypt Certificate signed by unknown authority
The initial implementation of Let's Encrypt integration only used the certificate, not the full certificate chain.
Starting in 10.5.4, the full certificate chain will be used. For installs which are already using a certificate, the switchover will not happen until the renewal logic indicates the certificate is near expiration. To force it sooner, run the following
# rm /etc/gitlab/ssl/HOSTNAME*
# gitlab-ctl reconfigure
Where HOSTNAME is the hostname of the certificate.
Let's Encrypt fails on reconfigure
When you reconfigure, there are common scenarios under which Let's Encrypt may fail:
-
Let's Encrypt may fail if your server isn't able to reach the Let's Encrypt verification servers or vice versa:
letsencrypt_certificate[gitlab.domain.com] (letsencrypt::http_authorization line 3) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: RuntimeError: [gitlab.domain.com] Validation failed for domain gitlab.domain.com
If you run into issues reconfiguring GitLab due to Let's Encrypt make sure you have ports 80 and 443 open and accessible.
-
Your domain's Certification Authority Authorization (CAA) record does not allow Let's Encrypt to issue a certificate for your domain. Look for the following error in the reconfigure output:
letsencrypt_certificate[gitlab.domain.net] (letsencrypt::http_authorization line 5) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: RuntimeError: ruby_block[create certificate for gitlab.domain.net] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [gitlab.domain.com] Validation failed, unable to request certificate
-
If you're using a test domain such as
gitlab.example.com
, without a certificate, you'll see theunable to request certificate
error shown above. In that case, disable Let's Encrypt by settingletsencrypt['enable'] = false
in/etc/gitlab/gitlab.rb
.
You can test your domain using the Let's Debug diagnostic tool. It can help you figure out why you can't issue a Let's Encrypt certificate.
Details on how GitLab and SSL work
GitLab-Omnibus includes its own library of OpenSSL and links all compiled
programs (e.g. Ruby, PostgreSQL, etc.) against this library. This library is
compiled to look for certificates in /opt/gitlab/embedded/ssl/certs
.
GitLab-Omnibus manages custom certificates by symlinking any certificate that
gets added to /etc/gitlab/trusted-certs/
to /opt/gitlab/embedded/ssl/certs
using the c_rehash
tool. For example, let's suppose we add customcacert.pem
to
/etc/gitlab/trusted-certs/
:
$ sudo ls -al /opt/gitlab/embedded/ssl/certs
total 272
drwxr-xr-x 2 root root 4096 Jul 12 04:19 .
drwxr-xr-x 4 root root 4096 Jul 6 04:00 ..
lrwxrwxrwx 1 root root 42 Jul 12 04:19 7f279c95.0 -> /etc/gitlab/trusted-certs/customcacert.pem
-rw-r--r-- 1 root root 263781 Jul 5 17:52 cacert.pem
-rw-r--r-- 1 root root 147 Feb 6 20:48 README
Here we see the fingerprint of the certificate is 7f279c95
, which links to
the custom certificate.
What happens when we make an HTTPS request? Let's take a simple Ruby program:
#!/opt/gitlab/embedded/bin/ruby
require 'openssl'
require 'net/http'
Net::HTTP.get(URI('https://www.google.com'))
This is what happens behind the scenes:
- The "require
openssl
" line causes the interpreter to load/opt/gitlab/embedded/lib/ruby/2.3.0/x86_64-linux/openssl.so
. - The
Net::HTTP
call then attempts to read the default certificate bundle in/opt/gitlab/embedded/ssl/certs/cacert.pem
. - SSL negotiation occurs.
- The server sends its SSL certificates.
- If the certificates that are sent are covered by the bundle, SSL finishes successfully.
- Otherwise, OpenSSL may validate other certificates by searching for files
that match their fingerprints inside the predefined certificate directory. For
example, if a certificate has the fingerprint
7f279c95
, OpenSSL will attempt to read/opt/gitlab/embedded/ssl/certs/7f279c95.0
.
Note that the OpenSSL library supports the definition of SSL_CERT_FILE
and
SSL_CERT_DIR
environment variables. The former defines the default
certificate bundle to load, while the latter defines a directory in which to
search for more certificates. These variables should not be necessary if you
have added certificates to the trusted-certs
directory. However, if for some
reason you need to set them, they can be defined as environment
variables. For example:
gitlab_rails['env'] = {"SSL_CERT_FILE" => "/usr/lib/ssl/private/customcacert.pem"}