352 lines
13 KiB
Markdown
352 lines
13 KiB
Markdown
---
|
|
stage: Systems
|
|
group: Distribution
|
|
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
|
|
---
|
|
|
|
# Logs on Linux package installations
|
|
|
|
DETAILS:
|
|
**Tier:** Free, Premium, Ultimate
|
|
**Offering:** Self-managed
|
|
|
|
GitLab includes an [advanced log system](https://docs.gitlab.com/ee/administration/logs/index.html) where every service and component within GitLab will output system logs.
|
|
Here are the configuration settings and tools for managing these logs on Linux package installations.
|
|
|
|
## Tail logs in a console on the server
|
|
|
|
If you want to 'tail', i.e. view live log updates of GitLab logs you can use
|
|
`gitlab-ctl tail`.
|
|
|
|
```shell
|
|
# Tail all logs; press Ctrl-C to exit
|
|
sudo gitlab-ctl tail
|
|
|
|
# Drill down to a sub-directory of /var/log/gitlab
|
|
sudo gitlab-ctl tail gitlab-rails
|
|
|
|
# Drill down to an individual file
|
|
sudo gitlab-ctl tail nginx/gitlab_error.log
|
|
```
|
|
|
|
### Tail logs in a console and save to a file
|
|
|
|
Oftentimes, it is useful to both display the logs in the console and save them to a file for later debugging/analysis. You can use the [`tee`](https://en.wikipedia.org/wiki/Tee_(command)) utility to accomplish this.
|
|
|
|
```shell
|
|
# Use 'tee' to tail all the logs to STDOUT and write to a file at the same time
|
|
sudo gitlab-ctl tail | tee --append /tmp/gitlab_tail.log
|
|
```
|
|
|
|
## Configure default log directories
|
|
|
|
In your `/etc/gitlab/gitlab.rb` file, there are many `log_directory` keys for
|
|
the various types of logs. Uncomment and update the values for all the logs
|
|
you want to place elsewhere:
|
|
|
|
```ruby
|
|
# For example:
|
|
gitlab_rails['log_directory'] = "/var/log/gitlab/gitlab-rails"
|
|
puma['log_directory'] = "/var/log/gitlab/puma"
|
|
registry['log_directory'] = "/var/log/gitlab/registry"
|
|
...
|
|
```
|
|
|
|
Gitaly and Mattermost have different log directory configs:
|
|
|
|
```ruby
|
|
gitaly['configuration'] = {
|
|
logging: {
|
|
dir: "/var/log/gitlab/registry"
|
|
}
|
|
}
|
|
mattermost['log_file_directory'] = "/var/log/gitlab/registry"
|
|
```
|
|
|
|
Run `sudo gitlab-ctl reconfigure` to configure your instance with these settings.
|
|
|
|
## runit logs
|
|
|
|
The [runit-managed](../architecture/index.md#runit) services in Linux package installations generate log data using
|
|
`svlogd`.
|
|
|
|
- Logs are written to a file called `current`.
|
|
- Periodically, this log is compressed and renamed using the TAI64N format, for
|
|
example: `@400000005f8eaf6f1a80ef5c.s`.
|
|
- The filesystem datestamp on the compressed logs will be consistent with the time
|
|
GitLab last wrote to that file.
|
|
- `zmore` and `zgrep` allow viewing and searching through both compressed or uncompressed logs.
|
|
|
|
Read the [svlogd documentation](https://smarden.org/runit/svlogd.8) for more information
|
|
about the files it generates.
|
|
|
|
You can modify svlogd settings via `/etc/gitlab/gitlab.rb` with the following settings:
|
|
|
|
```ruby
|
|
# Below are the default values
|
|
logging['svlogd_size'] = 200 * 1024 * 1024 # rotate after 200 MB of log data
|
|
logging['svlogd_num'] = 30 # keep 30 rotated log files
|
|
logging['svlogd_timeout'] = 24 * 60 * 60 # rotate after 24 hours
|
|
logging['svlogd_filter'] = "gzip" # compress logs with gzip
|
|
logging['svlogd_udp'] = nil # transmit log messages via UDP
|
|
logging['svlogd_prefix'] = nil # custom prefix for log messages
|
|
|
|
# Optionally, you can override the prefix for e.g. Nginx
|
|
nginx['svlogd_prefix'] = "nginx"
|
|
```
|
|
|
|
## Logrotate
|
|
|
|
The **logrotate** service built into GitLab manages all logs except those
|
|
captured by **runit**. This service will rotate, compress, and eventually delete
|
|
the log data such as `gitlab-rails/production.log` and
|
|
`nginx/gitlab_access.log`. You can configure common logrotate settings,
|
|
configure per-service logrotate settings, and completely disable logrotate
|
|
with `/etc/gitlab/gitlab.rb`.
|
|
|
|
### Configuring common logrotate settings
|
|
|
|
Settings common to all **logrotate** services can be set in the
|
|
`/etc/gitlab/gitlab.rb` file. These settings correspond to configuration options
|
|
in the logrotate configuration files for each service. See the logrotate man
|
|
page (`man logrotate`) for details.
|
|
|
|
```ruby
|
|
logging['logrotate_frequency'] = "daily" # rotate logs daily
|
|
logging['logrotate_maxsize'] = nil # logs will be rotated when they grow bigger than size specified for `maxsize`, even before the specified time interval (daily, weekly, monthly, or yearly)
|
|
logging['logrotate_size'] = nil # do not rotate by size by default
|
|
logging['logrotate_rotate'] = 30 # keep 30 rotated logs
|
|
logging['logrotate_compress'] = "compress" # see 'man logrotate'
|
|
logging['logrotate_method'] = "copytruncate" # see 'man logrotate'
|
|
logging['logrotate_postrotate'] = nil # no postrotate command by default
|
|
logging['logrotate_dateformat'] = nil # use date extensions for rotated files rather than numbers e.g. a value of "-%Y-%m-%d" would give rotated files like production.log-2016-03-09.gz
|
|
```
|
|
|
|
### Configuring individual service logrotate settings
|
|
|
|
You can customize logrotate settings for each individual service by using
|
|
`/etc/gitlab/gitlab.rb`. For example, to customize logrotate frequency and size
|
|
for the `nginx` service, use:
|
|
|
|
```ruby
|
|
nginx['logrotate_frequency'] = nil
|
|
nginx['logrotate_size'] = "200M"
|
|
```
|
|
|
|
### Disabling logrotate
|
|
|
|
You can also disable the built-in logrotate service with the following setting
|
|
in `/etc/gitlab/gitlab.rb`:
|
|
|
|
```ruby
|
|
logrotate['enable'] = false
|
|
```
|
|
|
|
### Logrotate `notifempty` setting
|
|
|
|
The logrotate service runs with a non-configurable default of `notifempty`, resolving
|
|
the following issues:
|
|
|
|
- Empty logs being rotated unnecessarily, and often many empty logs being stored.
|
|
- One-off logs that are useful for long term troubleshooting being deleted after 30 days, such as database migration logs.
|
|
|
|
### Logrotate one-off and empty log handling
|
|
|
|
Logs are now rotated and recreated by **logrotate** as needed, and one-off logs
|
|
are only rotated when they change. With this setting in place, some tidying can be done:
|
|
|
|
- Empty one-off logs such as `gitlab-rails/gitlab-rails-db-migrate*.log` can be deleted.
|
|
- Empty logs which were rotated and compressed by older versions of GitLab. These
|
|
empty logs are usually 20 bytes in size.
|
|
|
|
### Run logrotate manually
|
|
|
|
Logrotate is a scheduled job but it can also be triggered on-demand.
|
|
|
|
To manually trigger GitLab log rotation with `logrotate`, use the following command:
|
|
|
|
```shell
|
|
/opt/gitlab/embedded/sbin/logrotate -fv -s /var/opt/gitlab/logrotate/logrotate.status /var/opt/gitlab/logrotate/logrotate.conf
|
|
```
|
|
|
|
### Increase how often logrotate is triggered
|
|
|
|
The logrotate script triggers every 50 minutes and waits for 10 minutes before attempting to rotate the logs.
|
|
|
|
To modify these values:
|
|
|
|
1. Edit `/etc/gitlab/gitlab.rb`:
|
|
|
|
```ruby
|
|
logrotate['pre_sleep'] = 600 # sleep 10 minutes before rotating after start-up
|
|
logrotate['post_sleep'] = 3000 # wait 50 minutes after rotating
|
|
```
|
|
|
|
1. Reconfigure GitLab:
|
|
|
|
```shell
|
|
sudo gitlab-ctl reconfigure
|
|
```
|
|
|
|
## UDP log forwarding
|
|
|
|
DETAILS:
|
|
**Tier:** Premium, Ultimate
|
|
**Offering:** Self-managed
|
|
|
|
Linux package installations can utilize the UDP logging feature in svlogd as well as sending non-svlogd logs to a syslog-compatible remote system using UDP.
|
|
To configure a Linux package installation to send syslog-protocol messages via UDP, use the following settings:
|
|
|
|
```ruby
|
|
logging['udp_log_shipping_host'] = '1.2.3.4' # Your syslog server
|
|
# logging['udp_log_shipping_hostname'] = nil # Optional, defaults the system hostname
|
|
logging['udp_log_shipping_port'] = 1514 # Optional, defaults to 514 (syslog)
|
|
```
|
|
|
|
NOTE:
|
|
Setting `udp_log_shipping_host` will [add a `svlogd_prefix`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/libraries/logging.rb)
|
|
for the specified hostname and service for each of the [runit-managed](../architecture/index.md#runit) services.
|
|
|
|
Example log messages:
|
|
|
|
```plaintext
|
|
Jun 26 06:33:46 ubuntu1204-test production.log: Started GET "/root/my-project/import" for 127.0.0.1 at 2014-06-26 06:33:46 -0700
|
|
Jun 26 06:33:46 ubuntu1204-test production.log: Processing by ProjectsController#import as HTML
|
|
Jun 26 06:33:46 ubuntu1204-test production.log: Parameters: {"id"=>"root/my-project"}
|
|
Jun 26 06:33:46 ubuntu1204-test production.log: Completed 200 OK in 122ms (Views: 71.9ms | ActiveRecord: 12.2ms)
|
|
Jun 26 06:33:46 ubuntu1204-test gitlab_access.log: 172.16.228.1 - - [26/Jun/2014:06:33:46 -0700] "GET /root/my-project/import HTTP/1.1" 200 5775 "https://172.16.228.169/root/my-project/import" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
|
|
2014-06-26_13:33:46.49866 ubuntu1204-test sidekiq: 2014-06-26T13:33:46Z 18107 TID-7nbj0 Sidekiq::Extensions::DelayedMailer JID-bbfb118dd1db20f6c39f5b50 INFO: start
|
|
2014-06-26_13:33:46.52608 ubuntu1204-test sidekiq: 2014-06-26T13:33:46Z 18107 TID-7muoc RepositoryImportWorker JID-57ee926c3655fcfa062338ae INFO: start
|
|
```
|
|
|
|
## Using a custom NGINX log format
|
|
|
|
By default the NGINX access logs will use a version of the 'combined' NGINX
|
|
format, designed to hide potentially sensitive information embedded in query strings.
|
|
If you want to use a custom log format string you can specify it
|
|
in `/etc/gitlab/gitlab.rb` - see
|
|
[the NGINX documentation](http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format)
|
|
for format details.
|
|
|
|
```ruby
|
|
nginx['log_format'] = 'my format string $foo $bar'
|
|
mattermost_nginx['log_format'] = 'my format string $foo $bar'
|
|
```
|
|
|
|
## JSON logging
|
|
|
|
Structured logs can be exported via JSON to be parsed by Elasticsearch,
|
|
Splunk, or another log management system.
|
|
The JSON format is enabled by default for all services that support it.
|
|
|
|
NOTE:
|
|
PostgreSQL does not support JSON logging without an
|
|
external plugin. However, it does support logging in CSV format:
|
|
|
|
```ruby
|
|
postgresql['log_destination'] = 'csvlog'
|
|
postgresql['logging_collector'] = 'on'
|
|
```
|
|
|
|
A restart of the database is required for this to take effect. For more
|
|
details, see the
|
|
[PostgreSQL documentation](https://www.postgresql.org/docs/12/runtime-config-logging.html).
|
|
|
|
## Text logging
|
|
|
|
Customers with established log ingestion systems may not wish to use the JSON
|
|
log format. Text formatting can be configured by setting the following
|
|
in `/etc/gitlab/gitlab.rb` and then running `gitlab-ctl reconfigure` afterward:
|
|
|
|
```ruby
|
|
gitaly['configuration'] = {
|
|
logging: {
|
|
format: ""
|
|
}
|
|
}
|
|
gitlab_shell['log_format'] = 'text'
|
|
gitlab_workhorse['log_format'] = 'text'
|
|
registry['log_formatter'] = 'text'
|
|
sidekiq['log_format'] = 'text'
|
|
gitlab_pages['log_format'] = 'text'
|
|
```
|
|
|
|
NOTE:
|
|
There are a few variations in attribute names for the log format depending on the service involved (for example, Container Registry uses `log_formatter`, Gitaly and Praefect both use `logging_format`). See [Issue #4280](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4280) for more details.
|
|
|
|
## rbtrace
|
|
|
|
GitLab ships with [`rbtrace`](https://github.com/tmm1/rbtrace), which
|
|
allows you to trace Ruby code, view all running threads, take memory dumps,
|
|
and more. However, this is not enabled by default. To enable it, define the
|
|
`ENABLE_RBTRACE` variable to the environment:
|
|
|
|
```ruby
|
|
gitlab_rails['env'] = {"ENABLE_RBTRACE" => "1"}
|
|
```
|
|
|
|
Then reconfigure the system and restart Puma and Sidekiq. To run this
|
|
in a Linux package installation, run as root:
|
|
|
|
```ruby
|
|
/opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/rbtrace
|
|
```
|
|
|
|
## Configuring log level/verbosity
|
|
|
|
You can configure the minimum log levels (verbosity) for GitLab Rails,
|
|
Container Registry, GitLab Shell and Gitaly:
|
|
|
|
1. Edit `/etc/gitlab/gitlab.rb` and set the log levels:
|
|
|
|
```ruby
|
|
gitlab_rails['env'] = {
|
|
"GITLAB_LOG_LEVEL" => "WARN",
|
|
}
|
|
registry['log_level'] = 'info'
|
|
gitlab_shell['log_level'] = 'INFO'
|
|
gitaly['configuration'] = {
|
|
logging: {
|
|
level: "warn"
|
|
}
|
|
}
|
|
```
|
|
|
|
1. Reconfigure GitLab:
|
|
|
|
```shell
|
|
sudo gitlab-ctl reconfigure
|
|
```
|
|
|
|
NOTE:
|
|
You [cannot edit](https://gitlab.com/groups/gitlab-org/-/epics/6034)
|
|
the `log_level` for certain GitLab logs, for example
|
|
`production_json.log`, `graphql_json.log`, and so on.
|
|
See also [Override default log level](https://docs.gitlab.com/ee/administration/logs/#override-default-log-level).
|
|
|
|
## Setting a custom log group
|
|
|
|
GitLab supports assigning a custom group to the configured [log directories](#configure-default-log-directories)
|
|
|
|
A global `logging['log_group']` setting in your `/etc/gitlab/gitlab.rb` file can
|
|
be configured as well as per-service `log_group` settings such as `gitaly['log_group']`.
|
|
You will need to run `sudo gitlab-ctl reconfigure` to configure your instance
|
|
when adding `log_group` settings.
|
|
|
|
Setting a global or per-service `log_group` will:
|
|
|
|
- Change the permissions on the per-service log directories (or all log directories
|
|
if using the global setting) to `0750` to allow the configured group members to
|
|
read the contents of the log directory.
|
|
|
|
- Configure [runit](#runit-logs) to write and rotate logs using the specified
|
|
`log_group` : either per-service or for all runit-managed services.
|
|
|
|
### Custom log group limitations
|
|
|
|
Logs for services not managed by runit (e.g. the `gitlab-rails` logs in
|
|
`/var/log/gitlab/gitlab-rails`) will not inherit the configured `log_group` setting.
|
|
|
|
The group must already exist on the host. Linux package installations don't create the group
|
|
when running `sudo gitlab-ctl reconfigure`.
|