Merge branch 'move-mattermost-files-to-mattermost-cookbook' into 'master'

Move Mattermost related files to Mattermost cookbook

See merge request https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6359

Merged-by: Balasankar 'Balu' C <balasankar@gitlab.com>

.gitlab-ci.yml

@@ -37,7 +37,7 @@ variables:
   # To be used for images that exist only on dev.gitlab.org
   DEV_BUILDER_IMAGE_REGISTRY: 'dev.gitlab.org:5005/cookbooks/gitlab-omnibus-builder'
   PUBLIC_BUILDER_IMAGE_REGISTRY: "registry.gitlab.com/gitlab-org/gitlab-omnibus-builder"
-  BUILDER_IMAGE_REVISION: "5.10.0"
+  BUILDER_IMAGE_REVISION: "5.12.0"
   # The registry to pull the assets image from
   ASSET_REGISTRY: "${CI_REGISTRY}"
   ASSET_SYNC_EXISTING_REMOTE_FILES: "keep"

config/software/exiftool.rb

@@ -15,7 +15,7 @@
 #
 
 name 'exiftool'
-version = Gitlab::Version.new('exiftool', '12.82')
+version = Gitlab::Version.new('exiftool', '12.84')
 default_version version.print(false)
 
 license 'GPL-1.0 or Artistic'

config/software/node-exporter.rb

@@ -18,7 +18,7 @@ require "#{Omnibus::Config.project_root}/lib/gitlab/version"
 require "#{Omnibus::Config.project_root}/lib/gitlab/prometheus_helper"
 
 name 'node-exporter'
-version = Gitlab::Version.new('node-exporter', '1.7.0')
+version = Gitlab::Version.new('node-exporter', '1.8.0')
 default_version version.print
 
 license 'APACHE-2.0'

doc/architecture/index.md

@@ -148,7 +148,7 @@ parsed correctly.
 
 ### runit
 
-GitLab uses [runit](http://smarden.org/runit/) recipes for
+GitLab uses [runit](https://smarden.org/runit/) recipes for
 service management and supervision. runit recipes do the job of identifying the
 init system used by the OS and performing basic service management tasks like
 creating necessary service files for GitLab, service enabling, and service

doc/architecture/multiple_database_support/index.md

@@ -82,7 +82,7 @@ cluster for all databases_ to _one cluster per database_.
 Therefore the HA configuration of logical databases must be independent of the
 deployment model.
 
-Consul [services](https://developer.hashicorp.com/consul/docs/srvices/configuration/services-configuration-reference)
+Consul [services](https://developer.hashicorp.com/consul/docs/services/configuration/services-configuration-reference)
 can have multiple health-checks and [watches](https://developer.hashicorp.com/consul/docs/dynamic-app-config/watches#service).
 At this level, Omnibus GitLab defines _a Consul service per database cluster_
 and _a service watch per logical database_.

doc/settings/logs.md

@@ -77,7 +77,7 @@ The [runit-managed](../architecture/index.md#runit) services in Linux package in
   GitLab last wrote to that file.
 - `zmore` and `zgrep` allow viewing and searching through both compressed or uncompressed logs.
 
-Read the [svlogd documentation](https://smarden.org/runit/svlogd.8.html) for more information
+Read the [svlogd documentation](https://smarden.org/runit/svlogd.8) for more information
 about the files it generates.
 
 You can modify svlogd settings via `/etc/gitlab/gitlab.rb` with the following settings:

doc/settings/memory_constrained_envs.md

@@ -145,14 +145,14 @@ We observed 100-400MB of memory usage reduction configuring Puma this way.
 ## Optimize Sidekiq
 
 Sidekiq is a background processing daemon. When configured with GitLab by default
-it runs with a high concurrency mode of `50`. This does impact how much memory it can
+it runs with a concurrency mode of `20`. This does impact how much memory it can
 allocate at a given time. It is advised to configure it to use a significantly
 smaller value of `5` or `10` (preferred).
 
 In `/etc/gitlab/gitlab.rb`:
 
 ```ruby
-sidekiq['max_concurrency'] = 10
+sidekiq['concurrency'] = 10
 ```
 
 ## Optimize Gitaly
@@ -249,7 +249,7 @@ and disable the Prometheus Metrics feature:
    ```ruby
    puma['worker_processes'] = 0
 
-   sidekiq['max_concurrency'] = 10
+   sidekiq['concurrency'] = 10
 
    prometheus_monitoring['enable'] = false
 

doc/settings/redis.md

@@ -120,7 +120,7 @@ redis['announce_ip_from_hostname'] = true
 ## Setting the Redis Cache instance as an LRU
 
 Using multiple Redis instances allows you to configure Redis as a
-[Least Recently Used cache](https://redis.io/docs/manual/eviction/). Note you should only
+[Least Recently Used cache](https://redis.io/docs/latest/operate/rs/databases/memory-performance/eviction-policy/). Note you should only
 do this for the Redis cache, rate-limiting, and repository cache instances; the Redis queues, shared
 state instances, and tracechunks instances should never be configured as an LRU,
 since they contain data (e.g. Sidekiq jobs) that is expected to be persistent.
@@ -160,7 +160,7 @@ You can configure Redis to run behind SSL.
 NOTE:
 Some `redis-cli` binaries are not built with support for directly connecting to a Redis server over TLS.
 If your `redis-cli` doesn't support the `--tls` flag, you will have to use something like
-[`stunnel`](https://redis.com/blog/stunnel-secure-redis-ssl/) to connect to the
+[`stunnel`](https://redis.io/blog/stunnel-secure-redis-ssl/) to connect to the
 Redis server using `redis-cli` for any debugging purposes.
 
 ### Make GitLab client connect to Redis server over SSL

doc/settings/rpi.md

@@ -55,7 +55,7 @@ some running processes:
    ```ruby
    # Reduce the number of running workers to the minimum in order to reduce memory usage
    puma['worker_processes'] = 2
-   sidekiq['max_concurrency'] = 9
+   sidekiq['concurrency'] = 9
    # Turn off monitoring to reduce idle cpu and disk usage
    prometheus_monitoring['enable'] = false
    ```

doc/settings/smtp.md

@@ -142,7 +142,7 @@ gitlab_rails['smtp_force_ssl'] = false
 NOTE:
 Gmail has [strict sending limits](https://support.google.com/a/answer/166852)
 that can impair functionality as your organization grows. We strongly recommend using a
-transactional service like [SendGrid](https://sendgrid.com/) or [Mailgun](https://www.mailgun.com/)
+transactional service like [SendGrid](https://sendgrid.com/en-us) or [Mailgun](https://www.mailgun.com/)
 for teams using SMTP configuration.
 
 ```ruby
@@ -1216,7 +1216,7 @@ gitlab_rails['gitlab_email_from'] = 'username@example.com'
 gitlab_rails['gitlab_email_reply_to'] = 'username@example.com'
 ```
 
-Check the SMTP user name and SMTP user password from the ESS [dashboard](https://pfs.nifcloud.com/help/ess/dashboard.htm).
+Check the SMTP user name and SMTP user password from the ESS [dashboard](https://docs.nifcloud.com/ess/help/dashboard.htm).
 `gitlab_email_from` and `gitlab_email_reply_to` must be ESS authenticated sender email addresses.
 
 ### Sina mail

files/gitlab-config-template/gitlab.rb.template

@@ -1280,9 +1280,7 @@ external_url 'GENERATED_EXTERNAL_URL'
 # sidekiq['log_format'] = "json"
 # sidekiq['shutdown_timeout'] = 4
 # sidekiq['interval'] = nil
-# sidekiq['concurrency'] = nil
-# sidekiq['max_concurrency'] = 20
-# sidekiq['min_concurrency'] = nil
+# sidekiq['concurrency'] = 20
 
 ##! GitLab allows route a job to a particular queue determined by an array of ##! routing rules.
 ##! Each routing rule is a tuple of queue selector query and corresponding queue. By default,

files/gitlab-cookbooks/gitlab/attributes/default.rb

@@ -728,9 +728,7 @@ default['gitlab']['sidekiq']['health_checks_listen_port'] = 8092
 
 # Cluster specific settings
 default['gitlab']['sidekiq']['interval'] = nil
-default['gitlab']['sidekiq']['concurrency'] = nil
-default['gitlab']['sidekiq']['max_concurrency'] = 20
-default['gitlab']['sidekiq']['min_concurrency'] = nil
+default['gitlab']['sidekiq']['concurrency'] = 20
 default['gitlab']['sidekiq']['queue_groups'] = ['*']
 default['gitlab']['sidekiq']['consul_service_name'] = 'sidekiq'
 default['gitlab']['sidekiq']['consul_service_meta'] = nil

files/gitlab-cookbooks/gitlab/recipes/nginx.rb

@@ -276,6 +276,7 @@ mattermost_nginx_vars['https'] = if mattermost_nginx_vars['listen_https'].nil?
 
 template gitlab_mattermost_http_conf do
   source "nginx-gitlab-mattermost-http.conf.erb"
+  cookbook "mattermost"
   owner "root"
   group "root"
   mode "0644"

files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

@@ -22,12 +22,6 @@ exec chpst -e /opt/gitlab/etc/gitlab-rails/env -P \
       <% if node['gitlab']['sidekiq']['concurrency'] %>
       -c <%= node['gitlab']['sidekiq']['concurrency'] %> \
       <% end %>
-      <% if node['gitlab']['sidekiq']['max_concurrency'] %>
-      -m <%= node['gitlab']['sidekiq']['max_concurrency'] %> \
-      <% end %>
-      <% if node['gitlab']['sidekiq']['min_concurrency'] %>
-      --min-concurrency <%= node['gitlab']['sidekiq']['min_concurrency'] %> \
-      <% end %>
       <% if node['gitlab']['sidekiq']['shutdown_timeout'] %>
       --timeout <%= node['gitlab']['sidekiq']['shutdown_timeout'] %> \
       <% end %>

files/gitlab-cookbooks/mattermost/libraries/gitlab_mattermost.rb

@@ -15,7 +15,7 @@
 # limitations under the License.
 #
 
-require_relative 'nginx.rb'
+require_relative '../../gitlab/libraries/nginx.rb'
 require_relative '../../package/libraries/deprecations'
 require_relative '../../package/libraries/helpers/logging_helper'
 require_relative '../../letsencrypt/libraries/helper'

files/gitlab-cookbooks/package/libraries/deprecations.rb

@@ -346,14 +346,14 @@ module Gitlab
           },
           {
             config_keys: %w(gitlab sidekiq min_concurrency),
-            deprecation: '16.9',
-            removal: '17.0',
+            deprecation: '16.9', # Remove message issue: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/8491
+            removal: '17.0', # Removal issue: https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/3422
             note: "Starting with GitLab 17.0, `sidekiq['min_concurrency']` will be removed. Please follow https://docs.gitlab.com/ee/administration/sidekiq/extra_sidekiq_processes.html#manage-thread-counts-explicitly to use `sidekiq['concurrency']` instead."
           },
           {
             config_keys: %w(gitlab sidekiq max_concurrency),
-            deprecation: '16.9',
-            removal: '17.0',
+            deprecation: '16.9', # Remove message issue: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/8491
+            removal: '17.0', # Removal issue: https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/3422
             note: "Starting with GitLab 17.0, `sidekiq['max_concurrency']` will be removed. Please follow https://docs.gitlab.com/ee/administration/sidekiq/extra_sidekiq_processes.html#manage-thread-counts-explicitly to use `sidekiq['concurrency']` instead."
           },
           {

spec/chef/cookbooks/gitlab/recipes/sidekiq_spec.rb

@@ -26,7 +26,7 @@ RSpec.describe 'gitlab::sidekiq' do
           expect(content).to match(/rubyopt=\"-W:no-experimental\"/)
           expect(content).to include(%(RUBYOPT="${rubyopt}"))
           expect(content).to match(%r{bin/sidekiq-cluster})
-          expect(content).to match(/-m 20/) # max_concurrency
+          expect(content).to match(/-c 20/) # concurrency
           expect(content).to match(/--timeout 25/) # shutdown timeout
           expect(content).to match(/\*/) # all queues
         }
@@ -64,11 +64,19 @@ RSpec.describe 'gitlab::sidekiq' do
       before do
         stub_gitlab_rb(
           sidekiq: {
-            log_group: 'fugee'
+            log_group: 'fugee',
+            concurrency: 42
           }
         )
       end
       it_behaves_like 'enabled logged service', 'sidekiq', true, { log_directory_owner: 'git', log_group: 'fugee' }
+
+      it 'correctly renders out the sidekiq service file' do
+        expect(chef_run).to render_file("/opt/gitlab/sv/sidekiq/run")
+          .with_content { |content|
+            expect(content).to match(/-c 42/) # concurrency
+          }
+      end
     end
   end