Merge branch 'docker-build' into 'master'

Add dockerfile to build gitlab-ce and gitlab-ee images

This is continuation of https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1399

I put it there, because building and releasing of Docker images requires Omnibus packages to be pushed to packagecloud.

1. This also introduces a separate sshd configuration to provide lockdown configuration.
2. Fixes problem with persisting ssh_host_key.
3. Creates tagged gitlab-ce and gitlab-ee images and pushes them to https://hub.docker.com/r/gitlab/gitlab-ce/ and https://hub.docker.com/r/gitlab/gitlab-ee/.
4. Supports pushing of nightly builds.
5. Requires a new runner with the access to docker build and access for pushing image (currently it sits on my home server) /cc @jeroen 
6. Creates and manages users before `gitlab-ctl reconfigure`. This fixes UID pinning and permission problem.
7. Introduces update-permissions script to fix potential permission problem when updating docker image.

/cc @marin @jacobvosmaer @sytse 


See merge request !21
This commit is contained in:
Kamil Trzciński 2015-09-29 20:23:11 +00:00
commit 9757575747
16 changed files with 549 additions and 0 deletions

1
.gitignore vendored
View File

@ -14,3 +14,4 @@ build.txt
Vagrantfile
.idea
*.log
docker/RELEASE

View File

@ -123,3 +123,21 @@ Raspberry Pi 2:
- RaspberryPi2-tag
except:
- branches
Docker master:
script:
- bundle install --binstubs --path ~/gems
- if ./support/is_gitlab_ee.sh; then exit 0; else make do_docker_master;fi
tags:
- docker-build
except:
- tags
Docker:
script:
- bundle install --binstubs --path ~/gems
- if ./support/is_gitlab_ee.sh; then exit 0; else make do_docker_release;fi
tags:
- docker-build
except:
- branches

View File

@ -6,6 +6,17 @@ PLATFORM_DIR:=$(shell bundle exec support/ohai-helper platform-dir)
PACKAGECLOUD_USER=gitlab
PACKAGECLOUD_REPO:=$(shell support/repo_name.sh)
PACKAGECLOUD_OS:=$(shell bundle exec support/ohai-helper repo-string)
ifeq ($(shell support/is_gitlab_ee.sh; echo $$?), 0)
RELEASE_PACKAGE=gitlab-ee
else
RELEASE_PACKAGE=gitlab-ce
endif
RELEASE_VERSION?=$(shell git describe | tr '+' '-')
ifdef NIGHTLY
DOCKER_TAG:=nightly
else
DOCKER_TAG:=$(RELEASE_VERSION)
endif
build:
bin/omnibus build ${PROJECT} --override append_timestamp:false --log-level info
@ -65,6 +76,31 @@ move_to_secret_dir:
&& mv ${SECRET_DIR} pkg/ \
; fi
docker_cleanup:
-docker ps -q -a | xargs docker rm -v
-docker images -f dangling=true -q | xargs docker rmi
-docker images | grep $(RELEASE_PACKAGE) | awk '{print $$3}' | xargs docker rmi
docker_build: docker_cleanup
echo PACKAGECLOUD_REPO=$(PACKAGECLOUD_REPO) > docker/RELEASE
echo RELEASE_PACKAGE=$(RELEASE_PACKAGE) >> docker/RELEASE
echo RELEASE_VERSION=$(RELEASE_VERSION) >> docker/RELEASE
docker build -t $(RELEASE_PACKAGE):latest -f docker/Dockerfile docker/
docker_push:
docker tag $(RELEASE_PACKAGE):latest gitlab/$(RELEASE_PACKAGE):$(DOCKER_TAG)
docker push gitlab/$(RELEASE_PACKAGE):$(DOCKER_TAG)
docker_push_latest:
docker tag $(RELEASE_PACKAGE):latest gitlab/$(RELEASE_PACKAGE):latest
docker push gitlab/$(RELEASE_PACKAGE):latest
do_docker_master:
ifdef NIGHTLY
do_docker_master: docker_build docker_push
endif
do_docker_release: no_changes on_tag docker_build docker_push docker_push_latest
md5:
find pkg -name '*.json' -exec cat {} \;

View File

@ -5,6 +5,7 @@
- [Package downloads page](https://about.gitlab.com/downloads/)
- [GitLab CI](gitlab-ci/README.md) Set up the GitLab CI coordinator that ships with Omnibus GitLab package.
- [GitLab Mattermost](gitlab-mattermost/README.md) Set up the Mattermost messaging app that ships with Omnibus GitLab package.
- [Docker](docker/README.md) Set up the GitLab in Docker container.
## Maintenance

25
doc/build/README.md vendored
View File

@ -60,3 +60,28 @@ Full help for the Omnibus command line interface can be accessed with the
```shell
$ bin/omnibus help
```
## Build Docker image
```shell
# Build with stable packagecloud packages
# This will build gitlab-ee (8.0.2-ee.1) using STABLE repo and tag it as gitlab-ee:latest
make docker_build RELEASE_VERSION=8.0.2-ee.1 PACKAGECLOUD_REPO=gitlab-ee RELEASE_PACKAGE=gitlab-ee
# Build with unstable packagecloud packages
# This will build gitlab-ce (8.0.2-ce.1) using UNSTABLE repo and tag it as gitlab-ce:latest
make docker_build RELEASE_VERSION=8.0.2-ce.1 PACKAGECLOUD_REPO=unstable RELEASE_PACKAGE=gitlab-ce
```
### Publish Docker image
```shell
# This will push gitlab-ee:latest as gitlab/gitlab-ee:8.0.2-ee.1
make docker_push RELEASE_PACKAGE=gitlab-ee RELEASE_VERSION=8.0.2-ee.1
# This will push gitlab-ce:latest as gitlab/gitlab-ce:8.0.2-ce.1
make docker_push RELEASE_PACKAGE=gitlab-ce RELEASE_VERSION=8.0.2-ce.1
# This will push gitlab-ce:latest as gitlab/gitlab-ce:latest
make docker_push_latest RELEASE_PACKAGE=gitlab-ce
```

170
doc/docker/README.md Normal file
View File

@ -0,0 +1,170 @@
# GitLab Docker images
The GitLab CE docker image is [available on Docker Hub](https://registry.hub.docker.com/u/gitlab/gitlab-ce/).
The GitLab EE docker image is [available on Docker Hub](https://registry.hub.docker.com/u/gitlab/gitlab-ce/).
To use GitLab EE instead of GitLab CE replace image name to `gitlab/gitlab-ee:latest`.
## Run the image
Run the image:
```bash
sudo docker run --detach \
--hostname gitlab.example.com \
--publish 8443:443 --publish 8080:80 --publish 2222:22 \
--name gitlab \
--restart always \
--volume /srv/gitlab/config:/etc/gitlab \
--volume /srv/gitlab/logs:/var/log/gitlab \
--volume /srv/gitlab/data:/var/opt/gitlab \
gitlab/gitlab-ce:latest
```
This will download and start GitLab CE container and publish ports needed to access SSH, HTTP and HTTPS.
All GitLab data will be stored as subdirectories of `/srv/gitlab/`.
The container will automatically `restart` after system reboot.
After this you can login to the web interface as explained above in 'After starting a container'.
## Where is the data stored?
The GitLab container uses host mounted volumes to store persistent data:
- `/srv/gitlab/data` mounted as `/var/opt/gitlab` in the container is used for storing *application data*
- `/srv/gitlab/logs` mounted as `/var/log/gitlab` in the container is used for storing *logs*
- `/srv/gitlab/config` mounted as `/etc/gitlab` in the container is used for storing *configuration*
You can fine tune these directories to meet your requirements.
### Configure GitLab
This container uses the official Omnibus GitLab package, so all configuration is done in the unique configuration file `/etc/gitlab/gitlab.rb`.
To access GitLab configuration, you can start an bash in a context of running container. This will allow you to browse all directories and use your favorite text editor:
```bash
sudo docker exec -it gitlab /bin/bash
```
You can also edit just `/etc/gitlab/gitlab.rb`:
```bash
sudo docker exec -it gitlab vi /etc/gitlab/gitlab.rb
```
**You should set the `external_url` to point to a valid URL.**
**You may also be interested in [Enabling HTTPS](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#enable-https).**
**To receive e-mails from GitLab you have to configure the [SMTP settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/smtp.md),
because Docker image doesn't have a SMTP server.**
**Note** that GitLab will reconfigure itself **at each container start.** You will need to restart the container to reconfigure your GitLab:
```bash
sudo docker restart gitlab
```
For more options for configuring the container please check [Omnibus GitLab documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#configuration).
## Diagnose potential problems
Read container logs:
```bash
sudo docker logs gitlab
```
Enter running container:
```bash
sudo docker exec -it gitlab /bin/bash
```
From within container you can administrer GitLab container as you would normally administer Omnibus installation: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md.
## After starting a container
After starting a container you can go to [http://localhost/](http://localhost/) or [http://192.168.59.103/](http://192.168.59.103/) if you use boot2docker.
It might take a while before the docker container is responding to queries.
You can check the status with something like `sudo docker logs -f gitlab`.
You can login to the web interface with username `root` and password `5iveL!fe`.
Next time, you can just use docker start and stop to run the container.
### Upgrade GitLab to newer version
To upgrade GitLab to new version you have to do:
1. pull new image,
```bash
sudo docker stop gitlab
```
1. stop running container,
```bash
sudo docker rm gitlab
```
1. remove existing container,
```bash
sudo docker pull gitlab/gitlab-ce:latest
```
1. create the container once again with previously specified options.
```bash
sudo docker run --detach \
--hostname gitlab.example.com \
--publish 8443:443 --publish 8080:80 --publish 2222:22 \
--name gitlab \
--restart always \
--volume /srv/gitlab/config:/etc/gitlab \
--volume /srv/gitlab/logs:/var/log/gitlab \
--volume /srv/gitlab/data:/var/opt/gitlab \
gitlab/gitlab-ce:latest
```
On the first run GitLab will reconfigure and update itself.
### Use tagged versions of GitLab
We provide tagged version of GitLab docker images.
To see all available tags check [GitLab-CE Tags](https://hub.docker.com/r/gitlab/gitlab-ce/tags/) and [GitLab-EE Tags](https://hub.docker.com/r/gitlab/gitlab-ce/tags/).
To use specific tagged version replace `gitlab/gitlab-ce:latest` with `gitlab/gitlab-ce:8.0.2`.
### Run GitLab CE on public IP address
You can make Docker to use your IP address and forward all traffic to the GitLab CE container.
You can do that by modifying the `--publish` ([Binding container ports to the host](https://docs.docker.com/articles/networking/#binding-ports)):
> --publish=[] : Publish a container᾿s port or a range of ports to the host format: ip:hostPort:containerPort | ip::containerPort | hostPort:containerPort | containerPort
To expose GitLab CE on IP 1.1.1.1:
```bash
sudo docker run --detach \
--hostname gitlab.example.com \
--publish 1.1.1.1:443:443 --publish 1.1.1.1:80:80 --publish 1.1.1.1:22:22 \
--name gitlab \
--restart always \
--volume /srv/gitlab/config:/etc/gitlab \
--volume /srv/gitlab/logs:/var/log/gitlab \
--volume /srv/gitlab/data:/var/opt/gitlab \
gitlab/gitlab-ce:latest
```
You can then access GitLab instance at http://1.1.1.1/ and https://1.1.1.1/.
## Troubleshooting
### Permission problems
When updating from older GitLab Docker images you might encounter permission problems.
This happens due to a fact that users in previous images were not preserved correctly.
There's script that fixes permissions for all files.
To fix your container, simply execute `update-permissions` script and restart container afterwards:
```
sudo docker exec gitlab update-permissions
sudo docker restart gitlab
```

1
docker/.dockerignore Normal file
View File

@ -0,0 +1 @@
*.md

33
docker/Dockerfile Normal file
View File

@ -0,0 +1,33 @@
FROM ubuntu:14.04
MAINTAINER Kamil Trzciński <kamil@gitlab.com>
# Install required packages
RUN apt-get update -q \
&& DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
ca-certificates \
openssh-server \
wget \
apt-transport-https \
vim \
nano \
patch
# Copy assets
COPY RELEASE /
COPY assets/ /assets/
RUN /assets/setup
# Allow to access embedded tools
ENV PATH /opt/gitlab/embedded/bin:/opt/gitlab/bin:/assets:$PATH
# Resolve error: TERM environment variable not set.
ENV TERM xterm
# Expose web & ssh
EXPOSE 443 80 22
# Define data volumes
VOLUME ["/etc/gitlab", "/var/opt/gitlab", "/var/log/gitlab"]
# Wrapper to handle signal, trigger runit and reconfigure GitLab
CMD ["/assets/wrapper"]

1
docker/README.md Normal file
View File

@ -0,0 +1 @@
The latest docker guide can be found here: [GitLab Docker images](/doc/docker/README.md).

View File

@ -0,0 +1,50 @@
--- etc/gitlab.rb.template.bak 2015-09-22 20:55:42.088202003 +0000
+++ etc/gitlab.rb.template 2015-09-22 20:55:50.888202003 +0000
@@ -3,7 +3,7 @@
## Url on which GitLab will be reachable.
## For more details on configuring external_url see:
## https://gitlab.com/gitlab-org/omnibus-gitlab/blob/629def0a7a26e7c2326566f0758d4a27857b52a3/README.md#configuring-the-external-url-for-gitlab
-external_url 'GENERATED_EXTERNAL_URL'
+# external_url 'GENERATED_EXTERNAL_URL' # default: http://hostname
## Note: configuration settings below are optional.
--- embedded/cookbooks/gitlab.bak/recipes/remove_accounts.rb 2015-09-22 20:50:46.964202003 +0000
+++ embedded/cookbooks/gitlab/recipes/remove_accounts.rb 2015-09-22 20:51:49.256202003 +0000
@@ -16,8 +16,8 @@
#
Gitlab[:node] = node
-if File.exists?("/etc/gitlab/gitlab.rb")
- Gitlab.from_file("/etc/gitlab/gitlab.rb")
+if File.exists?("/assets/gitlab.rb")
+ Gitlab.from_file("/assets/gitlab.rb")
end
node.consume_attributes(Gitlab.generate_config(node['fqdn']))
--- embedded/cookbooks/gitlab.bak/recipes/show_config.rb 2015-09-22 20:50:46.964202003 +0000
+++ embedded/cookbooks/gitlab/recipes/show_config.rb 2015-09-22 20:52:02.716202003 +0000
@@ -16,9 +16,9 @@
# limitations under the License.
#
-if File.exists?("/etc/gitlab/gitlab.rb")
+if File.exists?("/assets/gitlab.rb")
Gitlab[:node] = node
- Gitlab.from_file("/etc/gitlab/gitlab.rb")
+ Gitlab.from_file("/assets/gitlab.rb")
end
config = Gitlab.generate_config(node['fqdn'])
--- embedded/cookbooks/gitlab.bak/recipes/default.rb 2015-09-22 20:50:46.964202003 +0000
+++ embedded/cookbooks/gitlab/recipes/default.rb 2015-09-22 20:52:13.880202003 +0000
@@ -31,8 +31,8 @@
end.run_action(:create)
Gitlab[:node] = node
-if File.exists?("/etc/gitlab/gitlab.rb")
- Gitlab.from_file("/etc/gitlab/gitlab.rb")
+if File.exists?("/assets/gitlab.rb")
+ Gitlab.from_file("/assets/gitlab.rb")
end
node.consume_attributes(Gitlab.generate_config(node['fqdn']))

13
docker/assets/gitlab.rb Normal file
View File

@ -0,0 +1,13 @@
# Docker options
## Prevent Postgres from trying to allocate 25% of total memory
postgresql['shared_buffers'] = '1MB'
# Manage accounts with docker
manage_accounts['enable'] = false
# Get hostname from shell
host = `hostname`.strip
external_url "http://#{host}"
# Load /etc/gitlab/gitlab.rb
from_file("/etc/gitlab/gitlab.rb")

41
docker/assets/setup Executable file
View File

@ -0,0 +1,41 @@
#!/bin/bash
set -xe
source /RELEASE
# Download & Install GitLab
echo "deb https://packages.gitlab.com/gitlab/${PACKAGECLOUD_REPO}/ubuntu/ `lsb_release -cs` main" > /etc/apt/sources.list.d/gitlab_${RELEASE_PACKAGE}.list
wget -q -O - https://packages.gitlab.com/gpg.key | apt-key add -
apt-get update
apt-get install -yq --no-install-recommends ${RELEASE_PACKAGE}=${RELEASE_VERSION}
# Create sshd daemon
mkdir -p /opt/gitlab/sv/sshd/supervise /opt/gitlab/sv/sshd/log/supervise
mkfifo /opt/gitlab/sv/sshd/supervise/ok /opt/gitlab/sv/sshd/log/supervise/ok
printf "#!/bin/sh\nexec 2>&1\numask 077\nexec /usr/sbin/sshd -D -f /assets/sshd_config -e" > /opt/gitlab/sv/sshd/run
printf "#!/bin/sh\nexec svlogd -tt /var/log/gitlab/sshd" > /opt/gitlab/sv/sshd/log/run
chmod a+x /opt/gitlab/sv/sshd/run /opt/gitlab/sv/sshd/log/run
mkdir -p /var/run/sshd
# Remove current gitlab.rb file
rm -f /etc/gitlab/gitlab.rb
# Patch omnibus package
patch -p0 -d /opt/gitlab < /assets/gitlab-rb-location.patch
# Create groups
groupadd -g 998 git
groupadd -g 999 gitlab-www
groupadd -g 997 gitlab-redis
groupadd -g 996 gitlab-psql
# groupadd -g 995 gitlab-ci
groupadd -g 994 mattermost
# Create accounts
useradd -m -u 998 -g git -m -s /bin/sh -d /var/opt/gitlab git
useradd -m -u 999 -g gitlab-www -m -s /bin/false -d /var/opt/gitlab/nginx gitlab-www
useradd -m -u 997 -g gitlab-redis -m -s /bin/nologin -d /var/opt/gitlab/redis gitlab-redis
useradd -m -u 996 -g gitlab-psql -m -s /bin/sh -d /var/opt/gitlab/postgresql gitlab-psql
# useradd -m -u 995 -g gitlab-ci -m -s /bin/sh -d /var/opt/gitlab/gitlab-ci gitlab-ci
useradd -m -u 994 -g mattermost -m -s /bin/sh -d /var/opt/gitlab/mattermost mattermost

18
docker/assets/sshd_config Normal file
View File

@ -0,0 +1,18 @@
Port 22
ChallengeResponseAuthentication no
HostKey /etc/gitlab/ssh_host_rsa_key
Protocol 2
PermitRootLogin no
PasswordAuthentication no
MaxStartups 100:30:200
AllowUsers git
PrintMotd no
PrintLastLog no
PubkeyAuthentication yes
# Fix: User username not allowed because account is locked
# With "UsePAM yes" the "!" is seen as a password disabled account and not fully locked so ssh public key login works
UsePAM yes
# Disabling use DNS in ssh since it tends to slow connecting
UseDNS no

View File

@ -0,0 +1,44 @@
#!/bin/bash
set -x
# Fix GitLab permissions
if id -u git; then
# Fix data storage
chown -R git:git /var/opt/gitlab/.ssh
chown -R git:git /var/opt/gitlab/.gitconfig
chown -R git:git /var/opt/gitlab/git-data
chown -R git:git /var/opt/gitlab/gitlab-ci/builds
chown -R git:git /var/opt/gitlab/gitlab-git-http-server
chown -R git:git /var/opt/gitlab/gitlab-rails
chown -R git:git /var/opt/gitlab/gitlab-shell
# Fix log storage
chown git /var/log/gitlab/gitlab-git-http-server
chown git /var/log/gitlab/gitlab-rails
chown git /var/log/gitlab/gitlab-shell
chown git /var/log/gitlab/sidekiq
chown git /var/log/gitlab/unicorn
chown gitlab-psql /var/log/gitlab/postgresql
chown gitlab-redis /var/log/gitlab/redis
# Update log files
chown -R git:git /var/log/gitlab/gitlab-rails/*.log
chown -R git:git /var/log/gitlab/gitlab-shell/*.log
chown -R git:git /var/log/gitlab/unicorn/*.log
fi
# Fix nginx buffering directory permission
if id -u gitlab-www; then
chown -R gitlab-www:gitlab-www /var/opt/gitlab/nginx/*_temp
fi
# Fix database storage
if id -u gitlab-psql; then
chown -R gitlab-psql:gitlab-psql /var/opt/gitlab/postgresql
fi
# Fix redis storage
if id -u gitlab-redis; then
chown gitlab-redis:gitlab-redis /var/opt/gitlab/redis
fi

66
docker/assets/wrapper Executable file
View File

@ -0,0 +1,66 @@
#!/bin/bash
set -e
function sigterm_handler() {
echo "SIGTERM signal received, try to gracefully shutdown all services..."
gitlab-ctl stop
}
trap "sigterm_handler; exit" TERM
source /RELEASE
echo "Thank you for using GitLab Docker Image!"
echo "Current version: $RELEASE_PACKAGE=$RELEASE_VERSION"
echo ""
if [[ "$PACKAGECLOUD_REPO" == "unstable" ]]; then
echo "You are using UNSTABLE version of $RELEASE_PACKAGE!"
echo ""
fi
echo "Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file"
echo "And restart this container to reload settings."
echo "To do it use docker exec:"
echo
echo " docker exec -it gitlab vim /etc/gitlab/gitlab.rb"
echo " docker restart gitlab"
echo
echo "For a comprehensive list of configuration options please see the Omnibus GitLab readme"
echo "https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md"
echo
echo "If this container fails to start due to permission problems try to fix it by executing:"
echo
echo " docker exec -it gitlab update-permissions"
echo " docker restart gitlab"
echo
sleep 3s
# Copy gitlab.rb for the first time
if [[ ! -e /etc/gitlab/gitlab.rb ]]; then
echo "Installing gitlab.rb config..."
cp /opt/gitlab/etc/gitlab.rb.template /etc/gitlab/gitlab.rb
chmod 0600 /etc/gitlab/gitlab.rb
fi
# Generate ssh host key for the first time
if [[ ! -f /etc/gitlab/ssh_host_rsa_key ]]; then
echo "Generating ssh_host_rsa_key..."
ssh-keygen -f /etc/gitlab/ssh_host_rsa_key -N '' -t rsa
chmod 0600 /etc/gitlab/ssh_host_rsa_key
fi
# Remove all services, the reconfigure will create them
echo "Preparing services..."
rm -f /opt/gitlab/service/*
ln -s /opt/gitlab/sv/sshd /opt/gitlab/service
mkdir -p /var/log/gitlab/sshd
# Start service manager
echo "Starting services..."
/opt/gitlab/embedded/bin/runsvdir-start &
# Configure gitlab package
echo "Configuring GitLab..."
gitlab-ctl reconfigure
# Tail all logs
gitlab-ctl tail

31
docker/marathon.json Normal file
View File

@ -0,0 +1,31 @@
{
"id": "/gitlab",
"ports": [0,0],
"cpus": 2,
"mem": 2048.0,
"disk": 10240.0,
"container": {
"type": "DOCKER",
"docker": {
"network": "HOST",
"image": "gitlab/gitlab-ce:latest"
},
"volumes": [
{
"containerPath": "/etc/gitlab",
"hostPath": "/var/data/etc/gitlab",
"mode": "RW"
},
{
"containerPath": "/var/opt/gitlab",
"hostPath": "/var/data/opt/gitlab",
"mode": "RW"
},
{
"containerPath": "/var/log/gitlab",
"hostPath": "/var/data/log/gitlab",
"mode": "RW"
}
]
}
}