Note: Currently new registrations are closed, if you want an account Contact us

Difference between revisions of "GitLab"

From Free Software Community of India
Jump to: navigation, search
(Maintenance: add qa tasks here)
 
(3 intermediate revisions by the same user not shown)
Line 3: Line 3:
 
==Hosting==
 
==Hosting==
  
Sponsored by [http://about.gitlab.com GitLab Inc].Hosted at gandi.net in France.
+
Sponsored by [http://about.gitlab.com GitLab Inc]. Hosted at gandi.net in France.
  
 
==Setup==
 
==Setup==
  
Running on debian gnu/linux 9 stretch with postgresql 9.6. Using gitlab package from https://people.debian.org/~praveen/gitlab/
+
Running on debian gnu/linux 10 buster with postgresql 11. Using gitlab package from http://fasttrack.debian.net
  
Note: We install gitlab dependencies from stretch-backports, so all apt commands should enable stretch-backports (''apt -t stretch-backports'').
+
Check https://wiki.debian.org/gitlab for installation/update instructions.
  
Debian package specific documentation -> https://salsa.debian.org/ruby-team/gitlab/raw/stretch-backports/debian/README.Debian
+
Debian package specific documentation -> https://salsa.debian.org/ruby-team/gitlab/raw/buster-fasttrack/debian/README.Debian
  
 
Letsencrypt domains:
 
Letsencrypt domains:
Line 45: Line 45:
 
==Backup==
 
==Backup==
  
Backup server is provided by Manu. Running on debian gnu/linux 9 stretch.
+
Backup server is provided by Manu (KVM virtual machine with 100 GB storage and 1 GB ram). Running on debian gnu/linux 10 buster.  
 +
 
 +
Documentation: https://linuxhint.com/setup_postgresql_replication/ and https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/
  
 
===Slave configuration: Step 1===
 
===Slave configuration: Step 1===
 
Install postgresql and rsync
 
Install postgresql and rsync
  
  # apt-get install postgresql-contrib-9.6 rsync
+
  # apt-get install postgresql-contrib-11 rsync
 +
 
 +
Check postgresql server is running
 +
 
 +
# su postgres -c psql
  
 
Make sure en_US.UTF-8 locale is available
 
Make sure en_US.UTF-8 locale is available
Line 59: Line 65:
  
 
  # su - postgres
 
  # su - postgres
  $ ssh-keygen
+
  $ ssh-keygen -t ed25519
  
Now copy /var/lib/postgresql/.ssh/id_rsa.pub to master postgres users' /var/lib/postgresql/.ssh/authorized_keys
+
Now copy /var/lib/postgresql/.ssh/id_ed25519.pub to master postgres users' /var/lib/postgresql/.ssh/authorized_keys
  
 
Stop postgresql before changing any configuration
 
Stop postgresql before changing any configuration
  $ pg_ctlcluster 9.6 main stop
+
  # systemctl stop postgresql@11-main
  
  cd /etc/postgresql/9.6/main
+
as postgres user
 +
  $ su - postgres
 +
$ cd /etc/postgresql/11/main
  
 
Open the postgres configuration file
 
Open the postgres configuration file
  
  nano postgresql.conf
+
  $ nano postgresql.conf
 +
 
 
Set the following configuration options in the postgresql.conf file
 
Set the following configuration options in the postgresql.conf file
  
 
  listen_addresses = 'localhost,192.168.0.115'
 
  listen_addresses = 'localhost,192.168.0.115'
 
  port=5432
 
  port=5432
  wal_level = 'hot_standby'
+
  wal_level = replica
 +
max_wal_senders = 1
 
  archive_mode = on
 
  archive_mode = on
 
  archive_command = 'cd .'
 
  archive_command = 'cd .'
max_wal_senders = 1
 
 
  hot_standby = on
 
  hot_standby = on
  
Line 96: Line 105:
 
Allow slave to connect to master using the user just created.
 
Allow slave to connect to master using the user just created.
  
  $ cd /etc/postgresql/9.6/main
+
  $ cd /etc/postgresql/11/main
  
 
  $ nano pg_hba.conf
 
  $ nano pg_hba.conf
Line 112: Line 121:
 
  listen_addresses = 'localhost,213.167.243.152'
 
  listen_addresses = 'localhost,213.167.243.152'
 
  port=5432
 
  port=5432
  wal_level = 'hot_standby'
+
  wal_level = replica
 +
max_wal_senders = 1
 
  archive_mode = on
 
  archive_mode = on
 
  archive_command = 'cd .'
 
  archive_command = 'cd .'
max_wal_senders = 1
 
 
  hot_standby = on
 
  hot_standby = on
  
 
Now, to activate your changes, reload the postgresql server
 
Now, to activate your changes, reload the postgresql server
  
  $ pg_ctlcluster 9.6 main reload
+
  # systemctl reload postgresql@11-main
  
 
You may need to restart it via systemd,
 
You may need to restart it via systemd,
Line 126: Line 135:
 
  # systemctl restart postgresql
 
  # systemctl restart postgresql
  
Start the backup process,
+
Open 5432 port in the firewall
  
  psql -c "select pg_start_backup('initial_backup');"
+
  # ufw allow from 62.210.83.200 to any port 5432 proto tcp
rsync -cva -e 'ssh -p 12022' --inplace --exclude=*pg_xlog* /var/lib/postgresql/9.6/main/ 62.210.83.200:/var/lib/postgresql/9.6/main/
 
psql -c "select pg_stop_backup();"
 
  
 
===Slave Configuration: Step 2===
 
===Slave Configuration: Step 2===
Create a recovery file called  recovery.conf and add following lines.
+
 
  standby_mode = 'on'
+
Copy data from master and create recovery.conf
primary_conninfo = 'host=213.167.243.152 port=5432 user=rep password=yourpassword'
+
  $ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R
  trigger_file = '/tmp/postgresql.trigger.5432'
 
  
 
Start the slave server
 
Start the slave server
  
  $ pg_ctlcluster 9.6 main start
+
  # systemctl start postgresql@11-main
  
 
===Replication Status===
 
===Replication Status===
Line 147: Line 153:
  
 
  $ ps -ef | grep sender
 
  $ ps -ef | grep sender
 +
$ psql -c "select * from pg_stat_activity where usename='rep';"
 +
 
On slave server,
 
On slave server,
  

Latest revision as of 22:47, 6 March 2020

Our public GitLab instance is https://git.fosscommunity.in

Hosting

Sponsored by GitLab Inc. Hosted at gandi.net in France.

Setup

Running on debian gnu/linux 10 buster with postgresql 11. Using gitlab package from http://fasttrack.debian.net

Check https://wiki.debian.org/gitlab for installation/update instructions.

Debian package specific documentation -> https://salsa.debian.org/ruby-team/gitlab/raw/buster-fasttrack/debian/README.Debian

Letsencrypt domains:

 letsencrypt --expand --webroot --webroot-path /usr/share/gitlab/public -d git.fosscommunity.in -d gitlabce.tk -d gitlab.debian.net -d wiki.fsci.org.in -d git.fsci.org.in  certonly

Mail Server Setup

Maintenance

Maintenance discussion at this loomio subgroup and this matrix chat room

Gitlab QA - running tests

<Create a user account for QA>

$ sudo gem install gitlab-qa

$ GITLAB_USERNAME=<username> GITLAB_PASSWORD=<password> gitlab-qa Test::Instance::Any gitlab/gitlab-ce-qa:<gitlab version> https://git.hacksk.xyz

Backup

Backup server is provided by Manu (KVM virtual machine with 100 GB storage and 1 GB ram). Running on debian gnu/linux 10 buster.

Documentation: https://linuxhint.com/setup_postgresql_replication/ and https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Slave configuration: Step 1

Install postgresql and rsync

# apt-get install postgresql-contrib-11 rsync

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Create ssh key for postgres user

# su - postgres
$ ssh-keygen -t ed25519

Now copy /var/lib/postgresql/.ssh/id_ed25519.pub to master postgres users' /var/lib/postgresql/.ssh/authorized_keys

Stop postgresql before changing any configuration

# systemctl stop postgresql@11-main

as postgres user

$ su - postgres
$ cd /etc/postgresql/11/main

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,192.168.0.115'
port=5432
wal_level = replica
max_wal_senders = 1
archive_mode = on
archive_command = 'cd .'
hot_standby = on

Master configuration

Create and copy ssh public key to slave like above.

# su - postgres
$ ssh-keygen

Now copy /var/lib/postgresql/.ssh/id_rsa.pub to slave postgres users' /var/lib/postgresql/.ssh/authorized_keys

Create postgresql user for replication.

$ psql -c "CREATE USER rep REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"

Allow slave to connect to master using the user just created.

$ cd /etc/postgresql/11/main
$ nano pg_hba.conf

Add below line to allow rep user to get access to the server

host    replication     rep     62.210.83.200/32   md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,213.167.243.152'
port=5432
wal_level = replica
max_wal_senders = 1
archive_mode = on
archive_command = 'cd .'
hot_standby = on

Now, to activate your changes, reload the postgresql server

# systemctl reload postgresql@11-main

You may need to restart it via systemd,

# systemctl restart postgresql

Open 5432 port in the firewall

# ufw allow from 62.210.83.200 to any port 5432 proto tcp

Slave Configuration: Step 2

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch  -R

Start the slave server

# systemctl start postgresql@11-main

Replication Status

On master server ,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On slave server,

$ ps -ef | grep receiver