Difference between revisions of "Poddery - Diaspora, Matrix and XMPP"

From Free Software Community of India
Jump to: navigation, search
(Coordination)
(Workers)
 
(87 intermediate revisions by 8 users not shown)
Line 1: Line 1:
We run a diaspora service at [http://poddery.com poddery.com]
+
We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.
  
== Environment ==
+
= Environment =
We use diaspora-installer-mysql package from  https://people.debian.org/~praveen/diaspora
+
== Hosting ==
See /usr/share/doc/diaspora-common/README for package specific configuration.
+
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:
  
'''System health check:'''
+
* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
 +
* 4TB HDD
 +
* 32GB DDR3 RAM
  
* There should be a data disk attached (added from cloud.scaleway.com)
+
== Operating System ==
* The attached disk (/dev/nbd2) should be an lvm physical volume (pvcreate /dev/nbd2). We cannot use it directly for encryption, so we use lvm.
+
* Debian Buster
* /dev/data is an lvm volume group created from /dev/nbd2 (vgcreate data /dev/nbd2).
 
* /dev/data/diaspora is an lvm logical volume (lvcreate -n diaspora /dev/data -L 93.5G).
 
* /dev/mapper/diaspora is an encrypted device (cryptsetup luksFormat /dev/data/diaspora; cryptsetup luksOpen /dev/data/diaspora diaspora)
 
* /dev/mapper/diaspora is an ext4 file system (mkfs.ext4 /dev/mapper/diaspora)
 
* /var/lib/diaspora should be mounted (mount /dev/mapper/diaspora /var/lib/diaspora)
 
* all critical data should be on /var/lib/diaspora.
 
** /etc/init.d/mysql stop; mv /var/lib/mysql /var/lib/diaspora; ln /var/lib/diaspora/mysql /var/lib/diaspora
 
** mkdir /var/lib/diaspora/uploads; chown -R diaspora: /var/lib/diaspora/uploads;ln -s /var/lib/diaspora/uploads /usr/share/diaspora/public/uploads
 
  
== Homepage ==
+
== User Visible Services ==
 +
=== Diaspora ===
 +
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
 +
* For live statistics see https://poddery.com/statistics
  
Homepage and other static pages are maintained in our gitlab instance. You can change it directly in the master branch or send pull requests. You can edit it via web as well.
+
=== Chat/XMPP ===
 +
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
 +
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
 +
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.
  
* poddery.com -> https://git.fosscommunity.in/community/poddery.com.
+
=== Chat/Matrix ===
 +
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
 +
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
 +
* Riot-web Matrix client is hosted at https://chat.poddery.com
  
 +
=== Homepage ===
 +
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
 +
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
 
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
 
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
 +
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery
  
save.poddery.com repo is maintained as a sub module in poddery.com repo. See this tutorial -> https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ for working with git submodules.
+
== Backend Services ==
 +
=== Web Server / Reverse Proxy ===
 +
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.
  
== SSL/TLS certificates ==
+
=== Database ===
 +
* PostgreSQL for Matrix
 +
* MySQL for Diaspora
  
# letsencrypt certonly --webroot -w /usr/share/diaspora/public  -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com
+
''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).
# cp  -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
 
# cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
 
# chown -R root:ssl-cert /etc/letsencrypt
 
# chmod g+r -R /etc/letsencrypt
 
# chmod g+x /etc/letsencrypt/*
 
  
# ls -l /etc/prosody/certs/
+
=== Email ===
total 0
+
* Exim
lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem
 
lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem
 
  
# crontab -e
+
=== SSL/TLS certificates ===
30 2 * * 1 letsencrypt renew  >> /var/log/le-renew.log
+
* Let's Encrypt
32 2 * * 1 /etc/init.d/nginx reload
 
34 2 * * 1 /etc/init.d/prosody reload
 
  
== XMPP/Chat ==
+
=== Firewall ===
 +
* UFW (Uncomplicated Firewall)
  
* We use prosody and steps for setting up prosody is given at -> https://wiki.debian.org/Diaspora/XMPP
+
=== Intrusion Prevention ===
* We have enabled all XEPs conversations expect. We use sslh to multiplex diaspora and prosody on port 443.
+
* Fail2ban
  
== Statistics ==
+
= Coordination =
 +
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
 +
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
 +
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks
  
* [https://poddery.com/statistics live statistics]
+
=== Contact ===
 +
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
 +
* The following people have their GPG keys in the [[#Server_Access|access file]]:
 +
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
 +
** ID: 0xB77D2E2E23735427 - Balasankar C
 +
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
 +
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
 +
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
 +
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
 +
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
 +
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
 +
** ID: 0x0B1955F40C691CCE - Kannan
 +
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
 +
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
 +
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
 +
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].
  
== Coordination ==
+
=== Server Access ===
 +
Maintained in a private git repo at https://git.fosscommunity.in/community/access
  
*[https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group loomio group] - we use this for decision making.
+
= Configuration and Maintenance =
* Hangout with us in our matrix room [http://matrix.to/#/poddery@matrix.org #poddery:matrix.org]
+
== Disk Partitioning ==
 +
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
 +
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
 +
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
 +
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
 +
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
 +
cryptsetup luksFormat /dev/mdX
 +
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
 +
cryptsetup luksOpen /dev/mdX poddery
 +
 +
# LVM Setup
 +
# Create physical volume named <code>poddery</code>
 +
pvcreate /dev/mapper/poddery
 +
# Create volume group named <code>data</code>
 +
vgcreate data /dev/mapper/poddery
 +
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
 +
lvcreate -n log /dev/data -L 50G
 +
lvcreate -n db /dev/data -L 500G
 +
# Assign remaining free space for static files
 +
lvcreate -n static /dev/data -l 100%FREE
 +
 +
# Setup filesystem on the logical volumes
 +
mkfs.ext4 /dev/data/log
 +
mkfs.ext4 /dev/data/db
 +
mkfs.ext4 /dev/data/static
 +
 +
# Create directories for mounting the encrypted partitions
 +
mkdir /var/lib/db /var/lib/static /var/log/poddery
 +
 +
# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
 +
mount /dev/data/db /var/lib/db
 +
mount /dev/data/static /var/lib/static
 +
mount /dev/data/log /var/log/poddery
  
== Contact ==
+
== Hardening checklist ==
 +
* SSH password based login disabled (allow only key based logins)
 +
* SSH login disabled for root user (use a normal user with sudo)
 +
# Check for the following settings in /etc/ssh/sshd_config:
 +
...
 +
PermitRootLogin no
 +
...
 +
PasswordAuthentication no
 +
...
  
Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
+
* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
 +
ufw default deny incoming
 +
ufw default allow outgoing
 +
ufw allow ssh
 +
ufw allow http/tcp
 +
ufw allow https/tcp
 +
ufw allow Turnserver
 +
ufw allow XMPP
 +
ufw allow 8448
  
The following people have their gpg keys in the password file.
+
ufw enable
 +
 +
# Verify everything is setup properly
 +
ufw status
 +
 +
# Enable ufw logging with default mode low
 +
ufw logging on
  
Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA)
+
* <code>fail2ban</code> configured against brute force attacks:
 +
# Check for the following line <code>/etc/ssh/sshd_config</code>
 +
...
 +
LogLevel VERBOSE
 +
...
 +
 +
# Restart SSH and enable fail2ban
 +
systemctl restart ssh
 +
systemctl enable fail2ban
 +
systemctl start fail2ban
 +
 +
# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
 +
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
 +
fail2ban-client set sshd unbanip <banned_ip>
  
== Server Access ==
+
== Diaspora ==
 +
* Install <code>diaspora-installer</code> from Debian Buster contrib:
 +
apt install diaspora-installer
  
Maintained in a private git repo at -> https://git.fosscommunity.in/community/access
+
* Move MySQL data to encrypted partition:
 +
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
 +
systemctl stop mysql
 +
systemctl disable mysql
 +
mv /var/lib/mysql /var/lib/db/
 +
ln -s /var/lib/db/mysql /var/lib/
 +
systemctl start mysql
  
== Setting up Backup ==
+
* Move static files to encrypted partition:
 +
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
 +
mkdir /var/lib/static/diaspora
 +
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
 +
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
 +
chown -R diaspora: /var/lib/static/diaspora
  
Backup is setup on a Scaleway C1 VPS (4 core ARM processor with 2GB RAM)
+
* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
 +
* Homepage configuration:
 +
# Make sure <code>git</code> and <code>acl</code> packages are installed
 +
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
 +
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public
 +
 +
# Clone poddery.com repo
 +
cd /usr/share/diaspora/public
 +
git clone https://git.fosscommunity.in/community/poddery.com.git
 +
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
 +
cd .. && rmdir poddery.com
  
Hostname (IP): backup.poddery.com (No public ip, access via scaleway.com web console). If you restart this machine, you may want to add poddery.com private ip in /etc/hosts
+
* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
 +
# Clone save.poddery.com repo
 +
cd /usr/share/diaspora/public/save
 +
git submodule init
 +
git submodule update
  
  # apt-get install lvm2 cryptsetup
+
== Matrix ==
 +
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
 +
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
 +
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
 +
* Move PostgreSQL data to encrypted partition:
 +
  # Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
 +
systemctl stop postgresql
 +
systemctl disable postgresql
 +
mv /var/lib/postgres /var/lib/db/
 +
ln -s /var/lib/db/postgres /var/lib/
 +
systemctl start postgresql
  
Directly creating luks volume on /dev/nbd1 is not working, so we use a logical volume
+
* Move static files to encrypted partition:
 +
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
 +
mkdir /var/lib/static/synapse
 +
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
 +
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
 +
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
 +
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
 +
chown -R matrix-synapse: /var/lib/static/synapse
  
# pvcreate /dev/nbd1
+
* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])
# vgcreate data /dev/nbd1
 
# lvcreate -n diaspora -L 46.5G /dev/data
 
  
# cryptsetup luksFormat /dev/data/diaspora
+
=== Workers ===
# cryptsetup luksOpen /dev/data/diaspora diaspora
+
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
 +
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
 +
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
 +
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
 +
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
 +
  enable_media_repo: False
 +
  send_federation: False
 +
  start_pushers: False
 +
  update_user_directory: false
  
and update /etc/crypttab
+
* These services must be enabled:
# <target name> <source device>        <key file>      <options>
 
diaspora /dev/data/diaspora none luks
 
  
 +
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service
  
# mkfs.ext4 /dev/mapper/diaspora
+
To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>
# mkdir /var/lib/diaspora
 
and update /etc/fstab
 
# UNCONFIGURED FSTAB FOR BASE SYSTEM
 
/dev/mapper/diaspora /var/lib/diaspora ext4 defaults 0 2
 
  
# mount -a
+
=== Synapse Updation ===
# apt-get install mysql-server
+
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>
  
Move mysql data directory to encrypted volume
+
=== Riot-web Updation ===
# /etc/init.d/mysql stop
+
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
  # mv /var/lib/mysql /var/lib/diaspora/
+
  /var/www/get-riot <version>
# ln -s /var/lib/diaspora/mysql /var/lib/mysql
 
  
Follow steps in https://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html for replication
+
== Chat/XMPP ==
 +
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
 +
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
 +
mysql -u root -p # Enter password from the access repo
 +
 +
CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
 +
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
 +
FLUSH PRIVILEGES;
 +
 +
systemctl restart prosody
  
Follow steps in https://www.howtoforge.com/how-to-set-up-mysql-database-replication-with-ssl-encryption-on-centos-5.4 for ssl (but ssl support is disabled in debian)
+
* Install plugins
 +
# Make sure <code>mercurial</code> is installed
 +
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules
  
Follow steps in http://www.networkcomputing.com/storage/how-set-ssh-encrypted-mysql-replication/1111882674 to use ssh port forwarding to have encrypted replication
+
=== Set Nginx Conf for BOSH URLS ===
 +
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
 +
upstream chat_cluster {
 +
  server localhost:5280;
 +
}
  
  # adduser sshtunnel --disabled-login
+
  location /http-bind {
  # su sshtunnel
+
  proxy_set_header X-Real-IP $remote_addr;
 +
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 +
  proxy_set_header Host $http_host;
 +
  proxy_set_header X-Forwarded-Proto https;
 +
  proxy_redirect off;
 +
  proxy_connect_timeout 5;
 +
  proxy_buffering      off;
 +
  proxy_read_timeout    70;
 +
  keepalive_timeout    70;
 +
  send_timeout          70;
 +
  client_max_body_size 4M;
 +
  client_body_buffer_size 128K;
 +
  proxy_pass http://chat_cluster;
 +
  }
  
Generate ssh key pair and copy public key to target system
+
* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].
$ ssh-keygen -t rsa
 
$ ssh -f sshtunnel@poddery.com -L 7777:127.0.0.1:3306 -N
 
  
Test the connectivity
+
== TLS ==
  # mysql -u poddery_backup -p -P 7777 -h 127.0.0.1
+
* Install <code>letsencrypt</code>.
 +
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
 +
chown -R root:ssl-cert /etc/letsencrypt
 +
chmod g+r -R /etc/letsencrypt
 +
chmod g+x /etc/letsencrypt/{archive,live}
 +
* Generate certificates. For more details see https://certbot.eff.org.
 +
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
 +
ls -l /etc/diaspora/ssl
 +
''total 0
 +
''lrwxrwxrwx 1 root root 47 Apr  2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
 +
''lrwxrwxrwx 1 root root 45 Apr  2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''
 +
 +
  # If you don't get the above output, then run the following:
 +
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
 +
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
  
Uploads are rsynced every hour
+
* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
 +
ls -l /etc/prosody/certs/
 +
''total 0''
 +
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
 +
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''
 +
 +
# If you don't get the above output, then run the following:
 +
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
 +
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key
  
  # crontab -e
+
* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
  # m h  dom mon dow  command
+
* Cron jobs:
0 * * * * pgrep rsync || rsync -av --delete root@poddery.com:/var/lib/diaspora/uploads/ /var/lib/diaspora/uploads/ >/var/lib/diaspora/rsync-uploads.log
+
  crontab -e
 +
  ''30 2 * * 1 letsencrypt renew  >> /var/log/le-renew.log''
 +
''32 2 * * 1 /etc/init.d/nginx reload''
 +
''34 2 * * 1 /etc/init.d/prosody reload''
  
 +
* Manually updating TLS certificate:
 +
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public  -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
 +
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
 +
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public --expand -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com
  
'''Note:''' Since we are not using a public ip (saves us money), backup.poddery.com connects to poddery.com via private ip. So if poddery.com is rebooted, the new ip address should be updated in /etc/hosts file of backup.poddery.com. To connect, use the web console from scaleway.com
+
= History =
 
+
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.
== Add more disk space ==
 
 
 
# Power off the machine with "ARCHIVE" option. It may take upto an hour for shutdown to complete on backup.poddery.com and poddery.com
 
# Add more disk from scaleway.com control panel . Volumes -> CREATE VOLUME
 
# Attach the newly created volume to server from Server page
 
# Power on the server
 
# Create physical volume (pvcreate /dev/nbdN)
 
# Expand volume group (vgextend data /dev/nbdN)
 
# Expand logical volume (lvresize --size=186G data/diaspora)
 
# Expand encrypted partition (cryptsetup resize diaspora)
 
# Resize file system (resize2fs /dev/mapper/diaspora)
 
 
 
== Maintenance history ==
 
This section holds maintenance/issue history for future tracking.
 
 
 
'''When updating diaspora-installer-mysql packages, remember to recreate /usr/share/diaspora/public/uploads symlink to /var/lib/diaspora/uploads'''.
 
 
 
1. Prosody error - Failed to load private key
 
 
 
certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Previous error (see logs), or other system error. (for poddery.com)
 
tls error  Unable to initialize TLS: error loading private key (system lib)
 
certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Check that the permissions allow Prosody to read this file.
 
 
 
This error is usually when ssl certificate in freshly installed or renewed. Prosody user is unable to access the key file due to lack of privileges.
 
 
 
Note that Poddery uses Letsencrypt for ssl.
 
 
 
Fix:
 
 
 
* Make sure that prosody user is in 'certs' group (this group may also be called ssl-certs as setup by Letencrypt)
 
* /etc/letsencrypt/ is the ssl directory.
 
* Prosody user should have permissions to all folders importantly archive and live folders in /etc/letsencrypt. Permissions to each folder must be 750.
 
* Troubleshoot by checking if you can switch to each folder in /etc/letsencrypt as prosody user and cat the files.
 
 
 
'''If replication fails, you can restart it following the instructions here'''
 
 
 
https://dba.stackexchange.com/questions/69394/mysql-replication-error-1594
 
  
 
[[Category:Services]]
 
[[Category:Services]]

Latest revision as of 14:28, 12 June 2019

We run decentralized and federated Diaspora social netowrk, XMPP and Matrix instant messaging services at poddery.com. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. chat.poddery.com provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

Environment

Hosting

Poddery is hosted at Hetzner with the following specs:

  • Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
  • 4TB HDD
  • 32GB DDR3 RAM

Operating System

  • Debian Buster

User Visible Services

Diaspora

Chat/XMPP

  • Prosody is used as the XMPP server which is modern and lightweight.
  • Currently installed version is 0.11.2 which is available in Debian Buster.
  • All XEPs are enabled which the Conversations app support.

Chat/Matrix

Homepage

Homepage and other static pages are maintained in FSCI GitLab instance.

Backend Services

Web Server / Reverse Proxy

  • Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

Database

  • PostgreSQL for Matrix
  • MySQL for Diaspora

TODO: Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

Email

  • Exim

SSL/TLS certificates

  • Let's Encrypt

Firewall

  • UFW (Uncomplicated Firewall)

Intrusion Prevention

  • Fail2ban

Coordination

Contact

  • Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
  • The following people have their GPG keys in the access file:
    • ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
    • ID: 0xB77D2E2E23735427 - Balasankar C
    • ID: 0x5D0064186AF037D9 - Manu Krishnan T V
    • ID: 0x51C954405D432381 - Fayad Fami (fayad)
    • ID: 0x863D4DF2ED9C28EF - Abhijith PA
    • ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
    • ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
    • ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
    • ID: 0x0B1955F40C691CCE - Kannan
    • ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
    • ID: 0xFBB7061C27CB70C1 - Ranjith Siji
    • ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
  • It's recommended to setup Vim GnuPG Plugin for transparent editing. Those who are new to GPG can follow this guide.

Server Access

Maintained in a private git repo at https://git.fosscommunity.in/community/access

Configuration and Maintenance

Disk Partitioning

  • RAID 1 setup on 2x2TB HDDs (sda and sdb).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
  • Separate partitions for swap (md0 - 16GB), boot (md1 - 512MB) and root (md2 - 50GB).
  • LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure lvm2, udev and cryptsetup packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the access repo
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named poddery
pvcreate /dev/mapper/poddery
# Create volume group named data
vgcreate data /dev/mapper/poddery
# Create logical volumes named log, db and static
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE 

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

Hardening checklist

  • SSH password based login disabled (allow only key based logins)
  • SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...
  • ufw firewall enabled with only the ports that needs to be opened (ufw tutorial):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448
ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on
  • fail2ban configured against brute force attacks:
# Check for the following line /etc/ssh/sshd_config
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check /var/log/fail2ban.log to get the banned IP and then run the following
# Here sshd is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

Diaspora

  • Install diaspora-installer from Debian Buster contrib:
apt install diaspora-installer
  • Move MySQL data to encrypted partition:
# Make sure /dev/data/db is mounted to /var/lib/db
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql
  • Move static files to encrypted partition:
# Make sure /dev/data/static is mounted to /var/lib/static
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora
  • Modify configuration files at /etc/diaspora and /etc/diaspora.conf as needed (backup of the current configuration files are available in the access repo).
  • Homepage configuration:
# Make sure git and acl packages are installed
# Grant rwx permissions for the ssh user to /usr/share/diaspora/public
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com
  • Save Poddery repo is maintained as a sub module in poddery.com repo. See this tutorial for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

Matrix

  • See the official installation guide of Synapse for installing from source.
  • Nginx is used as reverse proxy to send requests that has /_matrix/* in URL to Synapse on port 8008. This is configured in /etc/nginx/sites-enabled/diaspora.
  • Shamil's Synapse Diaspora Auth script is used to authenticate Synapse with Diaspora database.
  • Move PostgreSQL data to encrypted partition:
# Make sure /dev/data/db is mounted to /var/lib/db
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql
  • Move static files to encrypted partition:
# Make sure /dev/data/static is mounted to /var/lib/static
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse
  • Install identity server mxisd (deb package available here)

Workers

  • For scalability, Poddery is running workers. Currently all workers specified in that page, expect synapse.app.appservice is running on poddery.com
  • A new service matrix-synapse@.service is installed for the workers (Save the synape_worker file somewhere like /usr/local/bin/ or something).
  • The worker config can be found at /etc/matrix-synapse/workers
  • Synapse needs to be put under a reverse proxy see /etc/nginx/sites-enabled/matrix. A lot of /_matrix/ urls needs to be overridden too see /etc/nginx/sites-enabled/diaspora
  • These lines must be added to homeserver.yaml as we are running media_repository, federation_sender, pusher, user_dir workers respectively:
 enable_media_repo: False
 send_federation: False
 start_pushers: False
 update_user_directory: false
  • These services must be enabled:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running matrix-synchrotron-balancer. It has a systemd file at /etc/systemd/system/matrix-synchrotron-balancer. The files are in /opt/matrix-synchrotron-balancer

Synapse Updation

  • First check synapse/UPGRADE.rst to see if anything extra needs to be done. Then, just run /root/upgrade-synapse

Riot-web Updation

  • Just run the following (make sure to replace <version> with a proper version number like v1.0.0):
/var/www/get-riot <version>

Chat/XMPP

# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody
  • Install plugins
# Make sure mercurial is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

Set Nginx Conf for BOSH URLS

  • Add the following in nginx configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
  server localhost:5280;
}
location /http-bind {
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_set_header X-Forwarded-Proto https;
  proxy_redirect off;
  proxy_connect_timeout 5;
  proxy_buffering       off;
  proxy_read_timeout    70;
  keepalive_timeout     70;
  send_timeout          70;
  client_max_body_size 4M;
  client_body_buffer_size 128K;
  proxy_pass http://chat_cluster;
}
  • See here for more details on nginx configuration. Alternatively, apache settings can be found here.

TLS

  • Install letsencrypt.
  • Ensure proper permissions are set for /etc/letsencrypt and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
  • Generate certificates. For more details see https://certbot.eff.org.
  • Make sure the certificates used by diaspora are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
total 0
lrwxrwxrwx 1 root root 47 Apr  2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem
lrwxrwxrwx 1 root root 45 Apr  2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
  • Make sure the certificates used by prosody are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
total 0
lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem
lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key
  • Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
  • Cron jobs:
crontab -e
30 2 * * 1 letsencrypt renew  >> /var/log/le-renew.log
32 2 * * 1 /etc/init.d/nginx reload
34 2 * * 1 /etc/init.d/prosody reload
  • Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public  -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
  • To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public --expand -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

History

  • See here for the archive of Poddery wiki page before the migration to Hetzner.