====== Information ======

  * Mastodon ((https://github.com/tootsuite/mastodon))
  * [[Information:Realm of Espionage]]
  * https://social.realmofespionage.xyz

===== Prerequisites =====

  * [[distros:ubuntu_server | Ubuntu Server]]
  * [[servers:nginx_php_php-fpm | nginx]]
  * [[servers:nginx:lets_encrypt | Let's Encrypt]]

====== Repositories ======

===== Node.js =====

  wget -O '/tmp/nodesource-gpg.key' 'https://deb.nodesource.com/gpgkey/nodesource.gpg.key' && sudo apt-key add '/tmp/nodesource-gpg.key' && echo -e "deb https://deb.nodesource.com/node_8.x bionic main\n# deb-src https://deb.nodesource.com/node_8.x bionic main" | sudo tee '/etc/apt/sources.list.d/nodesource.list' > '/dev/null' && sudo apt update && rm '/tmp/nodesource-gpg.key' && sync

  sudo -e '/etc/apt/sources.list.d/nodesource.list'

===== Yarn =====

  wget -O '/tmp/yarn-gpg.key' 'https://dl.yarnpkg.com/debian/pubkey.gpg' && sudo apt-key add '/tmp/yarn-gpg.key' && echo -e "deb https://dl.yarnpkg.com/debian/ stable main\n# deb-src https://dl.yarnpkg.com/debian/ stable main" | sudo tee '/etc/apt/sources.list.d/yarn.list' > '/dev/null' && sudo apt update && rm '/tmp/yarn-gpg.key' && sync

  sudo -e '/etc/apt/sources.list.d/yarn.list'

====== Dependencies ======

****

  sudo apt install imagemagick ffmpeg libpq-dev libxml2-dev libxslt1-dev file git g++ libprotobuf-dev protobuf-compiler pkg-config nodejs gcc autoconf bison build-essential libssl-dev libyaml-dev libreadline-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm5 libgdbm-dev redis-server redis-tools postgresql postgresql-contrib yarn libidn11-dev libicu-dev libjemalloc-dev

====== Create User ======

****

  sudo useradd --comment 'Mastodon' --home-dir '/var/lib/mastodon' --create-home --system 'mastodon'

====== Download Source ======

===== Switch User =====

****

  sudo su 'mastodon' -s '/bin/bash'

===== Download Source =====

  * Clones into ''/var/lib/mastodon/live''

  cd ~ && git clone --branch 'master' --depth '1' --recurse-submodules 'https://github.com/tootsuite/mastodon.git' ~/'live' && sync

====== Additional Dependencies ======

===== Switch User =====

****

  sudo su 'mastodon' -s '/bin/bash'

===== rbenv =====

==== Download Source ====

****

  cd ~ && git clone --branch 'master' --depth '1' --recurse-submodules 'https://github.com/rbenv/rbenv.git' ~/'.rbenv' && sync

==== Compile ====

****

  cd ~/'.rbenv' && ~/'.rbenv/src/configure' && make --directory=~/'.rbenv/src'

==== PATH ====

****

  echo -e '\n# rbenv\nexport PATH="$HOME/.rbenv/bin:$PATH"\neval "$(rbenv init -)"' | tee --append ~/'.bashrc' > '/dev/null' && exec bash

===== Ruby =====

==== Download Source ====

****

  cd ~ && git clone --branch 'master' --depth '1' --recurse-submodules 'https://github.com/rbenv/ruby-build.git' ~/'.rbenv/plugins/ruby-build' && sync

==== Compile ====

  * Change ''2.6.0'' to whatever version is reported in [[https://raw.githubusercontent.com/tootsuite/mastodon/master/.ruby-version|.ruby_version]]

  RUBY_CONFIGURE_OPTS='--with-jemalloc' rbenv install '2.6.0' && rbenv global '2.6.0'

===== Gem =====

****

  gem update --system

===== Bundler =====

==== Install ====

  * :!: Mastodon's notes say to install this, even though it appears to be covered by ''gem update''; this command failed because ''bundler'' already exists

  gem install bundler --no-document

==== Dependencies ====

****

  cd ~/'live' && bundle install --jobs='2' --deployment --without 'development' 'test' && sync

===== Yarn =====

****

  cd ~/'live' && yarn install --pure-lockfile && sync

===== Exit User =====

****

  exit

====== Database ======

===== Service =====

****

  sudo systemctl enable 'postgresql' --now

===== Database =====

  sudo -u 'postgres' psql

  CREATE USER mastodon CREATEDB;

  \q

====== Redis ======

===== Service =====

  * TODO: See if this is needed; Mastodon's production notes don't mention it and this is a carry-over from openSUSE

  sudo systemctl enable 'redis-server' --now

====== nginx Configuration ======

===== Mastodon Proxies =====

  sudo -e '/etc/nginx/snippets/mastodon.conf'

<code>
location @proxy {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header Proxy "";
    proxy_pass_header Server;

    proxy_pass http://127.0.0.1:3000;
    proxy_buffering on;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    proxy_cache CACHE;
    proxy_cache_valid 200 7d;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    add_header X-Cached $upstream_cache_status;
    add_header Strict-Transport-Security "max-age=31536000";

    tcp_nodelay on;
}

location /api/v1/streaming {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header Proxy "";

    proxy_pass http://127.0.0.1:4000;
    proxy_buffering off;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    tcp_nodelay on;
}</code>

===== Server Block =====

  sudo -e '/etc/nginx/sites-available/social.conf'

<code>
map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=CACHE:10m inactive=7d max_size=1g;

server {
    listen '443' 'ssl' 'http2';
    server_name 'social.realmofespionage.xyz';
    root '/var/lib/mastodon/live/public';

    include '/etc/nginx/snippets/mastodon.conf';
    include '/etc/nginx/snippets/headers.conf';

    client_max_body_size '100M';

    #access_log /var/log/nginx/mastodon-access.log;
    #error_log /var/log/nginx/mastodon-error.log;

    location / {
        try_files $uri @proxy;
    }

    location ~ ^/(emoji|packs|system/accounts/avatars|system/media_attachments/files) {
        try_files $uri @proxy;
    }

    location /sw.js {
        try_files $uri @proxy;
    }

    error_page 500 501 502 503 504 /500.html;
}</code>

==== Enable Server Block ====

****

  sudo rm -f '/etc/nginx/sites-enabled/social.conf' && sudo ln -s '/etc/nginx/sites-available/social.conf' '/etc/nginx/sites-enabled' && sudo systemctl reload 'nginx'

====== Initial Setup ======

===== Switch User =====

****

  sudo su 'mastodon' -s '/bin/bash'

===== Initial Setup =====

****

  cd ~/'live' && RAILS_ENV=production bundle exec rake mastodon:setup

==== Settings ====

  * Domain name: ''social.realmofespionage.xyz''
  * Yes single user mode
  * No Docker
  * Default PostgresSQL details
  * Default Redis details
  * No uploaded files on cloud
  * No emails from localhost

====== Post Setup ======

===== Logo =====

==== Hero image ====

  * 600x100
  * RoE | Social text centered in font ''DejaVu Sans Bold'' at ''70px'' size
  * Gray background with HTML color notation ''#484848''
  * White text for ''RoE'' and ''Social''
  * Blue text for ''|'' with HTML color notation ''#3D8BFF''

==== Instance thumbnail ====

  * 1200x630
  * ''R'' at 450px in White
  * ''|'' at 525px in HTML color notation ''#3D8BFF''
  * ''S'' at 450px in White
  * ''DejaVu Sans Bold'' font for all characters
  * Gray background with HTML color notation ''#484848''

===== Documentation =====

  * https://github.com/tootsuite/documentation/tree/master/Running-Mastodon

===== Rake Tasks =====

  * https://github.com/tootsuite/documentation/blob/master/Running-Mastodon/List-of-Rake-tasks.md

====== Services ======

===== Web Workers =====

  * Background daemon ran at system startup

  sudo -e '/etc/systemd/system/mastodon-web.service' && sudo systemctl daemon-reload && sudo systemctl enable 'mastodon-web' --now && sudo systemctl status 'mastodon-web' -l

<code>
[Unit]
Description=Mastodon Web Workers
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=mastodon
Group=mastodon
WorkingDirectory=/var/lib/mastodon/live
Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1'
Environment='RAILS_ENV=production'
Environment='PORT=3000'
ExecStart='/var/lib/mastodon/.rbenv/shims/bundle' exec 'puma' -C '/var/lib/mastodon/live/config/puma.rb'
ExecReload=/bin/kill -SIGUSR1 $MAINPID
TimeoutSec=15
Restart=always

[Install]
WantedBy=multi-user.target</code>

===== Background Queue =====

  * :!: ''DB_POOL'' and ''sidekiq -c'' control how many threads are assigned; increase with specs as-needed

  sudo -e '/etc/systemd/system/mastodon-sidekiq.service' && sudo systemctl daemon-reload && sudo systemctl enable 'mastodon-sidekiq' --now && sudo systemctl status 'mastodon-sidekiq' -l

<code>
[Unit]
Description=Mastodon Background Queue
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=mastodon
Group=mastodon
WorkingDirectory=/var/lib/mastodon/live
Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1'
Environment='RAILS_ENV=production'
Environment='DB_POOL=2'
Environment='MALLOC_ARENA_MAX=2'
ExecStart='/var/lib/mastodon/.rbenv/shims/bundle' exec 'sidekiq' -c '2'
TimeoutSec=15
Restart=always

[Install]
WantedBy=multi-user.target</code>

===== Streaming API =====

  * Background daemon ran at system startup

  sudo -e '/etc/systemd/system/mastodon-streaming.service' && sudo systemctl daemon-reload && sudo systemctl enable 'mastodon-streaming' --now && sudo systemctl status 'mastodon-streaming' -l

<code>
[Unit]
Description=Mastodon Streaming API
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=mastodon
Group=mastodon
WorkingDirectory=/var/lib/mastodon/live
Environment='NODE_ENV=production'
Environment='PORT=4000'
Environment='STREAMING_CLUSTER_NUM=1'
ExecStart='/usr/bin/npm' run 'start'
TimeoutSec=15
Restart=always

[Install]
WantedBy=multi-user.target</code>

===== Updater =====

==== Service ====

  sudo -e '/etc/systemd/system/mastodon-up.service'

<code>
[Service]
User=mastodon
Group=mastodon
Type=oneshot
WorkingDirectory=/var/lib/mastodon/live
Environment='RAILS_ENV=production'
ExecStart='/usr/bin/git' -C '/var/lib/mastodon/live' pull origin 'master'
ExecStart='/var/lib/mastodon/.rbenv/shims/bundle' install --deployment --without 'development' 'test'
ExecStart='/usr/bin/yarn' install --pure-lockfile
ExecStart='/var/lib/mastodon/.rbenv/shims/bundle' exec rails db:migrate
ExecStart='/var/lib/mastodon/.rbenv/shims/bundle' exec rails assets:precompile
ExecStartPost='/bin/sync'</code>



==== Timer ====

  * Every day at ''02:00:00''

  sudo -e '/etc/systemd/system/mastodon-up.timer' && sudo systemctl daemon-reload && sudo systemctl enable 'mastodon-up.timer' --now && sudo systemctl start 'mastodon-up' && sudo systemctl status 'mastodon-up' -l

<code>
[Unit]
Description=Mastodon Updater
After=network-online.target
Wants=network-online.target

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target</code>

===== Ruby Updater =====

==== Service ====

  sudo -e '/etc/systemd/system/mastodon-ruby-up.service'

<code>
[Service]
User=mastodon
Group=mastodon
Type=oneshot
WorkingDirectory=/var/lib/mastodon/.rbenv
Environment='RUBY_CONFIGURE_OPTS=--with-jemalloc'
ExecStart='/usr/bin/git' -C '/var/lib/mastodon/.rbenv' pull origin 'master'
ExecStart='/usr/bin/git' -C '/var/lib/mastodon/.rbenv/plugins/ruby-build' pull origin 'master'
ExecStart='/var/lib/mastodon/.rbenv/src/configure'
ExecStart='/usr/bin/make' --directory='/var/lib/mastodon/.rbenv/src'
ExecStart='/bin/bash' -c '"/var/lib/mastodon/.rbenv/bin/rbenv" install --skip-existing $(cat "/var/lib/mastodon/live/.ruby-version")'
ExecStart='/bin/bash' -c '"/var/lib/mastodon/.rbenv/bin/rbenv" global $(cat "/var/lib/mastodon/live/.ruby-version")'
ExecStartPost='/bin/sync'</code>

==== Timer ====

  * Every day at ''01:30:00''

  sudo -e '/etc/systemd/system/mastodon-ruby-up.timer' && sudo systemctl daemon-reload && sudo systemctl enable 'mastodon-ruby-up.timer' --now && sudo systemctl start 'mastodon-ruby-up' && sudo systemctl status 'mastodon-ruby-up' -l

<code>
[Unit]
Description=Mastodon Ruby Updater
After=network-online.target
Wants=network-online.target

[Timer]
OnCalendar=*-*-* 01:30:00
Persistent=true

[Install]
WantedBy=timers.target</code>

===== Maintenance =====

==== Service ====

  sudo -e '/etc/systemd/system/mastodon-m.service'

<code>
[Service]
User=mastodon
Group=mastodon
Type=oneshot
ExecStart='/usr/bin/git' -C '/var/lib/mastodon/live' gc --aggressive --prune='all'
ExecStart='/usr/bin/git' -C '/var/lib/mastodon/live' fsck --full --strict
ExecStartPost='/bin/sync'</code>

==== Timer ====

  * ''01'' day of every month at ''02:20:00''

  sudo -e '/etc/systemd/system/mastodon-m.timer' && sudo systemctl daemon-reload && sudo systemctl enable 'mastodon-m.timer' --now

<code>
[Unit]
Description=Mastodon Maintenance
After=network-online.target
Wants=network-online.target

[Timer]
OnCalendar=*-*-01 02:20:00
Persistent=true

[Install]
WantedBy=timers.target</code>

===== Restarter =====

==== Service ====

  sudo -e '/etc/systemd/system/mastodon-re.service'

<code>
[Service]
Type=oneshot
ExecStartPre='/bin/sync'
ExecStart='/bin/systemctl' stop 'mastodon-web'
ExecStart='/bin/systemctl' stop 'mastodon-sidekiq'
ExecStart='/bin/systemctl' stop 'mastodon-streaming'
ExecStartPost='/bin/sync'
ExecStartPost='/bin/systemctl' start 'mastodon-web'
ExecStartPost='/bin/systemctl' start 'mastodon-sidekiq'
ExecStartPost='/bin/systemctl' start 'mastodon-streaming'</code>

==== Timer ====

  * Every day at ''02:10:00''

  sudo -e '/etc/systemd/system/mastodon-re.timer' && sudo systemctl daemon-reload && sudo systemctl enable 'mastodon-re.timer' --now && sudo systemctl start 'mastodon-re' && sudo systemctl status 'mastodon-re' -l

<code>
[Unit]
Description=Mastodon Service Restarter
After=network-online.target
Wants=network-online.target

[Timer]
OnCalendar=*-*-* 02:10:00
Persistent=true

[Install]
WantedBy=timers.target</code>

===== Backup =====

==== Information ====

  * :!: This differs from the usual backup scheme and **should not** be used copy/paste-style unless dealing with something using PostgreSQL in a similar manner
  * Because ''pg_dump'' needs to be ran as the ''mastodon'' user, it doesn't have access to my home folder
  * So to avoid having to run another service just to move the database dump, it's integrated into the files backup service

==== Database ====

=== Service ===

  sudo -e '/etc/systemd/system/mastodon-db.service'

<code>
[Service]
Type=oneshot
User=mastodon
Group=mastodon
ExecStart='/usr/bin/pg_dump' --format='custom' 'mastodon_production' --file='/var/lib/mastodon/mastodon-db.dump'
ExecStartPost='/bin/sync'</code>

=== Timer ===

  * Every day at ''02:45:00''

  sudo -e '/etc/systemd/system/mastodon-db.timer' && sudo systemctl daemon-reload && sudo systemctl enable 'mastodon-db.timer' --now && sudo systemctl start 'mastodon-db' && sudo systemctl status 'mastodon-db' -l

<code>
[Unit]
Description=Mastodon Database Dump
After=postgresql.service

[Timer]
OnCalendar=*-*-* 02:45:00
Persistent=true

[Install]
WantedBy=timers.target</code>

==== Files ====

=== Service ===

  mkdir -p ~/'backups' && sudo -e '/etc/systemd/system/mastodon-fb.service' && sudo sed -i 's/CHANGEME/'$USER'/g' '/etc/systemd/system/mastodon-fb.service'

<code>
[Service]
Type=oneshot
WorkingDirectory=/var/lib/mastodon/live/public
ExecStart='/bin/bash' -c '"/bin/tar" -cvzf "/home/CHANGEME/backups/mastodon-files-auto-"$$(date +%%Y-%%m-%%d)".tar.gz" "system" ../".env.production" ../../../"redis/dump.rdb" ../../"mastodon-db.dump"'
ExecStartPost='/bin/sync'</code>

=== Timer ===

  * Every day at ''02:50:00''

  sudo -e '/etc/systemd/system/mastodon-fb.timer' && sudo systemctl daemon-reload && sudo systemctl enable 'mastodon-fb.timer' --now && sudo systemctl start 'mastodon-fb' && sudo systemctl status 'mastodon-fb' -l

<code>
[Unit]
Description=Mastodon Files and Database Backup

[Timer]
OnCalendar=*-*-* 02:50:00
Persistent=true

[Install]
WantedBy=timers.target</code>

====== Backup ======

  * Create backup archive on server and transfer to client computer

===== Server =====

==== Stop Services ====

****

  sudo systemctl stop mastodon-web mastodon-sidekiq mastodon-streaming redis-server

==== Backup Files ====

****

  cd '/var/lib/mastodon/live/public' && sudo tar -cvzf ~/'mastodon-files-manual-'$(date +%Y-%m-%d)'.tar.gz' 'system' ../'.env.production' ../../../'redis/dump.rdb' && sync

==== Backup Database ====

=== Backup Database ===

****

  sudo -u 'mastodon' pg_dump --format='custom' 'mastodon_production' --file='/tmp/mastodon-db.dump' && sudo mv '/tmp/mastodon-db.dump' ~/'mastodon-database-manual-'$(date +%Y-%m-%d)'.dump'

==== Start Services ====

****

  sudo systemctl start mastodon-web mastodon-sidekiq mastodon-streaming redis-server

===== Client =====

==== Transfer Files To Client ====

****

  scp espionage724@192.168.1.153:~/'mastodon-files-'*'.tar.gz' espionage724@192.168.1.153:~/'mastodon-database-'*'.dump' ~/'Downloads' && sync

====== Restore ======

  * :!: This is untested as of 2019/01/18

===== Client =====

==== Transfer Files To Server ====

****

  scp ~/'Downloads/mastodon-files-'*'.tar.gz' ~/'Downloads/mastodon-database-'*'.dump' espionage724@192.168.1.153:~

==== Remove Files ====

****

  rm -f ~/'Downloads/mastodon-files-'*'.tar.gz' ~/'Downloads/mastodon-database-'*'.dump' && sync

===== Server =====

==== Stop Services ====

****

  sudo systemctl stop mastodon-web mastodon-sidekiq mastodon-streaming redis-server

==== Restore Files ====

=== Extract ===

****

  mkdir -p '/tmp/mastodon' && cd '/tmp/mastodon' && tar -xvzf ~/'mastodon-files-'*'.tar.gz' && sync

=== Move ===

== system Folder ==

****

  sudo rm -Rf '/var/lib/mastodon/live/public/system' && sudo mv '/tmp/mastodon/system' '/var/lib/mastodon/live/public' && sudo chown -R 'mastodon':'mastodon' '/var/lib/mastodon/live/public/system' && sync

== Settings ==

****

  sudo rm -f '/var/lib/mastodon/live/.env.production' && sudo mv '/tmp/mastodon/.env.production' '/var/lib/mastodon/live' && sudo chown 'mastodon':'mastodon' '/var/lib/mastodon/live/.env.production' && sync

== Redis Database ==

****

  sudo rm -f '/var/lib/redis/dump.rdb' && sudo mv '/tmp/mastodon/redis/dump.rdb' '/var/lib/redis' && sudo chown 'redis':'redis' '/var/lib/redis/dump.rdb' && sync

==== Restore Database ====

=== Create Database ===

****

  sudo -u 'mastodon' createdb --template='template0' 'mastodon_production'

=== Restore ===

****

  sudo -u 'mastodon' pg_restore --username='mastodon' --schema='public' --no-owner --role='mastodon' --dbname='mastodon_production' ~/'mastodon-database-'*'.dump'

==== Start Services ====

****

  sudo systemctl start mastodon-web mastodon-sidekiq mastodon-streaming redis-server

==== Remove Backups ====

  * Verify that Mastodon works before running

  rm ~/'mastodon-files-'*'.tar.gz' ~/'mastodon-database-'*'.dump' && sync