Gangkasthosting

class: center, middle

# Gangkasthosting:<br>die Übertragung

a.k.a. i no teh lunix workshop for 1337 haxxors

---

# Agenda

* Overview
* Configuration Management
* Containers
* Logging
* Monitoring
* Backup
* Deployment
* Shell
* Toekomstmuziek
* Workshop gangkasthosting.nl

---

# Overview{1}

.center[![Overview](img/overview.svg)]

---

# Overview{2}

* Debian GNU/Linux jessie
* Backports
  * `kernel`, `btrfs`, `nginx`, `docker`, `systemd`, `certbot`
* Unattended upgrades
* User `ops` met sudo
* UFW

---

# Configuration Management

* Ansible™ by Red Hat™ 
* Role per service
* Playbook per "site"
  * Hosts
  * Roles
  * Overrides
* Dependencies door role meta dependencies
* Secrets
  * Ansible Vault `group_vars/all`
  * `lookup('password', 'creds/something')`
* Beheerd op centrale servers
  * `floki` (is)
  * `ukko` (fi)
* Fungeren ook als bastion
* Mercurial (REGEL 7: NIET CLONEN!)

---

# dsu-staging.yml

```yaml
---
- name: dsu staging application
  hosts: constructors-staging.wasda.nl
  become: yes
  roles:
    - role: dsu
      dsu_domain: dsu-staging.constructors.nl
      dsu_api: "{{ dsu_api_test }}"
      dsu_env: staging
      dsu_branch: staging
      dsu_sentry_dsn: "{{ dsu_stag_sentry_dsn }}"
  tasks:
    - include: tasks/slack.yml
      vars:
        playbook_name: DSU staging application
        slack_channel: "#dsu"
        icon: black_square_button
```

---

# dsu.yml

```yaml
---
- name: dsu production application
  hosts: constructors3.wasda.nl
  become: yes
  roles:
    - role: dsu
      dsu_domain: www.dsu-nl.com
      dsu_aliases:
        - dsu-nl.com
        - test.dsu-nl.com
        - dsu.constructors.nl
      dsu_api: "{{ dsu_api_production }}"
      dsu_smtp_address: "{{ dsu_prod_smtp_address }}"
      dsu_smtp_port: "{{ dsu_prod_smtp_port }}"
      dsu_smtp_username: "{{ dsu_prod_smtp_username }}"
      dsu_smtp_password: "{{ dsu_prod_smtp_password }}"
      dsu_smtp_starttls_auto: "{{ dsu_prod_smtp_starttls_auto }}"
      dsu_smtp_auth: "{{ dsu_prod_smtp_auth }}"
      dsu_sentry_dsn: "{{ dsu_prod_sentry_dsn }}"
  tasks:
    - include: tasks/slack.yml
      ..etc
```

---

# roles/dsu/defaults/main.yml

```yaml
---
dsu_branch: master
dsu_env: production
dsu_domain: dsu.constructors.nl
dsu_aliases: []
dsu_port: 5006
dsu_smtp_address: "{{ mx_server }}"
dsu_smtp_port: 25
dsu_smtp_username: "{{ mx_user }}"
dsu_smtp_password: "{{ mx_pass }}"
dsu_smtp_auth: plain
dsu_smtp_starttls_auto: true
dsu_sentry_dsn: ""
```

---

# roles/dsu/vars/main.yml{1}

```yaml
dsu_db_port: "{{ postgresql_port }}"
dsu_db_name: dsu
dsu_db_user: dsu
dsu_db_pass: "{{ lookup('password', 'creds/passwordfile_dsu chars=ascii_letters,digits,hexdigits') }}"
dsu_timeout: 300
memcached_name: dsu
memcached_port: 11211
nspawn_container_name: dsu
nspawn_container_port: "{{ dsu_port }}"
nspawn_container_repo: git@bitbucket.org:constructorsehv/dsu-webshop.git
nspawn_container_repo_branch: "{{ dsu_branch }}"
nspawn_container_buildsteps:
  - ln -s /storage uploads
  - bundle install --path vendor/bundle --deployment --clean
  - bundle exec rake assets:precompile
  - bundle exec rake db:migrate
```

---

# roles/dsu/vars/main.yml{2}

```yaml
nspawn_container_env:
  RAILS_ENV: "{{ dsu_env }}"
  PORT: "{{ dsu_port }}"
  SECRET_KEY_BASE: "{{ lookup('password', 'creds/secret_dsu chars=ascii_letters,digits,hexdigits') }}"
  DATABASE_URL: postgres://{{ dsu_db_user }}:{{ dsu_db_pass }}@{{ dsu_db_host }}:{{ dsu_db_port }}/{{ dsu_db_name }}
  MEMCACHE_SERVERS: "{{ memcached_address }}:{{ memcached_port }}"
  SMTP_ADDRESS: "{{ dsu_smtp_address }}"
  SMTP_PORT: "{{ dsu_smtp_port }}"
  SMTP_USERNAME: "{{ dsu_smtp_username }}"
  SMTP_PASSWORD: "{{ dsu_smtp_password }}"
  SMTP_AUTH: "{{ dsu_smtp_auth }}"
  SMTP_STARTTLS_AUTO: "{{ dsu_smtp_starttls_auto }}"
  DSU_API: "{{ dsu_api }}"
  LOGHOST: "{{ graylog_loghost }}"
  LOGPORT: "{{ graylog_logport }}"
  WEB_TIMEOUT: "{{ dsu_timeout }}"
  SENTRY_DSN: "{{ dsu_sentry_dsn }}"
```

---

# roles/dsu/meta/main.yml

```yaml
---
dependencies:
  - role: nginx
  - role: postgresql
  - role: memcached
  - role: nspawn-container
```

---

# roles/dsu/tasks/main.yml

```yaml
---
- name: add database user
  postgresql_user: name={{ dsu_db_user }} password={{ dsu_db_pass }}
  become_user: postgres
- name: add database
  postgresql_db: name={{ dsu_db_name }} owner={{ dsu_db_user }}
  become_user: postgres
- include: tasks/trigger_container_rebuild.yml
- include: tasks/letsencrypt.yml
  vars:
    letsencrypt_domains:
      - "{{ dsu_domain }}"
- include: tasks/letsencrypt.yml
  vars:
    letsencrypt_domains:
      - "{{ item }}"
  with_items: "{{ dsu_aliases }}"
- name: configure nginx
  template: src=nginx.j2 dest=/etc/nginx/sites-available/dsu owner=root mode=0644
  notify: reload nginx
- name: enable nginx config
  file: state=link src=/etc/nginx/sites-available/dsu dest=/etc/nginx/sites-enabled/dsu owner=root mode=0644
  notify: reload nginx

```

---

# Containers

* Niet een soort van VM
* Toch een soort van VM
* Die Docker, die!
* `systemd-nspawn` en `machinectl`

---

# Docker

* Wordt gebruikt door legacy-systemen
  * Eigenlijk alles behalve DSU
* Backwards compatiblity bestaat niet
* Security backports bestaan niet
* Geen stable version; altijd latest and greatest
* Security-nachtmerrie
* Snapt taken PID 1 niet (zombies FTW!)
* Docker Hub dependency
* https://github.com/constructors/docker

---

# systemd-nspawn

* Onderdeel van Debian stable
* Integreert in default init, networking, etc.
* Geen externe dependencies
* Identieke kernel-functionaliteiten als Docker
* Integratie btrfs copy-on-write
* PID 1 goed opgelost (`STUBINIT`)
* Filesystem image met `debootstrap`; secure

---

# Foreman en Process Management

* Foreman is een process manager
* Processes staan in een `Procfile`
* Hosting maakt gebruik van `shoreman`
* Process tree:

```
/usr/bin/systemd-nspawn
\_ STUBINIT                                                                                 
   \_ /bin/bash /usr/local/bin/shoreman
      \_ rails master -b 127.0.0.1 -p 5006 -P /tmp/dsu-rails.pid         
      |   \_ rails worker[0] -b 127.0.0.1 -p 5006 -P /tmp/dsu-rails.pid      
      |   \_ rails worker[1] -b 127.0.0.1 -p 5006 -P /tmp/dsu-rails.pid      
      |   \_ rails worker[2] -b 127.0.0.1 -p 5006 -P /tmp/dsu-rails.pid      
      \_ /bin/bash /usr/local/bin/shoreman
      \_ ruby2.3 /app/app/vendor/bundle/ruby/2.3.0/bin/rake jobs:work
      \_ /bin/bash /usr/local/bin/shoreman
      \_ sh bin/cron
      |   \_ sleep 86400
      \_ /bin/bash /usr/local/bin/shoreman

```

---

# Logging{1}

* Systemd journal forward naar syslog
* RSyslog transport
* Centrale RSyslog logger (`logger.wasda.nl`)
* Storage in plain text met logrotate
* Forward naar Graylog

---

# Logging{2}

![Logging](img/logging.svg)

---

# Graylog

* Log indexer
* Java ding met elasticsearch
* Demo time!

https://logs.constructors.nl/

---

# Monitoring

* Bereikbaarheid systemen
* Bereikbaarheid diensten
* Gezondheid van systemen
* Dienst-specifieke metrics

---

# Collectd

* Draait op iedere machine
* Verzamelt gegevens en gooit ze over het netwerk

---

# Prometheus

* Lief open source van Soundcloud
* Ja, veel beter dan Nagios/Icinga
* Time series database

https://prometheus.constructors.nl/

---

# Alertmanager

* Default alert manager van Prometheus
* Silences, rate limiting
* Praat tegen Slack

https://alertmanager.constructors.nl/

---

# Grafana

* Maakt mooie grafiekjes
* Praat tegen Prometheus
* Kan veel meer

https://grafana.constructors.nl/

---

# Monitoring

![Monitoring](img/monitoring.svg)

---

# Backup

* `rdiff-backup` backupt incrementeel
* Aparte user met `sudo` voor alleen `rdiff-backup`
* Beperkt tot `/var/backupper` "chroot" op alle machines
* Backup-dirs worden erin gemount

---

# rdiff-backup

```
ops@backup:~$ sudo -s -u backupper
backupper@backup:/home/ops$ cd ~/binsbergen.wasda.nl/
backupper@backup:~/binsbergen.wasda.nl$ rdiff-backup -l . | tail -n 5
    increments.2017-05-25T00:06:11Z.dir   Thu May 25 00:06:11 2017
    increments.2017-05-25T04:06:12Z.dir   Thu May 25 04:06:12 2017
    increments.2017-05-25T08:06:10Z.dir   Thu May 25 08:06:10 2017
    increments.2017-05-25T12:06:08Z.dir   Thu May 25 12:06:08 2017
Current mirror: Thu May 25 16:06:09 2017
backupper@backup:~/binsbergen.wasda.nl$ ls
home  mail  rdiff-backup-data
backupper@backup:~/binsbergen.wasda.nl$ rdiff-backup -r 1D \
  home/jorrizza/.notmuch-config \
  binsbergen.wasda.nl::/var/backupper/home/jorrizza/.notmuch-config
```

---

# Deployment

* Bastion host (operations machine)
* Ruby script doet authz
* Heeft eigen SSH keypair met toegang tot repo's
* SSH key forwarding naar host waarop deploy draait
* Via sudo en een wrapper wordt een Ansible playbook aangeroepen

---

# Shell

* Net als deployment
* Bastion host (operations machine)
* Ruby script doet authz     
* Heeft eigen SSH keypair met toegang tot alle machines voor user `shell`
* Die user kan alleen shells starten
* Een shell is een shell-script dat of via Docker of via nspawn een container shell opent

---

# Deployment en Shell authz

* Configuratie in Ansible YAML
  * roles/deploy-server/vars/main.yml
  * roles/shell-server/vars/main.yml
* Ruby script
  * Wordt aangeroepen via `.ssh/authorized_keys`
  * Leest een YAML file met wie waarbij mag

---

# Toekomstmuziek

Uit `ops/TODO`:

* Remove Docker everywhere and replace with nspawn
* Use btrfs on new nodes
* Use a secret store other than Ansible Vault so users can add env vars (vaultproject.io)
* Replace shoreman with systemd in containers (use foreman export)
* Boot containers with --boot
* Upgrade to Debian Stretch
* Use SMS for alerts next to Slack
* Use hardware tokens and 2FA
* Somehow monitor "systemctl status" for degraded state
* Replace collectd with prometheus agent
* Use cental build machine and transport images instead of debootstrapping everywhere
* Instead of meta dependencies, use playbooks with includes; this will allow for faster deploys with streamlined playbooks

---

class: center, middle

# Gangkasthosting:<br>die Werkstatt

We gaan zorgen dat deze link werkt: https://www.gangkasthosting.nl/