Category Archives: Ubuntu

Updating Intel’s i40e driver in Ubuntu 18.10

No, Canonical does not provide the latest stable Intel i40e driver in the latest version of Ubuntu Server. In fact it’s 12 months old.

No, Intel does not keep the most current version of the i40e driver on Intel.com. It’s on SourceForge.

/HeadDesk

Let’s look at all of the REALLY IMPORTANT changes made to the i40e driver since version 2.3.2 that’s installed by default in Ubuntu 18.10:

Changelog for i40e 2.7.11
===========================================================================
Implement ADq feature
Add support for configuring VFs through sysfs
Add XDP support
Add support Forward Error Correction (FEC) on 25G links
Allow disabling FW LLDP (requires updated firmware)
Fix bug where link could be forced down after MTU change
Add support for new kernel APIs
Fix WARN_ON() if MSI-X not available
Fix issue with lost link when forcing link mode on some adapters
Add support to allow the FW update tool to upgrade/downgrade from any FW version to any other FW version
   Note: this may require a power cycle as an intermediate step.
Fix kernel panic with repeated quick enable/disable of VFs
Fix possible TX timeout with DCB on some kernels
Re-enable TSO after MTU change
Fix potential panic on shutdown with many MSI-X vectors
Add VEB stats to ethtool
Fix panic on suspend with RDMA client driver active
Fix timing issue with assigning VFs to VMs using virsh
Restore TCPv4 input set when re-enabling ATR
Performance fixes with Adaptive ITR
Code cleanups
Minor bug fixes
Add support for RHEL 7.5
Add support for SLES12SP3

Source: README.txt, updated 2018-10-26
Changelog for i40e 2.4.10
===========================================================================
Add support for RHEL 7.5
Fix synchronization issue with simultaneous resets on multiple ports

Source: README.txt, updated 2018-06-08
Changelog for i40e 2.4.6
===========================================================================
- fix unnecessary PF resets issue
- update Flow Director User documentation
- fix compilation error on kernel 4.15.1

Source: README.txt, updated 2018-04-03
Changelog for i40e 2.4.3
===========================================================================
- fix issue on X722 with netdev close removing RDMA instance
- performance updates
- fix some issues with link reporting
- fix issue with tx timeouts on systems with > 128 cores
- fix issue with promiscuous mode disappearing across resets
- fix issue with flow director filters not working on fragmented IPv4
- fix issue with duplicate flow director filters not being updated to HW

Source: README.txt, updated 2017-12-07
Changelog for i40e-linux-2.3.6
===========================================================================

- Fix mac filter removal timing issue
- Sync i40e_ethtool.c with upstream
- Fixes for TX hangs
- Some fixes for reset of VFs
- Fix build error with packet split disabled
- Fix memory leak related to filter programming status
- Add and modify branding strings
- Fix kdump failure
- Implement an ethtool private flag to stop LLDP in FW
- Add delay after EMP reset for firmware to recover
- Fix incorrect default ITR values on driver load
- Fixes for programming cloud filters
- Some performance improvements
- Enable XPS with QoS on newer kernels
- Enable support for VF VLAN tag stripping control
- Build fixes to force perl to load specific ./SpecSetup.pm file
- Fix the updating of pci.ids
- Use 16 byte descriptors by default
- Fixes for DCB
- Don't close client in debug mode
- Add change MTU log in VF driver
- Fix for adding multiple ethtool filters on the same location
- Add new branding strings for OCP XXV710 devices
- Remove X722 Support for Destination IP Cloud Filter
- Allow turning off offloads when the VF has VLAN set

Source: README.txt, updated 2017-10-31

/HeadDesk

I have already updated the X710-DA4 firmware (to 6.01) prior to updating the driver, but I have not read anywhere that updating the firmware first is necessary.

I started by looking at my installed drivers, but this is not needed, and does not show the versions of the driver:

ls -R /lib/modules/`uname -r`/kernel/ > drivers.log

Show current i40e driver:

ethtool -i enp1s0f0

Output:

driver: i40e
version: 2.3.2-k
firmware-version: 6.01 0x800035ce 1.1747.0
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Install dependencies:

sudo apt install make gcc

Download the latest driver:

wget https://cfhcable.dl.sourceforge.net/project/e1000/i40e%20stable/2.7.11/i40e-2.7.11.tar.gz

Unzip:

tar -xf i40e-2.7.11.tar.gz

Instructions are in the README — read them:

vim i40e-2.7.11/README

Install. This was performed while my X710-DA4 was not being used nor had assigned IPs:

sudo su
cd i40e-2.7.11/src/
make install
rmmod i40e; modprobe i40e
update-initramfs -u
exit

Confirm updated driver:

ethtool -i enp1s0f0

Output:

driver: i40e
version: 2.7.11
firmware-version: 6.01 0x800035ce 1.1747.0
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Don’t forget to read the included README for performance tuning options, and the over 2 year old Intel Ethernet Controller X710/XL710 and Intel Ethernet Converged Network Adapter X710/XL710 Family Linux Performance Tuning Guide from March 2016:

https://www.intel.com/content/dam/www/public/us/en/documents/reference-guides/xl710-x710-performance-tuning-linux-guide.pdf

Oh and:

sudo vim /etc/default/grub

Like this:

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=2
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt pcie_aspm=off"
GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt pcie_aspm=off"
sudo update-grub
sudo shutdown -r now

Setting up DPDK 18.08 in Ubuntu 18.04

I’m just testing, this post isn’t complete nor should it be used.

sudo apt install gcc gcc-multilib libnuma-dev python python3.7 python-dev python-pip dpdk hugepages

sudo vim /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT=default_hugepagesz=1G hugepagesz=1G hugepages=4
GRUB_CMDLINE_LINUX=default_hugepagesz=1GB hugepagesz=1G hugepages=4

sudo vim /etc/sysctl.conf

kernel.shmmax = 8589934592
vm.hugetlb_shm_group = 0
vm.nr_hugepages = 8

sudo vim /etc/security/limits.conf

* soft nproc 65535
* hard nproc 65535
* soft nofile 65535
* hard nofile 65535
* soft memlock 8388608
* hard memlock 8388608

sudo shutdown -r now

git clone https://dpdk.org/git/dpdk 

 cd dpdk

make config T=x86_64-native-linuxapp-gcc 

 make T=x86_64-native-linuxapp-gcc 

 make install T=x86_64-native-linuxapp-gcc

These steps are still not persistent after reboots:

sudo modprobe uio i40e vfio-pci

sudo insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko

./usertools/dpdk-devbind.py --status |grep Eth

sudo ./usertools/dpdk-devbind.py --bind=igb_uio enp1s0f0

./usertools/dpdk-devbind.py --status

Tuning a fast Tor relay (0.3.5.3-alpha) on Ubuntu 18.04

Rule: don’t enable iptables / ufw

as root:

ip link set eth0 txqueuelen 300000

These are just temporary:

sysctl -w fs.file-max=65535
sysctl -w net.core.optmem_max=25165824
sysctl -w net.core.netdev_max_backlog=65536
sysctl -w net.core.rmem_default=25165824
sysctl -w net.core.rmem_max=25165824
sysctl -w net.core.somaxconn=20480
sysctl -w net.core.wmem_default=25165824
sysctl -w net.core.wmem_max=25165824
sysctl -w net.ipv4.ip_local_port_range='1024 65535'
sysctl -w net.ipv4.route.flush=1
sysctl -w net.ipv4.tcp_congestion_control=cubic
sysctl -w net.ipv4.tcp_fin_timeout=15
sysctl -w net.ipv4.tcp_keepalive_intvl=15
sysctl -w net.ipv4.tcp_keepalive_probes=5
sysctl -w net.ipv4.tcp_keepalive_time=1200
sysctl -w net.ipv4.tcp_max_syn_backlog=65536
sysctl -w net.ipv4.tcp_max_tw_buckets=1440000
sysctl -w net.ipv4.tcp_mem='65536 131072 262144'
sysctl -w net.ipv4.tcp_moderate_rcvbuf=1
sysctl -w net.ipv4.tcp_mtu_probing=1
sysctl -w net.ipv4.tcp_no_metrics_save=1
sysctl -w net.ipv4.tcp_rfc1337=1
sysctl -w net.ipv4.tcp_rmem='20480 12582912 25165824'
sysctl -w net.ipv4.tcp_synack_retries=2
sysctl -w net.ipv4.tcp_tw_recycle=1
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.ipv4.tcp_window_scaling=1
sysctl -w net.ipv4.tcp_wmem='20480 12582912 25165824'
echo "* soft nproc 65535" >> /etc/security/limits.conf
echo "* hard nproc 65535" >> /etc/security/limits.conf
echo "* soft nofile 65535" >> /etc/security/limits.conf
echo "* hard nofile 65535" >> /etc/security/limits.conf
echo "root soft nproc 65535" >> /etc/security/limits.conf
echo "root hard nproc 65535" >> /etc/security/limits.conf
echo "root soft nofile 65535" >> /etc/security/limits.conf
echo "root hard nofile 65535" >> /etc/security/limits.conf
echo "session required pam_limits.so" >> /etc/pam.d/common-session
shutdown -r now

Using overclocked speeds with an Intel i7-8086K at 5.2GHz in Ubuntu Server

This tutorial demonstrates how to assure that Ubuntu Server 18.04 x64 will operate at a specific CPU frequency overclock. I will also demonstrate how to validate the frequency per CPU core and monitor the temperature of each CPU core in near real time.

This tutorial presumes you have already installed and updated the OS, in addition to having a stable overclock. In my case, I have found that my Intel i7-8086K operates at 5.2GHz at 1.392 volts while my RAM is using its XMP profile. This tutorial, however, only focuses on CPU frequency settings. Hyper-threading is disabled in BIOS so Ubuntu sees 6 cores and 6 threads.

sudo apt install cpufrequtils lm-sensors -y

sudo vim /etc/init.d/cpufrequtils

Edit and enable these lines to your target overclocked frequency:

ENABLE="true"
GOVERNOR="performance"
MAX_SPEED="5200000"
MIN_SPEED="5200000"

5.0GHz = 5000000
5.2GHz = 5200000
5.4GHz = 5400000

Note: If AVX is set to 2 in BIOS, even with this MIN_SPEED setting, Ubuntu will cap itself at 5.0GHz. Set AVX to 0 to assure Ubuntu will stay at 5.2GHz constantly.

sudo systemctl daemon-reload

Open a new terminal and watch the CPU frequencies for cores 0-5 (6 cores)

watch -n.1 "cat /proc/cpuinfo | grep "^[c]pu MHz""

Find the available temp sensors on your system and say YES to everything:

sudo sensors-detect

Open a new terminal and watch the CPU temps:

watch -n.1 "sensors"

We can use OpenSSL to burn the CPU for testing… this “-multi 6” flag will burn all 6 of my cores (be sure you have proper cooling on your CPU!):

openssl speed -multi 6

Note: Like in Windows, if my CPU is not properly cooled resulting in CPU temps at or above 100 degree C, my multiplier will automatically throttle down in order to keep the core temps down below critical. If you’re needing to test single-threaded performance, “-multi 1” will just burn one core. Burning half the available cores or less will keep temps down quite a bit since heat dissipation on the IHS is cut in half, meaning the multiplier should stay at the set x52. This is all dependent on proper cooling.

Also checkout CoreFreq for seeing greater CPU details like your core multipliers:

https://github.com/cyring/CoreFreq

sudo apt install dkms git libpthread-stubs0-dev -y

git clone https://github.com/cyring/CoreFreq.git

cd CoreFreq

make

Load the kernel module:

sudo insmod corefreqk.ko

Load the daemon

sudo ./corefreqd

In a new window:

./CoreFreq/corefreq-cli

Configurations that didn’t help me, especially after reboots:

sudo bash -c 'for i in {0..5}; do cpufreq-set -c $i -g performance; done'

for x in /sys/devices/system/cpu/*/cpufreq/; do echo 5200000 | sudo tee $x/scaling_max_freq; done

for x in /sys/devices/system/cpu/*/cpufreq/; do echo 5200000 | sudo tee $x/scaling_min_freq; done

sudo cpupower frequency-set -d 5.2GHz -u 5.2GHz -g performance -r

sudo cpufreq-set -d 5.2GHz -u 5.2GHz -g performance -r

building caddy from source (and updating caddy on xenial)

sudo add-apt-repository ppa:gophers/archive
sudo vim /etc/apt/sources.list.d/gophers-ubuntu-archive-xenial.list

deb tor+http://ppa.launchpad.net/gophers/archive/ubuntu xenial main

sudo apt update && sudo apt install golang-1.10*
sudo apt remove golang-1.6*
sudo apt update && sudo apt dist-upgrade -V && sudo apt autoremove -y && sudo apt autoclean
/usr/lib/go-1.10/bin/go version
echo 'PATH="/usr/lib/go-1.10/bin:$PATH"' >> ~/.profile
source ~/.profile
go version

go version go1.10 linux/amd64

go get -u github.com/mholt/caddy
go get -u github.com/caddyserver/builds
cd ~/go/src/github.com/mholt/caddy/caddy
go run build.go -goos=linux -goarch=amd64
sudo service caddy stop
sudo cp /usr/local/bin/caddy /usr/local/bin/caddy.bak
cp ~/go/src/github.com/mholt/caddy/caddy/caddy /usr/local/bin
sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/caddy
caddy --version

Caddy 0.11.0 (+60a0208) (unofficial)

sudo shutdown -r now

A+ TLS config for ubuntu + nginx

These are my config notes for getting a brand new Xenial + nginx server online.

Install Tor:

sudo apt install tor apt-transport-tor
sudo gpg --keyserver keys.gnupg.net --recv 886DDD89

sudo gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add -

Edit the sources list by removing all the lines and adding these:

sudo vim /etc/apt/sources.list
deb tor+https://deb.torproject.org/torproject.org xenial main
deb tor+https://mirrors.wikimedia.org/ubuntu/ xenial main restricted universe multiverse
deb tor+https://mirrors.wikimedia.org/ubuntu/ xenial-updates main restricted universe multiverse
deb tor+https://mirrors.wikimedia.org/ubuntu/ xenial-security main restricted universe multiverse

Update the repos:

sudo add-apt-repository ppa:nginx/development
sudo add-apt-repository ppa:ondrej/nginx
sudo add-apt-repository ppa:ondrej/php
sudo add-apt-repository ppa:certbot/certbot

Add “tor+” to all of the above sources files in /etc/apt/sources.list.d/*

Update and restart:

sudo apt update && sudo apt upgrade -V && sudo apt autoremove -y && sudo shutdown -r now

Install nginx + certbot:

sudo apt install python-certbot-nginx -V

Add server_name to (replacing “_”):

sudo vim /etc/nginx/sites-available/default
server_name domain.net;

Get Let’s Encrypt cert for nginx:

sudo certbot --nginx -d domain.net --redirect --rsa-key-size 4096

Further harden the TLS config:

sudo vim /etc/letsencrypt/options-ssl-nginx.conf
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:!3DES:!aNULL:!DES:!DSS:!eNULL:!EXP:!IDEA:!LOW:!MD5:!PSK:!RC4:!SEED";

Delete the “SSL” config:

sudo vim /etc/nginx/nginx.conf

Edit the nginx config:

sudo vim /etc/nginx/sites-available/default

replace “domain.net”

server {
        listen 80 default_server;
        listen [::]:80 default_server;

        server_name domain.net www.domain.net;
        return 301 https://$host$request_uri;

        server_tokens off;
        add_header X-Frame-Options SAMEORIGIN;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header Referrer-Policy "no-referrer";
}

server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;

        server_name domain.net www.domain.net;
        root /var/www;
        index index.php index.html index.htm;

        ssl_certificate /etc/letsencrypt/live/domain.net/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/domain.net/privkey.pem;
        include /etc/letsencrypt/options-ssl-nginx.conf;
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

        server_tokens off;
        add_header Strict-Transport-Security 'max-age=31536000; includeSubDomains; preload';
        add_header X-Frame-Options SAMEORIGIN;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header Referrer-Policy "no-referrer";

        resolver 8.8.8.8 8.8.4.4 valid=300s;

# For WordPress

        location / {
        try_files $uri $uri/ /index.php?$args;
        }

        location ~ .php$ {
                include snippets/fastcgi-php.conf;
                fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }
}

Validate the nginx config:

sudo nginx -t

Restart nginx:

sudo service nginx restart

Add inbound and outbound firewall rules:

sudo ufw limit 22/tcp && sudo ufw allow 443/tcp && sudo ufw allow out 22/tcp && sudo ufw allow out 25/tcp && sudo ufw allow out 53/udp && sudo ufw allow out 443/tcp && sudo ufw allow out 9050/tcp && sudo ufw deny out to any && sudo ufw enable && sudo ufw status verbose

Updating Caddy to PHP 7.2

This was wonderfully easy.

sudo apt update && sudo apt dist-upgrade -y && sudo apt autoremove -y && sudo apt autoclean
sudo apt install php7.2-cli php7.2-common php7.2-fpm php7.2-json php7.2-mysql php7.2-opcache php7.2-readline

Update Caddyfile:

sudo vim /etc/caddy/Caddyfile
www.yawnbox.com {
        redir https://yawnbox.com{uri}
        }

yawnbox.com {
        root /var/www/yawnbox/
        log stdout
        errors stderr

header / {
        Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline';"
        Expect-CT "enforce; max-age=7073887;"
        Referrer-Policy "strict-origin, strict-origin-when-cross-origin"
        Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
        X-XSS-Protection "1; mode=block"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        }

fastcgi / /var/run/php/php7.2-fpm.sock {
        ext .php
        split .php
        index index.php
        }

rewrite / {
        to {path} {path}/ /index.php?{query}
        }

tls {
        protocols tls1.2
        curves p384
        key_type p384
        }
sudo service caddy restart
sudo apt remove php-mysql php7.1-cli php7.1-common php7.1-fpm php7.1-json php7.1-mysql php7.1-opcache php7.1-readline

Ubuntu SSH crypto hardening

Fix the security and privacy of your Ubuntu 16.04, 16.10, and 17.04 web server access. Fuck global mass surveillance.

Special thanks to @stribika for writing a very similar guide two years ago.

From “man sshd_config”

allowable ciphers

Run “ssh -Q cipher” for validating usable “Ciphers” on clients and servers

Specifies the ciphers allowed.  Multiple ciphers must be comma-separated.  If the specified value begins with a ‘+’ character, then the specified ciphers will be appended to the default set instead of replacing them.

allowable message authentication code algorithms

Run “ssh -Q mac” for validating usable “MACs” on clients and servers

Specifies the available MAC (message authentication code) algorithms.  The MAC algorithm is used for data integrity protection. Multiple algorithms must be comma-separated.  If the specified value begins with a ‘+’ character, then the specified algorithms will be appended to the default set instead of replacing them. The algorithms that contain "-etm" calculate the MAC after encryption (encrypt-then-mac).  These are considered safer and their use recommended.

allowable key exchange algorithms

Run “ssh -Q kex” for validating usable “KexAlgorithms” on clients and servers

Specifies the available KEX (Key Exchange) algorithms.  Multiple algorithms must be comma-separated.  Alternately if the specified value begins with a ‘+’ character, then the specified methods will be appended to the default set instead of replacing them.

allowable server key algorithms

Run “ssh -Q key” for validating usable “HostKeyAlgorithms” on servers

Specifies the host key algorithms that the server offers.

allowable key authentication types

Run “ssh -Q key” for validating usable “HostbasedAcceptedKeyTypes” on servers

Specifies the key types that will be accepted for hostbased authentication as a comma-separated pattern list.  Alternately if the specified value begins with a ‘+’ character, then the specified key types will be appended to the default set instead of replacing them.

allowable public key authentication types

Run “ssh -Q key” for validating usable “PubkeyAcceptedKeyTypes” on servers

Specifies the key types that will be accepted for public key authentication as a comma-separated pattern list.  Alternately if the specified value begins with a ‘+’ character, then the specified key types will be appended to the default set instead of replacing them.

Fix your server keys

cd /etc/ssh

sudo rm ssh_host_*key*

sudo ssh-keygen -t ed25519 -f ssh_host_ed25519_key -N "" < /dev/null

Amending sshd

sudo vim /etc/ssh/sshd_config

Only use the ed25519 key (delete the others):

HostKey /etc/ssh/ssh_host_ed25519_key

Add these lines (tailor them down based on what you know your client and server have available (see above for “ssh -Q x” options)):

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256

KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521

HostKeyAlgorithms ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519

HostbasedAcceptedKeyTypes ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519

PubkeyAcceptedKeyTypes ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519

Restart sshd:

sudo service ssh restart

ssh over Tor

Fix metadata leaks by using Tor as your second end-to-end encrypted tunnel if you don’t mind a mildly delayed CLI due to added latency.

Install Tor by first fixing apt sources and adding Tor’s repo:

sudo vim /etc/apt/sources.list

Delete all lines and use these (replace “zesty” if needed):

deb https://mirrors.wikimedia.org/ubuntu/ zesty main restricted universe multiverse
deb https://mirrors.wikimedia.org/ubuntu/ zesty-updates main restricted universe multiverse
deb https://mirrors.wikimedia.org/ubuntu/ zesty-backports main restricted universe multiverse
deb https://mirrors.wikimedia.org/ubuntu/ zesty-security main restricted universe multiverse
deb https://deb.torproject.org/torproject.org zesty main

Install Tor’s signing key:

sudo gpg --keyserver keys.gnupg.net --recv A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89

sudo gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add -

Update and install Tor:

sudo apt-get update && sudo apt-get install tor deb.torproject.org-keyring -y

Configure Tor for an onion:

sudo vim /etc/tor/torrc

Delete all lines and add these:

HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 22 127.0.0.1:22

Restart Tor:

sudo service tor restart

Find your new dot-onion address:

sudo cat /var/lib/tor/hidden_service/hostname

Configure sshd to only listen via Tor (and not exposed on the clear net):

sudo vim /etc/ssh/sshd_config

Add (or change) this line:

ListenAddress 127.0.0.1:22

Restart sshd:

sudo service ssh restart

Firewall everything

Presuming you are only hosting a web server over ports 80 and 443:

sudo ufw allow 80/tcp && sudo ufw allow 443/tcp && sudo ufw allow out 53/udp && sudo ufw allow out 80/tcp && sudo ufw allow out 123/udp && sudo ufw allow out 443/tcp && sudo ufw allow out 9050/tcp && sudo ufw deny out to any && sudo ufw enable && sudo ufw status verbose

80 for http
443 for https
53 out for DNS
123 out for NTP
9050 out for Tor

Deny everything else.

client side for Tor

sudo vim /etc/ssh/ssh_config

Add these lines under “Host *” (tailor the Ciphers, MACs, and Kex down based on what you know your client and server have available (see above for “ssh -Q x” options)):

UseRoaming no

proxyCommand ncat -v --proxy localhost:9050 --proxy-type socks5 %h %p

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256

KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521

Generate client keys:

ssh-keygen -t ed25519 -o -a 100

Restart ssh:

sudo service ssh restart

Send the client public key to the server:

ssh-copy-id yawnbox@2vytis5xf5djnaoo.onion

Connect to the server with debug to verify hardened crypto:

ssh -v yawnbox@2vytis5xf5djnaoo.onion

You will find this info buried:

debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC:  compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC:  compression: none

debug2: key: /home/yawnbox/.ssh/id_rsa ((nil))
debug2: key: /home/yawnbox/.ssh/id_dsa ((nil))
debug2: key: /home/yawnbox/.ssh/id_ecdsa ((nil))
debug2: key: /home/yawnbox/.ssh/id_ed25519 (0x55zg8nba8f20)

Cheers

bonus server config script


#!/bin/bash

sudo apt-get update

sudo apt-get upgrade -y

sudo apt-get dist-upgrade -y

sudo apt-get install tor openssh-server -y

sudo apt-get autoremove -y

sudo apt-get autoclean

cd /etc/ssh

sudo rm ssh_host_*key*

sudo ssh-keygen -t ed25519 -f ssh_host_ed25519_key -N "" > /etc/ssh/sshd_config

sudo echo "HostKey /etc/ssh/ssh_host_ed25519_key" >> /etc/ssh/sshd_config

sudo echo "SyslogFacility AUTH" >> /etc/ssh/sshd_config

sudo echo "LogLevel INFO" >> /etc/ssh/sshd_config

sudo echo "LoginGraceTime 30" >> /etc/ssh/sshd_config

sudo echo "PermitRootLogin no" >> /etc/ssh/sshd_config

sudo echo "StrictModes yes" >> /etc/ssh/sshd_config

sudo echo "MaxAuthTries 5" >> /etc/ssh/sshd_config

sudo echo "MaxSessions 5" >> /etc/ssh/sshd_config

sudo echo "PasswordAuthentication no" >> /etc/ssh/sshd_config

sudo echo "PermitEmptyPasswords no" >> /etc/ssh/sshd_config

sudo echo "ChallengeResponseAuthentication no" >> /etc/ssh/sshd_config

sudo echo "UsePAM yes" >> /etc/ssh/sshd_config

sudo echo "X11Forwarding no" >> /etc/ssh/sshd_config

sudo echo "PrintMotd no" >> /etc/ssh/sshd_config

sudo echo "AcceptEnv LANG LC_*" >> /etc/ssh/sshd_config

sudo echo "Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr" >> /etc/ssh/sshd_config

sudo echo "MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256" >> /home/cs/test_sshd

sudo echo "KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521" >> /etc/ssh/sshd_config

sudo echo "HostKeyAlgorithms ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519" >> /etc/ssh/sshd_config

sudo echo "HostbasedAcceptedKeyTypes ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519" >> /etc/ssh/sshd_config

sudo echo "PubkeyAcceptedKeyTypes ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519" >> /etc/ssh/sshd_config

sudo service ssh restart

sudo mv /etc/apt/sources.list /etc/apt/sources1.bak

sudo touch /etc/apt/sources.list

sudo echo "deb https://mirrors.wikimedia.org/ubuntu/ xenial main restricted universe multiverse" >> /etc/apt/sources.list

sudo echo "deb https://mirrors.wikimedia.org/ubuntu/ xenial-updates main restricted universe multiverse" >> /etc/apt/sources.list

sudo echo "deb https://mirrors.wikimedia.org/ubuntu/ xenial-backports main restricted universe multiverse" >> /etc/apt/sources.list

sudo echo "deb https://mirrors.wikimedia.org/ubuntu/ xenial-security main restricted universe multiverse" >> /etc/apt/sources.list

sudo echo "deb https://deb.torproject.org/torproject.org xenial main" >> /etc/apt/sources.list

sudo torify gpg --keyserver keys.gnupg.net --recv A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89

sudo torify gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add -

sudo apt-get update

sudo apt-get install tor deb.torproject.org-keyring apt-transport-tor -y

sudo mv /etc/apt/sources.list /etc/apt/sources2.bak

sudo touch /etc/apt/sources.list

sudo echo "deb tor+https://mirrors.wikimedia.org/ubuntu/ xenial main restricted universe multiverse" >> /etc/apt/sources.list

sudo echo "deb tor+https://mirrors.wikimedia.org/ubuntu/ xenial-updates main restricted universe multiverse" >> /etc/apt/sources.list

sudo echo "deb tor+https://mirrors.wikimedia.org/ubuntu/ xenial-backports main restricted universe multiverse" >> /etc/apt/sources.list

sudo echo "deb tor+https://mirrors.wikimedia.org/ubuntu/ xenial-security main restricted universe multiverse" >> /etc/apt/sources.list

sudo echo "deb tor+https://deb.torproject.org/torproject.org xenial main" >> /etc/apt/sources.list

sudo echo "capability dac_read_search," >> /etc/apparmor.d/abstractions/tor

sudo /etc/init.d/apparmor reload

sudo mv /etc/tor/torrc /etc/tor/torrc.bak

sudo touch /etc/tor/torrc

sudo echo "HiddenServiceDir /var/lib/tor/hidden_service/" >> /etc/tor/torrc

sudo echo "HiddenServicePort 22 127.0.0.1:22" >> /etc/tor/torrc

sudo service tor restart

sudo touch ~/onion.txt

sudo cat /var/lib/tor/hidden_service/hostname >> ~/onion.txt

sudo echo "ListenAddress 127.0.0.1:22" >> /etc/ssh/sshd_config

sudo service ssh restart

sudo ufw allow 22/tcp

sudo ufw allow 80/tcp

sudo ufw allow 443/tcp

sudo ufw allow out 22/tcp

sudo ufw allow out 53/udp

sudo ufw allow out 80/tcp

sudo ufw allow out 123/udp

sudo ufw allow out 443/tcp

sudo ufw allow out 9050/tcp

sudo ufw deny out to any

sudo ufw enable

sudo ufw status verbose

sudo cat ~/onion.txt

Tor onion service config fails due to apparmor

Thanks for the help, Will.

After installing Tor on a new host and configuring an onion service, Tor fails due to AppArmor.

Hosts:

Xenial server
Zesty server

Tor versions:

0.3.0.9
0.3.1.4-alpha

Errors:

/var/log/kern.log |grep tor

Jul 20 19:25:58 zesty kernel: [   50.173406] audit: type=1400 audit(1500578758.127:16): apparmor="DENIED" operation="capable" profile="system_tor" pid=2148 comm="tor" capability=2  capname="dac_read_search"

/var/log/syslog |grep tor

Jul 20 19:26:00 zesty tor[2190]: Jul 20 19:26:00.111 [notice] Tor 0.3.1.4-alpha (git-c3fe257c709bb814) running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.2g, Zlib 1.2.11, Liblzma N/A, and Libzstd N/A.
Jul 20 19:26:00 zesty tor[2190]: Jul 20 19:26:00.112 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jul 20 19:26:00 zesty tor[2190]: Jul 20 19:26:00.113 [notice] This version is not a stable Tor release. Expect more bugs than usual.
Jul 20 19:26:00 zesty tor[2190]: Jul 20 19:26:00.114 [notice] Read configuration file "/usr/share/tor/tor-service-defaults-torrc".
Jul 20 19:26:00 zesty tor[2190]: Jul 20 19:26:00.114 [notice] Read configuration file "/etc/tor/torrc".
Jul 20 19:26:00 zesty tor[2190]: Configuration was valid
Jul 20 19:26:00 zesty tor[2193]: Jul 20 19:26:00.223 [notice] Tor 0.3.1.4-alpha (git-c3fe257c709bb814) running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.2g, Zlib 1.2.11, Liblzma N/A, and Libzstd N/A.
Jul 20 19:26:00 zesty tor[2193]: Jul 20 19:26:00.224 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jul 20 19:26:00 zesty tor[2193]: Jul 20 19:26:00.225 [notice] This version is not a stable Tor release. Expect more bugs than usual.
Jul 20 19:26:00 zesty tor[2193]: Jul 20 19:26:00.225 [notice] Read configuration file "/usr/share/tor/tor-service-defaults-torrc".
Jul 20 19:26:00 zesty tor[2193]: Jul 20 19:26:00.226 [notice] Read configuration file "/etc/tor/torrc".
Jul 20 19:26:00 zesty tor[2193]: Jul 20 19:26:00.233 [warn] Directory /var/lib/tor/hidden_service/ cannot be read: Permission denied
Jul 20 19:26:00 zesty tor[2193]: Jul 20 19:26:00.234 [warn] Failed to parse/validate config: Failed to configure rendezvous options. See logs for details.
Jul 20 19:26:00 zesty tor[2193]: Jul 20 19:26:00.235 [err] Reading config failed--see warnings above.
Jul 20 19:26:00 zesty systemd[1]: tor@default.service: Main process exited, code=exited, status=1/FAILURE
Jul 20 19:26:00 zesty systemd[1]: tor@default.service: Unit entered failed state.
Jul 20 19:26:00 zesty systemd[1]: tor@default.service: Failed with result 'exit-code'.
Jul 20 19:26:00 zesty systemd[1]: tor@default.service: Service hold-off time over, scheduling restart.
Jul 20 19:26:00 zesty systemd[1]: tor@default.service: Start request repeated too quickly.
Jul 20 19:26:00 zesty systemd[1]: tor@default.service: Unit entered failed state.
Jul 20 19:26:00 zesty systemd[1]: tor@default.service: Failed with result 'exit-code'.

Solution

sudo vim /etc/apparmor.d/abstractions/tor

add this line to capabilities:

capability dac_read_search,

reload:

sudo /etc/init.d/apparmor reload
sudo service tor restart

Ubuntu Server + Caddy + Mediawiki

Ubuntu Server 16.04 + Caddy 0.10.2 + Mediawiki 1.28.2

For use on your private LAN (no LetsEncrypt).

sudo ufw limit 22/tcp && sudo ufw allow 80/tcp && sudo ufw allow out 22/tcp && sudo ufw allow out 25/tcp && sudo ufw allow out 53/udp && sudo ufw allow out 80/tcp && sudo ufw deny out to any && sudo ufw enable && sudo ufw status verbose
sudo vim /etc/apt/sources.list

Delete all lines, use these instead:

deb https://mirrors.wikimedia.org/ubuntu/ xenial main restricted universe multiverse
deb https://mirrors.wikimedia.org/ubuntu/ xenial-updates main restricted universe multiverse
deb https://mirrors.wikimedia.org/ubuntu/ xenial-backports main restricted universe multiverse
deb https://mirrors.wikimedia.org/ubuntu/ xenial-security main restricted universe multiverse
sudo apt-get update && sudo apt-get upgrade -y && sudo apt-get dist-upgrade -y && sudo shutdown -r now
sudo apt-get install mysql-server
mysql -u root -p
CREATE DATABASE mediawiki_db DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;

At the least, change the user and password:

GRANT ALL ON mediawiki_db.* TO 'mediawiki_user'@'localhost' IDENTIFIED BY 'mediawiki_pass';
FLUSH PRIVILEGES;
EXIT;
sudo add-apt-repository ppa:ondrej/php
sudo apt-get update && sudo apt-get install php7.1-fpm php7.1-cli php-mysql php7.1-intl php7.1-curl php7.1-gd php7.1-mbstring php7.1-xml
curl https://getcaddy.com | bash && sudo chown root:root /usr/local/bin/caddy && sudo chmod 755 /usr/local/bin/caddy && sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/caddy && sudo mkdir /etc/caddy && sudo chown -R root:www-data /etc/caddy && sudo mkdir /etc/ssl/caddy && sudo chown -R www-data:root /etc/ssl/caddy && sudo chmod 0770 /etc/ssl/caddy
sudo vim /etc/caddy/Caddyfile
*:80 {
        root /var/www/
        log stdout
        errors stderr

header / {
        Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'"
        Referrer-Policy "strict-origin, strict-origin-when-cross-origin"
        X-XSS-Protection "1; mode=block"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        }

fastcgi / /var/run/php/php7.1-fpm.sock {
        ext .php
        split .php
        index index.php
        }

rewrite / {
        to {path} {path}/ /index.php?{query}
        }
}
sudo chown www-data:www-data /etc/caddy/Caddyfile && sudo chmod 444 /etc/caddy/Caddyfile && sudo mkdir /var/www && sudo chown -R www-data:www-data /var/www && sudo chmod -R 555 /var/www && cd /tmp && wget https://releases.wikimedia.org/mediawiki/1.28/mediawiki-1.28.2.tar.gz && tar -xvzf mediawiki-1.28.2.tar.gz && sudo mv mediawiki-1.28.2/* /var/www/ && sudo chown www-data:www-data -R /var/www/
cd ~ && wget https://raw.githubusercontent.com/mholt/caddy/master/dist/init/linux-systemd/caddy.service && sudo cp caddy.service /etc/systemd/system/ && sudo chown root:root /etc/systemd/system/caddy.service && sudo chmod 644 /etc/systemd/system/caddy.service && sudo systemctl daemon-reload && sudo systemctl enable caddy.service && sudo /usr/local/bin/caddy -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/var/tmp

Setup Mediawiki by navigating to the internal IP address of the server. Upon completion, download LocalSettings.php then paste its contents to:

sudo vim /var/www/LocalSettings.php

Restart the server:

sudo shutdown -r now

Caddy will automatically start and you can start using your wiki!