All posts by yawnbox

Updating Intel’s i40e driver in Ubuntu 18.10

No, Canonical does not provide the latest stable Intel i40e driver in the latest version of Ubuntu Server. In fact it’s 12 months old.

No, Intel does not keep the most current version of the i40e driver on Intel.com. It’s on SourceForge.

/HeadDesk

Let’s look at all of the REALLY IMPORTANT changes made to the i40e driver since version 2.3.2 that’s installed by default in Ubuntu 18.10:

Changelog for i40e 2.7.11
===========================================================================
Implement ADq feature
Add support for configuring VFs through sysfs
Add XDP support
Add support Forward Error Correction (FEC) on 25G links
Allow disabling FW LLDP (requires updated firmware)
Fix bug where link could be forced down after MTU change
Add support for new kernel APIs
Fix WARN_ON() if MSI-X not available
Fix issue with lost link when forcing link mode on some adapters
Add support to allow the FW update tool to upgrade/downgrade from any FW version to any other FW version
   Note: this may require a power cycle as an intermediate step.
Fix kernel panic with repeated quick enable/disable of VFs
Fix possible TX timeout with DCB on some kernels
Re-enable TSO after MTU change
Fix potential panic on shutdown with many MSI-X vectors
Add VEB stats to ethtool
Fix panic on suspend with RDMA client driver active
Fix timing issue with assigning VFs to VMs using virsh
Restore TCPv4 input set when re-enabling ATR
Performance fixes with Adaptive ITR
Code cleanups
Minor bug fixes
Add support for RHEL 7.5
Add support for SLES12SP3

Source: README.txt, updated 2018-10-26
Changelog for i40e 2.4.10
===========================================================================
Add support for RHEL 7.5
Fix synchronization issue with simultaneous resets on multiple ports

Source: README.txt, updated 2018-06-08
Changelog for i40e 2.4.6
===========================================================================
- fix unnecessary PF resets issue
- update Flow Director User documentation
- fix compilation error on kernel 4.15.1

Source: README.txt, updated 2018-04-03
Changelog for i40e 2.4.3
===========================================================================
- fix issue on X722 with netdev close removing RDMA instance
- performance updates
- fix some issues with link reporting
- fix issue with tx timeouts on systems with > 128 cores
- fix issue with promiscuous mode disappearing across resets
- fix issue with flow director filters not working on fragmented IPv4
- fix issue with duplicate flow director filters not being updated to HW

Source: README.txt, updated 2017-12-07
Changelog for i40e-linux-2.3.6
===========================================================================

- Fix mac filter removal timing issue
- Sync i40e_ethtool.c with upstream
- Fixes for TX hangs
- Some fixes for reset of VFs
- Fix build error with packet split disabled
- Fix memory leak related to filter programming status
- Add and modify branding strings
- Fix kdump failure
- Implement an ethtool private flag to stop LLDP in FW
- Add delay after EMP reset for firmware to recover
- Fix incorrect default ITR values on driver load
- Fixes for programming cloud filters
- Some performance improvements
- Enable XPS with QoS on newer kernels
- Enable support for VF VLAN tag stripping control
- Build fixes to force perl to load specific ./SpecSetup.pm file
- Fix the updating of pci.ids
- Use 16 byte descriptors by default
- Fixes for DCB
- Don't close client in debug mode
- Add change MTU log in VF driver
- Fix for adding multiple ethtool filters on the same location
- Add new branding strings for OCP XXV710 devices
- Remove X722 Support for Destination IP Cloud Filter
- Allow turning off offloads when the VF has VLAN set

Source: README.txt, updated 2017-10-31

/HeadDesk

I have already updated the X710-DA4 firmware (to 6.01) prior to updating the driver, but I have not read anywhere that updating the firmware first is necessary.

I started by looking at my installed drivers, but this is not needed, and does not show the versions of the driver:

ls -R /lib/modules/`uname -r`/kernel/ > drivers.log

Show current i40e driver:

ethtool -i enp1s0f0

Output:

driver: i40e
version: 2.3.2-k
firmware-version: 6.01 0x800035ce 1.1747.0
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Install dependencies:

sudo apt install make gcc

Download the latest driver:

wget https://cfhcable.dl.sourceforge.net/project/e1000/i40e%20stable/2.7.11/i40e-2.7.11.tar.gz

Unzip:

tar -xf i40e-2.7.11.tar.gz

Instructions are in the README — read them:

vim i40e-2.7.11/README

Install. This was performed while my X710-DA4 was not being used nor had assigned IPs:

sudo su
cd i40e-2.7.11/src/
make install
rmmod i40e; modprobe i40e
update-initramfs -u
exit

Confirm updated driver:

ethtool -i enp1s0f0

Output:

driver: i40e
version: 2.7.11
firmware-version: 6.01 0x800035ce 1.1747.0
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Don’t forget to read the included README for performance tuning options, and the over 2 year old Intel Ethernet Controller X710/XL710 and Intel Ethernet Converged Network Adapter X710/XL710 Family Linux Performance Tuning Guide from March 2016:

https://www.intel.com/content/dam/www/public/us/en/documents/reference-guides/xl710-x710-performance-tuning-linux-guide.pdf

Oh and:

sudo vim /etc/default/grub

Like this:

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=2
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt pcie_aspm=off"
GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt pcie_aspm=off"
sudo update-grub
sudo shutdown -r now
Advertisements

Setting up DPDK 18.08 in Ubuntu 18.04

I’m just testing, this post isn’t complete nor should it be used.

sudo apt install gcc gcc-multilib libnuma-dev python python3.7 python-dev python-pip dpdk hugepages

sudo vim /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT=default_hugepagesz=1G hugepagesz=1G hugepages=4
GRUB_CMDLINE_LINUX=default_hugepagesz=1GB hugepagesz=1G hugepages=4

sudo vim /etc/sysctl.conf

kernel.shmmax = 8589934592
vm.hugetlb_shm_group = 0
vm.nr_hugepages = 8

sudo vim /etc/security/limits.conf

* soft nproc 65535
* hard nproc 65535
* soft nofile 65535
* hard nofile 65535
* soft memlock 8388608
* hard memlock 8388608

sudo shutdown -r now

git clone https://dpdk.org/git/dpdk 

 cd dpdk

make config T=x86_64-native-linuxapp-gcc 

 make T=x86_64-native-linuxapp-gcc 

 make install T=x86_64-native-linuxapp-gcc

These steps are still not persistent after reboots:

sudo modprobe uio i40e vfio-pci

sudo insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko

./usertools/dpdk-devbind.py --status |grep Eth

sudo ./usertools/dpdk-devbind.py --bind=igb_uio enp1s0f0

./usertools/dpdk-devbind.py --status

Tuning a fast Tor relay (0.3.5.3-alpha) on Ubuntu 18.04

Rule: don’t enable iptables / ufw

as root:

ip link set eth0 txqueuelen 300000

These are just temporary:

sysctl -w fs.file-max=65535
sysctl -w net.core.optmem_max=25165824
sysctl -w net.core.netdev_max_backlog=65536
sysctl -w net.core.rmem_default=25165824
sysctl -w net.core.rmem_max=25165824
sysctl -w net.core.somaxconn=20480
sysctl -w net.core.wmem_default=25165824
sysctl -w net.core.wmem_max=25165824
sysctl -w net.ipv4.ip_local_port_range='1024 65535'
sysctl -w net.ipv4.route.flush=1
sysctl -w net.ipv4.tcp_congestion_control=cubic
sysctl -w net.ipv4.tcp_fin_timeout=15
sysctl -w net.ipv4.tcp_keepalive_intvl=15
sysctl -w net.ipv4.tcp_keepalive_probes=5
sysctl -w net.ipv4.tcp_keepalive_time=1200
sysctl -w net.ipv4.tcp_max_syn_backlog=65536
sysctl -w net.ipv4.tcp_max_tw_buckets=1440000
sysctl -w net.ipv4.tcp_mem='65536 131072 262144'
sysctl -w net.ipv4.tcp_moderate_rcvbuf=1
sysctl -w net.ipv4.tcp_mtu_probing=1
sysctl -w net.ipv4.tcp_no_metrics_save=1
sysctl -w net.ipv4.tcp_rfc1337=1
sysctl -w net.ipv4.tcp_rmem='20480 12582912 25165824'
sysctl -w net.ipv4.tcp_synack_retries=2
sysctl -w net.ipv4.tcp_tw_recycle=1
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.ipv4.tcp_window_scaling=1
sysctl -w net.ipv4.tcp_wmem='20480 12582912 25165824'
echo "* soft nproc 65535" >> /etc/security/limits.conf
echo "* hard nproc 65535" >> /etc/security/limits.conf
echo "* soft nofile 65535" >> /etc/security/limits.conf
echo "* hard nofile 65535" >> /etc/security/limits.conf
echo "root soft nproc 65535" >> /etc/security/limits.conf
echo "root hard nproc 65535" >> /etc/security/limits.conf
echo "root soft nofile 65535" >> /etc/security/limits.conf
echo "root hard nofile 65535" >> /etc/security/limits.conf
echo "session required pam_limits.so" >> /etc/pam.d/common-session
shutdown -r now

Using overclocked speeds with an Intel i7-8086K at 5.2GHz in Ubuntu Server

This tutorial demonstrates how to assure that Ubuntu Server 18.04 x64 will operate at a specific CPU frequency overclock. I will also demonstrate how to validate the frequency per CPU core and monitor the temperature of each CPU core in near real time.

This tutorial presumes you have already installed and updated the OS, in addition to having a stable overclock. In my case, I have found that my Intel i7-8086K operates at 5.2GHz at 1.392 volts while my RAM is using its XMP profile. This tutorial, however, only focuses on CPU frequency settings. Hyper-threading is disabled in BIOS so Ubuntu sees 6 cores and 6 threads.

sudo apt install cpufrequtils lm-sensors -y

sudo vim /etc/init.d/cpufrequtils

Edit and enable these lines to your target overclocked frequency:

ENABLE="true"
GOVERNOR="performance"
MAX_SPEED="5200000"
MIN_SPEED="5200000"

5.0GHz = 5000000
5.2GHz = 5200000
5.4GHz = 5400000

Note: If AVX is set to 2 in BIOS, even with this MIN_SPEED setting, Ubuntu will cap itself at 5.0GHz. Set AVX to 0 to assure Ubuntu will stay at 5.2GHz constantly.

sudo systemctl daemon-reload

Open a new terminal and watch the CPU frequencies for cores 0-5 (6 cores)

watch -n.1 "cat /proc/cpuinfo | grep "^[c]pu MHz""

Find the available temp sensors on your system and say YES to everything:

sudo sensors-detect

Open a new terminal and watch the CPU temps:

watch -n.1 "sensors"

We can use OpenSSL to burn the CPU for testing… this “-multi 6” flag will burn all 6 of my cores (be sure you have proper cooling on your CPU!):

openssl speed -multi 6

Note: Like in Windows, if my CPU is not properly cooled resulting in CPU temps at or above 100 degree C, my multiplier will automatically throttle down in order to keep the core temps down below critical. If you’re needing to test single-threaded performance, “-multi 1” will just burn one core. Burning half the available cores or less will keep temps down quite a bit since heat dissipation on the IHS is cut in half, meaning the multiplier should stay at the set x52. This is all dependent on proper cooling.

Also checkout CoreFreq for seeing greater CPU details like your core multipliers:

https://github.com/cyring/CoreFreq

sudo apt install dkms git libpthread-stubs0-dev -y

git clone https://github.com/cyring/CoreFreq.git

cd CoreFreq

make

Load the kernel module:

sudo insmod corefreqk.ko

Load the daemon

sudo ./corefreqd

In a new window:

./CoreFreq/corefreq-cli

Configurations that didn’t help me, especially after reboots:

sudo bash -c 'for i in {0..5}; do cpufreq-set -c $i -g performance; done'

for x in /sys/devices/system/cpu/*/cpufreq/; do echo 5200000 | sudo tee $x/scaling_max_freq; done

for x in /sys/devices/system/cpu/*/cpufreq/; do echo 5200000 | sudo tee $x/scaling_min_freq; done

sudo cpupower frequency-set -d 5.2GHz -u 5.2GHz -g performance -r

sudo cpufreq-set -d 5.2GHz -u 5.2GHz -g performance -r

building caddy from source (and updating caddy on xenial)

sudo add-apt-repository ppa:gophers/archive
sudo vim /etc/apt/sources.list.d/gophers-ubuntu-archive-xenial.list

deb tor+http://ppa.launchpad.net/gophers/archive/ubuntu xenial main

sudo apt update && sudo apt install golang-1.10*
sudo apt remove golang-1.6*
sudo apt update && sudo apt dist-upgrade -V && sudo apt autoremove -y && sudo apt autoclean
/usr/lib/go-1.10/bin/go version
echo 'PATH="/usr/lib/go-1.10/bin:$PATH"' >> ~/.profile
source ~/.profile
go version

go version go1.10 linux/amd64

go get -u github.com/mholt/caddy
go get -u github.com/caddyserver/builds
cd ~/go/src/github.com/mholt/caddy/caddy
go run build.go -goos=linux -goarch=amd64
sudo service caddy stop
sudo cp /usr/local/bin/caddy /usr/local/bin/caddy.bak
cp ~/go/src/github.com/mholt/caddy/caddy/caddy /usr/local/bin
sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/caddy
caddy --version

Caddy 0.11.0 (+60a0208) (unofficial)

sudo shutdown -r now

A+ TLS config for ubuntu + nginx

These are my config notes for getting a brand new Xenial + nginx server online.

Install Tor:

sudo apt install tor apt-transport-tor
sudo gpg --keyserver keys.gnupg.net --recv 886DDD89

sudo gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add -

Edit the sources list by removing all the lines and adding these:

sudo vim /etc/apt/sources.list
deb tor+https://deb.torproject.org/torproject.org xenial main
deb tor+https://mirrors.wikimedia.org/ubuntu/ xenial main restricted universe multiverse
deb tor+https://mirrors.wikimedia.org/ubuntu/ xenial-updates main restricted universe multiverse
deb tor+https://mirrors.wikimedia.org/ubuntu/ xenial-security main restricted universe multiverse

Update the repos:

sudo add-apt-repository ppa:nginx/development
sudo add-apt-repository ppa:ondrej/nginx
sudo add-apt-repository ppa:ondrej/php
sudo add-apt-repository ppa:certbot/certbot

Add “tor+” to all of the above sources files in /etc/apt/sources.list.d/*

Update and restart:

sudo apt update && sudo apt upgrade -V && sudo apt autoremove -y && sudo shutdown -r now

Install nginx + certbot:

sudo apt install python-certbot-nginx -V

Add server_name to (replacing “_”):

sudo vim /etc/nginx/sites-available/default
server_name domain.net;

Get Let’s Encrypt cert for nginx:

sudo certbot --nginx -d domain.net --redirect --rsa-key-size 4096

Further harden the TLS config:

sudo vim /etc/letsencrypt/options-ssl-nginx.conf
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:!3DES:!aNULL:!DES:!DSS:!eNULL:!EXP:!IDEA:!LOW:!MD5:!PSK:!RC4:!SEED";

Delete the “SSL” config:

sudo vim /etc/nginx/nginx.conf

Edit the nginx config:

sudo vim /etc/nginx/sites-available/default

replace “domain.net”

server {
        listen 80 default_server;
        listen [::]:80 default_server;

        server_name domain.net www.domain.net;
        return 301 https://$host$request_uri;

        server_tokens off;
        add_header X-Frame-Options SAMEORIGIN;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header Referrer-Policy "no-referrer";
}

server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;

        server_name domain.net www.domain.net;
        root /var/www;
        index index.php index.html index.htm;

        ssl_certificate /etc/letsencrypt/live/domain.net/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/domain.net/privkey.pem;
        include /etc/letsencrypt/options-ssl-nginx.conf;
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

        server_tokens off;
        add_header Strict-Transport-Security 'max-age=31536000; includeSubDomains; preload';
        add_header X-Frame-Options SAMEORIGIN;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header Referrer-Policy "no-referrer";

        resolver 8.8.8.8 8.8.4.4 valid=300s;

# For WordPress

        location / {
        try_files $uri $uri/ /index.php?$args;
        }

        location ~ .php$ {
                include snippets/fastcgi-php.conf;
                fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }
}

Validate the nginx config:

sudo nginx -t

Restart nginx:

sudo service nginx restart

Add inbound and outbound firewall rules:

sudo ufw limit 22/tcp && sudo ufw allow 443/tcp && sudo ufw allow out 22/tcp && sudo ufw allow out 25/tcp && sudo ufw allow out 53/udp && sudo ufw allow out 443/tcp && sudo ufw allow out 9050/tcp && sudo ufw deny out to any && sudo ufw enable && sudo ufw status verbose