Category Archives: Ubuntu

Windows Server 2012, Hyper-V, Ubuntu+ZFS VM for Backups

I set this up only to test ZFS with deduplication as an alternative to using a standard hardware RAID configuration for storing SQL backups.

Hardware used:

  • Dell R720xd
  • Dell Perc H800 RAID controller
  • Dell PowerVault MD1220
  1. Configure RAID on the Dell R720xd / Perc H800 controller. All 24 disks (2.5″ 900GB SAS in my case) as an independent RAID-0 logical volume since the H800 (as far as I can tell) doesn’t support an easy JBOD option.
    Untitled3
    Untitled1
    Untitled2
  2. In Windows Server 2012 Computer Management, I set each of the 24 volumes to GPT partition table when asked, and then set all 24 volumes “offline”. Doing so allows Hyper-V to access each disk directly, so that Windows cannot.
    Untitled5
  3. Install the Hyper-V role and rebooted.
  4. I went into the VMs settings. First I added a SCSI controller, since IDE Controllers are limited to 2 devices, while SCSI can support up to 64 devices. Then I added the 24 volumes as “physical hard disks”, matching the SCSI location number 0 – 23 to the 24 volume target number 0 – 23.
    Untitled4
  5. Also, make sure to install a virtual switch and configure your network interface for the Ubuntu VM.
  6. Downloaded Ubuntu Server 12.04 x64, installed Ubuntu with OpenSSH and Samba features. Started the Ubuntu VM.
  7. Verify Ubuntu can see the 24 volumes:
    sudo lshw -C disk

    Untitled6

  8. sudo apt-get update
  9. sudo apt-get install python-software-properties software-properties-common -y
  10. sudo add-apt-repository ppa:zfs-native/stable -y
  11. sudo apt-get update && sudo apt-get dist-upgrade -y
  12. sudo apt-get install ubuntu-zfs -y
  13. dmesg | grep ZFS
    [    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.11.0-15-generic root=/dev/mapper/DPMHOST--ZFS--vg-root ro
     [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.11.0-15-generic root=/dev/mapper/DPMHOST--ZFS--vg-root ro
     [    8.529432] Adding 4190204k swap on /dev/mapper/DPMHOST--ZFS--vg-swap_1.  Priority:-1 extents:1 across:4190204k SSFS
  14. sudo vim /etc/modules
  15. Add these lines:
    spl
    zavl
    znvpair
    zunicode
    zcommon
    zfs
  16. Incorporate new modules into the boot files:
    sudo update-initramfs -u
  17. sudo reboot
  18. I created a ZFS pool called “zfs0” using raidz3 which can lose up to 3 disks using all 24 volumes:
    sudo zpool create zfs0 raidz3 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy -f
  19. sudo zpool status
    pool: zfs0
     state: ONLINE
     scan: none requested
     config:
    
    NAME        STATE     READ WRITE CKSUM
     zfs0      ONLINE       0     0     0
     raidz3-0  ONLINE       0     0     0
     sdb     ONLINE       0     0     0
     sdc     ONLINE       0     0     0
     sdd     ONLINE       0     0     0
     sde     ONLINE       0     0     0
     sdf     ONLINE       0     0     0
     sdg     ONLINE       0     0     0
     sdh     ONLINE       0     0     0
     sdi     ONLINE       0     0     0
     sdj     ONLINE       0     0     0
     sdk     ONLINE       0     0     0
     sdl     ONLINE       0     0     0
     sdm     ONLINE       0     0     0
     sdn     ONLINE       0     0     0
     sdo     ONLINE       0     0     0
     sdp     ONLINE       0     0     0
     sdq     ONLINE       0     0     0
     sdr     ONLINE       0     0     0
     sds     ONLINE       0     0     0
     sdt     ONLINE       0     0     0
     sdu     ONLINE       0     0     0
     sdv     ONLINE       0     0     0
     sdw     ONLINE       0     0     0
     sdx     ONLINE       0     0     0
     sdy     ONLINE       0     0     0
    
    errors: No known data errors
  20. sudo zfs list
    NAME   USED  AVAIL  REFER  MOUNTPOINT
     zfs0   297K  16.7T  89.8K  /zfs0
  21. df -h
    Filesystem                         Size  Used Avail Use% Mounted on
     /dev/mapper/DPMHOST--ZFS--vg-root   15G  1.5G   13G  11% /
     udev                               2.0G  4.0K  2.0G   1% /dev
     tmpfs                              790M  668K  789M   1% /run
     none                               5.0M     0  5.0M   0% /run/lock
     none                               2.0G     0  2.0G   0% /run/shm
     /dev/sda1                          236M   32M  192M  14% /boot
     zfs0                                17T  128K   17T   1% /zfs0
  22. Configure Samba:
    sudo vim /etc/samba/smb.conf
  23. sudo zfs set sharesmb=on zfs0/backuptest1
  24. sudo chmod 0777 /zfs0/backuptest1
  25. sudo service smbd restart
  26. sudo zfs get sharesmb,sharenfs
    NAME              PROPERTY  VALUE     SOURCE
    zfs0              sharesmb  off       default
    zfs0              sharenfs  off       default
    zfs0/backuptest1  sharesmb  on        local
    zfs0/backuptest1  sharenfs  off       default
  27. I set compression as LZ4, which does wonders for raw SQL files:
    sudo zfs set compression=lz4 zfs0/backuptest1

Enjoy!

New PGP key for email

Heya

Used Seahorse today to manage my new PGP key for my email address yawnbox@gmail.com. My really good password was generated using passwordsgenerator.net and securely stored.

A531D73D

Other (older) keys are not valid (EBF3C1FD and 2FB08023), so please do not use them.

I followed RiseUp’s Howto Setup OpenPGP Keys. I then made a backup of my private key in a Tomb.

Exported to pgp.mit.edu for ease of use:

gpg --list-keys
gpg --export -a A531D73D > mypubkey.asc
gpg --send-keys --keyserver pgp.mit.edu A531D73D

Verifying output:

gpg: sending key A531D73D to hkp server pgp.mit.edu

Happy encrypting!

Strip PNG metadata using Ubuntu 13.11: pngcrush and optipng

PNG optimizing tools reduce size my getting rid of “extra” stuff. Some of that extra stuff is the metadata that can be used to identify who took the picture. I’m no professional metadata-remover, I just did this testing for fun.

sudo apt-get install -y pngcrush libimage-exiftool-perl
pngcrush -rem allb -brute -reduce original.png optimized.png && optipng -o7 optimized.png

pngcrush – run the pngcrush program

-rem allb – remove all extraneous data

-brute – attempt all optimization methods

-reduce – eliminate unused colors and reduce bit-depth 4

original.png – the name of the original (unoptimized) PNG file

optimized.png – the name of the new, optimized PNG file

&& – command #2 will be executed if and only if command #1 returns exit status zero

optipng – run the optipng program

-o7 – optimize the image at the highest possible level

optimized.png – the already pngcrush-optimized PNG file that will be further optimized (if possible) with optipng

Let’s test!

Here’s an image that’s CC-BY-SA from Wikipedia: http://upload.wikimedia.org/wikipedia/commons/8/89/Tenaya_Lake_Yosemite_National_Park.png

pngcrush -rem allb -brute -reduce Tenaya_Lake_Yosemite_National_Park.png Tenaya_Lake_Yosemite_National_Park2.png && optipng -o7 Tenaya_Lake_Yosemite_National_Park2.png

Then to check the metadata:

identify -verbose Tenaya_Lake_Yosemite_National_Park.png

Image: Tenaya_Lake_Yosemite_National_Park.png
Format: PNG (Portable Network Graphics)
Class: DirectClass
Geometry: 2048x1536+0+0
Resolution: 70.87x70.87
Print size: 28.898x21.6735
Units: PixelsPerCentimeter
Type: TrueColor
Endianess: Undefined
Colorspace: sRGB
Depth: 8-bit
Channel depth:
red: 8-bit
green: 8-bit
blue: 8-bit
Channel statistics:
Red:
min: 0 (0)
max: 255 (1)
mean: 111.14 (0.435842)
standard deviation: 42.8511 (0.168043)
kurtosis: 0.724192
skewness: 1.17595
Green:
min: 0 (0)
max: 255 (1)
mean: 130.885 (0.513273)
standard deviation: 38.549 (0.151173)
kurtosis: -0.384294
skewness: 0.458422
Blue:
min: 0 (0)
max: 255 (1)
mean: 155.366 (0.609278)
standard deviation: 48.5428 (0.190364)
kurtosis: -0.907909
skewness: -0.00882359
Image statistics:
Overall:
min: 0 (0)
max: 255 (1)
mean: 132.463 (0.519464)
standard deviation: 43.5073 (0.170617)
kurtosis: 0.23089
skewness: 0.637459
Rendering intent: Perceptual
Gamma: 0.454545
Chromaticity:
red primary: (0.64,0.33)
green primary: (0.3,0.6)
blue primary: (0.15,0.06)
white point: (0.3127,0.329)
Interlace: None
Background color: white
Border color: srgb(223,223,223)
Matte color: grey74
Transparent color: black
Compose: Over
Page geometry: 2048x1536+0+0
Dispose: Undefined
Iterations: 0
Compression: Zip
Orientation: Undefined
Properties:
date:create: 2014-01-15T19:41:21-08:00
date:modify: 2014-01-15T19:41:21-08:00
png:cHRM : chunk was found (see Chromaticity, above)
png:gAMA : gamma=0.45454544 (See Gamma, above)
png:iCCP : chunk was found
png:IHDR.bit_depth : 8
png:IHDR.color_type : 2 (Truecolor)
png:IHDR.interlace_method: 0 (Not interlaced)
png:IHDR.width,height : 2048, 1536
png:pHYs : x_res=7087, y_res=7087, units=1
png:sRGB : intent=0 (See Rendering intent)
signature: 4be08d8b3f54c63739c5653a38dd4f817da97114025dddaccbf7e9e533396d56
Profiles:
Profile-icc: 1352 bytes
Description: Camera RGB Profile
Manufacturer: Camera RGB Profile
Model: Camera RGB Profile
Copyright: Copyright 2003 Apple Computer Inc., all rights reserved.
Artifacts:
filename: Tenaya_Lake_Yosemite_National_Park.png
verbose: true
Tainted: False
Filesize: 4.961MB
Number pixels: 3.146M
Pixels per second: 10.49MB
User time: 0.290u
Elapsed time: 0:01.300
Version: ImageMagick 6.7.7-10 2013-09-10 Q16 http://www.imagemagick.org

And compare the optimized copy:

identify -verbose Tenaya_Lake_Yosemite_National_Park2.png

Image: Tenaya_Lake_Yosemite_National_Park2.png
Format: PNG (Portable Network Graphics)
Class: DirectClass
Geometry: 2048x1536+0+0
Resolution: 72x72
Print size: 28.4444x21.3333
Units: Undefined
Type: TrueColor
Endianess: Undefined
Colorspace: sRGB
Depth: 8-bit
Channel depth:
red: 8-bit
green: 8-bit
blue: 8-bit
Channel statistics:
Red:
min: 0 (0)
max: 255 (1)
mean: 111.14 (0.435842)
standard deviation: 42.8511 (0.168043)
kurtosis: 0.724192
skewness: 1.17595
Green:
min: 0 (0)
max: 255 (1)
mean: 130.885 (0.513273)
standard deviation: 38.549 (0.151173)
kurtosis: -0.384294
skewness: 0.458422
Blue:
min: 0 (0)
max: 255 (1)
mean: 155.366 (0.609278)
standard deviation: 48.5428 (0.190364)
kurtosis: -0.907909
skewness: -0.00882359
Image statistics:
Overall:
min: 0 (0)
max: 255 (1)
mean: 132.463 (0.519464)
standard deviation: 43.5073 (0.170617)
kurtosis: 0.23089
skewness: 0.637459
Rendering intent: Perceptual
Gamma: 0.454545
Chromaticity:
red primary: (0.64,0.33)
green primary: (0.3,0.6)
blue primary: (0.15,0.06)
white point: (0.3127,0.329)
Interlace: None
Background color: white
Border color: srgb(223,223,223)
Matte color: grey74
Transparent color: black
Compose: Over
Page geometry: 2048x1536+0+0
Dispose: Undefined
Iterations: 0
Compression: Zip
Orientation: Undefined
Properties:
date:create: 2014-01-15T19:58:34-08:00
date:modify: 2014-01-15T19:58:34-08:00
png:cHRM : chunk was found (see Chromaticity, above)
png:gAMA : gamma=0.45454544 (See Gamma, above)
png:IHDR.bit_depth : 8
png:IHDR.color_type : 2 (Truecolor)
png:IHDR.interlace_method: 0 (Not interlaced)
png:IHDR.width,height : 2048, 1536
png:sRGB : intent=0 (See Rendering intent)
signature: 4be08d8b3f54c63739c5653a38dd4f817da97114025dddaccbf7e9e533396d56
Artifacts:
filename: Tenaya_Lake_Yosemite_National_Park2.png
verbose: true
Tainted: False
Filesize: 4.454MB
Number pixels: 3.146M
Pixels per second: 14.98MB
User time: 0.210u
Elapsed time: 0:01.209
Version: ImageMagick 6.7.7-10 2013-09-10 Q16 http://www.imagemagick.org

Setting up OpenVPN Access Server for Ubuntu 13.11

About OpenVPN Access Server: https://openvpn.net/index.php/access-server/overview.html. I use OpenVPN-AS to self-host a really easy to use VPN for Windows, Linux, and Android devices.

Access Server release notes for 2.0.3: http://openvpn.net/index.php/access-server/download-openvpn-as-sw/532-release-notes-v200.html

on the server side:

sudo apt-get install openvpn bridge-utils openvpn-blacklist
openvpn --version

You should get (or later): “OpenVPN 2.3.2 x86_64-pc-linux-gnu”.

Check to verify that you will be downloading and installing the latest version of OpenVPN-AS by visiting this page and selecting your OS: https://openvpn.net/index.php/access-server/download-openvpn-as-sw.html

wget http://swupdate.openvpn.org/as/openvpn-as-2.0.3-Ubuntu13.amd64.deb
sudo dpkg -i openvpn-as-2.0.3-Ubuntu13.amd_64.deb

I use Ubuntu’s “Uncomplicated Firewall” (https://help.ubuntu.com/community/UFW) to mange my server-side iptables firewall. I added a rule to allow incoming TCP traffic over port 1194.

sudo ufw allow 1194/tcp
sudo ufw reload

Create a user on your server that won’t have administrative rights, that you’ll use to access your VPN:

sudo adduser ovpnuser

then on the client side:

Unfortunately, at the time of writing, “.ovpn” files are not supported through the gnome GUI as described here: http://askubuntu.com/questions/187511/how-can-i-use-a-ovpn-file-with-network-manager. So you will have to connect via command-line.

  1. Go to https://your_static_ip:1194 in your web browser.
  2. Log in with the above user credentials that you created.
  3. Click: “Yourself (user-locked profile)” to download the “client.ovpn” file.
  4. Open a terminal window and enter:
sudo openvpn --config /home/your_user/Downloads/client.ovpn

5. Verify that you’re using your remote IP address: http://ipchicken.com/

Ubuntu 13.11 + ZFS / raidz2 Samba share

These are the steps that I took and what works for me. I hope it helps someone else. Configure the RAID controller as either JBOD or as each HDD being an independent RAID-0 logical volume. Then install Ubuntu server 13.11 x64 with OpenSSH and Samba.

sudo add-apt-repository ppa:zfs-native/stable
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install ubuntu-zfs python-software-properties
dmesg | grep ZFS
sudo vim /etc/modules

(add the following…)

spl
zavl
znvpair
zunicode
zcommon
zfs

(then run…)

sudo update-initramfs -u
sudo reboot
sudo zpool status
sudo zpool create zfsshare raidz2 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf -f
sudo zfs list
sudo zfs create zfsshare/backup
sudo zpool status
sudo vim /etc/samba/smb.conf

(configured smb.conf…)

sudo zfs set sharesmb=on zfsshare/backup
sudo chmod 0777 /zfsshare/backup
sudo service samba restart
sudo zfs get sharesmb,sharenfs
sudo zfs set compression=lz4 zfsshare/backup
sudo zdb -b zfsshare
sudo zfs set dedup=on zfsshare/backup

(after copying SQL .bak files, etc, to the share…)

ls -alh /zfsshare/backup/
sudo zfs get compressratio zfsshare/backup
sudo zfs get all |grep comp

Setup Nagios 4.02 in Ubuntu 12.04 LTS

Install Ubuntu 12.04, dist-update, install openssh-server. Then as root:

apt-get install -y vim apache2 libapache2-mod-php5 build-essential libgd2-xpm-dev libssl-dev sendmail-bin sendmail heirloom-mailx wget curl daemon apt-file libnet-snmp-perl libperl5.14 libpq5 libradius1 libsensors4 libsnmp-base libsnmp15 libtalloc2 libtdb1 libwbclient0 samba-common samba-common-bin smbclient snmp whois libmysqlclient15-dev && groupadd -g 3000 nagios && groupadd -g 3001 nagcmd && useradd -u 3000 -g nagios -G nagcmd -d /usr/local/nagios -c 'Nagios Admin' nagios && adduser www-data nagcmd && mkdir -p /var/www/nagios && cd /opt && wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-4.0.2.tar.gz && tar xf nagios-4.0.2.tar.gz && cd nagios-4.0.2 && mkdir -p /usr/local/nagios/share/{stylesheets,images} && ./configure --prefix=/usr/local/nagios --with-nagios-user=nagios --with-nagios-group=nagios --with-command-user=nagios --with-command-group=nagcmd && make all && make install && make install-init && make install-config && make install-commandmode && make install-webconf && make install-exfoliation && mkdir -p /usr/local/nagios/nagios-plugins && cd /usr/local/nagios/nagios-plugins && wget https://www.nagios-plugins.org/download/nagios-plugins-1.5.tar.gz && tar -xf nagios-plugins-1.5.tar.gz && cd nagios-plugins-1.5 && ./configure --with-nagios-user=nagios --with-nagios-group=nagios --with-openssl=/usr/bin/openssl --enable-perl-modules --enable-libtap && make && make install && mkdir -p /usr/local/nagios/nrpe && cd /usr/local/nagios/nrpe && wget http://kent.dl.sourceforge.net/project/nagios/nrpe-2.x/nrpe-2.15/nrpe-2.15.tar.gz && tar -xf nrpe-2.15.tar.gz && cd nrpe-2.15 && ./configure --with-ssl=/usr/bin/openssl --with-ssl-lib=/usr/lib/x86_64-linux-gnu && make all && make install && cd && rm /etc/init.d/nagios && vim /etc/init/nagios.conf

# Nagios 4.02
# by Remy Van Elst at https://raymii.org/s/tutorials/Nagios_Core_4_Installation_on_Ubuntu_12.04.html
description "nagios monitoring system"
start on virtual-filesystems
stop on runlevel [06]
respawn
respawn limit 5 30
limit nofile 65550 65550
chdir /usr/local/nagios/
setuid nagios
setgid nagios
console log
script
exec bin/nagios etc/nagios.cfg
end script

sudo initctl reload-configuration && vim /etc/apache2/httpd.conf

ScriptAlias /nagios/cgi-bin /usr/local/nagios/sbin
<Directory "/usr/local/nagios/sbin">
AllowOverride None
Options ExecCGI
Order allow,deny
Allow from all
Alias /nagios /usr/local/nagios/share
<Directory "/usr/local/nagios/share">
AllowOverride None
Options None
Order allow,deny
Allow from all

htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin
chown nagios:nagcmd /usr/local/nagios/etc/htpasswd.users && service apache2 restart && start nagios

Very special thank yous:

Get Tomb 1.4 up and running on Ubuntu 13.10

Tomb is an excellent command line tool for maintaining encrypted files. Tomb files can be stored on an Internet-facing server so that they can be accessed from anywhere in the world using any SSH client. An adversary would have to compromise said server, gain administrative privileges, and brute force the Tombs (if they have the key files) in order to recover the contents of said Tombs. Someone that is more “at risk” than me should invoke an air gap between the Internet and their Tombs. Managing your Tomb’s key files is a different matter that I’ll discuss later.

Read about Tomb here: http://www.dyne.org/software/tomb/

Download Tomb onto your Ubuntu server.

wget https://files.dyne.org/.xsend.php?file=tomb/releases/Tomb-1.4.tar.gz

Rename the downloaded file.

mv .xsend.php?file=tomb%2Freleases%2FTomb-1.4.tar.gz Tomb-1.4.tar.gz

Download the SHA hash/checksum file.

wget https://files.dyne.org/tomb/releases/Tomb-1.4.tar.gz.sha

View the Tomb tar file’s SHA hash.

cat Tomb-1.4.tar.gz.sha

2621ac6b9180321e69743dc899645449b2b958c6aa46e4b2601c2e89131bbf29  Tomb-1.4.tar.gz

View the Tomb tar file’s SHA checksum and compare it to the above hash–if they’re the same, continue with installation.

sha256sum Tomb-1.4.tar.gz

2621ac6b9180321e69743dc899645449b2b958c6aa46e4b2601c2e89131bbf29  Tomb-1.4.tar.gz

Unzip the Tomb tar file.

sudo tar -zxvf Tomb-1.4.tar.gz

Change into the newly created Tomb Directory.

cd Tomb-1.4/

Install Tomb.

sudo make install

Check that Tomb installed by checking its version.

tomb -v

Tomb 1.4 – a strong and gentle undertaker for your secrets

Copyright (C) 2007-2013 Dyne.org Foundation, License GNU GPL v3+
This is free software: you are free to change and redistribute it
The latest Tomb sourcecode is published on
This source code is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Please refer to the GNU Public License for more details.

System utils:

Sudo version 1.8.6p3
cryptsetup 1.4.3
pinentry-gtk2 0.8.1
gpg (GnuPG) 1.4.14 – key forging algorithms (GnuPG symmetric ciphers):
IDEA 3DES CAST5 BLOWFISH AES AES192 AES256 TWOFISH CAMELLIA128 CAMELLIA192 CAMELLIA256

Be sure to “shred” your Tombs or Tomb key files if you ever want to move them or delete them. If you’re moving your files, copy them first then shred the unwanted files. Do not simply move them.

sudo shred -f -v -z -u tomb.tomb.key