MergerFS on Ubuntu 20.04

This post for my own references, is made thanks to this great article.

https://www.teknophiles.com/2018/02/19/disk-pooling-in-linux-with-mergerfs/

Start by downloading the latest version, for me it is the mergerfs_2.31.0.ubuntu-bionic_amd64.deb file:

wget https://github.com/trapexit/mergerfs/releases/download/2.31.0/mergerfs_2.31.0.ubuntu-bionic_amd64.deb

Then simply install it using dpkg:

dpkg -i mergerfs_2.31.0.ubuntu-bionic_amd64.deb 

Following the install, you can easily mount separate mounted HDDs into a common containing folder.

root@lisa:/mnt# 
root@lisa:/mnt# 
root@lisa:/mnt# pwd
/mnt
root@lisa:/mnt# ls -l
total 8
drwxr-xr-x 3 root root 4096 Oct 13 15:14 SATA.2.WD-Red.3TB-1
drwxr-xr-x 3 root root 4096 Oct 13 15:14 SATA.3.WD-Red.3TB-2
root@lisa:/mnt# mkdir virtual
root@lisa:/mnt# mergerfs -o defaults,allow_other,use_ino,fsname=mergerFS /mnt/SATA.2.WD-Red.3TB-1:/mnt/SATA.3.WD-Red.3TB-2 /mnt/virtual
root@lisa:/mnt# 
root@lisa:/mnt# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           795M  1.1M  794M   1% /run
/dev/sda2       229G  6.6G  210G   4% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/loop0       98M   98M     0 100% /snap/core/10126
/dev/loop1       89M   89M     0 100% /snap/core/7270
/dev/sdb1       2.7T   89M  2.6T   1% /mnt/SATA.2.WD-Red.3TB-1
/dev/sdc1       2.7T   89M  2.6T   1% /mnt/SATA.3.WD-Red.3TB-2
tmpfs           795M     0  795M   0% /run/user/1000
mergerFS        5.4T  177M  5.1T   1% /mnt/virtual
root@lisa:/mnt# 

So far using mergerFS has proved easy, and very flexible. Finally we can use /etc/fstab to mount these at boot time. We can use wildcards to include the drive mount points as below:

root@lisa:/mnt# more /etc/fstab
UUID=42e23971-b35d2-4b5f-a5a5-2ade6bf39eee / ext4 defaults 0 0
# SDB1 on Slot 2
UUID=38175b20-4394-4c42-b14f-cdefw3bf4524 /mnt/SATA.2.WD-Red.3TB-1 ext4 defaults 0 0
# SDC1 on Slot 3
UUID=66ef6909-b715-4c80-ec91-acf6e734bf15 /mnt/SATA.3.WD-Red.3TB-2 ext4 defaults 0 0

# <file system>           <mount point>      <type>         <options>                                      <dump>  <pass>
/mnt/SATA.?.WD-Red.3TB-?  /mnt/virtual       fuse.mergerfs  defaults,allow_other,use_ino,fsname=mergerFS   0       0

Please consider visiting https://www.teknophiles.com/2018/02/19/disk-pooling-in-linux-with-mergerfs/ for a complete introduction.

Shaarli Bookmark – missing icons.

In a few versions of Shaarli there appears to be a bug which stops some of the icons appearing for your sites on the picturewall. I believe this has been fixed in later versions, but this simple workaround solved the problem for me.

From you shaarli directory, simply edit the css file

root@ubuntu:/var/www/shaarli# vi ./assets/default/scss/shaarli.scss

and add the following attributes to the b-lazy class.

.b-lazy {
         min-width: 1px;
         min-height: 1px;
        }

Some of my sites still do not get an icon, but I think that is a different problem. Anyway, worked for me.

Ubuntu Server + Xubuntu-core

Simple HowTo for adding a graphical display to a base Ubuntu Server 20.04 system.

user@ubuntu:~$ sudo apt update && apt upgrade -y
user@ubuntu:~$ sudo apt install lightdm tasksel
user@ubuntu:~$ sudo tasksel install xubuntu-core
user@ubuntu:~$ reboot

When the system has rebooted you will be able to log into the xubuntu desktop. The -core indicates just the core desktop environment rather than the numerous recommended / associated apps.

Other desktops are just as easy, want mate, then its ubuntu-mate-core. Lubuntu is lubuntu-core. You get the idea.

Ubuntu Samba Install

Following on from my efforts building a dedicated data recovery box, I decided to use Samba as an easy way of looking through the recovered data, in addition to the local client disks.

First of all, a quick update as always to check the latest packages in the repo:

root@moe:~# apt update && apt upgrade -y

Next the samba install:

root@moe:~# apt install samba -y

The samba setup required is very simple, I want one account with write access, then a guest account with read access for everyone else. Make sure you understand the implications of this insecure configuration before blindly following it. My folder structure is very basic, with everything from the /media directory being visible.

Rather than wade through the sea of options in the default config file, I simply backed it up and started from a blank slate.

root@moe:~# mv /etc/samba/smb.conf /etc/samba/smb.conf.original
root@moe:~# vi /etc/samba/smb.conf

I then used the following config in the new smb.conf file

======================= Global Settings =======================
[global]
workgroup = WORKGROUP
dns proxy = no
load printers = no
printcap name = /dev/null
disable spoolss = yes

#### Networking ####
interfaces = 127.0.0.0/8
bind interfaces only = yes

#### Debugging/Accounting ####
log file = /var/log/samba/log.%m
max log size = 1000

####### Authentication #######
server role = standalone server
passdb backend = tdbsam
obey pam restrictions = yes
unix password sync = yes
passwd program = /usr/bin/passwd %u
passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
pam password change = yes
map to guest = bad user

############ Misc ############
usershare allow guests = yes

#======================= Share Definitions =======================
[media]
comment = Media Share on Moe
path = /media
valid users = "jon"
write list = "jon"
guest ok = no
browseable = no


[store]
comment = Data Store on Moe
path = /media/store/
read only = yes
guest ok = yes

Once you have saved the file, use the testparm command to check for configuration errors. Then simply restart as below:

root@moe:~# service smbd restart
root@moe:~# service smbd status
● smbd.service - Samba SMB Daemon
   Loaded: loaded (/lib/systemd/system/smbd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-05-07 11:04:22 UTC; 4s ago
     Docs: man:smbd(8)
           man:samba(7)
           man:smb.conf(5)
 Main PID: 28444 (smbd)
   Status: "smbd: ready to serve connections..."
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/smbd.service
           ├─28444 /usr/sbin/smbd --foreground --no-process-group
           ├─28479 /usr/sbin/smbd --foreground --no-process-group
           └─28480 /usr/sbin/smbd --foreground --no-process-group


May 07 11:04:22 moe systemd[1]: Starting Samba SMB Daemon...
May 07 11:04:22 moe systemd[1]: Started Samba SMB Daemon.
May 07 11:04:22 moe smbd[28444]: [2020/05/07 11:04:22.166574,  0] ../lib/util/become_daemon.c:124(daemon_ready)
May 07 11:04:22 moe smbd[28444]:   STATUS=daemon 'smbd' finished starting up and ready to serve connections
root@moe:~# 
root@moe:~# 

I can’t stress enough that this is far from a secure or recommended setup. However in my single use case it is fine, I’m only working on data believed to be lost, and this isn’t my livelyhood.

Disk Recovery Ubuntu Box

The Background

After years of having multiple floppies, SCSI, IDE, SATA, USB drives and sticks kicking around the office I’ve decided it can’t carry on like this. I’ve fallen into the habit of buying the biggest USB disk I could afford to shuffle data back and forth between new and old machines as I nuke and pave my way through life.

I’ve also had numerous requests from family and friends over the years to rebuild machines or recover data for them which has just added to my problem. I always like to have two copies of the data when doing anything like this for security against mistakes (of which there have been a few).

The end result is more disks and sticks kicking around with no order or structure. Every time I’ve started to have a sort out in the past, I’ve always ran out of space or needed the device I was loading disks in for real work.

Whilst having a sort out I have come across an old HP xw4600 workstation which has a ton of space inside, 4 SATA ports and even IDE It isn’t the most powerful on the processor front, but it will be perfect for a box I can use as a dedicated platform for this task.

So after digging through the old disks and finding an empty 120G SSD from some unknown source, I connected it up to SATA0 and started installing Ubuntu Server. Even though I’m planning on doing most of the work on the command line via SSH, I decided to install LightDM and LXDE just incase I needed some GUI tools later.

20 minutes later, I have a box which I can wake up remotely from a so called magic packet. In less than 25 seconds later, I can log in locally through the GUI or remotely via SSH, ideal. Now for some more interesting bits. Obviously as with all devices these days some precautions on the security front should be taken to protect against the evils on the Internet.

The Customisation Journey

I say Journey, as I never seem to get to a fully configured server, but once the basic ubuntu server build is there, I’ll add some tools etc I require. First off I’ll add the usb auto mount functionality as used in the ubuntu desktop variants along with support for NTFS as I know there will be some old MS Windows disks to go through. Finally here I’ll get the testdisk utilities installed which also provides PhotoRec.

jon@moe:~$ sudo apt update && sudo apt install usbmount ntfs-3g testdisk

Now when I plug a USB Device (disk or stick) it gets auto mounted under /dev/usb? somewhere. This just safes me the bother of doing it manually.

I decided to add a GUI as some of the tools I may use in the future may require it. I went for a simple default lxde core.

jon@moe:~$ sudo apt update && sudo apt install lightdm tasksel -y
<snip>
jon@moe:~$ sudo tasksel install Lubuntu-core
<snip>

Data Recovery Example

A grand total of 45 minutes since starting I’ve got a platform built, now with two extra drives connected. A blank data disk, and a drive which was inadvertently formatted that I need to recover data from. Some 3 hours later, and PhotoRec has already recovered 800+ jpg files.

PhotoRec 7.0, Data Recovery Utility, April 2015
Christophe GRENIER <grenier@cgsecurity.org>
http://www.cgsecurity.org

Disk /dev/sdb - 4000 GB / 3726 GiB (RO) - WDC WD40PURZ-85TUZV0
     Partition                  Start        End    Size in sectors
     No partition             0   0  1 486401  80 63 7814037168 [Whole disk]

Pass 1 - Reading sector  439340400/7814037168, 843 files found
Elapsed time 3h47m26s - Estimated time to completion 63h37m39
jpg: 843 recovered

It is now well past half way, according to the disk geometry, but I will let it run through to its conclusion. Just for the record, it didn’t take the estimated 63 hours, it completed overnight.

Grub Default Last Session

Grub has been around for years, sitting in the boot process allowing us to choose which OS to boot from. One option which I find particularly useful is the ability to remember the last selection and default to that selection. If like me you have ever needed windows in a hurry to realise its half way through installing updates, but never finished because it rebooted into Linux this may be of help.

Edit the /etc/default/grub file and add the following text:

GRUB_DEFAULT=saved
GRUB_SAVEDEFAULT=true

Once you have added those lines, simply issue the command below to update your Grub settings:

sudo update-grub

Simply reboot, and from now on, it will remember your last selection and default to it accordingly. Of course, you still need to boot into windows to start the update process, but at least now you don’t have to watch over it whilst it reboots to install them. 🙂

Disabling Lets-encrypt TLSv1.0 & TLSv1.1 on Nginx & Apache

Following some maintenance work which included the movement of some web sites around between various hosts, I visited https://www.ssllabs.com/ to sanity check a few things. I was surprised to find that all my sites were rated as B because the still supported TLSv1.0 & TLSv1.1.

TLS is a more recent or continuation of SSL, TLS (Transport Layer Security) and SSL (Secure Socket Layers) are both cryptographic protocols. These protocols are used to authenticate and encrypt data on the Internet.

I’m no professional web guy, but knowing that SSL1,SSL2,SSL3 along with TLSv1 and TLSv1.1 all had various vulnerabilities. I took care and disabled them on each of the Virtual Hosts as I moved them. I was surprised that they appeared in the ssllab report, a short while later, with a bit of looking through Apache & Nginx config files I found it was the included Lets-encrypt config file.

In order to get an A rating on the https://www.ssllabs.com/ checker I needed to disable them. Fortunately this was simple enough once I knew where to look. Then it’s simple to amend the appropriate file followed by a restart of the appropriate service.

For Apache2:

root@Apache2-Host: vi /etc/letdencrypt/options-ssl-apache.conf

Edit the file, find the relevant line, comment it out, and make the changes below:

#SSLProtocol             all -SSLv2 -SSLv3
 SSLProtocol             -ALL +TLSv1.2

Followed by a simple restart:

root@Apache2-Host: systemctl restart apache2

For Nginx:

root@NginX-Host: vi /etc/letsencrypt/options-ssl-nginx.conf

Similar to Apache2, find the relevant line, comment it out, and make the changes below:

#ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_protocols TLSv1.2;

Then just restart the Nginx server:

root@NginX-Host: systemctl restart nginx

Heading back to https://www.ssllabs.com/ and restarting to check gave me the positive result:

I did find it slightly ironic, that it was in a Lets Encrypt config file that these legacy protocols were enabled. I am full of admiration for https://letsencrypt.org I think they have done the world of good to increase security of the internet.

Working Ubuntu Linux & WPA2e (enterprise) Configuration

Following on from my recent entry WPA2 Enterprise on a Raspberry PI I was asked if I could offer some guidance for a couple of linux distributions, in particular Ubuntu Mate and Elementary OS. The instructions here may be applicable to other Linux distributions, but I’ve only confirmed them on Ubuntu Mate 18.04 and Elementary Junos.

So starting with ubuntu Mate, select the WPA2 enterprise protected network in the normal manner from the desktop. Forgive the images, I’ve hidden some of the network names.

Once you have clicked on the relevant WPA2e protected network, you will see a dialogue box as below. Choose all the options in WiFi security, and others as shown below. Before going on to choose the option “select from file” under the CA certificate option. If you select the CA cert to early other options may become inaccessible. We will be returned to this shortly so if an option isn’t available now, it will be soon.

Once the dialogue box below pops open, navigate to the directory /etc/ssl/certs go and choose the ca-certificates.crt as shown below.

Once you have selected the ca-certificates.crt file, and have returned to the earlier dialogue box, make sure PEAP Version is set to Automatic and the inner authentication is MSCHAPv2 if they are not already set.

Finally make sure you put your username and password in the relevant spaces. This screenshot was taken where Active Directory usernames were used hence the fields being populated with appropriate ad prefixed strings.

Finally click the NO CA Certificate is required. If you click this radio button early, it prevents you entering your username and password.

Finally click connect and it will attempt to connect to your chosen WPA2 enterprise protected network.

In Elementary OS Junos, it doesn’t let you click on the WPA2e protected network, in which case you will need to click on network settings.

Once in the Network Settings dialogue box, you can then select the WPA2 enterprise protected network.

After selecting your network, you should be able to mimic the Ubuntu-Mate instructions above.

Working Raspberry Pi & WPA2e (enterprise) Configuration.

I’ve seen numerous posts with many different approaches to getting the working. I’ve had to do this recently so did some testing along what way and made some notes. I had the opportunity to test on a number of Pi’s so should it should work with any WPA2e network regardless of the commercial entity or University. One final point here, just because I’ve detailed how to connect, you may still need permission from the appropriate Network Security body before connecting your device to WPA2e protected networks.

The instructions were tested on both the Raspberry Pi 3b and Raspberry Pi 4, running any of the images below:

  • Raspbian Buster with desktop and recommended software(September 2019)
  • Raspbian Buster with desktop (September 2019)
  • Raspbian Buster Lite (September 2019)

I didn’t do any updates to the base image, just so I knew I had a common starting point. I don’t expect any issues if I had done the updates, and did try at the time with no issues. However over time there will obviously be many updates that were not included in my testing.

So from a fresh boot using a new image, after the normal re-sizing of the root partition etc, the first thing we need to determine is a hashed version of the password for the WPA2 enterprise network. We could use clear text instead, but given we are going to be storing it in a config file in /etc a hashed password is obviously best practice. The following command (using the correct password) will give us the cached password.

Through the snippets below, always be aware the it may appear word wrapped on you screen.

pi@raspberry:~ $ echo -n 'WiFi-Password' | iconv -t utf16le | openssl md4 | cut -d " " -f2
01c5a3f0c2cad4e614d5e3c3d92906f6

It’s the string 01c5a3f0c2cad4e614d5e3c3d92906f6 we need later so keep it safe.

Next we create a new file:

pi@raspberry:~ $ vi /etc/network/interfaces.d/wpa2enterprise

With the following text:

auto wlan0

iface wlan0 inet dhcp
  pre-up wpa_supplicant -B -Dwext -i wlan0  -c/etc/wpa_supplicant/wpa_supplicant.conf
  post-down killall -q wpa_supplicant

Next we create the new referenced `/etc/wpa_supplicant/wpa_supplicant.conf` with the following details:

  • Wi-Fi Network name (The SSID to connect to)
  • Your Username to connect to the network with.
  • The Hashed Password (generated earlier)
pi@raspberry:~ $ vi /etc/wpa_supplicant/wpa_supplicant.conf

With the following content, updating the details as appropriate (marked with —).

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
 ssid="---WIFI-NAME---"
 proto=RSN
 key_mgmt=WPA-EAP
 auth_alg=OPEN
 eap=PEAP
 identity="---USERNAME---"
 password=hash:---PASSWORD-HASHED-VALUE---
 phase1="peaplabel=0"
 phase2="auth=MSCHAPV2"
 priority=1
}

Once you have created the files as specified above, with the appropriate details updated. Reboot the Raspberry Pi and it should auto connect to the WPA 2 enterprise (WPA2e) network.

Finally, when you password is changed, you just need to generate a new hash and update the /etc/wpa_supplicant/wpa_supplicant.conf file as appropriate.

SMS Texting using 3G dongle

I initially did this on a Raspberry Pi, but have since moved to various other Linux flavours and platforms. The instructions below can still be used if the similar issue is experienced.

The problem I’m having to solve is that the 3G Dongle isn’t being recognised as a serial port, it is being recognised as a storage device. Even though the description obtained from lsusb clearly identifies it as a modem.

  root@rasp-storage:~# lsusb
  Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp.
  Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
  Bus 001 Device 004: ID 1a40:0101 Terminus Technology Inc. 4-Port HUB
  Bus 001 Device 005: ID 1a40:0101 Terminus Technology Inc. 4-Port HUB
  Bus 001 Device 006: ID 12d1:1446 Huawei Technologies Co., Ltd.E1552/E1800/E173
  (HSPA modem)

Anyway, before we get started ensure the system is up to date. I’m using the latest Raspian so its a simple case of: 

  apt-get update 
  apt-get upgrade

The tool we are going to be using is called usb-modeswitch.

  root@rasp-storage:~# apt-cache search modeswitch 
  usb-modeswitch - mode switching tool for controlling "flip flop" USB devices 
  usb-modeswitch-data - mode switching data for usb-modeswitch 
  root@rasp-storage:~# apt-get install usb-modeswitch usb-modeswitch-data

I found the information for the usb_modeswitch elements by googling the “Huawei E173 modem”, the details below are unlikely to work on other 3G modems. Create a file named 12d1:1446 under /etc/usb_modeswitch.d/ which contains:

DefaultVendor=0x12d1
DefaultProduct=0x1446
  MessageContent="55534243123456780000000000000011062000000100000000000000000000" 

Create a file /usr/local/sbin/3gmodemswitch containing.

  usb_modeswitch -I -W -c /etc/usb_modeswitch.d/12d1\:1446 

Create a file /etc/udev/rules.d/99-3gmodem.rules which contains:  

ACTION=="add", ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="1446", RUN+="/usr/local/sbin/3gmodemswitch"

So when the modem is plugged in, udev rules pick it up and execute the 3gmodemswitch script, which in turn calls usb_modeswitch. If we have a look at the output from lsusb now, we see:

  lsusb Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp. 
  Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub 
  Bus 001 Device 009: ID 12d1:1506 Huawei Technologies Co., Ltd. E398 LTE/UMTS/GSM Modem/Networkcard 
  Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. 
  Bus 001 Device 004: ID 0781:5571 SanDisk Corp. 
  Bus 001 Device 005: ID 1a40:0101 Terminus Technology Inc. 4-Port HUB 
  Bus 001 Device 006: ID 1a40:0101 Terminus Technology Inc. 4-Port HUB 
  root@rasp-sms:/etc/usb_modeswitch.d# 

I was able to reboot, and do a complete power down restart, and the modem continued to be recognized as a 3G modem. It should be noted I have in the past experienced an issue where the root filesystem was not on the SD card, but a USB drive. However when starting the Pi in that configuration sometimes the built in storage of the 3G modem was identified as SDA rather than the USB thumb drive. To fix this situation I made sure to use the root=PARTUUID=XXXXX notation in the /boot/cmdline.txt file. Finally, install smstools using:

  apt-get install smstools 

Following which edit the /etc/smsd.conf file and append the following to the bottom (using the correct ttyUSB device:- 

  [GSM1]
  #init = 
  device = /dev/ttyUSB0 
  incoming = yes 
  #pin = 
  baudrate = 19200

Now when I drop a file into the /var/spool/sms/outgoing/ directory in the format:

  To: 444797575B134

Hello From My Raspberry Pi. :)


Incidentally the number above is random, furthermore I put a letter B in the text on purpose just to save some poor soul getting unsolicited texts by anyone reading this and simply copying the text.

So with a little more scripting, I now use this Raspberry Pi as a notification server, it polls other devices (predominately network equipment) and services and text’s me if there are problems. I can also sent it texts to perform simple actions, but that element is still very much a work in progress.