Anthony J. Martinez

Librem 14 + USB-C PD Hub in Qubes OS

A little more than a week ago, I picked up a few USB-C items to try out with my Librem 14 laptop...

Librem 14 displaying 4K video over a VAVA VA-UC020 Hub

The BatPower P120B works great for powering both my Librem 14, and my work laptop. The two USB-A power ports are great as I've now freed up space on my power strip that was previously occupied by a dual-port USB-A charger. It has also lightened my load on the road, as I no longer require a laptop charger and USB chargers for other devices.

All of the features of the VAVA UC020 are working in Qubes OS. Both my USB keyboard and mouse are plugged into my 4K monitor, which is connected to one of the VAVA's USB-A ports and its HDMI port. Early in the boot process, the panel came alive and has stayed that way ever since. As soon as I plugged in my wired headset, the audio device was available for assignment.

Switching between my work laptop and the Librem 14 is now as easy as swapping one cable!

SmartCards and Fedora

Attempting to use my second Librem Key with Fedora presented some challenges in dealing with pcscd. The root cause is that polkit does not allow normal users access to pcsc or the smartcard itself. This can be resolved with a single rule:

In /etc/polkit-1/rules.d/42-pcsc.rules:

polkit.addRule(
  function(action, subject) {
    if ((action.id == "org.debian.pcsc-lite.access_pcsc" ||
        action.id == "org.debian.pcsc-lite.access_card") &&
        subject.isInGroup("wheel")) {
          return polkit.Result.YES;
        }
});

For the subject.isInGroup condition, I used the group wheel as I am the only member of that group on the system in question. Use your own descretion here, or use an even more specific condition to allow only one user like subject.user == "foo".

Additional Points

While this does allow access through pkcs11 and pkcs15 tools or gpg, I have not yet found the magic potion that will allow me to use both. Whichever tools are used first have a monopoly on the device. That said, on a modern Linux distro just using pkcs11 ought to do the trick.

Encryption

Use -engine pkcs11 with openssl subcommands that support it:

openssl rsautl -engine pkcs11 -keyform e -inkey <KEY_ID> -encrypt -in <INPUT> -out <OUTPUT>

SSH

Use "pkcs11:id=%<KEY_ID>?pin-value=<PIN>" as the identity file argument for ssh either on the command line, or in an ssh_config file. You will likely wish to get the PIN value itself from somewhere so it's not just in plaintext in your history:

ssh -i "pkcs11:id=%03?pin-value=123456" user@host

Or in an ssh_config file:

Host host
  IdentityFile "pkcs11:id=%03?pin-value=123456"
  User user

Adding SSH Agent Support to Split GPG

Split GPG is a very cool feature of Qubes OS but it leaves out one critical feature: enabling SSH support so the GPG backend qube can make use of an authentication subkey. There are a few different ways to solve this, and this guide provided some of the inspiration for what follows.

The Landscape

Here are the requirements for what follows:

Qubes RPC Policy

The first step is to configure an appropriate Qubes RPC Policy. A basic, and generally sane option, is to use a default configuration that asks the user to approve all requests and allows any qube to target any other qube with such a request. In my own configuration there are explicit allow rules for specific qubes where I use SSH frequently for admin purposes.

In dom0 create /etc/qubes-rpc/policy/qubes.SshAgent:

admin personal-gpg allow
@anyvm @anyvm ask

Actions in the Split GPG VM

The following actions all take place in the qube configured to act as the GPG backend for a Split GPG configuration.

Enable SSH support for gpg-agent:

$ echo "enable-ssh-support" >> /home/user/.gnupg/gpg-agent.conf

Update .bash_profile to use the gpg-agent socket as SSH_AUTH_SOCK by appending:

unset SSH_AUTH_SOCK
if [ "${gnupg_SSH_AUTH_SOCK_by:-0}" -ne $$ ]; then
	export SSH_AUTH_SOCK="$(gpgconf --list-dirs agent-ssh-socket)"
fi
export GPG_TTY=$(tty)
gpg-connect-agent updatestartuptty /bye >/dev/null

Create /rw/config/qubes.SshAgent with the following content, and make it executable:

#!/bin/sh
# Qubes Split SSH Script

# Notification for requests
notify-send "[`qubesdb-read /name`] SSH Agent access from: $QREXEC_REMOTE_DOMAIN"

# SSH connection
socat - UNIX-CONNECT:$SSH_AUTH_SOCK

Update /rw/config/rc.local appending the following:

ln -s /rw/config/qubes.SshAgent /etc/qubes-rpc/qubes.SshAgent

Sourcing .bash_profile and /rw/config/rc.local should put the qube in a state where, if available, a GPG authetication subkey will be available to ssh-agent:

Example from my system:

[user@personal-gpg ~]$ ssh-add -l
4096 SHA256:V2KMVlJjPOn86z6a2srEcnMQj78OujEXJ597PJ6+wyY (none) (RSA)

Template VM Modifications

For my tastes it made the most sense to make a systemd service available to all qubes using my f33-dev template, and then start that service from /rw/config/rc.local on qubes I want to use the new feature.

In the approprite Template VM create a service similar to the following, but replace personal-gpg with the name of your Split GPG backend qube.

/etc/systemd/system/split-ssh.service:

[Unit]
Description=Qubes Split SSH
StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
Type=simple
User=user
Group=user
Restart=on-failure
RestartSec=5s
WorkingDirectory=/home/user
Environment="AGENT_SOCK=/run/user/1000/SSHAgent" "AGENT_VM=personal-gpg"
ExecStart=socat "UNIX-LISTEN:${AGENT_SOCK},fork" "EXEC:qrexec-client-vm ${AGENT_VM} qubes.SshAgent"

[Install]
WantedBy=multi-user.target

Once this has been added run the following, and shut the template qube down:

sudo systemctl daemon-reload

The Client Side

In the actual SSH client qubes, there are a few actions required to complete the loop.

Append the following to .bashrc - make sure this matches the AGENT_SOCK in your systemd service:

### Split SSH Config
export SSH_AUTH_SOCK="/run/user/1000/SSHAgent"

In /rw/config/rc.local append the following to start the service:

systemctl start split-ssh

Source .bashrc and /rw/config/rc.local and with the split GPG backend qube running test that your key is available:

[user@admin ~]$ ssh-add -l
4096 SHA256:V2KMVlJjPOn86z6a2srEcnMQj78OujEXJ597PJ6+wyY (none) (RSA)

Since my Qubes RPC policy allows the admin qubes to reach personal-gpg without my confirmation, a system notification appears stating:

[personal-gpg] SSH Agent access from: admin

Conclusion

With a few simple steps the power of Split GPG can be extended to include SSH Agent support. As a result, network-attached qubes used for administration of remote assets no longer directly store the private key material used for authentication and the attack surface is that much smaller. There are a few ways to get the pubkey to add to remote ~/.ssh/authorized_keys but the easiest way is probably ssh-add -L.

Librem 14, Librem Keys, and Qubes OS

With the arrival of my second Librem Key, I thought now would be a good time to go over how I use Qubes OS features along with some more products from Purism for various signing, encryption, and authentication tasks.

The Landscape

Here are the various components at play:

The base of all but one of my qubes is Fedora.

Security Device

Getting Started

The new Librem Key needs to have its PIN set, and since my Qubes OS configuration uses a USB qube it will be necessary to give my running disposable VM access to the key itself:

In dom0, where my target vm is disp4632 and my BACKEND:DEVID is sys-usb:2-1:

$ qvm-usb attach disp4632 sys-usb:2-1

In the disposable VM run:

[user@disp4632 ~]$ gpg --card-status
Reader ...........: Purism, SPC Librem Key (000000000000000000009BB1) 00 00
Application ID ...: D276000124010303000500009BB10000
Application type .: OpenPGP
Version ..........: 3.3
Manufacturer .....: ZeitControl
Serial number ....: 00009BB1
Name of cardholder: [not set]
Language prefs ...: de
Salutation .......: 
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 64 64 64
PIN retry counter : 3 0 3
Signature counter : 0
KDF setting ......: off
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

Change the PINs

  1. gpg --card-edit in the disposable VM
  2. admin at the gpg/card> prompt
  3. passwd at the gpg/card> prompt
  4. Select 1 and follow the prompts, where the first PIN is the default: 123456
  5. Select 3 and follow the prompts, where the first Admin PIN is the default: 12345678
  6. Select q and quit.

Initialize the New Librem Key

Remove the new Librem Key

In dom0 run:

qvm-usb detach disp4632 sys-usb:2-1
Insert the original Librem Key

In dom0 run:

# Assuming you plugged the original key into the same port
qvm-usb attach disp4632 sys-usb:2-1
Insert and mount the Librem Vault

In dom0 find the appropriate block device, and attach it to the disposable VM:

qvm-block list
...

qvm-block attach disp4632 sys-usb:sdb1

In the disposable VM find the attached disk (likely /dev/xvdi)

[user@disp4632 ~]$ lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
--- SNIP ---
xvdi    202:128  1 28.9G  0 disk

Then mount the disk:

[user@disp4632 ~]$ udisksctl mount -b /dev/xvdi
Mounted /dev/xvdi at /mnt/removable

Note that I did not sudo mount /dev/xvdi /mnt/removable as the operation does not require root, and we do not use powers we do not need do we?!

Extract the encrypted backup from the Librem Vault
[user@disp4632 ~]$ cp /mnt/removable/gpg-backup/backup* .
Unmount and Remove the Librem Vault
[user@disp4632 ~]$ udisksctl unmount -b /dev/xvdi
Unmounted /dev/xvdi.

In dom0:

qvm-block detach disp4632 sys-usb:sdb1
Decrypt the backup

This assumes you have installed opensc and have pkcs15-tool and pkcs11 drivers.

First, find the Key ID for encryption key on your existing Librem Key:

[user@disp4632 ~]$ pkcs15-tool -D
Using reader with a card: Purism, SPC Librem Key (000000000000000000009BB4) 00 00
PKCS#15 Card [OpenPGP card]:
        Version        : 0
        Serial number  : 000500009bb4
        Manufacturer ID: ZeitControl
        Language       : de
        Flags          : PRN generation, EID compliant

// SNIP

Private RSA Key [Encryption key]
        Object Flags   : [0x03], private, modifiable
        Usage          : [0x22], decrypt, unwrap
        Access Flags   : [0x1D], sensitive, alwaysSensitive, neverExtract, local
        Algo_refs      : 0
        ModLength      : 4096
        Key ref        : 1 (0x01)
        Native         : yes
        Auth ID        : 02
        ID             : 02 <-- THIS ID
        MD:guid        : ee23dccc-fc38-2dc2-3bc8-bb5f859168d4

// SNIP

Now use it to decrypt the pbkdf2 key used to encrypt the GPG backup tarball itself. This hybrid encryption scheme allows securely storing data of arbitrary sizes and using pbkdf2 with randomly generated secrets and then encrypting those secrets with the Librem Key's encryption key.

Decrypting the pbkdf2 password file with the Librem Key:

[user@disp4632 ~]$ openssl rsautl -engine pkcs11 -keyform e -decrypt -inkey 02 -in backup.key.enc -out backup.key
engine "pkcs11" set.
Enter PKCS#11 token PIN for OpenPGP card (User PIN):

Decrypting the GPG backup with the pbkdf2 password file:

[user@disp4632 ~]$ openssl enc -chacha20 -pbkdf2 -pass file:backup.key -d -in backup.tar.gz.enc -out backup.tar.gz
Extract the backup
tar xf backup.tar.gz
Verify the keyring is in tact
[user@disp4632 ~]$ gpg -k
/home/user/.gnupg/pubring.kbx
-----------------------------
pub   rsa4096 2021-05-08 [C]
      FCBF31FDB34C8555027AD1AF0AD2E8529F5D85E1
uid           [ultimate] Anthony J. Martinez <@ajmartinez:txrx.staart.one>
sub   rsa4096 2021-05-08 [S]
sub   rsa4096 2021-05-08 [E]
sub   rsa4096 2021-05-08 [A]

Remove the original Librem Key

In dom0:

qvm-usb detach disp4632 sys-usb:2-1
Insert the new Librem Key again

In dom0:

qvm-usb attach disp4632 sys-usb:2-1
Export the signing, encryption, and authentication subkeys to the Librem Key

Edit the key in expert mode:

[user@disp4632 ~]$ gpg --expert --edit-key FCBF31FDB34C8555027AD1AF0AD2E8529F5D85E1

In the gpg> prompt select each subkey and use the keytocard command.

Example, using the signing key (key 1):

gpg> key 1

sec  rsa4096/0AD2E8529F5D85E1
     created: 2021-05-08  expires: never       usage: C   
     trust: ultimate      validity: ultimate
ssb* rsa4096/A2206FDD769DBCFC <-- NOTICE THE * HERE - this key is selected
     created: 2021-05-08  expires: never       usage: S   
ssb  rsa4096/6BE6910237B3B233
     created: 2021-05-08  expires: never       usage: E   
ssb  rsa4096/FD94BDD7BED5E262
     created: 2021-05-08  expires: never       usage: A   
[ultimate] (1). Anthony J. Martinez <anthony@ajmartinez.com>
[ultimate] (2)  Anthony J. Martinez <@ajmartinez:txrx.staart.one>

gpg> keytocard
gpg> key 1 <-- this is to deselect key 1

Repeat the above for key 2 and 3.

Verify the card status
[user@disp4632 ~]$ gpg --card-edit

Reader ...........: Purism, SPC Librem Key (000000000000000000009BB1) 00 00
Application ID ...: D276000124010303000500009BB10000
Application type .: OpenPGP
Version ..........: 3.3
Manufacturer .....: ZeitControl
Serial number ....: 00009BB1
Name of cardholder: [not set]
Language prefs ...: de
Salutation .......: 
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa4096 rsa4096 rsa4096
Max. PIN lengths .: 64 64 64
PIN retry counter : 3 0 3
Signature counter : 0
KDF setting ......: off
Signature key ....: C9ED 41D4 EB62 80BB E61F  0E59 A220 6FDD 769D BCFC
      created ....: 2021-05-08 11:43:52
Encryption key....: 335D C8BC E4A6 8FFF B9B5  CBEF 6BE6 9102 37B3 B233
      created ....: 2021-05-08 11:44:54
Authentication key: D157 68B9 CCCF 4FB5 6FC2  971E FD94 BDD7 BED5 E262
      created ....: 2021-05-08 11:45:39

Cleanup

From here the new Librem Key is configured, and the disposable VM is of no further use. Since disposable VMs are destroyed when the application they were created to run is stopped, the only cleanup necessary is to close the terminal to the disposable VM.

Additional Notes

On my system, I also have vault and personal-gpg qubes. These are both network-isolated and function much the same way the physical key does. The personal-gpg qube holds the very same subkeys as both Librem Keys, and through the use of Split GPG allows for a smartcard-like use of the qube from my other qubes. In a later post, I will detail how I use QubesRPC in personal-gpg to also serve as my ssh-agent for using the authentication subkey in things like my admin qube to prevent me from needing dozens of copies of my SSH private keys everywhere. The vault qube is home to the master secret key, and as such never has any data fed in to it.

The process used to decrypt data can be reversed to encrypt data as well. I will leave that as an exercise for the reader, but the short version is that instead of the decrypt option(s) for the openssl tools use their encrypt counterparts. If you wish to generate a random secret to use with pbkdf2 the following should do the trick:

openssl rand -base64 -out secret.key 32

Another week with the Librem 14

Another week has passed, and I am liking the Librem 14 quite a lot overall. Having now mostly pounded the keyboard into submission it is much more tolerable. The A and S keys are the most disobedient of the bunch, and the oddly located R Shift results in some random profanity when I end up a line higher in my menu or code when that is not at all what I wanted.

Most of what I have worked on was simplifying my home network. Thanks to the presence of a physical LAN port, and some of the finer points of Qubes OS NetVMs it was easy to setup multiple VMs each assigned my wired interface. This allowed me to verify tagged and untagged VLAN settings on a switch the configuraton of which I forgot long ago. All of this took place on battery with me sitting on the floor of my closet without much fear that I would soon need to race across the house to get my charger.

For much of the last week, I also tested out using awesomewm with Qubes OS. Basic tiling was fine, and helped me handle the lower 1080p resultion offered by the Librem 14. In the end, I went back to XFCE with some tiling enabled by keyboard shortcuts.

No buyer regret here. This machine does everything I need it to do and it does it much better than the machine it replaced. When I return to The Netherlands that machine will change roles and continue life, but my main machine will definitely be this Librem 14 for several years to come.

Librem 14 Battery Life While Running Qubes OS

Here is a first look at the kind of battery life one might expect while using a Purism Librem 14 as shipped with the 4-cell battery while running a normal workload in Qubes OS.

TL;DR - The Chart

Condtions

Given a steamy day in Texas, one Librem 14 running Qubes OS, and a few general tasks to accomplish I set out to find out how long I might expect to spend away from a wall outlet if I:

Process

Once the system was running, I started a loop to give me battery stats every 60s:

while true; do
    upower -i /org/freedesktop/UPower/devices/battery_BAT0 > $(date +%s)_bat-info.txt
    sleep 60
done

From here, I just went about my business. TemplateVMs on my system are primarily based on Fedora. As a result very nearly every boot means there are updates available. Updating was probably the most strenuous task executed during the test run. In fact, I do not recall the fans turning on for anything else throughout the day except for maybe the one time I started one of my more substantial VMs I use for development. For the most part I had at least (7) VMs running, one of which maintained a WireGuard VPN connection to my cloud environment. General tasks amounted to:

Conclusion

This machine lasts at least as long with light use as my previous system. No one runs towards Qubes OS with the hopes of marathon battery life, and I am pretty happy with near 5hrs of battery runtime on WiFi with a persistent VPN to my cloud resources. Next time, I may shut off WiFi and see how much purely local heavy use I can squeeze out before the battery dies.

First Impressions - Librem 14

For my birthday last year, I ordered the Purism Librem 14 to serve in place of my aging Lenovo T460s. Slightly less than a year later I got my new laptop, and a few weeks after delivery I was able to fly home on vacation to finally get started using it. To most of my friends, waiting a year for a laptop to be delivered is utter madness, but for me there are almost no new laptops I am even willing to consider. Having a physical RJ45 jack is a hard requirement for me, and today it seems this typically requires a willingness to carry a "laptop" that weighs more than a healthy newborn human. Give me power, give robust networking options, and give me the RAM I need to run Qubes OS to its fullest. The Librem 14 offers this all, on paper, and over the coming month I will find out exactly how that all pans out in reality.

First Boot

My machine was ordered with the following specs:

Since there are not any good defaults for installing Qubes OS on an OEM device one needs to install it on their own. Knowing this I performed my first boot into Pure OS using encryption passphrases and user passwords no one should ever use on system they plan to actually use. The process for using PureBoot was pretty clear and straight forward, and the Librem Key that shipped with the Librem 14 flashed green a expected when I made my first boot. It also flashed red, as expected, when I booted the second time having updated the kernel.

My time in Pure OS was limited to two tasks:

  1. Generating new GnuPG keys to store on the Librem Key
  2. Making a USB boot drive from the latest Qubes OS release

The Next 20 Boots

My initial installation of Qube was mising one critical point: an encrypted root partition. Use of PureBoot requires an unencrypted /boot and that is fine given that I am notified of any changes and can opt to sign them if they were expected after an update. Lacking encryption of the rest of the disk is not an acceptable scenario for me, so I reinstalled. And reinstalled again. And again. And again.

Since I am writing this on vacation, I will have to come back to a determination of the root cause later. For now, just know that selecting the defaults in the Qube OS installer - which regrettably includes wasting 15GB of disk on swap I will never use - was the only partition scheme that resulted in an encrypted root partition. To be clear, I do not think this has anything to do with the Librem 14. When I do find the root cause, the appropriate project will get a detailed bug report. At any rate, once I was up and running the next task was restoring my qubes (VMs) from a backup taken on my T460s right before shutting it down last. This was done from a USB3 SSD, and the speed was outstanding. The restore completed much faster than the backup was made, likely owing to both faster USB controllers and the gap between Intel's i5-6300U and i7-10710U.

Normal Use

So far, I have not done much more than restore my qubes and fix a few remote issues I caused late last week that made SELinux angry. Everything is running smoothly, and the few times I have run a CPU intensive task like Rust compilation I have been very pleased with the performance of the machine. The only thing I do not like is the keyboard. I am far from the only person who regards older Thinkpad keyboards as being top of class, and coming from such a keyboard to this does not please me much. While typing this, my backspace key (and vim motions) have been flexed. My a, s, b, and l seem to particularly hate their existence. For some, this could be grounds for a return, but I have had other machines in the past with keyboards I found infuriating at first. Usually, the force of my typing tends to smooth things out in short order. If that is the case here all will be will. If not, it will probably also be fine since most of my use is on a desk plugged in to a Drop ALT.

Qubes and extended battery life are rarely thoughts one has concurrently, and I am pleased to note that the Librem 14 managed more than 4 hours on WiFi doing a mixed load of tasks. These included spawning at least 40 Disposable VMs, upgrading all of my Template VMs, compiling two different versions of the engine rendering and serving this page several times, and for reasons I still do not understand, playing the official music video for Enya's "Only Time" which I have not linked lest anyone else inexplicably end up with it stuck in their head.

So far so good. Tomorrow, I head for the mountains. When I return, I will put the Librem 14 through its paces as I work on some personal Rust projects.

For Next Time

I will try to: