This guide will cover: IPv6 prefix delegation; Creating VLAN interfaces in OPNsense; DHCP configuration; Router advertisements; Firewall isolation rules; Managed switch VLAN tagging; and Omada VLAN Wi-Fi configuration.

Table of contents

1. Introduction
   1. Hardware used
   2. Why you should put IoT devices on a separate VLAN
   3. Before: The original network layout
   4. Afterwards: The improved network layout
2. Get your ISP to provision you a /60 or /56 block of IPv6 addresses
   1. Requesting the change from the ISP
   2. Validating the prefix your ISP is issuing
3. OPNsense VLAN setup
   1. Create the IoT VLAN in OPNsense
   2. Assign the VLAN interface
   3. Configure DHCP for the VLAN
   4. Enable IPv6 router advertisements
   5. Create firewall rules
4. Configure VLANs on your managed switch
5. Configure VLANs in TP-Link Omada
6. Common problems/Troubleshooting
   1. Error: 'IPv6 prefix ID is out of range'
   2. VLAN Wi-Fi devices are not receiving IP addresses
   3. IoT devices cannot see each other

Introduction

I decided to tackle a project I had been putting off for a while: Building out a separate IoT VLAN for my home network.

Prior to this, I had been using Omada’s guest network for IoT devices in my home. But doing so caused me some headaches:

• It prevented the devices on that network from talking to each other.
• It was cluttered: Both visitor devices and IoT devices were part of the same WLAN.
• It kept me from having full visibility into the network’s firewall rules and hid activity for those devices on my network.

The Omada guest network feature is great, and I want to keep using it for when I have visitors over. I want guests to have access to our Wi-Fi, but not to our home network. And I don’t want anything running on their devices to get to see anything on our trusted network. No funny business!

Instead of using the Omada guest network for IoT devices, creating a dedicated VLAN provides better firewall control, device visibility, and network segmentation.

On a personal note, prior to this I really didn’t know much about IPv6. Frankly, I just disabled it more often than not, because I didn’t want to deal with it. It turns out that giving each device on your network a truly-global address unlocks a lot of simplicity and allows services like ZeroTier, Syncthing, and other direct-connection services to work quickly without dealing with weird legacy workarounds like IPv4 NAT.

Turns out setting this up helped me learn a bunch about IPv6, too. It was a long and arduous project but I’m glad I hunkered down and tackled it!

Hardware used

In my home network, I have the following services and devices:

• Xfinity/Comcast ISP (Residential cable)
• Motorola Nighthawk CM2000 cable modem
• OPNsense firewall/router running on an HP T730 thin client with an Intel I340-T4 NIC
• MikroTik CSS318-16G-2S+IN managed switch
• TP-Link Omada Wi-Fi APs (2x EAP610-v3, 1x EAP650-v1)
• And a bunch of devices all connected in via ethernet and over Wi-Fi

Why you should put IoT devices on a separate VLAN

I’m fairly security-minded and want to help protect the devices on my network. Smart-home devices can offer useful functionality, but I don’t believe should have full access to my trusted home LAN. How many times have we heard news stories about botnets and other things running amok in things like Smart Light Bulbs?

IoT devices should be isolated from your trusted home network because many of them:

• Receive infrequent security updates and/or run outdated firmware.
• Do naughty things, like scan your network and phone home all sorts of telemetry.
• Have glaring issues like using hardcoded admin credentials that can eventually be discovered and exploited.

Isolating IoT devices lets you use them for their convenience while preventing them from accessing your trusted network. That limits the damage blast radius in the event they’re compromised.

Before: The original network layout

Here’s the original network layout before I started.

• For simplicity’s sake, I’m only illustrating the Wi-Fi parts of the network. In reality, there’s a bunch of hard-wired devices connected up to the network.
• Network diagram icons courtesy of vrt-sheet-for-dia on GitHub, with a little of my own zhuzhing up.

As you can see, everything non-trusted was just all tossed onto the Omada guest WLAN. There wasn’t any way to segment the devices in any meaningful way… it was either an “Fully trusted” or “Guest only” setup.

Afterwards: The improved network layout

In the improved network layout:

• IoT devices are now on their own VLAN’d WLAN, isolated from the home network. They can now see each other.
• Physical network devices that are IoT related, or that I otherwise felt should be isolated (printers, etc.),are also tagged on the same VLAN to keep them isolated from the trusted network.

Get your ISP to provision you a /60 or /56 block of IPv6 addresses

As I started going through the process to enable VLANs in OPNsense, you’ll need to assign separate IPv6 address blocks for each VLAN. But this isn’t something you have full control over… these addresses aren’t something you can create yourself. They come from your ISP.

Your ISP probably delegates you a single /64 block of IPv6 addresses, because (a) it’s easy for them, and (b) most residential customers won’t need anything different as they’ll just have everything on one network anyway.

Your ISP may listen for and honor requests to issue different prefixes. Unfortunately for me, this wasn’t the case. No matter what I tried, I was only ever issued a single /64 network.

If you don’t have the /60 or /56 prefix delegation, you’ll get an error in OPNsense at the point where you enable the VLAN interface’s IPv6 Identity Association. You won’t be able to assign a Prefix ID and will get the error You specified an IPv6 prefix ID that is out of range.

Requesting the change from the ISP

Since this was only something my ISP could change, I had to work with them to request the change. This ended up taking the most time and being the most frustrating part of my journey. Working with Xfinity/Comcast’s Tier-1 support proved to be a challenge because they aren’t trained to help with IPv6 prefix delegation. Escalation paths with their support team are also tricky. I was told several times by support reps that, “The matter was being worked on and that I’d hear back from the team in 3-4 hours.” But after a day went by I had to follow up with support again, starting all over in the explanation each time. I had to repeat that cycle for several days before I got a ticket number for my request.

In total, it took about a dozen calls to Xfinity support over the course of one week to get this changed. Eventually, they did provision me a /60, and I was up and running!

Validating the prefix your ISP is issuing

To validate the IPv6 prefix you’re receiving, go to OPNsense and navigate to System > Log Files > General. Search for keyword prefix under log level Notice. Look at the most recent log entries and look at the last digits to see what network prefix you’re being issued. You want to see either /60 or /56.

dhcp6c_script: REQUEST on igb0 prefix now 2001:db8:1234:5670::/56    # Good
dhcp6c_script: REQUEST on igb0 prefix now 2001:db8:1234:5670::/60    # Good
dhcp6c_script: REQUEST on igb0 prefix now 2001:db8:1234:5670::/64    # Not good

A /56 is good, too! That means you’re being issued 256 different network groups. I requested a /60 to be more conservative, as the 16 different networks provided by the /60 should be more than sufficient for my needs. I also wanted to be conservative with my request to Xfinity in case they pushed back on the larger allocation.

OPNsense VLAN setup

This section walks through the complete process of configuring a VLAN in OPNsense. We’ll create the VLAN interface, assign IPv4 and IPv6 networks, enable DHCP, and configure router advertisements so devices can obtain IPv6 addresses.

The steps below assume you already have:

• A managed switch capable of VLAN tagging
• Wi-Fi access points that support VLANs
• An IPv6 prefix delegation from your ISP (see above)

Once your ISP is providing you a /56 or /60, you can continue actually setting up the VLANs on your network.

We’ll start by building that out in OPNsense, then roll out the VLAN elsewhere on the network.

Create the IoT VLAN in OPNsense

Within OPNsense, navigate to Interfaces > Devices > VLAN.

• Parent interface: igb1 (LAN)
  • (Or whatever your LAN interface is)
• VLAN Tag: 10
  • (I used 10 here, but you can use something different if you’d like. Just be sure you keep this value consistent with the steps later on in this guide.)

Add the new VLAN interfaces and enable them.

Assign the VLAN interface

First, add the new interface.

Navigate to Interfaces > Assignments.

• Assign a new interface
  • Device: vlan01

Then, enable the interface.

Navigate to Interfaces > VLAN01.

• Basic
  • Enable interface = Checked
• Generic
  • IPv4 Config Type = Static IP
  • IPv6 Config Type = Identity association
• Static IPv4 Config
  • IPv4 address = Provide an IP range here to use for the VLAN devices, like 192.168.10.1/24
• IPv6 Identity Association
  • Parent interface = WAN
  • Assign Prefix ID = 1

If at this step you get the error “You specified an IPv6 prefix ID that is out of range,” then your ISP might be provisioning you a /64 IPv6 instead of the /60 or /56.

Configure DHCP for the VLAN

Adapt the values here to suit your needs. For this example, we’re using 192.168.10.1/24 as the address pool, and I’m giving .100-.254 to be open slots of the DHCP pool. We’ll keep .2-.99 for static IP reservations. You can of course change that allocation however you want!

Navigate to Services > Dnsmasq DNS & DHCP > DHCP ranges.

• Interface = VLAN10
  • Start address = 192.168.10.100
  • End address = 192.168.10.254

Enable DHCP server/service on new interfaces

Navigate to Services > Dnsmasq DNS & DHCP > General.

• Interface: LAN + Add new VLAN network (it’s a multi-select field)

Enable IPv6 router advertisements

Turns out, the default OPNsense configuration doesn’t enable this for you on the LAN. Enabling this (even just for the LAN network) fixed a bunch of weird issues I was seeing on my network!

Next, we’ll enable router advertisements (which provides IPv6 routing and address information for devices on your network.

Navigate to Services > Router Advertisements.

Make an RA for the base LAN.

New:

• Enabled = Checked
• Interface = Your LAN
• Mode = Assisted

Then, make an RA for each VLAN you created earlier.

New:

• Enabled = Checked
• Interface = VLAN10
• Mode = Assisted

Create firewall rules

There’s a few schools of thought here on how you can secure the VLAN traffic. We’ll of course prevent the VLAN from initiating connections to the trusted network, but as far as outbound traffic goes, we could…

1. Prevent all outbound traffic on all ports and protocols, except on specific ports.
   • The most secure, but the most time consuming to set up and maintain over time.
2. Allow all outbound traffic on all ports and protocols, except to protected networks and addresses.
   • Secure as long as you are careful!

I wanted to naturally go with “total blacklist except what I specifically allow,” but then I realized I’d be forever maintaining the list of ports that need to be open for all these random IoT devices to operate. So, I opted for the second choice.

OPNsense allows for the creation of Aliases so we can assign a single place to store all the trusted network addresses and network interface names. Aliases can only hold items of a certain type, so we’ll be making three of them:

1. A list of all trusted networks.
2. A list of all trusted IP addresses.
3. A joint list combining all items in #1 and #2, which we’ll apply in the firewall rules.

First, we’ll create an Alias for networks to trust.

Navigate to Firewall > Aliases > Create.

• Enabled = Checked
• Name = TRUSTED_NETWORKS
• Type = Network(s)
• Content = __lan_network
• Description = Specific networks to trust

Then, the Alias for IP addresses.

Navigate to Firewall > Aliases > Create.

• Enabled = Checked
• Name = TRUSTED_ADDRS
• Type = Host(s)
• Content = 192.168.1.1 (or your OPNsense router management IP), then the IP address of any other management devices or other things you want to block.
• Description = Specific IP addresses to trust

Create an Alias to join those two.

Navigate to Firewall > Aliases > Create.

• Enabled = Checked
• Name = TRUSTED_COMBINED
• Type = Network Group
• Content = TRUSTED_ADDRS, TRUSTED_NETWORKS
• Description = The combined group of all trusted entities

Next, we’ll set up the firewall rules.

Start by allowing traffic out of VLAN.

Navigate to Firewall > Rules > [Your VLAN Interface].

Add rule:

• Action = PASS
• Interface = [Your VLAN Interface]
• Direction = IN
• TCP/IP Version = IPv4+IPv6
• Protocol = ANY
• Source = [Your VLAN Interface] net
• Destination = ANY
• Description = Allow [Your VLAN Interface] out to any internet destination

Then, block traffic from the VLAN to the joint alias of protected network resources.

Navigate to Firewall > Rules > [Your VLAN Interface].

Add rule:

• Action = Either REJECT or BLOCK (see note below)
• Interface = [Your VLAN Interface]
• Direction = IN
• TCP/IP Version = IPv4+IPv6
• Protocol = ANY
• Source = [Your VLAN Interface] net
• Destination = TRUSTED_COMBINED
• Description = Block [Your VLAN Interface] to protected network entities

Then, move the block rule above the allow rule.

About the block rule:

• Make sure that the block rule is ABOVE the allow rule.
• Use REJECT if you want fast rejections upon accessing protected resources. That’s good for devices on internal networks so the devices don’t hang forever waiting to timeout when they hit the firewall. Use BLOCK if you just want devices trying to reach protected resources to never hear back. It’s more secure per-se, but could lead to long timeouts and possible network congestion from IoT devices waiting on hold.

Configure VLANs on your managed switch

We probably have different devices here, so you’ll need to follow whatever instructions your switch’s vendor provides for setting up the VLANs.

The most important parts are:

• Set up a base VLAN for trusted traffic (I used VLAN 1).
• Tag any physical ports for devices that must be VLAN’d (I put our network printer on the IoT VLAN.)
• Tag any physical ports for Omada APs to have both VLAN 1 and 10.

Configure VLANs in TP-Link Omada

To get Wi-Fi devices onto this new VLAN, you’ll need to create a new WLAN for them. Use a completely separate, strong password for this network.

While setting up the new WLAN, under Advanced Settings, use:

• VLAN = Custom
• Add VLAN = By VLAN ID, 10

When you connect devices, check that their IPv4 addresses are part of the new VLAN range. You’ll also want to check the IPv6 addresses and see that their addresses are part of the VLAN’s group, indicated by the different prefix (in this case, b500 for devices in the trusted LAN, and b501 for devices in the IoT VLAN with prefix 1).

Device on Trusted LAN    2001:db8:100:b500::25
Device on IoT VLAN       2001:db8:100:b501::25

Common problems/Troubleshooting

Error: “IPv6 prefix ID is out of range”

This usually means your ISP is only delegating a /64 IPv6 prefix instead of a /60 or /56. OPNsense requires a larger delegated prefix when assigning IPv6 subnets to multiple VLAN interfaces.

Solution:

• Contact your ISP and request a /60 or /56 prefix delegation (see above).
• Confirm the prefix using System > Log Files > General in OPNsense (see above).

VLAN Wi-Fi devices are not receiving IP addresses

Check the following:

• The VLAN ID configured on the Omada WLAN matches the VLAN tag on the switch.
• The switch port connected to the AP is configured as a tagged trunk.
• DHCP is enabled for the VLAN interface in OPNsense.

IoT devices cannot see each other

Make sure Client Isolation is disabled on the VLAN WLAN in Omada.

First, review the official upgrade notes. There have been some changes to be aware of, including my.cnf option deprecations that could throw errors after the upgrade.

Upgrade process

Launch a root console.

su -

Backup all databases:

mysqldump --all-databases > /root/mysql-upgrade-backup-$(date +%Y%m%d).sql

Back up all config files:

tar -cvzf /root/mysql_config_backup_$(date +%Y%m%d).tar.gz /usr/local/etc/mysql

Stop the MariaDB service:

service mysql-server stop

There’s no need to remove the old MariaDB packages. They’ll be automatically removed when the new version is installed.

Install the new version (this also installs the -client package):

pkg install mariadb118-server

You’ll see output like the following, showing the old packages will be removed. Enter y to proceed.

Checking integrity... done (3 conflicting)
  - mariadb118-client-11.8.5 [FreeBSD] conflicts with mariadb106-client-10.6.24 [installed] on /usr/local/bin/mariadb
  - mariadb118-client-11.8.5 [FreeBSD] conflicts with mariadb106-server-10.6.24 [installed] on /usr/local/bin/mariadb-dumpslow
  - mariadb118-server-11.8.5 [FreeBSD] conflicts with mariadb106-server-10.6.24 [installed] on /usr/local/bin/aria_chk
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 5 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  libfmt: 12.1.0 [FreeBSD]
  mariadb118-client: 11.8.5 [FreeBSD]
  mariadb118-server: 11.8.5 [FreeBSD]

Installed packages to be REMOVED:
  mariadb106-client: 10.6.24
  mariadb106-server: 10.6.24

Number of packages to be removed: 2
Number of packages to be installed: 3

The process will require 52 MiB more space.

Proceed with this action? [y/N]:

Start the MariaDB service:

service mysql-server start

Run the post-install upgrade:

mariadb-upgrade

Inspect your sites and applications to validate the systems are working as intended!

• Typing delays: Typed characters would show up only after other characters were typed.
• Screen refresh delays:
  • Instead of showing immediately on hover, Openbox’s main menu submenus would only show after the cursor was moved around inside the parent item’s menu a bit.
  • vim error messages like “No write since last change” wouldn’t show up until additional keys were pressed after :q<enter>. The computer sat motionless for more than 10 seconds and then only showed the error message after I pressed an arrow key!

These issues happened in MATE and Openbox, and even plagued basic tty consoles. It didn’t make sense, because the system was not under load and is utterly minimal (just plain-old Xorg running lightdm and Openbox with no other programs running). It’s a a Dell E7470 with an i5-6200U Skylake CPU with Intel HD Graphics 520… plenty of power for this DE.

The lag went away when booted into single-user mode, or if I disabled i915 and had Xorg use scfb.

After some hunting around, turns out that PSR (Panel Self Refresh) was to blame. Disabling PSR fixed the lag issues immediately. But, finding out the exact sysctl settings was tricky because they have to go under the compat.linuxkpi section, and they have a unique naming convention that wasn’t immediately apparent from the documentation I could find online.

The fix

In /boot/loader.conf, set:

# Disable PSR (panel self-refresh)
compat.linuxkpi.i915_enable_psr="0"

Reboot and check things out. That fixed it for me!

What’s PSR?

PSR is a power-saving feature that helps put the GPU into a low-power state if it believes nothing on-screen has changed and, thus, the screen doesn’t need a redraw. Apparently, the PSR implementation from Skylake-era systems was immature in hardware and in software, leading to buggy issues like this.

This setting to disable PSR is likely provided by some distros or pre-built environments… but since I’m rolling my own barebones Xorg environment, I had to learn about the setting the hard way!

Unfortunately, the only way to correct this is to destroy the pool and start over. With ZFS, it’s actually not as painful as it sounds, and with ZFS’s built-in checksumming and scrubbing features you can help ensure that your data transfers without a hitch.

I’ll admit, I’ve had to rebuild or recreate pools a few times now for various reasons. It almost feels like a right of passage as a sysadmin.

Here’s some tips to help you do this process safely and thoroughly. Please consider this a starting point for your own project… there might be some areas where you need to adapt these steps to better suit your individual circumstances. And hey, this is meant to be a friendly guide of tips and suggestions from one storage geek to another. So, be careful… and don’t sue me if you nuke your pool!

What you’ll need

• A backup storage device with enough capacity to store the entire pool’s data. In my case, I had two spare drives laying around that were the same capacity as my pool, so I made a new temporary ZFS mirror pool using them.
• A tested, working UPS. You’re already using one for this server, right?
• (Optional) A second copy of your data. I did an rsync of all the actual files in my pool to another external drive, for yet one more layer of redundancy in case something messed up. No such thing as being too careful here!

Before you begin

• I assume you have intermediate knowledge of ZFS. This guide doesn’t cover the deeper nuances of ZFS exhaustively. For instance, I assume you understand what tank is. I’ll do my best to walk you through the process below, but before you begin you might want to consider reading FreeBSD Mastery: ZFS by Michael W Lucas, or at a bare minimum the ZFS section of the FreeBSD Handbook.

• Decide on how many snapshots you want to keep. I decided to not save all of my pool’s daily snapshots, and instead transfer a single point-in-time migration snapshot. Decide if you want to retain your daily snapshots or if this single snapshot is sufficient.

• The transfer will take lots of time. It took about 5½ hours to transfer about 3.1 TB of data out of the pool, and another 5½ hours to transfer it back. Make sure that when you do this process you aren’t rushed and have ample time to do the job slowly and carefully.

Some additional thoughts

• ECC RAM helped give me confidence in the transfer job. During the transfer, htop reported that the anonymous ARC cache was being used heavily to shuttle data around. I could hear the drive activity and confirm that there was a lot of read pre-fetching going on, and that when the copy commands ended the ARC took a few moments to “empty out” before the target drive stopped working. I know there’s a lot of debate out there about ECC RAM not being necessary for ZFS, but in this case with so much data flying back and forth through RAM, having ECC helped reassure me that that random bit flips wouldn’t happen and introduce issues. (I’m sure a ZFS expert could weigh in here and reassure me more about checksum validations that might happen during data transport… If you are one, please contact me!)

• I avoided using USB drives and went direct SATA → SATA for speed and stability. I suppose there’s nothing wrong with using an external USB drive if you must, but if this is your primary storage pool then at least plug that drive into a UPS! Your goal is to minimize external issues and safeguard your data during this delicate process.

• This pool only had datasets, but volumes work basically the same way. If you need to export data from volumes, the process is mostly the same. The instructions below focus on datasets only, but you could easily adapt this process to include volumes.

How to rebuild a ZFS pool

Stop services and cronjobs

First, stop anything running on the server that might allow access to the source pool. This includes file shares, FTP servers, external SSH connections by people other than you, etc.

Also, consider stopping any cronjobs on the system. Since this process may take several hours, you don’t want your nightly backup job or a weekly scrub kicking off in the middle of things.

Collect data

Let’s collect some data that will be handy to have as we work. Save the results of these commands somewhere.

Start out by exporting a list of all datasets, their sizes, and mountpoints.

zfs list -r tank

Export a list of just the dataset names to help build scripts to go over all the datasets. Also useful to have if you enjoy making checklists!

zfs list -o name -r tank

Save the result of these commands too. They’ll help provide context for the disk layout and mountpoint names, permissions, etc.

• lsblk
• ls -lha of the pool root directory
• zpool status tank
• Properties for the pool root zfs get all tank and all datasets (run this command for each dataset: zfs get all tank/dataset). Pipe that output to text files and keep that info for reference.

Set up the new temporary storage

Making another ZFS storage pool and dataset is best because the ZFS exports we’ll do below will almost certainly exceed the maximum filesize of many filesystems.

When building the new temporary storage pool, be sure to label the disks with gpart labels so that you can be absolutely sure about which disks you’re addressing. (DO NOT rely on /dev/ labels like ada0 to address your disks, as those labels may change across reboots.)

Create snapshots

Now it’s time to create snapshots of the datasets to capture the data as it exists at this point in time. For each dataset, run:

zfs snapshot tank/dataset@YYYY-MM-DD-MIGRATION

You can adapt that snapshot naming scheme to something meaningful to you. I like using the YYYY-MM-DD format because it sorts nicely, and then use a name in ALL CAPS so this important snapshot stands out when you’re looking through a list of hundreds of snapshots.

Transfer snapshots to the temporary storage

PRO TIP: Are you SSH’ing in to the box you’re working on? If so, from here on out you should be running your commands in a tmux session so that the jobs you’re running are protected from an accidental network interruption or your terminal going to sleep.

Each snapshot now needs to be sent to the temporary storage drive. For each dataset, you’ll want to run:

zfs send tank/dataset@YYYY-MM-DD-MIGRATION > /mnt/temporary/dataset.zfs

As this process may take hours to run, consider batching up all of those commands in a shell script. This will allow you to leave the house, go to sleep, etc. while the job runs dutifully in the background.

Verify everything was copied

Use the list of datasets you wrote down earlier in the first steps of this process. Did you transfer all of them to the temporary storage location? Did you see any errors during the zfs send jobs? Do you want to send a second copy of the data somewhere, just in case? You could scrub the temporary location if you really want, but that’s probably overkill…

The point is, take your time here. Double- and triple-check everything, because we’re about to destroy the source pool. There’s no going back after that!

Destroy and rebuild the pool

First, you’ll want to export the source pool, to tell ZFS to stop using it.

zpool export tank

If that command gives you any guff about the pool being in use, make sure you’re not in a working directory inside the pool, that all services that might be using the pool are stopped, etc. Sometimes also the pool just takes a minute to free up before it can be exported, so try again after a minute.

If you’ve passed entire disks to ZFS, you’ll need to clear their labels to dissociate the disks from the pool.

Run this command on each drive, being very careful to supply the proper /dev/ names (or, try using GPT labels, etc.)

zpool labelclear /dev/x
zpool labelclear /dev/y

Once the labels are cleared from the drives, you’re ready to create the new pool. Follow the steps appropriate to the pool setup you want.

In my case, I wanted a mirror of two disks, so I ran:

zpool create tank mirror /dev/label/x /dev/label/y

Before you proceed, check the status of the newly-created pool.

zpool status tank

Were you trying to make a mirror? Be sure right now, right this very instant, that the output looks like this:

pool: tank
state: ONLINE
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0    <--
        ada0    ONLINE       0     0     0
        ada1    ONLINE       0     0     0

Make sure that line marked with the <-- arrow is there, otherwise you have a stripe! This is what bit me earlier… it’s very easy to miss.

After doing a sanity check on the pool architecture, set up any pool-specific settings, like: mountpoint, ACL settings, and other ZFS properties you want to configure at the pool level.

Transfer snapshots from the temporary storage

Now it’s time to repopulate the new pool with the snapshots you sent to the temporary storage. You’ll need to run this command for each dataset, so consider making a shell script, and make sure you’re executing it from within tmux if you’re working remotely.

cat /mnt/temporary/dataset.zfs | zfs receive tank/dataset

Once all the data is transferred, you’ll want to check and set:

• Dataset properties
• Mountpoints
• Owners, groups, and permissions

You can use the information exported in the first few steps to ensure everything is set up as you had it.

Verify the pool data

At this point, I ran a scrub on the new pool to make sure everything was in order. Scrubbing my pool took 4½ hours, so this is another opportunity to let the system run unattended until it’s done.

Re-start services, cronjobs, and the server

Make sure you restart any services you shut off and any cronjobs you commented out earlier.

Once that’s done, give the system a reboot. It’s not absolutely necessary, but it helps validate that you have everything in order so you can prove that the next boot will succeed.

Remove the temporary storage drives and hang on to them for a bit until you are sure everything’s working.

All that’s left is to test your system, make sure the various services are working, and then you’re done – one more pool rebuild is now under your belt!

I’m excited to announce Bildaro: A new self-hosted, minimalist, teeny tiny, mobile-friendly photo gallery and image processor/uploader.

Bildaro is a small-footprint Jekyll blog that, when combined with AWS S3 and AWS Amplify, provides a simple, performant, serverless way to self-host your photo collection at a very low cost.

You retain control over the platform and your content, and your site visitors get a fast, streamlined, simple experience.

The small codebase is easy to understand and welcomes tinkering so you can adjust it to your needs:

• 30 lines of JS to power the image lightbox modal.
• 260 lines of CSS (Sass) that Jekyll will compile for you on build.
• 170 lines of code for the image processor and file uploader, written as a simple Bash script with no dependencies other than the tools it uses (imagemagick and aws cli via pip).
• Other dependencies are minimal and are only needed if you want to run Jekyll locally (ruby and a few small Ruby gems).

And just to reiterate: The numbers above are not KB of code, they’re individual lines of code. Bildaro is smol.

Why a self-hosted serverless photo gallery?

Running your own web server requires expertise, takes considerable effort to get running, and requires an investment in long-term maintenance.

Many popular self-hosted solutions, like my previous all-time favorite Lychee, have grown into behemoths that require all sorts of dependencies just for a baseline install. And since they run on a live system exposed to the internet, you’ll have to be vigilant and keep current with updates to keep your system secure.

A static website: doesn’t require any maintenance, is practically immune to almost all of the typical vulnerability channels websites face because it doesn’t have any “moving parts,” and can be served quickly and at massive scale for very little money.

The backstory

I’ve hosted a photo gallery in the past, but I was tired of maintaining it. I took it offline for a while to see if Dropbox would be sufficient. Turns out Dropbox is “just fine,” but it’s not an ideal experience. The web interface is heavy and bulky, full of log in dialogs, cookie notices, upsells, and now all this AI bullshit. It left me wanting to self-host again.

This past weekend I spun up a new web server to host Lychee, and it quickly became apparent that the project had just gotten really big for my rather basic needs. Composer, Laravel, Artisan, code linters, local development tools, CSS frameworks, Vue, Typescript, Font Awesome, gesture libraries… All in all about 400 MB of dependencies to download just to get started.

On top of that, don’t forget that you have the complexity of the baseline OS install to support the software stack in the first place. You’ll need Apache/nginx/lighttpd, PHP, and a database engine. Do you want MariaDB or SQLite? You have to keep all of these things updated. And backed up. Then there’s SSH, ZeroTier, Certbot and SSL certs and renewals, software dependencies…

That’s a lot of moving parts for something that feels so simple.

It got me thinking that there’s gotta be a better way.

My brain started turning:

• My photo gallery website doesn’t change, except when I add or remove photos.
• So if it rarely changes, why do I need a dynamically-generated, database-backed, full-blown web server running with dedicated CPU and RAM just to host a basic static site?
• Could this be a static website in disguise? Should it have been one this entire time?

Jekyll to the rescue

Jekyll feels like an old friend. I’ve used the platform for years to create this blog, and other websites for business and hobbies. It’s a platform I know and trust. It’s performant and has powerful and useful features. It does a thing with focus and it does it well. And… it took on absolute superpowers when served with AWS Amplify.

As soon as I realized the photo gallery site could be static, the flood happened. Everything fell into place.

• The photo albums can just be posts, with YAML data storing all of the images in the album along with some basic metadata.
• The images can live on AWS S3, where they can take up as much space as I need, and it’ll cost next-to-nothing. They can be decoupled from the blog, so I don’t have to worry about working with the images in the repo.
• AWS Amplify can host the Jekyll site in a serverless fashion, like I do with this site. And it’ll be served from a CDN edge, with more performance than I could ever get out of the web server. (Frankly, more performance than I’ll ever need for my simple site.) And best of all, I won’t have to manage a single thing.
• I can write a simple Bash script to convert the source JPGs into web-optimized webp thumbnails and the display size variants, create the album zip download file for people that want it, have it export an image manifest, and even upload all the files to S3 for me.
• Changes happen as repo commits and can trigger build events, updating the gallery site.
• The site can just fucking sit there and serve and serve and serve for pennies until I’m ready to change it. Like how this blog does. No moving parts. No worrying about software updates, Certbot renewal triggers, or anything that comes with running your own server. Ever.

So, I got to work!

Bonvenon, Bildaro!

I wrote Bildaro in the course of two days. Two good “sit down and focus” sessions is all it took. It all came together very quickly, because the premise is so simple: Bildaro is basically a Jekyll blog, nothing more.

Albums are simply posts, and they contain a YAML list of the photos in the album. The view page simply iterates over those images and creates the photo gallery output. The image URLs get prepended with the S3 bucket URL. And even better, this file is generated from the Bash script. All you need to do is edit the title and the cover image, commit the changes, and Amplify takes care of the rest.

Example album _posts/2023-09-17-satsop-nuclear-plant.md

---
title: Satsop Nuclear Plant
cover: DSC07949.jpg
download_size: 4264
pictures:
  - DSC07949.jpg
  - DSC08116.jpg
  - DSC08264.jpg
  - DSC08378.jpg
  - DSC08392.jpg
  - DSC08403.jpg
  - DSC08426.jpg
  - DSC08429.jpg
---

Since the page is static and written from scratch, it’s lightweight and performant. Check out these Pagespeed scores and see just how lightweight the gallery homepage is:

I’m sure there are other Jekyll photo galleries out there, but it felt so rewarding to write one for myself, from scratch. Tuned to do exactly what I need it to do, and nothing more.

“This is my gallery. There are many like it, but this one is mine.”

Bildaro is open source, released under the permissive MIT license. It’s given me great pleasure to write this little thing. You’re welcome to take it for a spin!

Take a look at my photo gallery to see it in action.

As I reflect back on this issue, it kept reminding me of this post on The Jurassic Park Effect by Scott Alan Miller. In the article, the author explains that NAS OS’s allow novices to create a production storage environment that’s “dangerously easy” to set up. And in doing so, these OS’s add unecessary complexity, functional limitations, upstream update lag, a possibility of abandonment, and… they shield their operators from understanding basic system fundamtenals necessary to triage and repair problems with the system. This is a perfect illustration of the out-of-the-loop performance problem. All of this has been on my mind recently.

I first found Miller’s article in 2021. And after these 3 years, I can tell you: He’s absolutely right.

My home file server (and, homelab in general) became a great case study for this topic.

A home file server is a relatively straightforward appliance. Running a simple *nix box and a Samba share isn’t tough. But when ZFS came along, I knew I wanted to use it, but I was worried I didn’t know how to use it properly, so I turned to NAS OS’s for expert guidance.

TrueNAS went through a total rewrite shortly after I started using it, and it got too bloated and “dashboardy.” The streamlined simplicity of XigmaNAS (née NAS4Free) caught my attention, and I used it for several years. It worked smoothly and the team didn’t load it up with flashy updates. I fixed up parts of their documentation site as a show of thanks to the project.

But after a while, I found that I needed to grow past what that OS could provide for me. Features in the OS started to turn into roadblocks. My education around ZFS was stunted because I was letting someone else make decisions for me. And I wanted to do more with the server than the pared-down image was built to allow.

Miller was absolutely correct when he said that a NAS OS is “a more complex and less functional version” of the underlying OS it’s built on top of. It’s true for any and every NAS OS out there. New ones are coming and going all the time, and none of them, even the ones built with simplicity and freedom in mind, can escape this truth.

I decided that the training wheels needed to come off.

Hunkering down

Over the course of a few months I taught myself slowly and carefully about all the things needed to replace these NAS OS’s. I spun up a barebones FreeBSD server, tested building ZFS pools, running Samba shares, FTP servers, playing with snapshots and transferring datasets around, and a lot more. It was challenging, and fun! And my tinkering didn’t stop there.

Just two weeks ago I completed a migration and consolidation of my entire home lab onto a single FreeBSD server. My dev environments, experiements, and various services are running in jails. I finally mastered bhyve to the point where I was able to retire my Proxmox host so I can virtualize different OS’s like Debian and Windows.

Not only has my setup becomed streamlined and simplified, but my personal knowledge and confidence around these topics has grown tremendously.

Looking back

I can’t ever imagine using a NAS OS ever again at this point. There’s no feature one of these systems could ever offer that would draw me in. Ultimately, I don’t want to be disconnected and kept out-of-the-loop.

I want to be in-the-loop.

I want to know how a vital system like a file server works. I want to know how to build it, configure it, maintain it, and most importantly — how to repair it.

But that takes time, effort, aptitude, and space. I can understand why people turn to NAS OS’s. Some people don’t want to bother with all of this work. And that’s totally fine! It’s just a decision that imposes some limitations and a certain amount of risk.

In my case, I didn’t roll my own because I was simply scared of what I didn’t know. And like the saying goes, I didn’t know what I didn’t know. When it comes to my data, I wanted to be conservative and err on the side of caution. I wanted expert help and guidance. But after a while, once I warmed up to the idea and had time and space to learn, I was able to “earn the knowledge for myself” (Ian Malcom, Jurassic Park). I’m truly glad I made this decision.

Symptoms included:

• Directory contents would take a very long time to load.
• Directory contents would disappear after browsing between sibling folders.
• Directories would show the contents of different sibling folders.

The problem was really annoying and took me quite a while to figure out.

At first, I thought it was an issue with Finder. The internet is full of people’s complaints about Finder sure, but I spend a lot of time in Finder and want to try and fix the problem there if possible. I did try using third-party file managers like Commander One and Marta, but even those programs weren’t totally immune to the issues either. That indicated the issue must be with something at a deeper system level. But, was it an issue with the server, the client, or a combination of problems on both sides?

Since I built the file server myself, I assumed it must be a problem I inadvertently caused. Perhaps some setting I had misconfigured, or some configuration value I had overlooked.

These issues didn’t happen on non-SMB protocols (FTP and SSH were working), so I figured it was an issue with Samba. But, non-MacOS clients didn’t have connection issues (Windows, Linux, and FreeBSD clients were working).

Some articles online suggested that Samba itself was to blame, or that MacOS’s implementation of the Samba client was buggy and hopelessly beyond repair. I even considered switching away from Samba to some other protocol, but that wouldn’t be ideal.

After doing a lot of reading and testing, I learned that Finder rather aggressively caches directory listings, and that cache system can be unreliable with large fileshares over Samba. And I also learned that out-of-the-box Samba configurations need to be tweaked slightly to improve connections for MacOS clients.

The fixes below were scoured from many places on the web, including Reddit posts, blog posts, Github repos, and a deep dive of the Samba configuration documentation. It’s taken me a few months to test out various settings combinations, and now things are working properly.

I’ve been working with these new settings for more than two weeks and have not experienced any further issues. Feels like the right time to document and publish the fix!

Server changes

On the server, make the following changes to smb.conf:

[global]

# Enforce a minimum of Samba v2 (Vista / Server 2008) for client connections
min protocol = SMB2
server min protocol = SMB2

# Improve compatibility with Macs
vfs objects = catia fruit streams_xattr
fruit:aapl = yes
fruit:nfs_aces = no
fruit:zero_file_id = yes
fruit:metadata = stream
fruit:encoding = native
spotlight backend = tracker

readdir_attr:aapl_rsize = no
readdir_attr:aapl_finder_info = no
readdir_attr:aapl_max_access = no

fruit:model = MacSamba
fruit:posix_rename = yes
fruit:veto_appledouble = no
fruit:wipe_intentionally_left_blank_rfork = yes
fruit:delete_empty_adfiles = yes

# (OPTIONAL) Don't allow MacOS clients to write .DS_Store files to server shares
veto files = /.snap/.sujournal/._.DS_Store/.DS_Store/.Trashes/.TemporaryItems/
delete veto files = yes

Spotlight searching of Samba shares is disabled by default, but you can add this line to each of your Samba share definitions in smb.conf to be absolutely sure:

[yourshare]
spotlight = no

Client changes

On the MacOS client computers, create /etc/nsmb.conf:

[default]

# Disable SMB v1
protocol_vers_map=6

# Disable NetBIOS
port445=no_netbios

# Use NTFS streams if supported
streams=yes

# Disable directory caching
dir_cache_max_cnt=0
dir_cache_max=0
dir_cache_off=yes

# Disable packet signing
signing_required=no

# Disable multi-channel connections and prioritize the wired ethernet connection
mc_prefer_wired=yes
mc_on=no

# Disable SMB session signing
validate_neg_off=yes

Applying the changes

• Restart the Samba service on the server.
• Restart each client computer.

When you reconnect to the server shares on your machines, these issues shouldn’t occur anymore.

Start off by listing all of the PHP-related packages on your system, including the PHP extensions you have installed.

pkg query -a %n | grep php8

We’ll use phpXX to represent the old, existing version of PHP on the machine.

In a text editor, take off the leading phpXX- from each extension’s name. Then, put them into a comma-separated format like so:

bcmath,ctype,curl,dom,…

You can use brace expansion to simplify the package install command. Be sure your shell supports brace expansion, otherwise you’ll get an error.

In the next command, let phpYY be the modern version you want to install (php83, php84, php85, etc.)

sudo pkg install phpYY phpYY-{bcmath,ctype,curl,dom,exif,extensions,fileinfo,filter,gd,iconv,mbstring,mysqli,opcache,pdo,pdo_mysql,pdo_odbc,pdo_sqlite,pecl-imagick,pecl-json_post,pecl-mcrypt,phar,posix,session,simplexml,sodium,sqlite3,tidy,tokenizer,xml,xmlreader,xmlwriter,zip,zlib}

You may be notified that the packages for the older version will be removed as part of the upgrade process. That’s helpful! Watch for any notices or errors during the upgrade.

If you aren’t told this, or want to be sure the packages are removed:

sudo pkg remove phpXX phpXX-{your list of packages}
sudo pkg remove mod_phpXX

Restart PHP and Apache and then you’re done!

sudo service php-fpm restart
sudo service apache24 restart

Update Feb 2, 2024: Recent changes to Visual Studio Code require versions of glibc that aren't provided by the Linux compatibility packages in FreeBSD. This change completely broke my workflow yesterday – a Monday morning and major launch day – resulting in the error The remote host may not meet VS Code Server's prerequisites for glibc and libstdc++.

After some substantial and understandable public outcry, they've decided to walk back the changes for 12 months.

Seeing the writing on the wall, I can assume that support for connecting to FreeBSD remote machines will not be a priority for the team. I want to limit my future risk, so I’ve decided to change my process over to using the SSH FS plugin by Kelvin Schoofs.

The good news is there won’t be any need to install Linux compatibility binaries on FreeBSD machines, so the steps below aren’t necessary. It always felt a bit too clunky of a solution, anyway. You can still connect to remotes, edit files, and even launch remote terminals.

But, the bad news is that some advanced VSCode functionality for remote file systems might not be accessible (Git file status indicators, etc.) Those are nice-to-have features, but not critical for my day-to-day work.

Visual Studio Code’s native SSH remote explorer connection is more than just an SSH tunnel — it needs to be able to execute some Linux binaries on the remote server in order for all of the fancy functionality to work.

When connecting to a FreeBSD server, VSCode may fail to connect and give you errors like:

[] > Unsupported platform: FreeBSD
> exitCode==35==
> osReleaseId==freebsd==
[] Failed to parse remote port from server output
[] Terminating local server
[] > local-server-1> ssh child died, shutting down

Luckily, getting VSCode connecting to FreeBSD isn’t tough.

How to connect VSCode to a FreeBSD remote server

First, on the FreeBSD server, enable Linux compatibility mode, and then install the Linux base core.

sudo sysrc linux_enable="YES"
sudo service linux start
sudo pkg install linux_base-c7

Then, on the machine you’re using VSCode on, edit .ssh/config and add these lines for the FreeBSD server’s entry.

Host xxx
  Hostname xxx
  RemoteCommand /compat/linux/usr/bin/bash
  RequestTTY force

Finally, find the Enable Remote Command setting in VSCode and enable it.

Voila! Head on over to the remote explorer and connect to the server. You’ll now be able to interact with the FreeBSD remote just like it was a Linux host, with all the helpful functionality in VSCode that you’re used to.

It took a long time to figure this one out. Credit to Mateusz Kwiatkowski and Oliver Giersch for the helpful comments I stumbled upon which explained the missing parts of this process.

Table of contents

1. Before we begin
2. Find hosting and provision an instance
3. Set up FreeBSD
4. Simplify outbound email handling
5. Install the Apache web server
6. Install PHP
7. Install the MariaDB database server
8. Final steps
9. Tips and tricks

Before we begin

• This guide is meant to supplement the FreeBSD handbook, which should be considered the canonical documentation source for FreeBSD.
• If you found this guide from the future, please be aware these instructions may be outdated.
• This guide assumes you have some basic experience with BSD-style servers. If you’re familiar with Linux, you should be able to follow along nicely!
• This guide isn’t meant to act as a tuning guide for Apache, PHP, or MariaDB. That’s beyond the scope of this article. I may write about tuning those services in future posts!

Step 1. Find hosting and provision an instance

You can create FreeBSD instances on:

• Vultr
• AWS Lightsail
• AWS EC2

FreeBSD is supported on Linode and DigitalOcean, but requires some extra steps in order to get an instance up and running.

Choose a provider and build an instance. If you’re not sure what size machine to get, start small and upgrade later if you need to. A $7/mo AWS Lightsail instance (1 GB RAM, 2 vCPU, 40 GB SSD) is more than sufficient to host several WordPress sites. Static content caching and a CDN can help reduce load on the server.

Step 2. Set up FreeBSD

When the machine is provisioned it’ll come with a baseline OS image. You’ll want to do a few things first to finish the install of FreeBSD and get the machine configured for use as your web server.

Update the system

The first thing to do is to log in as root and update the system software.

Start by updating the FreeBSD core.

# freebsd-update fetch install

Then, update the non-core software packages.

# pkg update
# pkg upgrade

Set the timezone

# tzsetup

Install some useful applications

# pkg install doas wget git bash bash-completion tmux

# pkg install vim

Set up doas (instead of sudo)

I’ve changed from recommending sudo to doas instead. doas operates the same way, yet is smaller and simpler.

/usr/local/etc/doas.conf (Ensure this file ends with a blank line)

permit nopass :wheel

Set the hostname

Verify that your hostname looks appropriate.

# hostname

If you need to change it, edit /etc/rc.conf and modify the hostname="" line. Then, run doas hostname [your-hostname] to update the hostname without rebooting. Validate your work again by running hostname.

Create a swapfile

Some VPS’s don’t create swapfiles when the instances are provisioned. It’s a good idea to create one, even a small one (512-1,024 MB).

First, create the swapfile and set permissions on it.

# dd if=/dev/zero of=/usr/swap0 bs=1m count=512
# chmod 600 /usr/swap0

Then, edit /etc/fstab and add this line:

md99   none   swap   sw,file=/usr/swap0,late   0   0

Next, activate the swapfile with:

# swapon -aL

Create your non-root user account

Create a user account for yourself.

# adduser

You’ll need to answer a dozen or so questions during this process. Simply press [Enter] unless you’re at one of these prompts:

1. Select a username (should be all lowercase, no symbols or spaces).
2. Provide the user’s full name.
3. Invite to other groups? Add to wheel group so user can use doas.
4. Shell? Choose bash if you want.
5. Provide a password, and enter it twice to make sure it’s entered correctly.
6. Review the entries.
7. Stop adding users if you’re done.

The process will look like this:

Username: btorres                                                       <--- (1)
Full name: B'Elanna Torres                                              <--- (2)
Uid (Leave empty for default):
Login group [btorres]:
Login group is btorres. Invite btorres into other groups? []: wheel     <--- (3)
Login class [default]:
Shell (sh csh tcsh bash rbash git-shell nologin) [sh]:                  <--- (4)
Home directory [/home/btorres]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]:
Enter password:                                                         <--- (5)
Enter password again:                                                   <--- (5)
Lock out the account after creation? [no]:
Username   : btorres
Password   : *****
Full Name  : B'Elanna Torres
Uid        : 1002
Class      :
Groups     : btorres wheel
Home       : /home/btorres
Home Mode  :
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes                                                       <--- (6)
adduser: INFO: Successfully added (btorres) to the user database.
Add another user? (yes/no): no                                          <--- (7)
Goodbye!

If you want to change an existing user’s shell to bash:

# chsh -s /usr/local/bin/bash [username]

And then add this to the bottom of the user’s ~/.profile so that the system will parse their ~/.bashrc on login:

# Load .bashrc on login
if [[ $- == *i* && -f ~/.bashrc ]]; then
    . ~/.bashrc
fi

Set up SSH

Ensure you’re familar with and are using SSH key-based authentication.

Switch to your new user account.

# su [username]

Add your SSH public keys to ~/.ssh/authorized_keys, then lock down that file’s permissions. That file might not exist for new users, so you’ll need to create it.

$ vim ~/.ssh/authorized_keys
  (paste in your public keys)
  (save and quit)

$ chmod 600 ~/.ssh/authorized_keys

Log out as your user and return to root with:

exit

Step 3. Simplify outbound email handling

FreeBSD comes with a feature-rich inbound/outbound email system via sendmail. It’s too complex for most server installs that only need to send outbound emails (such as system alerts, cron job results, password reset emails via PHP, etc.).

We’ll replace sendmail with the lightweight, outbound-only ssmtp.

Disable sendmail

Edit /etc/rc.conf and add this to the bottom:

# Disable sendmail (we're using ssmtp)
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

Then, stop the running sendmail service with:

service sendmail stop

Install ssmtp

Follow these steps to install ssmtp.

Step 4. Install the Apache web server

We’ll use Apache here, but there are several other web servers you could choose, including Nginx or Lighttpd.

Install Apache

Find and install the latest version of Apache.

# pkg search apache | grep -i server
apache24-2.4.66                Version 2.4.x of Apache web server

# pkg install apache24

Ping your hostname and ensure that it resolves to the machine’s IP address. If you need to make changes, check either /etc/nsswitch.conf or /etc/hosts.

# ping [hostname]

Enable Apache at boot:

# sysrc apache24_enable="YES"

Configure Apache

First, make a backup copy of /usr/local/etc/apache24/httpd.conf, then edit the file.

/usr/local/etc/apache24/httpd.conf

• Set ServerName to either a FQDN or the server’s IP address
• Uncomment these lines:

Include etc/apache24/extra/httpd-ssl.conf
Include etc/apache24/extra/httpd-mpm.conf

# Uncomment any modules you want to enable
LoadModule authn_socache_module libexec/apache24/mod_authn_socache.so
LoadModule socache_shmcb_module libexec/apache24/mod_socache_shmcb.so
LoadModule ssl_module libexec/apache24/mod_ssl.so
LoadModule deflate_module libexec/apache24/mod_deflate.so
LoadModule expires_module libexec/apache24/mod_expires.so
LoadModule headers_module libexec/apache24/mod_headers.so
LoadModule speling_module libexec/apache24/mod_speling.so
LoadModule alias_module libexec/apache24/mod_alias.so
LoadModule rewrite_module libexec/apache24/mod_rewrite.so

Make a backup copy of /usr/local/etc/apache24/extra/httpd-ssl.conf, then edit the file.

/usr/local/etc/apache24/extra/httpd-ssl.conf

• Comment out line Listen 443
• Remove the default SSL virtualhost example at the bottom of the file. (Hint: it starts with <VirtualHost _default_:443> and continues for about 170 lines until the closing </VirtualHost>. Remove all of those lines.)
• Add these lines (and follow the link in your web browser, ensure the protocols listed below are still up-to-date):

SSLProtocol -all +TLSv1.2 +TLSv1.3
SSLProxyProtocol -all +TLSv1.2 +TLSv1.3

Later, you’ll add your virtualhost files and other common configs to /usr/local/etc/apache24/Includes/.

Validate the configuration

Let’s ensure our configuration files are set up properly. Run:

# apachectl -t

If Apache is running and you want to validate the configuration files, run:

# service apache24 configtest

If you get the Could not reliably determine the server's fully qualified domain name error, try adding a ServerName parameter in /usr/local/etc/apache24/httpd.conf

If there are no errors, start Apache:

# service apache24 start

Browse to the machine’s IP address (or hostname) and make sure you see the Apache “It works!” message. If you don’t see anything, ensure that Apache is running, and monitor /var/log/httpd-error.log and /var/log/httpd-apache.log for hints that can help you identify problems.

If you’re stuck, perhaps there’s some misconfiguration somewhere. Apache’s default configuration file pathway is, in order (all of the files below are in /usr/local/etc/apache24/):

1. httpd.conf
2. extra/httpd-mpm.conf
3. extra/proxy-html.conf
4. extra/httpd-ssl.conf
5. Include/*.conf

Step 5. Install PHP

Find the latest versions of PHP:

# pkg search php | grep -i scripting
php82-8.2.30                   PHP Scripting Language (8.2.X branch)
php83-8.3.29                   PHP Scripting Language (8.3.X branch)
php84-8.4.16                   PHP Scripting Language (8.4.X branch)
php85-8.5.1                    PHP Scripting Language (8.5.X branch)

Install PHP and related packages (we’ll use v8.3 here):

# pkg install php83 php83-mysqli php83-pdo

# For WordPress support, also install:
# pkg install php83-{curl,dom,exif,fileinfo,filter,gd,iconv,mbstring,phar,simplexml,sodium,xml,xmlreader,zip,zlib}

Validate that php-fpm was installed.

php-fpm -v

Put a php.ini file into place.

FreeBSD’s PHP doesn’t come with a php.ini file. There are two templates provided, one for production machines, and one for development machines. You’ll need to select one to start with and then customize from there.

The development template has settings that reduce or eliminate OPcode caching, so code changes will be visible immediately, at the expense of page rendering speed.

# cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini
-- OR --
# cp /usr/local/etc/php.ini-development /usr/local/etc/php.ini

At the end of the [PHP] section:

; (2026) This might not be necessary anymore. Leave it commented out unless necessary.
; Don't serve suggested files in misspelled URL requests.
; cgi.fix_pathinfo=0

; Dev server? Reload php files instantly
revalidate_freq=0

; Big or complex site? Divi/Elementor?
memory_limit = 256M
; memory_limit = 512M

; Need big uploads?
post_max_size = 32M
upload_max_filesize = 32M

At the end of the [Pdo_mysql] section:

[Pdo_mysql]
pdo_mysql.default_socket=/var/run/mysql/mysql.sock

Next, edit /usr/local/etc/php-fpm.d/www.conf

• Make note of the pool name (defaults to www)
• Make a note of the user/group (defaults to www)
• Change listen from 127.0.0.1:9000 to /var/run/php-fpm.sock
• Uncomment the socket permissions section (listen.owner, listen.group, and listen.mode)

Validate the configs:

php-fpm -t

Update Apache to look for and execute index.php files by editing /usr/local/etc/apache24/httpd.conf and adding index.php to DirectoryIndex, like so:

DirectoryIndex index.php index.html index.htm

Next, connect Apache with PHP by editing (or creating) /usr/local/etc/apache24/Includes/php.conf:

<FilesMatch "\.php$">
    SetHandler "proxy:unix:/var/run/php-fpm.sock|fcgi://localhost/"
</FilesMatch>

Enable PHP at boot:

# sysrc php_fpm_enable=YES

Create a demo page to show that PHP is working at /usr/local/www/apache24/data/index.php and put in these contents:

<html><body><?php phpinfo(); ?></body></html>

Then, start PHP:

# service php_fpm start

Browse to your web server’s hostname or IP address and you should see a long PHP information page.

If you don’t see anything, or if you get any errors, monitor /var/log/php-fpm.log for hints that can help you identify problems. Also, check the Apache log files /var/log/httpd-error.log and /var/log/httpd-apache.log for clues.

Step 6. Install the MariaDB database server

Find the latest version of MariaDB, then install it.

# pkg search mariadb | grep -i server

mariadb1011-server-10.11.15    Multithreaded SQL database (server)
mariadb106-server-10.6.24      Multithreaded SQL database (server)
mariadb114-server-11.4.9       Multithreaded SQL database (server)
mariadb118-server-11.8.5       Multithreaded SQL database (server)          <---

# pkg install mariadb118-server

Set MariaDB to start at boot and start the service.

# sysrc mysql_enable=YES
# service mysql-server start

Secure the installation

# /usr/local/bin/mysql_secure_installation

When prompted:

• Do not provide a root password, just press [Enter]
• Use Unix socket authentication
• Remove test users and databases
• Prevent remote login
• Flush the tables to save settings

Validate that the server is running

# mysql

You should see the MySQL prompt. Type exit to quit.

Final steps

After every server setup is complete, restart the system to make sure all systems come back online as expected. This is a great time to make sure all services come back online at system boot.

Log in as your non-root user and try some doas commands to ensure you can escalate permissions when needed.

Tips and tricks

Silence the login tips (fortunes)

Every time you log in to FreeBSD, it displays fun and helpful tips to familiarize you with the system. As you become proficient, you may want to turn these off.

Edit ~/.profile and comment out the line with /usr/bin/fortune.

To completely silence the rest of the login messages, add a blank file in your home directory named ~/.hushlogin.

Disable the daily system email reports

The server will send you daily reports with system statuses and security reports. Instead of cluttering up your inbox, let’s send the reports to files instead.

First, create a folder to store the files:

doas mkdir /var/log/periodic

Then, edit /etc/periodic.conf and add these lines to the bottom of the file:

# Instead of spamming yourself with daily email reports,
# output the results of these reports to a file.

# daily_output="root"
daily_output="/var/log/periodic/$(date +%Y%m%d)-daily.log"

# daily_status_security_output="root"
daily_status_security_output="/var/log/periodic/$(date +%Y%m%d)-daily-security.log"

# weekly_output="root"
weekly_output="/var/log/periodic/$(date +%Y%m%d)-weekly.log"

# weekly_status_security_output="root"
weekly_status_security_output="/var/log/periodic/$(date +%Y%m%d)-weekly-security.log"

# monthly_output="root"
monthly_output="/var/log/periodic/$(date +%Y%m%d)-monthly.log"

# monthly_status_security_output="root"
monthly_status_security_output="/var/log/periodic/$(date +%Y%m%d)-monthly-security.log"

Enable locate

locate lets you quickly find files on your machine. It’s installed by default, but unusuble until you prime the database:

/etc/periodic/weekly/310.locate

To automatically refresh the database weekly:

doas sysrc weekly_locate_enable="YES"

Then, you can quickly find any file on your machine with:

$ locate rc.conf
/etc/defaults/rc.conf
/etc/rc.conf
/etc/rc.conf.d
/usr/share/examples/jails/rc.conf.jails
/usr/share/man/man5/rc.conf.5.gz
/usr/share/man/man5/rc.conf.local.5.gz
/usr/share/man/man5/src.conf.5.gz
/var/db/etcupdate/current/etc/defaults/rc.conf

Since the database is only updated weekly, you might want to re-prime it after installing new software or making big changes to the system. Simply run the first command above to re-prime the database and scan your filesystem for new files.