469 lines
14 KiB
Markdown
469 lines
14 KiB
Markdown
# TrueNAS SCALE
|
|
|
|
###### guide-by-example
|
|
|
|
|
|
|
|
# Purpose & Overview
|
|
|
|
Network storage operating system managed through web GUI.<br>
|
|
|
|
* [Official site](https://www.truenas.com/truenas-scale/)
|
|
* [Forums](https://www.truenas.com/community/forums/truenas-scale-discussion/)
|
|
|
|
Based on debian linux with ZFS file system is at the core.
|
|
Running nginx and using python and django for the web interface.
|
|
|
|
**note** - There are links to the official documentation in subsections,
|
|
its of decent quality, with pictures and videos and it should be up-to-date.
|
|
|
|
[ZFS for Dummies](https://blog.victormendonca.com/2020/11/03/zfs-for-dummies/)
|
|
|
|
# My specific use case
|
|
|
|
My home server runs ESXi.<br>
|
|
TrueNAS is one of the virtual machines,
|
|
with Fujitsu D3307 [flashed in to IT Mode](https://forums.servethehome.com/index.php?threads/the-versatile-sas3008-chipset-my-vendor-crossflashing-adventures.28297/page-4#post-319106)
|
|
and pass-through in to the VM so that truenas has direct access to the disks,
|
|
without any in between layer. If I would have not lucked out with the HBA card,
|
|
I would be buying Fujitsu 9211-8i from ebay.
|
|
|
|
I hold strong opinion on backups > raid.<br>
|
|
And I also have only few drives.
|
|
So I make just little use of zfs raid features and use it just as nice web GUI
|
|
for samba and nfs sharing.
|
|
|
|
Good alterntive worth mention - [openmediavault](https://www.openmediavault.org/),
|
|
but truenas seems a bigger player.
|
|
|
|
<details>
|
|
<summary><h1>Installation as a VM in VMware ESXi</h1></summary>
|
|
|
|
|
|
|
|
[The official documentation.](https://www.truenas.com/docs/scale/gettingstarted/install/installingscale/)
|
|
|
|
Make sure the Esxi host has correc time and ntp sync.
|
|
|
|
* [download ISO](https://www.truenas.com/download-truenas-scale/)
|
|
* upload it to ESXi datastore
|
|
* create new VM
|
|
* Guest OS family - linux
|
|
* Guest OS version - Debian <latest> 64-bit
|
|
* give it 2 cpu cores
|
|
* give it 4GB RAM with sub-setting: `Reserve all guest memory (All locked)`
|
|
* give it 50GB disk space
|
|
* mount ISO in to the dvd drive
|
|
* SCSI Controller was left at default - vmware paravirtual
|
|
* switch tab and change boot from bios to uefi
|
|
* click through the Installation
|
|
* pick admin user and set password
|
|
* login, shutdown
|
|
* ESXi - edit VM, add other device, PCI device,
|
|
should be listed HBA card thats passthrough
|
|
so that truenas has direct disks access
|
|
|
|
</details>
|
|
|
|
---
|
|
---
|
|
|
|
<details>
|
|
<summary><h1>Basic Setup</h1></summary>
|
|
|
|
### Static IP address
|
|
|
|
* turn off dhcp and set static ip and mask<br>
|
|
Network > Interfaces<br>
|
|
uncheck DHCP; Add Aliases, IP address=10.0.19.11; mask=24<br>
|
|
on save it asks for the gateway IP
|
|
* set hostname, DNS server and enable netbios discovery<br>
|
|
Network > Global Configuration > Settings<br>
|
|
check `NetBIOS-NS`; set hostname; set dns if it's not
|
|
|
|
### Set time
|
|
|
|
* Set time zone and date format<br>
|
|
System Settings > General > Localization > Settings<br>
|
|
Timezone=Europe/Bratislava; Date Format=2 Jan 2023
|
|
|
|
If there are issues with the time... enable ssh service, ssh in to the truenas
|
|
check few things
|
|
|
|
* `timedatectl` - general time info
|
|
* `sudo ntpq -p` - lists configured ntp servers, the symbols in the first column
|
|
`+, -, *` [note the use](https://web.archive.org/web/20230102105411/https://detailed.wordpress.com/2017/10/22/understanding-ntpq-output/)
|
|
* `sudo ntpq -c sysinfo` - operational summary
|
|
* `sudo ntpd -g -x -q pool.ntp.org` - force sync to a pool
|
|
* `sudo sntp pool.ntp.org` - force sync to a pool
|
|
* `systemctl status ntp.service` - check service status
|
|
* `sudo journalctl -u ntp.service` - check journal info of the service
|
|
* `sudo systemctl restart ntp.service` - restart the service
|
|
* `cat /etc/ntp.conf` - check the config
|
|
* `sudo hwclock --systohc --utc` - set utc time to rtc clock, hardware clock runnin in bios
|
|
|
|
|
|
|
|
I faced an issue of time being out of sync after restarts and ntpq command
|
|
failing to connect. Reason for this was Esxi host not having default gateway
|
|
correctly set and so never being able to sync its time.
|
|
|
|
### Pools and Datasets
|
|
|
|
|
|
|
|
### Pool
|
|
|
|
[The official documentation.](https://www.truenas.com/docs/core/coretutorials/storage/pools/poolcreate/)
|
|
|
|
Pool is like a virtual unformated hard drive. Can't be mounted,
|
|
cant be used without *"partitioning"* it first.
|
|
But it is at the creation of pool where "raid" is set.
|
|
|
|
* start creating a pool<br>
|
|
Storage > Create Pool button<br>
|
|
name it; I prefer to not encrypt, that comes with datasets
|
|
* assign physical disks to the pool's default VDev,
|
|
if needed, more VDevs can be added<br>
|
|
select "raid" type for the VDev - stripe, mirror
|
|
* Create
|
|
|
|
For destruction of a pool - Storage > Export/Disconnect button
|
|
|
|
### Dataset
|
|
|
|
[The official documentation.](https://www.truenas.com/docs/core/coretutorials/storage/pools/datasets/)
|
|
|
|
`Dataset` is like a partition in the classical terms. It's where filesystem
|
|
actually comes to play, with all the good stuff like mount, access, quotas,
|
|
compression, snapshots,...
|
|
|
|
* start creating a dataset<br>
|
|
Datasets > Add Dataset button<br>
|
|
name it; I prefer to turn off compression
|
|
* set encryption to passphrase if desired<br>
|
|
this encryption prevents access to the data after shutdown,
|
|
nothing to do with sharing
|
|
* set Case sensitivity to `Insensitive` if windows will be accessing this dataset
|
|
* set Share Type to `SMB` if planning to share with SMB, which is the most used
|
|
way to share, especially for windows or mixed access
|
|
|
|
### Zvol
|
|
|
|
`Zvol` is a direct alternative to dataset.<br>
|
|
When planning to use iScsi with its approach of mounting network storage
|
|
as a block device.
|
|
This provides great speeds with small files, but at the cost of space.
|
|
|
|
</details>
|
|
|
|
---
|
|
---
|
|
|
|
<details>
|
|
<summary><h1>SMB share</h1></summary>
|
|
|
|
Should be go-to for most cases, as all systems(win, linux, mac,
|
|
android, ios) have mature reliable smb clients.
|
|
|
|
To see connected users, SSH in and `sudo smbstatus -b`
|
|
|
|
* Windows (SMB) Shares > Add
|
|
* set path to the dataset to share
|
|
* set the name under which it will be shared
|
|
* set Purpose if there is a special case
|
|
* on save the service will be enabled, if its not already
|
|
|
|
Now to deal with the permissions<br>
|
|
There are two type of permissions accessible through icons in the share view
|
|
|
|
* Share ACL - set to allow everyone by default
|
|
* Edit Filesystem ACL - where one actually wants to control permissions
|
|
|
|
Create smb user and allow the access to the share
|
|
|
|
* Credentials > Local Users > Add
|
|
* set user name, for example: smb_usr<br>
|
|
note the default UID for very first account added manually being `3000`
|
|
* set password
|
|
* switch to Shares > Edit Filesystem ACL (shield icon)
|
|
* in Edit ACL > Add Item > smb_usr
|
|
* set desired permissions
|
|
|
|
Trying to access the IP of truenas instance with the now set credentials
|
|
should allow full access to the share.
|
|
|
|
Worth noting that it's the UID number that identifies users,
|
|
not the username.
|
|
|
|
### SMB share for everyone
|
|
|
|
One might think that just allowing group `everyone@` access is enough.
|
|
But when someone connects to a share, there must be a username used.
|
|
For this a guest account needs to be enabled,
|
|
which under the hood is named `nobody`
|
|
|
|
* in Shares > Windows (SMB) Shares > edit the share
|
|
* Advanced Options > Allow Guest Access
|
|
|
|
### Mounting network share at boot
|
|
|
|
Using systemd. And the instructions from [arch wiki.](https://wiki.archlinux.org/title/samba#As_systemd_unit)
|
|
|
|
I prefer setting permissions in the unit
|
|
|
|
check your user `id $whoami` for uid and gid
|
|
|
|
`/etc/systemd/system/mnt-bigdisk.mount`
|
|
```ini
|
|
[Unit]
|
|
Description=12TB truenas mount
|
|
|
|
[Mount]
|
|
What=//10.0.19.11/Dataset-01
|
|
Where=/mnt/bigdisk
|
|
Options=rw,username=bastard,password=lalala,file_mode=0644,dir_mode=0755,uid=1000,gid=1000
|
|
Type=cifs
|
|
TimeoutSec=10
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
`/etc/systemd/system/mnt-bigdisk.automount`
|
|
```ini
|
|
[Unit]
|
|
Description=12TB truenas mount
|
|
|
|
[Automount]
|
|
Where=/mnt/bigdisk
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
</details>
|
|
|
|
---
|
|
---
|
|
|
|
<details>
|
|
<summary><h1>NFS share</h1></summary>
|
|
|
|
Linux to linux file sharing. Simple.
|
|
|
|
Before creation of nfs share, a linux permission group should be planned to use.<br>
|
|
Lets say that a group named `nfs` with gid `1100`
|
|
|
|
on linux client machine
|
|
|
|
- `sudo groupadd -g 1100 nfs` - create the group
|
|
- `sudo gpasswd -a bastard nfs` - add the user in to the group
|
|
- log out, log in, check with `id`
|
|
|
|
Now on truenas the new group is created and set for the dataset
|
|
and NFSv4 share is set.
|
|
|
|
* create nfs group with guid 1100<br>
|
|
Credentials > Local Groups > Add > GID = 1100; Name = nfs
|
|
* create new Dataset<br>
|
|
Datasets > Add Dataset > Name it; keep defaults
|
|
* set `nfs` group for this dataset root
|
|
Datasets > Permissions (scroll down, bottom right) > Edit<br>
|
|
Group = nfs; check `Apply Group`; check `Apply permissions recursively`<br>
|
|
Save
|
|
* switch NFS to version 4<br>
|
|
Shares > UNIX (NFS) Shares > three dots > Config Service<br>
|
|
check `Enable NFSv4`; check `NFSv3 ownership model for NFSv4`<br>
|
|
Save
|
|
* Set nfs share<br>
|
|
Shares > UNIX (NFS) Shares > Add<br>
|
|
pick path to the dataset<br>
|
|
Save
|
|
|
|
|
|
Test mounting on client machine, in my case arch linux machine,
|
|
[here](https://wiki.archlinux.org/title/NFS#Client) is wiki on nfs
|
|
|
|
* check you see the share `showmount -e 10.0.19.11`
|
|
* mount the share `sudo mount 10.0.19.11:/mnt/Pool-02/sun/ ~/temp`
|
|
* should work can check version using `nfsstat -m` or `rpcinfo -p 10.0.19.11`
|
|
|
|
### Mounting network share at boot
|
|
|
|
Using systemd. And the instructions from [arch wiki.](https://wiki.archlinux.org/title/NFS#As_systemd_unit)
|
|
|
|
`/etc/systemd/system/mnt-truenas.mount`
|
|
```ini
|
|
[Unit]
|
|
Description=Truenas 6TB in stripe
|
|
|
|
[Mount]
|
|
What=10.0.19.11:/mnt/Pool-02/sun
|
|
Where=/mnt/truenas
|
|
Options=vers=4
|
|
Type=nfs
|
|
TimeoutSec=10
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
`/etc/systemd/system/mnt-truenas.automount`
|
|
```ini
|
|
[Unit]
|
|
Description=Truenas 6TB in stripe
|
|
|
|
[Automount]
|
|
Where=/mnt/truenas
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
</details>
|
|
|
|
---
|
|
---
|
|
|
|
<details>
|
|
<summary><h1>iSCSI share</h1></summary>
|
|
|
|
[The official documentation.](https://www.truenas.com/docs/scale/scaletutorials/shares/iscsi/addingiscsishares/)
|
|
|
|
Sharing disk space as a block device over network.
|
|
Great perfromance, especially if lot of I/O small files stuff.
|
|
Only single client can work with the block device at once.
|
|
|
|
* **target** - a storage we want to make available over network
|
|
* **initiator** - a device connecting to a target
|
|
* **portal** - they say IP and port pair, but part of it is also authentication
|
|
*
|
|
|
|
both target and initiator must be assigned IQN - iSCSI Qualified Name<br>
|
|
name format: iqn.yyyy-mm.naming-authority:unique name<br>
|
|
examples:<br>
|
|
`iqn.2016-04.com.open-iscsi:4ab2905b66ca`<br>
|
|
`iqn.2005-10.org.freenas.ctl:garbage`<br>
|
|
`iqn.1991-05.com.microsoft:tester-81`<br>
|
|
|
|
|
|
assuming all sections (portals, Initators groups, Authgorized access, targets, extents,..) are empty and doing it first time
|
|
|
|
* create a new Zvol<br>
|
|
Datasets > Add Zvol button<br>
|
|
set Name; set Size, they recommend less than 80% of the pool but can be forced higher;
|
|
|
|
* click through iSCSI share wizzard or do the manual setup<br>
|
|
Shares > Block (iSCSI) Shares Targets > ...<br>
|
|
|
|
Manual setup
|
|
|
|
* Target Global Configuration<br>
|
|
nothing really worth changing
|
|
* Portals<br>
|
|
add some description and set IP of the truenas<br>
|
|
* Initiator<br>
|
|
add some description and for now check Allow All Initiators
|
|
* Authorized Access<br>
|
|
skip
|
|
* Targets<br>
|
|
set name; set portal group; set initiator group; authentication kept none
|
|
* Extents<br>
|
|
set name; device=some zvol; Logical Block Size=4096
|
|
* Associated Targets <br>
|
|
set target; LUN ID=0; set extent
|
|
|
|
Enable iSCSI service.
|
|
|
|
### Mounting the Share
|
|
|
|
**Windows**
|
|
|
|
* run `iscsicpl.exe` - iSCSI Initiator
|
|
* Set trueNAS ip as `Target`; Quick connect
|
|
|
|
This connects the share as block device and ads it to Favorite Targets,
|
|
meaning the share is remounted on boot.
|
|
|
|
**Linux**
|
|
|
|
Arch linux wiki has [detailed instructions](https://wiki.archlinux.org/title/Open-iSCSI)
|
|
|
|
* `sudo pacman -S open-iscsi` - install
|
|
* `sudo systemctl enable --now iscsi.service` - start and enable it on boot
|
|
* `sudo iscsiadm --mode discovery --portal 10.0.19.11 --type sendtargets` - discover shares
|
|
* edit `/var/lib/iscsi/nodes/../default` and set `node.startup = automatic`
|
|
* `sudo iscsiadm -m node -L all` - login to all available targets
|
|
|
|
Of note is a recent change of configs location from /etc/iscsi to /var/lib/iscsi
|
|
|
|
### Encryption setup using fs
|
|
|
|
[very well written arch wiki page](https://wiki.archlinux.org/title/Fscrypt)
|
|
|
|
* format the iscsi disk<br>
|
|
`sudo mkfs.ext4 -O encrypt /dev/sdb1`<br>
|
|
or enable it with `sudo tune2fs -O encrypt /dev/device`
|
|
* mount it lets say `/mnt/target1`
|
|
* install fscrypt<br>
|
|
`sudo pacman -S fscrypt`
|
|
* enable it on the system `fscrypt setup`
|
|
* enable it on the mounted partition `sudo fscrypt setup /mnt/target1`
|
|
* create a directory there as you cant encrypt root of a partition
|
|
* encrypt the directory `fscrypt encrypt /mnt/target1/homework`
|
|
* lock `fscrypt lock /mnt/target1/homework`
|
|
* lock `fscrypt unlock /mnt/target1/homework`
|
|
|
|
systemd mount files
|
|
|
|
`/etc/systemd/system/mnt-target1.mount`
|
|
```ini
|
|
[Unit]
|
|
Before=remote-fs.target
|
|
After=iscsi.service
|
|
Requires=iscsi.service
|
|
Description=iscasi test share
|
|
|
|
[Mount]
|
|
What=/dev/disk/by-uuid/58b83770-2c68-463e-9ea4-6f62ef8c001d
|
|
Where=/mnt/target1
|
|
Type=ext4
|
|
Options=_netdev,noatime
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
`/etc/systemd/system/mnt-bigdisk.automount`
|
|
```ini
|
|
[Unit]
|
|
Description=iscasi test share
|
|
|
|
[Automount]
|
|
Where=/mnt/target1
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
* `/etc/iscsi/nodes` - where targets are added
|
|
* `/etc/iscsi/initiatorname.iscsi` - machines id
|
|
* `/etc/iscsi/iscsid.conf` - general config
|
|
|
|
|
|
</details>
|
|
|
|
---
|
|
---
|
|
|
|
### Data protection settings
|
|
|
|
* enable autoamtic smart short tests<br>
|
|
Data Protection > S.M.A.R.T. Tests > Add > all disks/short/weekly
|
|
* enable autoamtic snapshots
|
|
|
|
Possible future chapters
|
|
|
|
* Testing access to ZFS disks on a desktop
|
|
* Reinstall and import of pools
|