By LeanCloud Wang Zi Ting

In 2016, I got the first NAS, THE DS215J of Qunhui. In fact, it didn’t come into much use for a long time after that, because I didn’t have much data, most of which was stored in the cloud, and I just wanted to experience the functions and workflow of NAS.

Until recently, I didn’t really make use of NAS, so I planned to upgrade it. However, considering that the cost performance of QUNhui is too low, and the configuration of Linux soft routing last year increased my confidence and interest in the open source solution based on “native Linux”, I decided to build a NAS by myself. Plan to address storage needs for the next ten years.

I still chose Ubuntu as my most familiar operating system and Ansible as my configuration management tool, so most of the configuration for this NAS can be found on my GitHub.

Note that the Ansible configurations in this repository are for reference only and are not recommended to run directly because I wrote them without sufficient consideration for compatibility and portability.

The file system

The most important thing for a NAS is, of course, the file system. ZFS doesn’t take much research to find — probably the single file system that puts the most effort into data reliability right now, so my whole selection revolves around ZFS.

ZFS is both a file system and an array (RAID) manager, which gives it some capabilities that other file systems cannot provide:

  • ZFS stores checksums for each block, while periodically scanning the entire drive to repair accidentally corrupted data (such as bit flips caused by cosmic rays) from other drives in the RAID.
  • On the basis of RAID, you can specify certain directories to store more copies of important data. Even if damaged hard drives exceed the limit of RAID, important data can still be retrieved.

ZFS also supports data encryption, compression, and deduplication, all of which work in a neat sequence that doesn’t conflict with each other, and all of which can be set at the directory level and changed at any time (only for new data).

Of course, ZFS also supports snapshots, which can be exported as binary streams and stored anywhere. This feature allows you to back up, transfer, and restore ZFS filesystems without losing any meta information.

hardware

I am not good at hardware, so I chose HPE MicroServer Gen10, a four-disk finished MicroServer, CPU is AMD X3421, 8G ECC memory, is also the standard x86 general hardware, should not be easy to encounter pits.

I installed an NVME SSD in the PCI-E slot with the adapter card and used it as a read cache for the system disk and ZFS (L2ARC, but not as much as I can see from the stats below), while the data disk was used as an older disk for the time being and was eventually upgraded to four 4T disks. One thing to note here is that ZFS does not support changing the structure of RAID, so you had to configure enough hard drives to fill space in the beginning and upgrade capacity later. I even used a USB drive to make up the difference.

ZFS

Since it is a four-disk bit, I used RAIDZ1 (RAID5), with one redundant disk as verification. If all disks are upgraded to 4TB eventually, the total available capacity is 12TB.

root@infinity:~# zpool status
  pool: storage
 state: ONLINE
config:
    NAME                                 STATE     READ WRITE CKSUM
    storage                              ONLINE       0     0     0
      raidz1-0                           ONLINE       0     0     0
        sda                              ONLINE       0     0     0
        sdb                              ONLINE       0     0     0
        sdc                              ONLINE       0     0     0
        sdd                              ONLINE       0     0     0
    cache
      nvme0n1p4                          ONLINE       0     0     0

root@infinity:~# zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT   FRAG    CAP  DEDUP    HEALTH
storage  7.27T  3.52T  3.75T        -    10%    48%  1.00x    ONLINE
Copy the code

RAID5 is generally considered to have a high risk of data loss due to the failure of a second disk. Or the data on the hard disk is corrupted due to bit reversal over time. However, considering that ZFS regularly performs data verification to ensure the correctness of data, and considering the number and capacity of disks, I think this risk is acceptable. Remote backup will also be mentioned later as a backstop measure.

I turned on ZFS encryption, but this created a problem: I couldn’t store the key in plaintext on the NAS system disk — otherwise the encryption would be meaningless if the key and ciphertext were placed together. Therefore, every time THE NAS restarts, I need to enter the password, mount the ZFS dataset, and then start other services that depend on the storage pool.

I also turned on data compression for ZFS. The default LZ4 uses a small amount of CPU but can improve IO performance in some cases because there is less data to read. I did not enable deduplication because deduplication requires high resource requirements, equivalent to creating an index for the entire disk to find duplicate blocks.

Some comments suggest that ZFS has high memory requirements and ECC memory must be used. This is a misconception: more memory can improve ZFS performance, and ECC can avoid memory errors for all applications in the system, but it is not necessary. Even without more memory or ECC, ZFS has the same performance and data integrity guarantee as other file systems.

Storage service

Fact: SMB is the most widely used LAN file sharing protocol. It has built-in support in all major operating systems. CIFS is a Microsoft (Windows) implementation of SMB, and Samba, which we will use, is another free software implementation of the SMB protocol.

As the core function of NAS is to provide storage services through SMB protocol, all finished NAS have rich options to configure SMB functions, but we have to directly edit the configuration file of Samba, Samba directly adopts the user and file permissions mechanism of Linux. It’s not too much trouble to configure:

[Home] path = /storage/private/homes/%U writeable [Home] path = /storage/private/homes/%U writeable = yes valid users = @staff # Samba creates files as login users by default, but NextCloud runs as www-data, [NextCloud] path = /storage/ NextCloud /data/%U/files writeable = yes valid users = @staff force User = WWW -data # With these Settings, macOS TimeMachine can also be backed up via SMB https://www.reddit.com/r/homelab/comments/83vkaz/howto_make_time_machine_backups_on_a_samba/ [TimeMachine] path = /storage/backups/timemachines/%U writable = yes valid users = @staff durable handles = yes kernel oplocks = no kernel share modes = no posix locking = no vfs objects = catia fruit streams_xattr ea support = yes inherit acls = yes Fruit :time machine = yes # For shared directories, you can use force group to overwrite the file's owning group, use create mask to overwrite the file's permission bit [VideoWorks] path = /storage/shares/VideoWorks writeable = yes valid users = @staff force group = staff create mask = 0775 # Visitors can read, can also set specified user group writable directory [Resources] public path = / storage/public/Resources guest ok = yes write list = @ staff force group = +staff create mask = 0775Copy the code

The shared directories are distributed in several different paths. In order to match different data types and facilitate the configuration of the directory level, I have divided several datasets:

  • dbStore the application database file and set recordSize to 8K (default 128K).
  • nextcloudThe NextCloud’s data directory is also accessible to SMBS.
  • privatePersonal files for each user.
  • sharesFiles (such as a video shot) shared within the family.
  • publicFiles that can be downloaded from the Internet do not participate in remote backup.
  • backupsBackup (Time Machine, etc.), do not participate in remote backup.
root@infinity:~# ZFS list NAME USED AVAIL REFER MOUNTPOINT Storage 2.27T 286G 169K /storage storage/backups 793G 286G 766G /storage/backups storage/db 741M 286G 339M /storage/db storage/nextcloud 207G 286G 207G /storage/nextcloud Storage/Private 62.2g 286G 62.2g /storage/ Private Storage/Public 648G 286G 613G /storage/ Public Storage/Shares 615G 286G  609G /storage/sharesCopy the code

application

First, I installed Netdata, an out-of-the-box monitoring tool that provides a large number of statistics with second-of-a-second accuracy with very few resources, perfect for monitoring performance bottlenecks on a single server.

I run the rest of the applications in Docker (docker-compose is used for management), which can isolate the operating environment of the application, improve the stability of the host computer, and make it more convenient to install, upgrade and uninstall the application.

One of the most important apps is NextCloud, which is an open source sync disk. I mainly like its iOS app and iOS integration, which can sync Live Photo correctly and can be called within the iOS file app.

The NextCloud server reads and writes files directly from the file system rather than storing them in a database, which means that the NextCloud data directory is also accessible through Samba, This is handy (though it requires a scheduled task to refresh the meta-information in the NextCloud database).

I also run these services in Docker, which are open source:

  • Miniflux, an RSS server, supports most RSS clients through Fever API.
  • Bitwarden (unofficial implementation), a password manager, provides clients and browser plug-ins for each platform.
  • Transmission, a BitTorrent client, provides a Web-based management interface.

External access

If you really want to use NAS instead of a web disk, you still need to be able to access files when you are not on your home Intranet.

The common practice is to use DDNS (dynamic DNS) to resolve a domain name to the IP of the home broadband, which requires that the home broadband has a public IP and the carrier allows Web services on port 80 or 443. I don’t want to rely on this, so I came up with the idea of using FRP for “reverse proxy”, or DDNS if you do have a public IP, which saves a relay server and gives you better speed.

In order to make NextCloud can have a fixed address (like https://nextcloud.example.com) I will domain name resolution in internal and external network respectively, resolution to the Intranet address at home, to parse out the transit server. Whether Intranet or extranet, data flows are encrypted by SSL using Let’s Encrypt, which eliminates the need for high security from the transfer server.

Although it is convenient to open NextCloud to the public network without having to dial a VPN first, it is not safe to open NextCloud on the public network. Some users in the community have requested that the NextCloud client support bidirectional SSL authentication. I am also looking forward to this feature, which can provide better security for public network access.

I also installed WireGuard on the NAS, which is a VPN module built into the Linux kernel and also exposed to the exet through FRP, for services other than NextCloud, Examples such as SMB, SSH, and Nextdata can all be accessed through WireGuard.

If you’re not committed to an open source solution, ZeroTier offers NAT penetration, allowing direct transmission between your device and NAS without a relay server, improving connection speeds.

Backup and data integrity

Based on RAIdz1, I set up a scheduled task for ZFS to generate a snapshot per day, and wrote a script to clean up backups using Time Machine-like rules: keep the daily snapshot of the last week, the weekly snapshot of the last month, the monthly snapshot of the last year, and the annual snapshot.

root@infinity:~# zfs list storage/nextcloud -t snapshot NAME USED AVAIL REFER MOUNTPOINT storage/nextcloud@2020-09-05 83.9m-182G-storage /nextcloud@2020-09-15 35.2m-207g-storage /nextcloud@2020-09-21 30.2m-207g - Storage /nextcloud@2020-09-23 29.7m-207g-storage /nextcloud@2020-09-26 29.3m-207g-storage /nextcloud@2020-09-27 28.2-207g-storage /nextcloud@2020-09-28 28.2-207g-storage /nextcloud@2020-09-29 29.1-207g - Storage /nextcloud@2020-09-30 33.5M-207G -Copy the code

Snapshots are designed to prevent manual error, and aside from simple, hand slips that can be spotted on the spot, sometimes you mistakenly delete a file thinking you don’t need it until much later.

At the same time, there is a weekly scheduled task to use Restic to back up a snapshot to Backblaze B2 as a remote backup, which is an inexpensive object storage, ideal for backup. Restic supports incremental snapshot backup as well as encryption. For cost reasons, the remote backup only includes data generated by me, not public and backups.

ZFS send/RECv supports sending a snapshot as a binary stream — without having to install any other tools remotely. Simply redirect the byte stream of ZFS send to the SSH command using the shell’s pipe operator. This was a very technically beautiful solution, but was abandoned considering that block storage was ten times more expensive than object storage.

Cost accounting

In fact, my budget is not tight on hardware, and the margin is relatively large. If I change some hardware with higher cost performance, the price can be reduced a lot.

  • Host (main board, CPU, memory, system disk) 3500 yuan
  • Hard disk (4 * 4T) 2200 YUAN (actually only bought one, the other three are old)

Considering that my previous Qunhui was used for five years, the service life of the new NAS design is set at ten years:

  • The hardware costs 570 yuan per year
  • Electricity (35W) 110 yuan per year
  • Remote access 100 yuan per year (domestic annual payment promotion server, if there is a public IP using DDNS is not required)
  • Remote backup RMB 415 per year (charge by volume, calculated according to 1T of data to be backed up in different places)

A total of 12 terabytes of capacity costs 1195 yuan per year, or 8 yuan per month for 1 terabyte, or 5 yuan per month for 1 terabyte if remote access and remote backup are excluded.

Why self-deploy

The first reason to use cloud services is, of course, the “feeling of control” over your data, and while there’s no hard reason why cloud services shouldn’t be secure, some people just like the feeling of control over their personal data.

Another technical reason is that NAS deployed on the home network can easily support some “online editing” through SMB, such as directly loading materials on the NAS for video clips, or even putting entire project files directly on the NAS. With cloud services, on the one hand, there is no SMB protocol support, and even support for latency is unacceptable for online editing.

Another topic that can’t be ignored is cost. In this case, we only consider the volume-based pricing schemes for web disk services. ICloud, Google Drive and Dropbox all have very similar pricing schemes. In more than 200 g (3) about this after a jump directly to the 2 t (3) about this after a jump directly to the 2 t (3) about this after a jump directly to the 2 t (10), then the cloud service volume, the advantage of paying actually has not had, is a switch to the deployment scheme of a good point in time, The one-time investment only takes 2-3 years to recoup.

Of course, the most important thing is interest, and there are a lot of decisions to be made, a lot of difficulties to be faced, and an almost unique self-deployment solution to be built. If you can find fun in the process, it’s definitely worth it; On the other hand, if you have no interest, self-deployment solutions will be very cost-effective, including the time investment.