Backup and restore via NFS

This recipe describes preparing NFS for the server commands export nfs and import nfs: cluster requirements, access rights, step-by-step configuration of the NFS server and clients, feature flags, performance, and diagnostics. See the reference for command syntax: export to NFS, import from NFS.

Prerequisites

The export nfs / import nfs operations are performed by server processes YDB, not by the CLI client. This means the file system must be accessible from each cluster node:

The NFS directory must be mounted on all nodes of the YDB cluster at the same absolute path.
The YDB process must have write permissions for export and read permissions for import to this directory.
The --fs-path path must be absolute. Relative paths, paths containing .. or ., and paths with null bytes will be rejected by the server.
The functionality must be enabled using the feature flag enable_fs_backups.

Warning

This functionality is compatible with the NFSv4 network file system protocol.

Access rights

File operations are performed by server processes YDB, not by the CLI client. Therefore, permissions to the NFS directory must be granted to the OS user under which the ydbd process runs.

POSIX permissions:

For export (writing files): rwx on the directory (read for traversal, write for creating files and subdirectories, execute for navigation) and rw on files.
For import (reading files): rx on the directory (read for traversal, execute for navigation) and r on files.

NFSv4 ACL:

NFSv4 uses an extended permission model instead of POSIX rwx. The same letter in an NFSv4 ACE has different meanings for files and directories:

Letter	For files	For directories
`r`	`read-data` — read content	`list-directory` — view content
`w`	`write-data` — write data	`create-file` — create files
`a`	`append-data` — append data	`create-subdirectory` — create subdirectories
`x`	`execute` — execute	`change-directory` — navigate
`D`	—	`delete-child` — delete child objects
`t`	`read-attributes`	`read-attributes`

Permissions on the NFS directory for export (creating files and subdirectories):

On the directory itself: w (create-file), a (create-subdirectory), x (change-directory), t (read-attributes), D (delete-child).
On created files: w (write-data), a (append-data), t (read-attributes).

Permissions on the NFS directory for import (reading files):

On the directory itself: r (list-directory), x (change-directory), t (read-attributes).
On files: r (read-data), t (read-attributes).

Since files are created by the ydbd process during export, permissions on them cannot be set in advance. In NFSv4, ACE inheritance is used for this — the f (file-inherit) and d (directory-inherit) flags on the parent directory's ACE. When a new file or subdirectory is created, the ACE with these flags is automatically copied to the new object.

Step-by-step setup NFS-server and clients

Below are instructions for configuring the NFS server and YDB client nodes for two permission management options: standard POSIX permissions and extended NFSv4 ACL.

Option 1: POSIX permissions

Configuring the NFS server

Install the NFS server package:
```
sudo apt install nfs-kernel-server
```
Create a directory for export and assign the owner:
```
sudo mkdir -p /mnt/ydb-backup
sudo chown ydb:ydb /mnt/ydb-backup
sudo chmod 755 /mnt/ydb-backup
```
Options of the chmod command:
- 7 (rwx) — the owner (ydb) can read, write, and enter the directory.
- 5 (r-x) — the group and others can read and enter, but not write.
Add the directory to /etc/exports:
```
/mnt/ydb-backup ydb-node-*.example.com(rw,sync,root_squash)
```
Export options:
- rw — allow read and write.
- sync — synchronous write (data is written to disk before acknowledging the client).
- root_squash — map root on the client to an anonymous user.
Apply the changes and start the NFS server:
```
sudo exportfs -arv
sudo systemctl enable --now nfs-server
```
Options of exportfs:
- -a — export all directories from /etc/exports.
- -r — re-export all directories (synchronization with /etc/exports).
- -v — verbose output.

Configuring clients (YDB nodes)

Install the NFS client package:
```
sudo apt install nfs-common
```
Create a mount point:
```
sudo mkdir -p /mnt/ydb-backup
```
Mount the NFS directory:
```
sudo mount -t nfs4 -o rw nfs-server.example.com:/mnt/ydb-backup /mnt/ydb-backup
```
Mount options:
- -t nfs4 — NFSv4 file system type.
- -o rw — mount with read and write permissions.
In this example, the host server does not belong to the YDB cluster, but backup and restore will work correctly if one of the cluster hosts is designated as the NFS server.
To enable automatic mounting at boot, add a line to /etc/fstab:
```
nfs-server.example.com:/mnt/ydb-backup /mnt/ydb-backup nfs4 rw,hard,timeo=600,retrans=2,_netdev 0 0
```
fstab options:
- rw — read and write.
- hard — retry requests indefinitely if the server is unavailable (do not lose data).
- timeo=600 — request timeout in tenths of a second (60 seconds).
- retrans=2 — number of retries before reporting an error.
- _netdev — mount only after network initialization.

Verification

On the server (or any client), create a test file as user ydb:

sudo -u ydb touch /mnt/ydb-backup/test-file

On all client nodes, verify that the file is visible:

sudo -u ydb ls /mnt/ydb-backup/test-file

After verification, delete the test file:

sudo -u ydb rm /mnt/ydb-backup/test-file

Option 2: NFSv4 ACL

NFSv4 ACLs allow you to set more granular access permissions and configure permission inheritance for newly created files and directories.

Configuring the NFS server

Install the NFS server package and utilities for working with NFSv4 ACLs:
```
sudo apt install nfs-kernel-server nfs4-acl-tools
```
Create a directory for export:
```
sudo mkdir -p /mnt/ydb-backup-server
```

Add the directory to /etc/exports:

/mnt/ydb-backup-server ydb-node-*.example.com(rw,sync,no_subtree_check,root_squash)

Apply the changes:

sudo exportfs -arv
sudo systemctl enable --now nfs-server

Configuring clients (YDB nodes)

Unlike POSIX permissions, which are set on the NFS server, NFSv4 ACLs are configured on each client after mounting the directory. If the NFS server is also a YDB cluster node, all the steps described below must be performed on it.

Install the NFS client packages and utilities for working with NFSv4 ACLs:
```
sudo apt install nfs-common nfs4-acl-tools
```
Create a mount point:
```
sudo mkdir -p /mnt/ydb-backup
```
Mount the NFS directory using the NFSv4 protocol:
```
sudo mount -t nfs4 -o rw nfs-server.example.com:/mnt/ydb-backup /mnt/ydb-backup
```
Options:
- -t nfs4 — explicitly specify the NFSv4 protocol (required for correct ACL operation).
- -o rw — mount with read and write permissions.

To enable automatic mounting, add a line to /etc/fstab:

nfs-server.example.com:/mnt/ydb-backup-server /mnt/ydb-backup nfs4 rw,hard,timeo=600,retrans=2,_netdev 0 0

Determine the UID of user ydb:
```
id -u ydb
```
Example output: 1871
Configure NFSv4 ACLs on the directory:
```
sudo nfs4_setfacl -s A::<uid>:rwaxtD,A:fi:OWNER@:rwat,A:di:OWNER@:rwaxtD /mnt/ydb-backup
```
ACE structure (A:flags:who:permissions):
- A — ACE type: Allow.
- Inheritance flags:
  - f (file-inherit) — inherit to created files.
  - d (directory-inherit) — inherit to created subdirectories.
  - i (inherit-only) — apply only to descendants, not to the directory itself.
- who — who the permissions are granted to:
  - 1871 — UID of user ydb.
  - OWNER@ — object owner.
- Permissions:
  - r — read-data / list-directory.
  - w — write-data / create-file.
  - a — append-data / create-subdirectory.
  - t — read-attributes.
  - D — delete-child.
This command sets:
- Permissions for user ydb on the directory itself.
- Inherited permissions for files and subdirectories (fi, di), which will apply only to descendants.
- Owner permissions (OWNER@) with inheritance to files and directories.

Verification

Check the configured ACLs:

nfs4_getfacl /mnt/ydb-backup

Create a test file as user ydb:

sudo -u ydb touch /mnt/ydb-backup/test-file

Verify that the file inherited the ACL:

nfs4_getfacl /mnt/ydb-backup/test-file

On all client nodes, check access:

sudo -u ydb ls /mnt/ydb-backup/test-file

After verification, delete the test file:

sudo -u ydb rm /mnt/ydb-backup/test-file

Feature flags

To use export and import via NFS, you need to enable the enable_fs_backups feature flag in the cluster configuration. By default, this flag is disabled.

Performance configuration

Two mechanisms affect the performance of export and import operations via NFS: the resource broker (manages the parallelism of scan and data load tasks) and the actor system IO pool (handles file write and read operations).

Resource broker

Export and import via NFS use the same resource broker queues as S3: queue_backup for export and queue_restore for import. Increasing the limits speeds up operations but increases CPU load. Default values and configuration method are described in the resource_broker_config section.

Actor system IO pool

File operations (write during export, read during import) are performed by actors in the pool of the actor system. By default, this pool contains 1 thread. Since file system operations are blocking, the IO pool can become a bottleneck during intensive export or import.

To increase the number of threads in the IO pool, change the parameter in the actor system configuration:

actor_system_config:
  executor:
  - name: IO
    threads: 3
    time_per_mailbox_micro_secs: 100
    type: IO

Tip

Recommendations for choosing the number of IO pool threads when using NFS export/import:

Start with 2 threads and monitor the load.
Keep in mind that the IO pool handles not only NFS operations but also other blocking tasks (e.g., writing logs).
Monitor the pool load via the YDB UI.

Logging

The progress of export and import operations is reflected in the YDB server logs. The main logging components are:

EXPORT — logging of the export process on SchemeShard: creating directories, copying tables, starting backup tasks, loading the schema.
IMPORT — logging of the import process on SchemeShard: reading metadata, creating tables, restoring data.
FS_WRAPPER — logging of file operations on each node. This component records I/O errors (file unavailability, insufficient space, locking errors), which helps identify issues with NFS mounting on specific nodes.

When diagnosing issues, it is recommended to:

Check the operation status via ydb operation get.
Search the server logs for entries with components EXPORT or IMPORT and the operation ID.
For file write or read errors, search the logs for entries with component FS_WRAPPER — they contain the file path and error description, which helps determine on which host and with which file the problem occurred.
Verify that the NFS directory is accessible and has enough free space on all cluster nodes.

Limitations

Export and import via NFS support the same object types as export to S3. A detailed list of supported objects is provided in the export command documentation.
Import always creates objects from scratch — you cannot import into existing tables.
The --list parameter for listing export objects is available only for import from S3 in the current version.
The operation is not supported on the Windows platform.

Was the article helpful?

Transfer — streaming NGINX access logs to a table

Streaming queries