TrueNAS SCALETrueNAS SCALE Nightly Development Documentation
This content follows experimental early release software. Use the Product and Version selectors above to view content specific to a stable software release.

Failover

The SCALE CLI guide is a work in progress! New namespace and command documentation is continually added and maintained, so check back here often to see what is new!

Failover Namespace

TrueNAS Enterprise
On licensed systems, use the Failover namespace to manage HA system controllers and failover operations for Enterprise customer. Systems without an Enterprise license do not use this namespace.

The failover namespace has 14 commands and is based on failover functions found in the SCALE API and web UI. It provides access to Enterprise HA customer management methods through the failover namespace commands.

Failover Commands

The following failover namespace commands allow you to perform failover operations for Enterprise customer systems.

You can enter commands from the main CLI prompt or from the failover namespace prompt.

Interactive Argument Editor (TUI)

Enter the -- flag following any CLI command to open the interactive arguments editor text-based user interface (TUI).

Click for more information

The interactive argument editor is a text user interface (TUI) that can help enter complex commands with multiple configurable properties. It shows expected properties, defaults, input types (string, boolean, integer, or array), and can include command instructions or warnings.

Optional properties, indicated by the # symbol, are disabled by default. Required properties are enabled. Do not disable properties that are enabled by default.

To configure required properties, enter a space after the colon then add the value.

To enable optional properties, delete # from the corresponding line.

Some required properties are disabled if they are part of a pair of properties where one or the other is required. Select one property to enable and enter a value.

Press F2 or click Save to save the modified file.

Press F10, Esc, or click Quit to exit the TUI. The command automatically executes upon exit.

Config Command

The config command displays failover configuration setting (ID, disabled status, timeout, and master/slave) information.

Using the Config Command

Description

config does not require entering properties or arguments. Enter the command then press Enter. The command returns a table with the id, disabled, timeout and master/slave status.

Usage

From the CLI prompt, enter:

system failover config

Command Example
system failover config
+----------+-------+
|       id | 1     |
| disabled | false |
|  timeout | 0     |
|   master | true  |
+----------+-------+

Control Command

The control command provides enabled/disabled status of failover control.

Using the Control Command

Description

control has two required properties, action and options. action has two values, ENABLE and DISABLE. options has one property argument, active with the default set to true. Enter the options default options={} to determine if control is active or disabled. Enter the command string then press Enter. The command returns true if enabled, false if not.

Usage

From the CLI prompt, enter:

system failover control action=<i>ENABLE</i> options={}

Where ENABLED is the failover control state.

Command Example
system failover control action=ENABLE options={}
false

Disabled Command

The disabled command returns a list of reasons if the system failover is disabled.

Using the Disabled Command

Description

disabled has one require property, reasons, but is not entered as an argument. Enter the command string then press Enter. The command returns a list of disabled reasons, or an empty list if the system is not disabled.

Disabled Reasons
ReasonDescription
NO_VOLUMEThere are no pools configured.
NO_VIPThere are no interfaces configured with Virtual IP. NO_SYSTEM_READY - Other storage controller has not finished booting.
NO_PONGOther storage controller is not communicable.
NO_FAILOVERFailover is administratively disabled.
NO_LICENSEOther storage controller has no license.
DISAGREE_VIPNodes Virtual IP states do not agree.
MISMATCH_DISKSThe storage controllers do not have the same quantity of disks.
MISMATCH_VERSIONSTrueNAS software versions do not match between storage controllers.
NO_CRITICAL_INTERFACESNo network interfaces are marked critical for failover.
NO_FENCEDZpools are imported but fenced isn’t running.
LOC_FAILOVER_ONGOINGThis node is currently processing a failover event.
REM_FAILOVER_ONGOINGOther node is currently processing a failover event.
NO_HEARTBEAT_IFACELocal heartbeat interface does not exist.
NO_CARRIER_ON_HEARTBEATLocal heartbeat interface is down.
(empty list)Returned if failover is enabled and functional.

Usage

From the CLI prompt, enter:

system failover disabled reasons

Command Example
system failover disabled reasons
(empty list)

Force_Master Command

The force_master command forces the current controller to become master if it is the standby.

Using the Force_Master Command

Description

force_master does not require entering properties or arguments. Enter the command then press Enter. The command returns true if the standby controller becomes master, false if the controller is already the master..

Usage

From the CLI prompt, enter:

system failover force_master

Command Example
system failover force_master
false

Get_Ips Command

The get_ips command returns a list of IP addresses configured for the failover service but not details on what they are assigned to.

Using the Get_Ips Command

Description

get_ips does not require entering properties or arguments. Enter the command then press Enter. The command returns the configured IP addresses but not details on what they are assigned to.

Usage

From the CLI prompt, enter:

system failover get_ips

Command Example
system failover get_ips
http://10.220.148.12
http://10.220.16.85
https://10.220.148.12
https://10.220.16.85

Hardware Command

The hardware command returns the failover configuration information.

Using the Hardware Command

Description

hardware does not require entering properties or arguments. Enter the command then press Enter. The command returns the hardware information.

Usage

From the CLI prompt, enter:

system failover hardware

Command Example
system failover hardware
ECHOWARP

In_Progress Command

The in_progress command provides verification of an in-progress failover event.

Using the In_Progress Command

Description

in_progress does not require entering properties or arguments. Enter the command then press Enter. The command returns true if the failover is in progress, false if not.

Usage

From the CLI prompt, enter:

system failover in_progress

Command Example
system failover in_progress
false

Licensed Command

The licensed command checks whether this system is licensed as an HA system.

Using the Licensed Command

Description

licensed does not require entering properties or arguments. Enter the command then press Enter. The command returns true if the system is licensed as an HA system, false if not.

Usage

From the CLI prompt, enter:

system failover licensed

Command Example
system failover licensed
true

Node Command

The node command returns the controller (node) status.

Using the Node Command

Description

node does not require entering properties or arguments. Enter the command then press Enter. The command returns the controller (node) designation (A or B). A is controller 1 or the primary, B is controller 2 or the standby.

Usage

From the CLI prompt, enter:

system failover node

Command Example
system failover node
A

Status Command

The status command returns the current HA status as either master or slave.

Using the Status Command

Description

status does not require entering properties or arguments. Enter the command then press Enter. The command returns MASTER for the primary controller, or SLAVE for the standby controller.

Usage

From the CLI prompt, enter:

system failover status

Command Example
system failover status
MASTER

Sync_From_Peer Command

The sync_from_peer command sync failover configuration settings from the other controller. This command is the same as the Sync From Peer function on the Failover Screen.

Using the Sync_From_Peer Command

Description

sync_from_peer does not require entering properties or arguments. Enter the command then press Enter. The command returns and emnpty line.

Usage

From the CLI prompt, enter:

system failover sync_from_peer

Command Example
system failover sync_from_peer

Sync_To_Peer Command

The sync_to_peer command initiates a sync operation that copies over the primary controller configuration to the standby controller.

This command is the same as the Sync To Peer function on the Failover Screen.

Using the Sync_To_Peer Command

Description

sync_to_peer has one required property, options. options has one required property argument, reboot. reboot is either true to reboot after syncing to peer, or false to not reboot. Enter the command then press Enter. The command returns an empty line.

Usage

From the CLI prompt, enter:

system failover sync_to_peer options={}

Command Example
system failover sync_to_peer

Upgrade_Pending Command

The upgrade_pending command verifies if HA upgrade is pending.

Use upgrade_finish to finish the HA upgrade process if this command returns true.

Using the Upgrade_pending Command

Description

upgrade_pending does not require entering properties or arguments. Enter the command then press Enter. The command returns true if the there is a pending upgrade, false if not.

Usage

From the CLI prompt, enter:

system failover upgrade_pending

Command Example
system failover upgrade_pending
false

Upgrade_Finish Command

The upgrade_finish command perform the last stage of an HA upgrade and activates the new boot environment on the standby controller after it reboots.

Using the Upgrade_Finish Command

Description

upgrade_finish does not require entering properties or arguments. Enter the command then press Enter.

Usage

From the CLI prompt, enter:

system failover upgrade_finish

Command Example
system failover upgrade_finish
[0%] ...
[0%] Ensuring the Standby Controller is booted...
[0%] Activating new boot environment...