Stone Computers Knowledgebase

Stone Equinox / Storage Spaces Direct (S2D) Hyper-V Cluster UPS Shutdown Script

Article ID: 820
Last updated: 20 Oct, 2020
Article ID: 820
Last updated: 20 Oct, 2020
Revision: 25
Views: 78
Posted: 24 Sep, 2020
by Andrew Sharrad
Updated: 20 Oct, 2020
by Andrew Sharrad

Stone Equinox / Storage Spaces Direct (S2D) Hyper-V Cluster UPS Shutdown Script

Storage Spaces Direct (S2D) nodes provide disk resiliency to each other as part of the storage network.

When an extended power outage occurs that exceeds the available battery runtime, it is important that a managed shutdown process is used:

  • To ensure that all virtual machines are saved or shutdown
  • To ensure that nodes do not start inadvertently start moving virtual machine loads between nodes
  • To ensure that any individual node does not switch off until all cluster workload has ceased - this ensures that the S2D disk storage is consistent and not damaged

A software package is now available to provide this shutdown facility. It is compatible with all UPS software that supports shutdown via a script.

See:

Script Installation Process

Pre-Requisites

A. Install your UPS Software as normal

Its a good idea to install your UPS Software first, as depending on the type of software installed, the software will be configured for you by the installation script.

B. Manual creation of a domain service account. Optional: If you do not follow this step to create a S2D Management domain account called svcUPs, the installation script will do it for you.

  • In your S2D Management domain, create a domain service account called svcUPS
    • This account should have domain administrator rights, as it needs to be able to control the cluster
      • Either copy the existing domain administrator account, or make a fresh, new account, and add it the Domain Admins group
    • Make sure the account is set so that the password never expires
    • Leave the account object in the Users container, or somewhere else unaffected by group policy
    • Set a secure password on the account and make a note of it

Installation on each S2D Host

  • Download the script and extract it to a suitable folder, for example C:\_STONE\S2DUPSShutdown
    • Do not install or extract to a network location
    • Do not install or extract to Cluster Storage (i.e. C:\Clusterstorage\...)
    • Do not install to a User specific folder (i.e. Documents, C:\Users\...)
    • We recommend that the folder path or name does not have any spaces in it
  • You should have 3 files in the folder
    • Install.CMD
    • Install_S2D_UPS_Shutdown_Script_(S2DUSS).ps1
    • ShutdownControl.txt
  • ShutdownControl.txt contains the shutdown settings. These can be customised but this is not normally necessary. See settings for more information.
  • Double click on Install.CMD. This starts the Install_S2D_UPS_Shutdown_Script_(S2DUSS).ps1 file and may prompt for administrator approval
  • If you have already created the svcUPS account, enter in the svcUPS password when prompted

  • Alternatively, if the account does not exist, you will be offered to create a new account with a new complex password
  • Type in the new password. The account will be created in the S2D Management Domain Active Directory

  • When you have supplied the password for the existing or new account, the script will complete the tool installation.
  • When it completes, note the exact name and location of the CMD file that the UPS software must call.
    • Copy the CMD filename into your clipboard to make it easier to use later
    • In the example below, the filename is C:\_STONE\S2DUPSShutdown\StartShutdown.CMD

Note: If Eaton Intelligent Power Protector (IPP) or Eaton Intelligent Power Manager (IPM) is running, the script will offer to configure the Eaton settings for you. See Configuring the UPS Software.
  • The installer will have renamed or created additional files:
    • StartShutdown.CMD (newly created) - this is the file that the UPS software must run
    • ShutdownParser.PS1 - this is used by StartShutdown.CMD to start the pre-installed Task from Task Scheduler
    • ShutdownControl.CMD (this was previously ShutdownControl.TXT) - this does the heavy lifting of the shutdown procedure
    • And a Logs directory, for debugging shutdown events
  • Remember to complete the script installation on all Nodes in the cluster, as well as UPS software installation and configuration

Verifying Installation

Check for a scheduled Task called UPShutdown in the \MANAGEMENT folder, in Task Scheduler

Configuring the UPS Software

Automatic Configuration

You should install your UPS software before running the UPS Shutdown script installer, as the script installer may be able to configure your software for you.

After installation, we recommend that manually check your UPS Software shutdown settings

Software Supported for Automatic Configuration

  • Eaton Intelligent Power Protector
  • Eaton Intelligent Power Manager
  • Riello Powershield 3
  • APC Powerchute Network Shutdown

Manual Configuration / Verifying your UPS Software Shutdown settings

  • Then, under the shutdown configuration, select that a script is used.
  • You must specify that a shutdown duration of 600 seconds (10 minutes) is used. This lets the UPS software allow plenty of UPS battery runtime to ensure the systems shutdown cleanly.

Reminder: You must configure the UPS software to use the shutdown script (for example C:\_STONE\S2DUPSShutdown\StartShutdown.CMD) and a Shutdown Duration of at least 600 seconds

Testing the System

You should arrange full systems downtime to test the shutdown sequence. You can then test the software either from the UPS software, or by running StartShutdown.CMD from an Administrative command prompt

  • Check the Logs directory later for Cluster and Host shutdown events
  • You can also check the Application log under Stone S2D UPS Shutdown Script

Recommended UPS Hardware Installation

  • When using Intel based servers, do not install UPS in parallel. For example, if you have two UPS and two nodes, dedicate one UPS to one Node
    • This is because of the active/passive nature of the system power supplies
    • In the event of one UPS battery lasting significantly longer than others, the Cluster (and all thus VMs) will still be shutdown by the first node triggered for shutdown
  • Always ensure your network switches between the Hosts and the UPS Network Management card are also UPS protected
  • Always periodically test your batteries using the run-time calibration feature
  • Batteries that are 3 years old or more may require replacement

Shutdown Settings

All of the settings in ShutdownControl.txt / ShutdownControl.PS1 are shown below.

Concepts:

  • LocalVMs: Virtual machines running on the local node. These may or may not be clustered. Clustered VMs are shutdown first
  • Save or shutdown: By default, Local VMs will be Saved unless they have a memory demand of more than 8192MB, in which case they will be shutdown
  • The starting of shutting down or saving of local VMs will be staggered over 2 minutes
  • When the script starts, it sets up a separate cluster shutdown job that will start after 2 minutes, during which time the local virtual machines will be saved or shutdown
  • During this time, in most situations, the same script running on other nodes will be saving or shutting down VMs local to those other nodes
  • By default, after 4 minutes of starting the cluster shutdown job (total 6 minutes), the script will see if it is safe to turn off the node:
    • When all local VMs (clustered and non-clustered) have completed shutdown/saving
    • When the cluster has shutdown
  • The remaining 4 minutes of the shutdown duration are to ensure the cluster and then the host have enough time to complete shutdown

# Stage 1 - Shutting down Local VMs on this host Only
#         - a) Clustered VM roles on this node only
#         - b) Non-Clustered VM roles on this node
[int]$LocalVMsStaggeredDelay = 5 # maximum number of seconds in between starting each VM shutdown
[int]$MaximumTimetoStartShuttingdownLocalVMs = 120 # maximum number of seconds to start shutting down local VMs. $LocalVMsStaggeredDelay might be reduced if you have lots of VMs
[int]$SaveLocalVMs=$true # $true to save local VMs, or $false to shut them down

# if $SaveLocalVMs=$true
[int]$OnlySaveLocalVMsBelowMemoryDemandThreshold=$true
[int]$SaveLocalVMsMemoryDemandThresholdMB=8192
# VMs above this threshold will be shutdown, instead of being saved

# Stage 2 - Start shutting down the Global cluster
$ShutdownCluster=$true
$OnlyInitiateClusterShutdownIfPoolOwner=$false
[int]$StartClusterShutdownDelay=120 # the amount of time before shutdown-cluster is issued
# The local UPS script on each node should have already started the shutdown the local VMs


# Stage 3 - Shutdown of this Node - in S2D, this needs to happen after all Cluster IO has ceased
$ClusterShutdownTimeout = 240 # this it the time in seconds, after Stage 2 has been started, that we will wait for the cluster shutdown job to complete before testing to see if we are ready to turn off the node 
$TurnOffThisNode=$true
$DontTurnOffNodeifLocalClusteredVMsStillRunning=$true
$DontTurnOffNodeifLocalNonClusteredVMsStillRunning=$true
$DontTurnOffNodeifClusterIsStillRunning=$true


# Total Shutdown time should be:
# The greater of:
# $MaximumTimetoStartShuttingdownLocalVMs + time to save or shutdown last VMs
# $StartClusterShutdownDelay + $ClusterShutdownTimeout
#
# Then a final optional checks for LocalClusteredVMs, LocalNon-ClusteredVMs, and then the Cluster itself
# Then issuing local Host Shutdown

Applies to:

  • Stone Computers Equinox
  • Stone Computers Elysium
  • Most S2D Hyperconverged Platforms

This article was:  
Article ID: 820
Last updated: 20 Oct, 2020
Revision: 25
Views: 78
Posted: 24 Sep, 2020 by Andrew Sharrad
Updated: 20 Oct, 2020 by Andrew Sharrad
Tags

Attached files