What you should consider when deploying a Nutanix cluster

This post focuses on the different aspects to be aware of to deploy a Nutanix Cluster. There are other considerations when we zoom out to look at large scale deployments, such as for Cloud Infrastructure transformation from traditional 3 tier storage deployments to Hyper-converged Infrastructure, but that will be another post for another day.

Nutanix installation is an activity that can be completed very quickly when good planning and preparation has been performed. Having a small cluster of up to 12 nodes set up, tested ready for further environmental integration in 2 days is something me and my team have done many times. In fact it is rather routine.

Here is what you need to know, and prepare for. Not every single options are covered, but the most commonly used ones are mentioned.

The Phases of Deployment

1. Preparation
2. Installation
3. Cluster Testing
4. Environmental Integration
5. Full Testing
6. Release to Production

Let’s go into the details next.

Phase 1 – Preparing for Nutanix Installation

In this phase, you will prepare everything needed for the upcoming installation. One thing I assume in this post is that you already have a design, and most (if not all) details have already been decided. Here are the key details.

1. Physical Environment Considerations
   • While Nutanix nodes, be it NX or otherwise are nothing more than standard x86 servers, and can be less demanding than a monster SAN Array, we ought to still put in the right due diligence to ensure we do not exceed any physical limits.
   • Rack Space – make sure you have enough Rack Units for all your nodes.
   • Power – sufficient and disparate power sockets are available for you to redundantly power up you nodes.
   • Power Connectors – confirm compatibility of the power sockets with what has been ordered with the nodes.
   • Cooling – there is enough cooling capacity at the location where the nodes are to be installed.
   • Load Bearing – making sure the floor underneath is rated sufficiently to hold the rack with everything in it.
   • Physical Security – building, room, rack security to limit access to only those who need to get to the nodes. Of course any necessary one off security arrangement for those who needs to get on site to do the installation.

2. Nutanix AOS Cluster Logical Properties
   • While it can vary a little with hypervisors, here are what we need
   • CVM Memory – this varies with capabilities you would like to enable. Most often they are seen to be deployed with 32GB of ram that are fully reserved. If more cache in RAM is desired, it can be increased to 64GB. In some environments, they can be small like 20GB. Understand what you need and choose the appropriate size.
   • IP Addresses – typically we need 3 IP addresses per physical node. Can be more, if network segmentation feature is needed, but the details are for another post on its own.
     1x Virtualization Host – Management IP [aka HOST IP]
     1x Nutanix Controller VM (CVM) [aka CVM IP]
     1x IPMI (iLO/XClarity/iDRAC) [aka IPMI IP]
     Important Rule > HOST IP and CVM IP must be in the same IP Subnet. IPMI IP can be in a different subnet.
   • Cluster Level IP Address – There are a few to be considered
     1x Virtual IP – this is a cluster VIP for all CVMs. Must be in the same subnet as HOST & CVM IP.
     1x iSCSI Data Services IP – [optional] this is also a VIP but it is used as the target iSCSI IP. You only need this if you intend to use the iSCSI storage service Must be in the same subnet as HOST & CVM IP.
     1x Failover Cluster IP – [Hyper-V Only] this is the VIP for Hyper-V Failover Cluster IP. Must be in the same subnet as HOST & CVM IP.
   • Network bonding (teaming) – to decide up front whether if the connections between the nodes and switches are going to be LAGs or not.
   • Port Groups / VLANs – determine all the VLANs that will be used by the hosts, CVM and virtual machines. For each “Port Group” know the VLAN ID, and a Name.
     HOST & CVM VLAN – this is the least that need to be determine. Optional, but preference is to use native VLANs where possible.
     User VMs (UVM) VLAN – these are optional at the time of deployment, but will be needed before any are deployed.
   • Storage Properties – these are various settings needed to be considered to configure the storage.
     Storage Pool Name – [optional] AOS deploys with a default name for the Storage Pool. It can be changed online anytime.
     Container – Container Name and Settings. Container names cannot be changed, however settings can be changed on the fly.
     Distributed Hot Spare Reservations – this is a concept to set aside some space in the physical disks to be used as rebuild storage if any failure is to occur.
   • Other settings – these are the standard details when installing just about any service. This is not a comprehensive listing, but a few common ones that you would want to be aware of.
     DNS Server IP – so that the CVMs and Hosts can perform name resolution. Particularly useful for systems that need to connect to AD/LDAP for authentication purposes, and also for any Internet connected systems, allows the following to work, Pulse (call home), 1-click upgrade / LCM can get to the cloud repository to download new software, 1-click PC deploy, etc.
     NTP Server IP – in my opinion, keeping time accurate and in sync in any enterprise environment should be mandatory.
     SNMP – if there is intention to use SNMP Traps for alerts, or SNMP Pool for metrics, you will prefer to use SNMPv3 for best security.
     SMTP – this allows the cluster to send email based alerts, and with Pulse, some alerts will also generate an auto ticket with Nutanix support. If your hardware is NX, this allows the Nutanix SRE to proactively reach out and say, schedule a time for Hardware replacement.
     Active Directory/LDAP – if using external service for Authentication, these details will need to be handy to configure the integration.

3. Physical Networking Preparation
   • Physical Network Switch Models – be aware of network devices that are not supported. For greater details read up the Nutanix best practice guide BP-2050 on Physical Networking, available in the Support Portal. Here are my rules of thumb
     Cisco Fabric Extenders – DO NOT USE, these do not even qualify as hubs, keep any sort of storage traffic away from these. DO NOT connect Hosts, CVM and any high performance VM to these devices. They are acceptable for IPMI, or low performance VMs, on their own uplinks.
     Datacenter Grade Switches – for best performance, you want good amount of buffers within the switch. At least 1MB per port is a guide number I use. Use switches that are at least 10GBps per port, high is better. Avoid oversubscribing switch uplinks.
   • Network architecture – this is important, especially when there are intentions to scale out the Nutanix cluster. Rule of thumb, do not exceed the maximum of three switch hops between any two Nutanix nodes that belong to the same cluster.
     Leaf-Spine – is the recommended for environments that will grow large.
     Core-Aggregation-Access – is the more traditional approach and is acceptable for small deployments, but with caveats. See the rule of thumb, and risk of oversubscription.
   • Cabling – running cables between the Nutanix nodes and switches can be a large topic. For ease of cabling, having Top-of-Rack (TOR) or Middle-of-Rack (MOR) placements are ideal. Also type of cables are also important considering the type of connectors the NIC and Switches have.
     Fibre – make sure they are of the right grade, for 10Gbps, the minimum grade is OM3. They are aqua coloured, and both ends are terminated with LC connectors.
     Copper UTP – if I were to just write “Copper” there are two interpretations of this term. To be precise, I added UTP. These are twisted pair cables, like what we are familiar with. However, the grade of the cable and RJ45 heads are important. Check cabling specifications to be at least Cat6 and can support 10Gbps. I have encountered Cat6 cables that only work up to 1Gbps, go figure.
     Twinax DAC – in the early days of 10Gbps networking, this is the only copper type cable that supports the speed, hence the confusion of the term “copper”. Be aware of length, typically from 5m length the “active” type of Twinax is preferred. They do get very thick with length though.
   • Connectors – this is important that both switch and node end will be using the same type of connectors. If there is a mismatch, there’s no way to hook them up.
     SFP+ – this applies for both Fibre and Twinax type connectivity. For Fibre, make sure the SFP+ module is compatible with the device they will be installed in. i.e. generally switch vendors prefer the same branded SFP+ modules. For Cisco Nexus switches, the SFP-10G-SR-S works just fine as we do not need FCoE function. On system side, check your with your HW vendor. There is no need for both ends to use the same model.
     RJ45 – this only applies when using Copper UTP cables, and 10GBase-T NICs and Switches. End-to-end quality of the cables, structured cabling (if any), keystones, connectors are important.
   • Switch port configuration – there are several recommendations on this one.
     Enable Portfast – this is only for the ports facing the Nutanix nodes. You would want to do this regardless if you are using LACP or not. More details available from an earlier post.
     Trunk/Access/Hybrid Port – this depends on what services on the nodes will be using the uplink. More details to come in a future post. My recommendation will either be Access or Hybrid, followed by Trunk. Have a read here to understand what are Access, Hybrid and Trunk ports.
   • LACP – if this is to be used, here are some switch side settings I recommend considering. Nutanix KB-3263 has more details. Also, make sure you are clear what LACP really is, read here and here.
     LACP Fallback – only supported by AHV, but it is useful that in case LACP negotiation fails, the switch will not block the port. Switch configurations are not called fallback though, see the KB above for examples on what settings there are. E.g. on Cisco Nexus no lacp suspend-individual is what you want.
     LACPDU Timer – prefer fast rate if supported by the switch. It is essentially a LACP heartbeat between host and switch. Shorter intervals allow failure detection to happen sooner.

4. Receiving of the equipment
   • it is an obvious point, but worthy of mentioning. Like anything else, it has been missed before. <grin>
   • make sure all the nodes, cables, SFP+ modules, switches, etc. have arrived

Phase 2 – Installation

Nutanix cluster installation is largely automated, all thanks for the Foundation tool that Nutanix developed. There are many ways the Foundation environment can be set up, look out for a future post on this.

Let’s break down the Installation process

• Unpacking the hardware
• Rack mounting the nodes and powering up
• Network connection to Foundation environment
• Kick of Foundation process
• Create Nutanix Cluster
• Configure network bonding and apply VLAN for CVM and Host
• Reconnect nodes to Production Network – if different from Foundation environment
• Test network connectivity for every single node-to-switch uplink for the following qualities portfast, VLAN accessibility and speed (iperf)
• Continue with AOS cluster configuration, Hypervisor clustering and configurations.
• Apply AOS License(s)

The foundation process installs the hypervisor and CVM on every node. Any necessary hypervisor configurations to allow the CVM to work well, will also be done at this point. The creation of AOS cluster can also be done by Foundation. If AHV is the choice of hypervisor, there is only a few other configuration needed. For ESXi and Hyper-V, there are several more steps to complete.

AHV – unlike ESXi and Hyper-V there isn’t a hypervisor level clustering per se. Think of it more like a true hyperconverged cluster, where everything is rolled and managed together. For example DNS and NTP settings, once set via Prism (or API), the settings are applied on to AHV Hosts and CVMs. The unique settings that have to be done for AHV are…
HA Reservation – this is simply a checkbox if you want to set aside some resource (memory) to guarantee HA restart of VMs. Without this setting, HA still exists, and operates on a best effort basis. If memory runs out, there will be VMs that cannot be powered on. This is essentially to the concept of Admission Control on vSphere clusters.
Hardening – there are ncli commands to set parameters around SCMA, AIDE, SNMP, Password Complexity and ssh logon Banners. Just like how we can apply them on CVMs, there is another set of commands just meant for AHV. This allows different variations of the settings between AHV hosts and CVM. Look up the Nutanix Security Guide for details.
Networking – If there are additional bridges to be set up, now is the time. Additionally configure the (Virtual) Networks for VMs. These are like port groups on vSphere.

ESXi – assuming there is already an existing vCenter Server, naturally the first thing to do is to add the hosts to vCenter Server and create a new vSphere Cluster. While it works to split the hypervisor into a few clusters, while maintaining a single Nutanix AOS cluster, I would recommend to always keep the vSphere to Nutanix cluster in a 1:1 relationship. Anything else increases complexities, and as of this writing, there are a few features that will not work in a split vSphere cluster scenario. Here are the other commonly applied settings.
HA and DRS – typically these are enabled, with admission controls, and advance settings to exclude DRS from HA and DRS calculations.
EVC – always turn this on during the creation of the cluster, even if it is to set to the same level as what the processors are today.
Networking – whether setting up more Standard vSwitch, or migrating to Distributed vSwitch, it is at this part of the process they are configured further.
DNS and NTP – Foundation doesn’t take care of this for ESXi hosts, it will have to be applied at this stage.
Hardening – if needed in your environment, do not apply the settings wholesale. Please review which should not be applied, or applied differently in a Nutanix platform. E.g. ssh settings, lockdown.

Hyper-V – the use of SCVMM is optional, and Prism has built in capability to help with the Hyper-V cluster set up. Few of the things that have to be done.
Join the AOS Storage Cluster to the Domain – AOS presents storage out via SMB to the Hyper-V hosts, and kerberos is used to authenticate access. Therefore the storage cluster will have to join the domain.
Create Windows Failover Cluster – This is where all the Hyper-V hosts are added to a new Windows Failover Cluster. Again, this can be automated by Prism. As part of the process, the hosts are first joined to the domain, then the Failover Cluster is created and all the nodes are added to it.
Adding the new cluster to SCVMM – This is optional, as Hyper-V cluster can be managed using a combination of Hyper-V Manager and Failover Cluster Manager.
Set Default Storage Path – this should be done from Prism, Storage dashboard. Select the container you like to be the default for Hyper-V, and update the properties to set all hosts to set that container as default. Do realise this that Hyper-V doesn’t mount the file shares like how ESXi mount NFS exports as datastores.
Networking – there isn’t a portgroup concept, you will need to add each required VLAN to every host’s ExternalSwitch.

Phase 3 – Cluster Testing

In this phase, key cluster functionalities are validated, making sure the cluster is correctly configured and functions as expected. Some aspects of performance can also be tested. I shall attempt to list out some of the tests to consider. Depending on the situation, not all tests can be performed.

• Access Validation – to validate logon to Prism, Hypervisor, SSH to CVM, IPMI, vCenter Server, etc.
• Hardware Inventory – to validate what has been set up is what was ordered. Node models & quantity, CPU models & quantity, RAM size, SSD capacities & quantity, HDD capacities & quantity, GPU models & quantity, NIC models & quantity, etc.
• AOS Configurations – double check all settings like DNS, NTP, AD, etc are correct. Also, validate that NTP on CVMs are syncing with the upstream NTP source.
• Storage Configuration – check the containers are created with the correct names, and advance settings like Replication Factor (RF), Compression, Deduplication, Erasure Coding, etc.
• Storage Rebuild – while most may approach this from a traditional storage angle by attempting to pull a random disk, I do encourage trying with an updated HCI method. Shut down a CVM instead.
• Network Testing – for those uplinks that have redundancy, perform testing to ensure failover and fail back work seamlessly. NIC speed & duplex are as expected and can deliver the desired performance. For a 10Gbps connection, an iperf test between nodes should yield at least 8Gbps.
• Network Switch Testing – where possible, perform a rolling power off test of the physical switches. Ensure that there is no interruption with network communications for the cluster.
• Hypervisor VM Live Migration – Essentially making sure VMs are able to Live Migrate/vMotion between hosts.
• Hypervisor HA – power off a host with at least one running UVM, and observe that the VM is able to restart on another host.
• AOS Diagnostics – this is a scripted test available on the CVMs. It helps to exercise the storage to observe the performance. It is not a benchmark test and certainly does not demonstrate the maximum possible storage performance the cluster can deliver. It simply deploys one VM per node in the cluster and run various iometer runs within each VM. Again, while the output can be impressive, it does not indicate the maximum the cluster can deliver.

Typically for most professional services led engagements, the job ends here. The cluster is handed over for the end customer to continue with their own environmental integration.

Phase 4 – Environmental Integration

This section attempts to list down a few common environmental integrations that should be considered.

• Monitoring – all well managed clusters have some sort of monitoring in place. Starting with adding the cluster details into CMDB, configuration and validation of monitoring such as using SMTP or SNMP. Ensuring alerts are generated and lands in the Incident Management / Ticketing system.
• Capacity Monitoring – this would be systems like Nutanix Prism Ops, VMware vROps, Microsoft System Center, and the likes. Storage monitoring, typically, is a new experience for systems team. It is important to know that the complete way to monitor storage utilisation has to be done at both the Hypervisor and Storage (AOS) level.
• Backup – However robust any solution in the world can be, there’s always a need to take off-host backups. Having another copy of the data in a disparate system is important. Never leave all the eggs in the same basket.
• Automation – In mature cloud environments where automated provisioning is in place, this would be one of the key steps to add the new cluster into the pool of resources.
• Process Updates – while this is not a technical integration, it is important to consider what sort of processes that may need to be updated. Especially if you are new to a Nutanix platform. Some practices may change, and they ought to be updated to suit.
• Training – again, if you are new to Nutanix, ensure the rest of the teams have some familiarity with operating/managing the environment. Understand what is different from a traditional environment, how some procedures and concepts are different. While Prism is easy to operate, some practices and concepts can be different. Do not leave to assumption, and always ask your friendly Nutanix contact, or support whenever you have any slightest doubts.

Phase 5 – Full Testing

The testing in this phase can be a repeat of some of the tests performed in Phase 3, plus all additional testing required due to the integrations in Phase 4. Some ideas for testing include.

• Repeat relevant Phase 3 tests – to ensure alerts are generated and show up in the ticketing system.
• Performing backup and restore – ensure that VMs on the new cluster can be backed up at the expected performance. A backup is not complete without a restoration test. Again, test for completeness of data recovered, as well as noting the performance. My favourite quote in this are is “Without restore testing, you do not have a backup; You only have hope.”
• Automation – If automation integration is done, do test the necessary automation procedures. For example, if there is provisioning, ensure that the provisioning script works, and new workloads are deployed with the appropriate performance.

Phase 6 – Release to Production

When you get to this point, you should have a good level of confidence that the cluster you deployed are in good working order, well monitored and ready to support the business.

Thank you for choosing Nutanix.