Easy way to change IP address for Prism Central VMs

It’s been a while since I had to change the IP address of a cluster. Last when I had to do it, it was very manual, having to edit entries within zookeeper, and various Linux config files. Fast forward to 2022, we now have a script that will do almost everything for you. It works, but just be aware of a couple of things.

I just had to change the IP of a scale out Prism Central PC.2022.4 deployment using the new, script based, method.

What I’m about to describe is not a replacement of the official procedure detailed within the official Prism Central documentation. I am adding some enhancements to enhance the experience. The document can always be found by hitting the Nutanix Support Portal https://portal.nutanix.com.

Specifics about the PC VM that I worked on

• It was a scale out Prism Central PC.2022.4 deployment consisting of 3 virtual machines.
• CMSP was not enabled on this PC.
• The PC VMs were deployed on AHV with an IPAM enabled Network.
• Each PC VM has an IP assigned statically via AHV.
• In other words within the guest OS, the NICs were configured with DHCP.

The goal was to change the subnet and VLAN for this Prism Central.

The procedure I followed

1. Preparation
   • Created the new IPAM enabled Network in AHV. This is the target subnet PC is to be migrated to.
   • Ensure that PC itself is healthy, by first running ncc and addressing any issues.
   • Eventually, shutdown the Prism Central service (but not the VMs)
     • $> cluster stop
2. On any of the PC (if scale out)
   • As a pre-caution, I executed next step via the VM Console, logged on as nutanix.
   • invoke the script
     • $> external_ip_reconfig
   • follow the steps and provide the new IP addresses as desired
   • Notice that it will ask for all IPs, except the VIP (we can change the VIP later)
   • it will eventually give a final summary before you acknowledge to proceed.
   • The process will begin, and will take between 5 to 10 minutes to complete.
     • this part is rather quiet, so if you’d like to follow the progress you’ll have to launch an ssh session to tail the log file.
       • $ > tail -f ~/data/logs/ip_reconfig.log
   • when it completes successfully, you should see an output similar to
     • External IP reconfig finished successfully. Restart all the CVMs and start the cluster.
3. As my PC VMs were configured with DHCP, I can’t reboot yet, and have to continue update the network configuration in Prism Element. I completed via the VM Dashboard in PE.
   • For each PCVM, select and invoke to Update the VM configuration.
   • Find the NIC, and switch it to the new Network, and supply the new IP address for the NIC.
4. After all 3 PCVMs have been updated, I can return and reboot all 3 of them.
5. Upon the reboot, you can login (via console or ssh) to each of the PCVMs and do some validations
   • notice that the hostname of each PCVM should have been updated with the new IP
   • you can check the contents of /etc/hosts and see that zk1, zk2 & zk3 all have been updated to the new IPs.
   • if you invoke svmips you may still see the previous IP addresses, you’ll have to wait for a few more minutes, as zookeeper processes start, they will eventually complete the final changes.
   • You can monitor the update simply using the following command
     • $ > watch svmips
   • There is a script you can call for a final verification
     • $ > python ~/cluster/bin/external_ip_reconfig_verify.py
   • it’s rather straightforward, and if checks passed, it will ask if you’d like to start the cluster.
   • Hit Y if desired and wait for cluster to start.
6. Final step, to update the PCVM VIP
   • Launch the browser, and hit Prism Central using one of the PCVM IPs (not the VIP)
   • Login, and go to Settings > Prism Central Management
   • then click on the old VIP to bring up the interface to update to the new IP.
7. that’s it, all IPs have been updated

Wrapping Up….

I’m glad engineering has provided an updated process and script for the procedure. Definitely will reduce chance of human error with the manual method. The only bit that felt uncertain was waiting for the script to run. Without looking at the log, I first thought something had gone wrong, luckily I looked at the log, and understood that a genesis restart happened and it will take time for genesis to apply the necessary updates. So grab a coffee while waiting, and it should be done soon after.