Skip to main content

ESXi 5.0 kickstart / scripted install

Years ago when I first started playing with Linux kickstart I was also dabbling in php; using the two I was able to come up with a single "ks.php" file that we used for all our Linux installs.  When I was given the opportunity to work with our VMware infrastructure (ESX 4.0), one of my first tasks was the ESX 4.1 installation.  When I saw it was Red Hat Linux I knew exactly what to do and pulled out my php scripts.  As ESXi has evolved so has my scripts so even though ESXi isn't Red Hat Linux the same approach still works.  I hope someone will find these useful.

You may be asking, why not just use Auto Deploy?  First, the infrastructure to support our scripted install is already in place.  There is nothing new to install except for new ESXi images.  Second, we had a concern about relying on additional infrastructure to get things back online in the event our entire datacenter went down.  Now that ESXi 5.1 has the "stateless cache" option I do plan on investigating Auto Deploy.

I'll get to the scripts and tidbits shortly but first I wanted to make sure to give a shout out to William Lam (@lamw).  You need to check out http://www.virtuallyghetto.com/2011/07/automating-esxi-5x-kickstart-tips.html to get the most out of your scripted install!  I don't discuss my specific firstboot script but if you check the link above then you should be able to get a fully-automated install with very little effort.

I expect the following directory structure under the ESXi 5.0 directory:
/config/ - contains the properties.php file and any other config files you need
/kickstart/ - contains the ks.php file
/scripts/ - contains firstboot scripts (you'll see below that you can have multiple firstboot scripts for different environments or different configurations)


If you've never used php before then it's important to note that anything between the <?php  ?> tags are code that is processed by the web server module and output as text  - anything outside those tags are treated like regular text.  These tags can be placed on multiple lines or they can be placed inline.


Now, to the kickstart!  The key here is dns - or more exactly reverse PTR records.  For the script to work correctly your web server needs to be able to resolve full-qualified domain names from ip addresses.  If your web server is unable to do that then it will return an empty kickstart config file and the install will fail.  In the top section of the ks.php script we get the ip address of the host that requested the page using php's built-in $_SERVER['REMOTE_ADDR'] variable.  From this we get the full-qualified domain name of the ESXi host (to be used later).


# get the ip address and check for fqdn
$ip = $_SERVER['REMOTE_ADDR'];
$fqdn=gethostbyaddr($ip);
if (empty($fqdn)) {
  exit;
}

For our particular installs, we have a multiple management networks for ESXi that are dedicated for development, testing, production and split between datacenters.  Based on the first three octects of the ip address we are able to set information about the particular environment - this is saved in a separate file called properties.php and is "included" in the ks.php file.


$ip_octets = explode(".", $ip);  # convert the ip address into an array
$vlan = $ip_octets[0] .'.'. $ip_octets[1] .'.'. $ip_octets[2];  # get the first 3 octets

$ks_dir='/var/www/html/os/esxi-5.0u2';  # the full-path to the ESXi directory on the web server - this is NOT the URL
include $ks_dir .'/config/properties.php';  #  the properties file - in the config directory under $ks_dir

The properties.php is simply a giant if-elseif-else statement.  We made the decision to create a separate so that no one was updating the master ks.php file.  You can just as easily replace the last two lines above with the contents of the properties.php file.  Here's the contents of the properties.php file (I'm including multiple vlans just to give you an idea of how the file would work with multiple management networks)  :


# properties.php
<?php

$firstboot="firstboot";
$root_pw='$1$EzSANjR3$Gdekotsvzx98U88eq.yCs/';
$ks_url="http://10.255.10.10/os/esxi-5.0u2";

if ($vlan == '10.255.10') {
    $root_pw='$1$EzSANjR3$Gvstokedzx98U88eq.yCs/';
    $firstboot="firstboot-10gig";

    $mgmtvlanid="10";
    $netmask="255.255.255.0";
    $gateway="10.255.10.1";
    $nameservers="10.255.10.10,10.255.10.11";
    $ntpservers="10.255.10.10";
    $loghost="10.255.10.10";
    $sshkeys="authorized_keys";

    $lickey="XXXXX-XXXXX-XXXXX-XXXXX-XXXXX";

    $portgroups = array(
      '11' => '10.255.11.0/24',
      '12' => '10.255.12.0/24'
    );

} elseif ($vlan == '10.255.13') {
    $firstboot="firstboot-1gig";

    $netmask="255.255.255.0";
    $gateway="10.255.13.1";
    $nameservers="10.255.10.10,10.255.10.11";
    $ntpservers="10.255.10.10";
    $loghost="10.255.10.10";
    $sshkeys="authorized_keys";

    $lickey="XXXXX-XXXXX-XXXXX-XXXXX-XXXXX";

    $portgroups = array(
      '14' => '10.255.14.0/24',
      '15' => '10.255.15.0/24'
    );

} else {
    $netmask="255.255.255.0";
    $gateway="10.255.10.1";
    $nameservers="10.255.10.10,10.255.10.11";
    $ntpservers="10.255.10.10";
    $loghost="10.255.10.10";
    $sshkeys="authorized_keys";

    $lickey="XXXXX-XXXXX-XXXXX-XXXXX-XXXXX";

    $portgroups = array(
      '0' => '10.255.9.0/24',
      '16' => '10.255.16.0/24'
    );

}
?>

The key here is that we are able to supply different values based on each vlan.  For example, you'll see $root_pw defined at the top of the file; if $root_pw is defined inside one of the vlans then that value overrides the value at the top of the file.  So, we are able to provide this to other VMware admins but allow them to set their own $root_pw.  This guarantees every server is built exactly the same but is still customized for different locations.

A couple of things to mention here; first is the $portgroups array.  The "key" for the array is the vlan id of the portgroup and the "value" is the name of the portgroup.  Yes, we are still using the standard switch in 5.0 because there isn't a way to export the distributed switch until you go to 5.1.  Second is a variable we don't need in our environment but it might be useful to someone - $mgmtvlanid.  If your physical switch doesn't have a native vlan set on your management network switchport then you can use $mgmtvlanid to add the --vlanid to the network section of the kickstart file.  This should set the vlan id on the Management Network vmkernel portgroup.


Now that you've seen the properties.php file, here's how we use them:

# the $root_pw comes from the properties.php file
# this is inline php code
rootpw --iscrypted  <?php echo $root_pw; ?>    

# each of these variables come from the properties.php file
# except for $ip which comes from the top of the ks.php file
network --bootproto=static --addvmportgroup=true --ip=<?php echo $ip; ?> --netmask=<?php echo $netmask; ?> --gateway=<?php echo $gateway; ?> <?php echo ((empty($mgmtvlanid)) ? '' : '--vlanid='.$mgmtvlanid);  ?> --nameserver=<?php echo $nameservers; ?> --hostname=<?php echo $fqdn; ?>   

PHP has an "empty" function which allows you to test whether or not a variable exists.  Using this we are able to check for the existence of variables and run the ESXi commands.

<?php
if (! empty($lickey)) {
?>
echo "using license <?php echo $lickey; ?> "   
vim-cmd vimsvc/license --set <?php echo $lickey; ?>   
<?php
}
?>

echo "configuring ntp"
/etc/init.d/ntpd stop
echo "Configuring NTP"
echo "restrict 127.0.0.1" > /etc/ntp.conf
<?php
if (! empty($ntpservers)) {
  $ntp_arr = split(",",$ntpservers);
  foreach ($ntp_arr as $ntpsvr) {
?>
echo "server <?php echo $ntpsvr; ?> " >> /etc/ntp.conf
<?php
  }
}
?>
echo "server 127.127.1.0        # local clock" >> /etc/ntp.conf
echo "fudge 127.127.1.0 stratum 10" >> /etc/ntp.conf
echo "driftfile /var/lib/ntp/drift" >> /etc/ntp.conf
echo "broadcastdelay  0.008" >> /etc/ntp.conf
/sbin/chkconfig ntpd on
/etc/init.d/ntpd start

<?php
if (! empty($loghost)) {
?>
echo "configuring syslog"
esxcli system syslog config set --default-rotate 20 --loghost <?php echo $loghost; ?>:514,udp://<?php echo $loghost; ?>:514    
<?php
}
?>

<?php
if (! empty($sshkeys)) {
?>
echo "Configure SSH keys"
busybox wget -q <?php echo $ks_url; ?>/sshkeys/<?php echo $sshkeys; ?> -O /etc/ssh/keys-root/authorized_keys   
<?php
}
?>

And finally, we can add portgroups to vSwitch0 based on the $portgroups array.  We don't need to use the "empty" function here because the foreach-loop will simply not iterate if the $portgroups array is empty.


echo "configuring portgroups"
<?php
foreach (array_keys($portgroups) as $vlan_id) {
  $pg_name = $portgroups[$vlan_id];
?>
echo "adding <?php echo $pg_name; ?>"   
esxcli network vswitch standard portgroup add --portgroup-name='<?php echo $pg_name; ?>' --vswitch-name=vSwitch0
esxcli network vswitch standard portgroup set --portgroup-name='<?php echo $pg_name; ?>' --vlan-id=<?php echo $vlan_id; ?>    
<?php
}
?>

I've made the assumption that you have all the workings for kickstart (i.e. dhcp and tftpd) in your environment.  There was one other change I made to get the scripted install to work courtesy of the blog I mentioned at the beginning of this post (Tip #1).  Set the kernelopt=ks=http://172.30.0.108/esxi5/ks.cfg in the boot.cfg file to kernelopt=ks=http://<web_server>/<ks_url>/kickstart/ks.php.  And finally, when you PXE boot you need to type the following at your boot: prompt so the networking is set correctly (your particular label in pxelinux default file and network settings may vary)

boot: esxi5u2 ip=10.255.10.20 netmask=255.255.255.0 gateway=10.255.10.1

Here are the complete ks.php file (complete properties.php file is above):


<?php

# get the ip address and make sure it resolves to a hostname
$ip = ( (empty($_GET["ip"])) ? $_SERVER['REMOTE_ADDR'] : $_GET["ip"] );
$fqdn=gethostbyaddr($ip);
if (empty($fqdn)) {
  exit;
}

# get the octets and build the vlan

$ip_octets = explode(".", $ip);
$vlan = $ip_octets[0] .'.'. $ip_octets[1] .'.'. $ip_octets[2];


# include the properties.php file to set other variables
$ks_dir='/var/www/html/os/esxi-5.0u2';
include $ks_dir .'/config/properties.php';

?>

vmaccepteula

rootpw --iscrypted  <?php echo $root_pw; ?>    

network --bootproto=static --addvmportgroup=true --ip=<?php echo $ip; ?> --netmask=<?php echo $netmask; ?> --gateway=<?php echo $gateway; ?> <?php echo ((empty($mgmtvlanid)) ? '' : '--vlanid='.$mgmtvlanid);  ?> --nameserver=<?php echo $nameservers; ?> --hostname=<?php echo $fqdn; ?>   

# install on local disk - this is valid for HP Servers G6/G7 so your mileage may vary
install --firstdisk=hpsa --overwritevmfs

# automatically reboot
reboot

%post --interpreter=busybox --ignorefailure=true

# download additional items
busybox wget -q <?php echo $ks_url; ?>/kickstart/ks.php -O /vmfs/volumes/datastore1/ks.cfg
busybox wget -q <?php echo $ks_url; ?>/scripts/<?php echo $firstboot; ?> -O /vmfs/volumes/datastore1/firstboot

%firstboot --interpreter=busybox

mkdir -p /store/instlog/
logfile=/store/instlog/ks-firstboot.log
exec > $logfile 2>&1

<?php
if (! empty($lickey)) {
?>
echo "using license <?php echo $lickey; ?> "   
vim-cmd vimsvc/license --set <?php echo $lickey; ?>   
<?php
}
?>

echo "configuring portgroups"
<?php
# this is the portsgroups variable from the properties.php file
foreach (array_keys($portgroups) as $vlan_id) {
  $pg_name = $portgroups[$vlan_id];
?>

echo "adding <?php echo $pg_name; ?>"   
esxcli network vswitch standard portgroup add --portgroup-name='<?php echo $pg_name; ?>' --vswitch-name=vSwitch0
/bin/sleep 3s
esxcli network vswitch standard portgroup set --portgroup-name='<?php echo $pg_name; ?>' --vlan-id=<?php echo $vlan_id; ?>    
<?php
}
?>

echo "configuring ntp"
/etc/init.d/ntpd stop
echo "Configuring NTP"
echo "restrict 127.0.0.1" > /etc/ntp.conf
<?php
if (! empty($ntpservers)) {
  $ntp_arr = split(",",$ntpservers);
  foreach ($ntp_arr as $ntpsvr) {
?>
echo "server <?php echo $ntpsvr; ?> " >> /etc/ntp.conf
<?php
  }
}
?>
echo "server 127.127.1.0        # local clock" >> /etc/ntp.conf
echo "fudge 127.127.1.0 stratum 10" >> /etc/ntp.conf
echo "driftfile /var/lib/ntp/drift" >> /etc/ntp.conf
echo "broadcastdelay  0.008" >> /etc/ntp.conf
/sbin/chkconfig ntpd on
/etc/init.d/ntpd start

echo "running firstboot script"
/bin/sh /vmfs/volumes/datastore1/firstboot

<?php
if (! empty($loghost)) {
?>
echo "configuring syslog"
esxcli system syslog config set --default-rotate 20 --loghost <?php echo $loghost; ?>:514,udp://<?php echo $loghost; ?>:514    
<?php
}
?>

<?php
if (! empty($sshkeys)) {
?>
echo "Configure SSH keys"
busybox wget -q <?php echo $ks_url; ?>/sshkeys/<?php echo $sshkeys; ?> -O /etc/ssh/keys-root/authorized_keys   
<?php
}
?>

# backup ESXi configuration to persist changes
/sbin/auto-backup.sh

# Needed for configuration changes that could not be performed in esxcli
/sbin/reboot

One more gotcha, if you end a line with the ?> tag then you need to make sure to add at least one blank character (i.e. space bar) so that the text will carriage return correctly.  And, one bonus feature I added.  You should see that the $ip = line at the top of the ks.php file is different from what I explained earlier.  This will allow you to generate the config for any ESXi host in your environment from your web broswer by browsing to http://<web_server>/<ks_url>/kickstart/ks.php?ip=<ip_address>.  Using this you can verify the kickstart script is being generated correctly without having to completely test the install.  If you ever get a blank page then that means either your dns isn't working or there is an error in your script - you'll need to check your web server error log to find out exactly what's wrong.  Good luck!

Comments

  1. Any way you can share your 4.1 ks scripting. I am working on a project to move our Altiris build to kickstart and your example here is great. Just need to see how different it is with classic ESX 4.1 compared to ESXi. After that then it is ESXi 5.5

    ReplyDelete
    Replies
    1. I'll have to dig through an old build server and see if I can find a copy.

      Delete

Post a Comment

Popular posts from this blog

Querying for nested folders with PowerCLI

Have you fought trying to query nested, duplicate-named folders?  Hopefully this will help solve the problem!  Suppose you have a VM folder-tree similar to this:


So, how do you get the "\dotcom\linux\dev" folder using PowerCLI?  If you query for just "dev" then you can get an array of folders.
You can parse through the array and, using the parent object, traverse the tree backwards validating the folder names.  But, what if you have 100s of folders?  In my opinion, this is not an optimal approach.  We really need to do this:

This is great case for recursion.  In my words, recursion is a "stack" of operations.  When an operation completes its result is used by the next operation in the "stack".  Most importantly there has to be base-case which causes the last operation in the stack to return a valid result.  Then each operation can be popped off the "stack" and its result can be used by the previous operation until the final result is obta…

Reconnecting ESXi servers with PowerCLI

We recently needed to re-ip our vCenter servers; each with ~200 ESXi servers which needed to be reconnected - thank goodness for PowerCLI.  Since there isn't a "Reconnect-VMHost" cmdlet provided by PowerCLI we needed to check the HostSystem Object at https://www.vmware.com/support/developer/converter-sdk/conv55_apireference/vim.HostSystem.html and to see what methods were available (hint: there's a ReconnectHost_Task method which will do the trick).  We can still leverage the "Get-VMHost" cmdlet to return the disconnected ESXi servers and then call the ReconnectHost_Task method to reconnect each ESXi server.  The code is fairly short:

Get-VMHost -state Disconnected | foreach-object {
  $vmhost = $_
  $connectSpec = New-Object VMware.Vim.HostConnectSpec
  $connectSpec.force = $true
  $connectSpec.hostName = $vmhost.name
  $connectSpec.userName = 'root'
  $connectSpec.password = 'MySecretPassword'
$vmhost.extensionData.ReconnectHost_Task($connectSpec,…

Parsing XML with VMware Orchestrator

For my first blog I thought I would start with something easy - parsing XML using VMware Orchestrator (a.k.a vCO)!  I started playing with vCO in September 2012 for a "Cloud" project so I still consider myself a newbie - if you happen across this post and find something incorrect or something that could be done better then please don't hesitate to speak up.

Since I can't post our actual XML, I'll be using the following XML which will give the gist of how to parse for elements & attributes.

<?xml version="1.0" encoding="UTF-8" ?>
<people>
  <person firstname="Jack" lastname="Smith" age="40">
    <phone type="home" number="1234567890" />
    <phone type="cell" number="1234567891" />
<sport name="basketball" position="shooting guard" />
  </person>
  <person firstname="Jill" lastname="Smith" age=&q…