Upgrade Issue on UCS Manager

 

++ UCSM and FI B is upgraded successfully but FI A is stuck at 65% with the following Message

UCS pre-upgrade check failed. Free space in the file system is below threshold

Note: None of UCS have stuck with this Message Unlinke N5K, N3K. This issue is similar to CSCun79792

Load the debug Plugin and Perform  the following (CHeck How to Load Debug Plugin at the End og this Section)

Linux(debug)# cd /var/tmp

Upgrade Issue on UCS Manager:

+++++++++++++++++++++++++++++

Versions Affected: https://www.cisco.com/c/en/us/support/docs/field-notices/640/fn64094.html

++ UCSM and FI B is upgraded successfully but FI A is stuck at 65% with the following Message

UCS pre-upgrade check failed. Free space in the file system is below threshold

Note: None of UCS have stuck with this Message Unlinke N5K, N3K. This issue is similar to CSCun79792

Load the debug Plugin and Perform  the following (CHeck How to Load Debug Plugin at the End og this Section)

Linux(debug)# cd /var/tmp

Linux(debug)# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/root             300M  255M   46M  85% /

none                  2.0M  4.0K  2.0M   1% /post

none                  300M  255M   46M  85% /var

none                  3.0G  1.1G  2.0G  35% /isan

none                  600M  180M  421M  30% /var/tmp   <————

Note: Check How much Percentage is /var/tmp used? In this case, 30% is used.

Most of the time, it is smm.log and auto_file_deletion_log.txt  that eats up the space.

Here is an example. After you have run “  df –h  “ , issue the following command to see which files are eating the space (below output shows SMM.LOG is eating 1.3M of space)

 

Linux(debug)# ls -lSh

total 2.2M

-rw-rw-rw- 1 root   root   1.3M Apr 22 14:20 smm.log    <<<———

-rw-rw-rw- 1 root   root   245K Apr 21 03:28 afm_srv_15.log.0

-rw-r–r– 1 root   root   196K Apr  8 11:47 sam_bridge_boot.log

-rw-rw-rw- 1 root   root    63K Apr 22 14:18 afm_srv_15.log

-rw-rw-rw- 1 root   root    60K Apr  8 11:50 syslogd_errlog.4886_4886

-rw-rw-rw- 1 root   root    21K Apr  8 11:47 fm_debug.log.vdc_1.4919

-rw-rw-rw- 1 root   root    18K Apr  8 11:49 fcoe_mgr_init.log

-rw-rw-rw- 1 root   root   5.2K Apr  8 11:49 first_setup.log

-rwxr-xr-x 1 root   root   4.4K Apr  8 12:25 iptab.sh

 

Since the files are in /var/tmp they are assumed to be safe to be deleted.  Rather than fully deleting the files, it is safer to echo 0 into the file to empty and reduce the file size, this will ensure any service that is dependent upon being able to write to this log file still can.

Linux(debug)# echo 0 > /var/tmp/smm.log

Linux(debug)# echo 0 > /var/tmp/auto_root_file_deletion_log.txt

After this, you can df -h again to confirm that /var/tmp is now less than 10% utilzation.  The upgrade should now be able to proceed.

++

After this workaround, I waited for 10 mins and still the FI is stuck at 65% and I reboot the FI and waited to 10 more mins after it came up – No Luck. So, I went ahead and re-activated the FI firmware image to 2.2(6c) with force option and the upgrade went successful.

 

Loading Debug Plugin in UCS

=======================

First Get the debug Plugin from : https://cspg-releng.cisco.com/

Note: You will need to select the appropriate version for the Plugin.

Below is the example for 2.2(6c) version

SELECT and CLICK on  2.2.6   Elcap_MR5    UNDER Release Builds on the lefthand side  (Dont select Commits)

NOW You will select FCSc   and   click on BUNDLES and then you will select DEBUG.

Note: You can upload this image to UCSM same as you upload Upgrade Images. Once you uplaod it to UCSM, login to CLI and then Perfomr the following steps

Load debug plugin

FI-A(local-mgmt)# copy  debug_plugin/ucs-dplug.5.0.3.N2.2.26c.gbin   x

FI-A(local-mgmt)# load  x

###############################################################

Warning: debug-plugin is for engineering internal use only!

For security reason, plugin image has been deleted.

###############################################################

Successfully loaded debug-plugin!!!

Detail List of Commands:

+++++++++++++++++++++

Connect Local-mgmt

FI-A(local-mgmt)# copy  debug_plugin/ucs-dplug.5.0.3.N2.2.26c.gbin   x

FI-A(local-mgmt)# load  x

Linux(debug)# cd /var/tmp

Linux(debug)# df -h

Linux(debug)# ls -lSh

Linux(debug)# echo 0 > /var/tmp/smm.log

Linux(debug)# echo 0 > /var/tmp/auto_root_file_deletion_log.txt

 

 

CHECK THE HARDWARE VERSION :

+++++++++++++++++++++++++++++++

5596# show sprom sup | inc H/W

If hardware version in 1.1:

Copy dplugin and update file,  Load Plugin

Copy the ucd-update.tar to the bootflash

Copy the debug plugin to the bootflash

Load the image via the load command

>>You might have to call support for the update.tar script

Linux(debug)# cp /bootflash/ucd-update.tar /tmp

Linux(debug)# cd /tmp/

Linux(debug)# tar -xvf ucd-update.tar

ucd.bin

Linux(debug)#

Step 2

———————————-

Run the ucd.bin file:

Linux(debug)# ./ucd.bin

You will see the prompt about updated version 1.1.

SAFESHUT:

++++++++++

>>Have to contact support for the safeshut.tar script

Copy tftp ://10.96.60.154/safeshut.tar workspace://safeshut.tar

Untar safeshut.tar (tar -xvf safeshut.tar)

Linux(debug)# ./safeshut.sh

md5sum for ./klm_sup_ctrl_mc.klm does not match – corrupted tarball?

Linux(debug)# ./safeshut.sh

md5sum for ./rebootsys.bin does not match – corrupted tarball?

Cd bootflash/ ————————————>Checks if the safeshut is updated ,look for the change in time of reboot it should be the current time

Cd /sbin

ls -la |grep “reboot”LS

Cd/bootflash

Ls -la |grep -I “klm_sup_ctrl_mc.klm”

Repeat for the peer FI

OPTIONAL:

++++++++++

echo 7 7 7 7 > /proc/sys/kernel/printk

 

Linux(debug)# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/root             300M  255M   46M  85% /

none                  2.0M  4.0K  2.0M   1% /post

none                  300M  255M   46M  85% /var

none                  3.0G  1.1G  2.0G  35% /isan

none                  600M  180M  421M  30% /var/tmp   <————

 

Note: Check How much Percentage is /var/tmp used? In this case, 30% is used.

Most of the time, it is smm.log and auto_file_deletion_log.txt  that eats up the space.

Here is an example. After you have run “  df –h  “ , issue the following command to see which files are eating the space (below output shows SMM.LOG is eating 1.3M of space)

Linux(debug)# ls -lSh

total 2.2M

-rw-rw-rw- 1 root   root   1.3M Apr 22 14:20 smm.log    <<<———

-rw-rw-rw- 1 root   root   245K Apr 21 03:28 afm_srv_15.log.0

-rw-r–r– 1 root   root   196K Apr  8 11:47 sam_bridge_boot.log

-rw-rw-rw- 1 root   root    63K Apr 22 14:18 afm_srv_15.log

-rw-rw-rw- 1 root   root    60K Apr  8 11:50 syslogd_errlog.4886_4886

-rw-rw-rw- 1 root   root    21K Apr  8 11:47 fm_debug.log.vdc_1.4919

-rw-rw-rw- 1 root   root    18K Apr  8 11:49 fcoe_mgr_init.log

-rw-rw-rw- 1 root   root   5.2K Apr  8 11:49 first_setup.log

-rwxr-xr-x 1 root   root   4.4K Apr  8 12:25 iptab.sh

Since the files are in /var/tmp they are assumed to be safe to be deleted.  Rather than fully deleting the files, it is safer to echo 0 into the file to empty and reduce the file size, this will ensure any service that is dependent upon being able to write to this log file still can.

Linux(debug)# echo 0 > /var/tmp/smm.log

Linux(debug)# echo 0 > /var/tmp/auto_root_file_deletion_log.txt

After this, you can df -h again to confirm that /var/tmp is now less than 10% utilzation.  The upgrade should now be able to proceed.

++ After this workaround, I waited for 10 mins and still the FI is stuck at 65% and I reboot the FI and waited to 10 more mins after it came up – No Luck. So, I went ahead and re-activated the FI firmware image to 2.2(6c) with force option and the upgrade went successful.

Loading Debug Plugin in UCS

=======================

First Get the debug Plugin from : https://cspg-releng.cisco.com/

Note: You will need to select the appropriate version for the Plugin.

Below is the example for 2.2(6c) version

SELECT and CLICK on  2.2.6   Elcap_MR5    UNDER Release Builds on the lefthand side  (Dont select Commits)

NOW You will select FCSc   and   click on BUNDLES and then you will select DEBUG.

Note: You can upload this image to UCSM same as you upload Upgrade Images. Once you uplaod it to UCSM, login to CLI and then Perfomr the following steps

Load debug plugin

FI-A(local-mgmt)# copy  debug_plugin/ucs-dplug.5.0.3.N2.2.26c.gbin   x

FI-A(local-mgmt)# load  x

###############################################################

Warning: debug-plugin is for engineering internal use only!

For security reason, plugin image has been deleted.

###############################################################

Successfully loaded debug-plugin!!!