ACI

Downgrading leaf firmware without APIC using CLI

5 min read
By prox
Photo by Brina Blum
Photo by Brina Blum / Unsplash

You may receive your new leaves with the firmware version newer than you currently use. There's a pretty simple way to upgrade leaves firmware in case if it's older than you use with the help of basic operations in APIC, but no well-documented steps on how to downgrade firmware. More to say, it's not possible with APIC (at least I wasn't able to find out how to do that).

Requirements

You will need:

  • a USB flash drive
  • n9k firmware file for ACI-mode
  • Console access to the leaf

Preparing USB flash drive

You need to have a FAT32 formatted flash drive. If you already have one with FAT32 it's fine, if not - format it. Copy downloaded firmware file on a flash. You can download firmware from the cisco.com site. Follow the link and select your leaf hardware type, then select NX-OS System Software-ACI software type and download firmware for the ACI version you're currently using.

Downgrading firmware

Insert the flash drive into a USB port on a leaf. Access the leaf via console. I'm assuming that we're dealing with the brand new clean leaf.

If your leaf is already booted up, you should see the login banner:

User Access Verification
(none) login:

Login as admin. No password will be requested. After login, reload the leaf.

After that, or if you just powered on the leaf, you will see the boot process:

CISCO MODULE 
BIOS Ver: 5.43
Switch G5
RC Revision:  02.03.00

Memory Information:
 MRC Revision:00.50.00
 Total  DRAM: 32768 MB
 Memory TOLM: 80000000
 PCIE   BASE: 80000000     Size : 10000000
 PCI32  BASE: 90000000     Limit: FBFFFFFF
 PCI64  BASE: 80000000000     Limit: 83FFFFFFFFF
 UC    START: 80000000000     End  : 84000000000
ME Operational Firmware Version: 06:3.0.3.100

DIMM Information:
 Clock Speed: 1067MHz
 Socket: 0x0 Channel: 0x0 Number: 0x0 Presence: Yes Size: 32GB
 Socket: 0x0 Channel: 0x0 Number: 0x1 Presence: No
 Socket: 0x0 Channel: 0x1 Number: 0x0 Presence: No
 Socket: 0x0 Channel: 0x1 Number: 0x1 Presence: No

This is provided as an example from my leaf switch, so the output in your case may differ.

Press Ctrl+C multiple times until the loader prompt will be displayed:

Aborting config file read and autoboot 
No autoboot or failed autoboot. falling to loader 



                Loader Version 5.43


loader >                                                 

Now run the dir command to get the flash drive designator and to make sure that the firmware file is in there:

loader > dir

usb1::  

 System Volume Information
 aci-n9000-dk9.14.2.4i.bin

bootflash::  

  aci-n9000-dk9.15.0.1k.bin
  CpuUsage.Log
  lxc
  disk_log.txt
  nxos.7.0.3.I7.3.bin
  auto-s
  libmon.logs
  .stats_pref.txt
  bios_bootup_scratch_not_cleared
 

As you can see, in my case the flash drive is named as usb1. Now we need to boot from the firmware located on a flash drive. Execute command boot <flash_name>:<firmware_name>, where <flash_name> is the name of the USB flash drive and <firmware_name> is the filename for your firmware located on a flash:

loader > boot usb1:aci-n9000-dk9.14.2.4i.bin

Security Lock
Booting usb1:aci-n9000-dk9.14.2.4i.bin 
Trying diskboot 
 Filesystem type is fat, partition type 0xc
Image valid


Image Signature verification was Successful.

Boot Time: 9/14/2021  11:54:21

Security Lock
...

Now the process has two paths:

  1. Leaf goes for a reboot after booting
  2. Leaf doesn't go for a reboot after booting

Leaf goes for a reboot

As it was in my case with the N9K-C93180YC-FX, the leaf went for a reboot after these messages:

[  336.389531] @@@cctrli: wrote 132 to scratch RR
[  336.443171] nvram_klm wrote rr=132 rr_str=Resetting switch. LPSS restore from SQL failed. to nvram
[  336.548899] Collected 9 ext4 filesystems 
[  336.599115] Freezing filesystems  
[  336.695232] Collected 0 ubi filesystems 
[  336.742224] Freezing filesystems  
[  336.782965] Done freezing filesystems  
[  336.828912] Putting SSD in stdby  
[  336.922668] Done putting SSD in stdby 0 
[  336.969662] Done offlining SSD 
[  337.007273] Writing reg=0x84 val=0x80000000

So we will see the boot process again, as it was when we started the leaf first. But now the boot process will fail, as we already have the new firmware file stored on a bootflash of a leaf. IDK why, but after we boot from the USB flash drive, the firmware is automatically copied to the bootflash and the original firmware image is deleted. Basically, the leaf tries to boot a non-existing image following the bootloader configuration.

Again you will see the loader prompt, but now you don't need to press Ctrl+C:

PGA SPI Flash Winbond W25Q128BV
Board type  4
IOFPGA @ 0xd8000000
SLOT_ID @ 0xf
Set fan speed to 60%
Initializing fan controller...
 Filesystem type is ext2fs, partition type 0x83
ACI chassis 
Trying to read config file /boot/grub/menu.lst.local from (hd0,0) 
 Filesystem type is fat, partition type 0xc
Trying to read config file /boot/grub/menu.lst.local from (hd1,4) 
 Filesystem type is ext2fs, partition type 0x83


Security Lock
Booting aci-n9000-dk9.15.0.1k.bin 
Trying diskboot 
 Filesystem type is ext2fs, partition type 0x83
Boot failed 
Booting from drive  failed 
Autoboot image boot failed. Trying recovery image 
Trying to read config file /boot/grub/menu.lst.recovery from (hd1,4) 
 Filesystem type is ext2fs, partition type 0x83


Security Lock
Booting aci-n9000-dk9.15.0.1k.bin 
Trying diskboot 
 Filesystem type is ext2fs, partition type 0x83
Boot failed 
Booting from drive  failed 
No autoboot or failed autoboot. falling to loader 



                Loader Version 5.43


loader > 

This time we will boot straight from the leaf bootflash. Execute boot bootflash:<firmware_name>:

loader > boot bootflash:aci-n9000-dk9.14.2.4i.bin

Security Lock
Booting bootflash:aci-n9000-dk9.14.2.4i.bin 
Trying diskboot 
 Filesystem type is ext2fs, partition type 0x83
Image valid


Image Signature verification was Successful.

Boot Time: 9/14/2021  12:1:31

Security Lock
...

Wait until the leaf will finish the boot process and you will see the login prompt. Scroll down to the Finalizing downgrade topic.

Leaf doesn't go for a reboot

That's good! I had the same behavior when downgrading the N9K-C9348G-FXP leaf. Nothing really you need to do in that case, scroll down to the Finalizing downgrade topic.

Finalizing downgrade

You should see the login prompt:

User Access Verification
(none) login:

Log in as admin without a password. After loging in, make sure the firmware is located on a bootflash:

(none)# dir bootflash
CpuUsage.Log   bios_bootup_scratch_not_cleared  lxc
aci-n9000-dk9.14.2.4i.bin  disk_log.txt    nxos.7.0.3.I7.3.bin
auto-s   libmon.logs

Now delete the auto-s file:

(none)# delete bootflash/auto-s
delete: remove write-protected regular file `bootflash/auto-s'? y

Change the active directory to the bootflash:

(none)# cd /bootflash

Execute setup-bootvar.sh script with the firmware filename as a parameter to create a new bootloader config:

(none)# setup-bootvar.sh aci-n9000-dk9.14.2.4i.bin
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
Done

Now execute another script to clean any possible configuration stored on a switch:

(none)# setup-clean-config.sh
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
Done

And finally reload the leaf to check if the new bootloader config is OK:

(none)# reload
This command will reload the chassis, Proceed (y/n)? [n]: y

You should see as leaf booting normally using the firmware image stored on a bootflash:

Bootable Disk is detected. Device Name: Micron_5300_MTFDDAV240TDS
Version 2.18.1260. Copyright (C) 2020 American Megatrends, Inc.


FPGA SPI Flash Winbond W25Q128BV
Board type  4
IOFPGA @ 0xd8000000
SLOT_ID @ 0xf
Set fan speed to 60%
Initializing fan controller...
 Filesystem type is ext2fs, partition type 0x83
ACI chassis 
Trying to read config file /boot/grub/menu.lst.local from (hd0,4) 
 Filesystem type is ext2fs, partition type 0x83


Security Lock
Booting aci-n9000-dk9.14.2.4i.bin 
Trying diskboot 
 Filesystem type is ext2fs, partition type 0x83
Image valid


Image Signature verification was Successful.

Boot Time: 9/14/2021  12:55:26

Security Lock
...

At this point the process is completed!