You may receive your new leaves with the firmware version newer than you currently use. There's a pretty simple way to upgrade leaves firmware in case if it's older than you use with the help of basic operations in APIC, but no well-documented steps on how to downgrade firmware. More to say, it's not possible with APIC (at least I wasn't able to find out how to do that).
Requirements
You will need:
- a USB flash drive
- n9k firmware file for ACI-mode
- Console access to the leaf
Preparing USB flash drive
You need to have a FAT32 formatted flash drive. If you already have one with FAT32 it's fine, if not - format it. Copy downloaded firmware file on a flash. You can download firmware from the cisco.com site. Follow the link and select your leaf hardware type, then select NX-OS System Software-ACI software type and download firmware for the ACI version you're currently using.
Downgrading firmware
Insert the flash drive into a USB port on a leaf. Access the leaf via console. I'm assuming that we're dealing with the brand new clean leaf.
If your leaf is already booted up, you should see the login banner:
User Access Verification
(none) login:Login as admin. No password will be requested. After login, reload the leaf.
After that, or if you just powered on the leaf, you will see the boot process:
CISCO MODULE
BIOS Ver: 5.43
Switch G5
RC Revision: 02.03.00
Memory Information:
MRC Revision:00.50.00
Total DRAM: 32768 MB
Memory TOLM: 80000000
PCIE BASE: 80000000 Size : 10000000
PCI32 BASE: 90000000 Limit: FBFFFFFF
PCI64 BASE: 80000000000 Limit: 83FFFFFFFFF
UC START: 80000000000 End : 84000000000
ME Operational Firmware Version: 06:3.0.3.100
DIMM Information:
Clock Speed: 1067MHz
Socket: 0x0 Channel: 0x0 Number: 0x0 Presence: Yes Size: 32GB
Socket: 0x0 Channel: 0x0 Number: 0x1 Presence: No
Socket: 0x0 Channel: 0x1 Number: 0x0 Presence: No
Socket: 0x0 Channel: 0x1 Number: 0x1 Presence: NoThis is provided as an example from my leaf switch, so the output in your case may differ.
Press Ctrl+C multiple times until the loader prompt will be displayed:
Aborting config file read and autoboot
No autoboot or failed autoboot. falling to loader
Loader Version 5.43
loader > Now run the dir command to get the flash drive designator and to make sure that the firmware file is in there:
loader > dir
usb1::
System Volume Information
aci-n9000-dk9.14.2.4i.bin
bootflash::
aci-n9000-dk9.15.0.1k.bin
CpuUsage.Log
lxc
disk_log.txt
nxos.7.0.3.I7.3.bin
auto-s
libmon.logs
.stats_pref.txt
bios_bootup_scratch_not_cleared
As you can see, in my case the flash drive is named as usb1. Now we need to boot from the firmware located on a flash drive. Execute command boot <flash_name>:<firmware_name>, where <flash_name> is the name of the USB flash drive and <firmware_name> is the filename for your firmware located on a flash:
loader > boot usb1:aci-n9000-dk9.14.2.4i.bin
Security Lock
Booting usb1:aci-n9000-dk9.14.2.4i.bin
Trying diskboot
Filesystem type is fat, partition type 0xc
Image valid
Image Signature verification was Successful.
Boot Time: 9/14/2021 11:54:21
Security Lock
...Now the process has two paths:
- Leaf goes for a reboot after booting
- Leaf doesn't go for a reboot after booting
Leaf goes for a reboot
As it was in my case with the N9K-C93180YC-FX, the leaf went for a reboot after these messages:
[ 336.389531] @@@cctrli: wrote 132 to scratch RR
[ 336.443171] nvram_klm wrote rr=132 rr_str=Resetting switch. LPSS restore from SQL failed. to nvram
[ 336.548899] Collected 9 ext4 filesystems
[ 336.599115] Freezing filesystems
[ 336.695232] Collected 0 ubi filesystems
[ 336.742224] Freezing filesystems
[ 336.782965] Done freezing filesystems
[ 336.828912] Putting SSD in stdby
[ 336.922668] Done putting SSD in stdby 0
[ 336.969662] Done offlining SSD
[ 337.007273] Writing reg=0x84 val=0x80000000So we will see the boot process again, as it was when we started the leaf first. But now the boot process will fail, as we already have the new firmware file stored on a bootflash of a leaf. IDK why, but after we boot from the USB flash drive, the firmware is automatically copied to the bootflash and the original firmware image is deleted. Basically, the leaf tries to boot a non-existing image following the bootloader configuration.
Again you will see the loader prompt, but now you don't need to press Ctrl+C:
PGA SPI Flash Winbond W25Q128BV
Board type 4
IOFPGA @ 0xd8000000
SLOT_ID @ 0xf
Set fan speed to 60%
Initializing fan controller...
Filesystem type is ext2fs, partition type 0x83
ACI chassis
Trying to read config file /boot/grub/menu.lst.local from (hd0,0)
Filesystem type is fat, partition type 0xc
Trying to read config file /boot/grub/menu.lst.local from (hd1,4)
Filesystem type is ext2fs, partition type 0x83
Security Lock
Booting aci-n9000-dk9.15.0.1k.bin
Trying diskboot
Filesystem type is ext2fs, partition type 0x83
Boot failed
Booting from drive failed
Autoboot image boot failed. Trying recovery image
Trying to read config file /boot/grub/menu.lst.recovery from (hd1,4)
Filesystem type is ext2fs, partition type 0x83
Security Lock
Booting aci-n9000-dk9.15.0.1k.bin
Trying diskboot
Filesystem type is ext2fs, partition type 0x83
Boot failed
Booting from drive failed
No autoboot or failed autoboot. falling to loader
Loader Version 5.43
loader > This time we will boot straight from the leaf bootflash. Execute boot bootflash:<firmware_name>:
loader > boot bootflash:aci-n9000-dk9.14.2.4i.bin
Security Lock
Booting bootflash:aci-n9000-dk9.14.2.4i.bin
Trying diskboot
Filesystem type is ext2fs, partition type 0x83
Image valid
Image Signature verification was Successful.
Boot Time: 9/14/2021 12:1:31
Security Lock
...Wait until the leaf will finish the boot process and you will see the login prompt. Scroll down to the Finalizing downgrade topic.
Leaf doesn't go for a reboot
That's good! I had the same behavior when downgrading the N9K-C9348G-FXP leaf. Nothing really you need to do in that case, scroll down to the Finalizing downgrade topic.
Finalizing downgrade
You should see the login prompt:
User Access Verification
(none) login:
Log in as admin without a password. After loging in, make sure the firmware is located on a bootflash:
(none)# dir bootflash
CpuUsage.Log bios_bootup_scratch_not_cleared lxc
aci-n9000-dk9.14.2.4i.bin disk_log.txt nxos.7.0.3.I7.3.bin
auto-s libmon.logsNow delete the auto-s file:
(none)# delete bootflash/auto-s
delete: remove write-protected regular file `bootflash/auto-s'? yChange the active directory to the bootflash:
(none)# cd /bootflash
Execute setup-bootvar.sh script with the firmware filename as a parameter to create a new bootloader config:
(none)# setup-bootvar.sh aci-n9000-dk9.14.2.4i.bin
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
Done
Now execute another script to clean any possible configuration stored on a switch:
(none)# setup-clean-config.sh
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
DoneAnd finally reload the leaf to check if the new bootloader config is OK:
(none)# reload
This command will reload the chassis, Proceed (y/n)? [n]: yYou should see as leaf booting normally using the firmware image stored on a bootflash:
Bootable Disk is detected. Device Name: Micron_5300_MTFDDAV240TDS
Version 2.18.1260. Copyright (C) 2020 American Megatrends, Inc.
FPGA SPI Flash Winbond W25Q128BV
Board type 4
IOFPGA @ 0xd8000000
SLOT_ID @ 0xf
Set fan speed to 60%
Initializing fan controller...
Filesystem type is ext2fs, partition type 0x83
ACI chassis
Trying to read config file /boot/grub/menu.lst.local from (hd0,4)
Filesystem type is ext2fs, partition type 0x83
Security Lock
Booting aci-n9000-dk9.14.2.4i.bin
Trying diskboot
Filesystem type is ext2fs, partition type 0x83
Image valid
Image Signature verification was Successful.
Boot Time: 9/14/2021 12:55:26
Security Lock
...At this point the process is completed!