Firmware

Firmware upgrading and downgrading sounds pretty simple, anyone can do it, right? The mark of a professional is not that they can do something correctly, or even do it correctly over and over again. A professional works in such a way that, if anything goes wrong they are prepared and able to quickly get things back to normal. Firmware updates can go wrong just like anything else. So a real professional does things in a way that minimizes their risk and follows some best practices, as listed below.

Firmware change management

Consider the following five points when performing firmware upgrades, not only in FortiOS but in general. This applies to pretty much any change you have to do in a production environment.

Understanding the new version first

Before attempting any changes in production, first make sure you set up a laboratory where you can freely play with the new features, and understand them with enough time and no pressure. Read the Release Notes, Manuals, and other documentation like presentations, videos, or podcasts about the new version.

You are ready to explain the need for an upgrade once you understand:

l The differences and the enhancements between the new version and the previous version(s). l The impact of the upgrade on customers and the users of the operating platform. l The known limitations that might affect your environment. l The potential risks when performing the upgrade. l The licensing changes that may apply.

Have a valid reason to upgrade

The reason can NOT be “Because I want to have the latest version”. The reason has to be explained in terms of business, technical, and/or operational improvement.

Affirmative answers to the following questions are valid reasons to upgrade:

• Does the new version have a feature that helps to ensure compliance?
• Does the new version have an enhancement that allows 40% decrease (40% improvement) on the time to perform a certain operation?
• Does the new feature correct a known defect/bug found on a previous version that affects the company business/operations?
• Will the new version allow your organization to deploy new services that will help to gain new customers or increase loyalty of existing ones? l Is the vendor cutting support for the version your organization is currently using?

If the best reason to upgrade is “Because the new features seem to be cool” or “Because I want to have the latest version”, a little more understanding and planning may be necessary.

Prepare an upgrade plan

If you choose to upgrade because you found a valid reason to do so, make sure you create a plan that covers business, technical, and operational aspects of the upgrade:

Business:

Proper planning and justification for an upgrade should be proportional to how critical the system is to the business.

• Make sure you can clearly articulate the benefits of the upgrade in business terms (time, money, and efficiency). l Understand the business processes that will be affected by the change.
• Make sure the upgrade maintenance window is not close to a business-critical process (such as quarterly or monthly business closure).
• Obtain executive and operational approval for the maintenance window. The approval must come from the owners of ALL the systems/information affected by the upgrade, not only from those that own the system being upgraded.

The approval must be done in a formal (written or e-mail) form.

Technical and operational:

• Re-read the Release Notes for the technology you are upgrading. Supported hardware models, upgrade paths, and known limitations should be clearly understood.
• Make sure your upgrade maintenance window does not overlap with any other maintenance window on your infrastructure.
• If you have any premium support offer (such as TAM, Premium Support), do a capacity planning exercise to ensure the new firmware/software version does not take more hardware resources than you currently have.
• Create a backup, whether or not you have scheduled backups. Create a new fresh backup. l Obtain offline copies of both the currently installed firmware and the new version.
• Create a list of systems with inter-dependencies to the system you are upgrading. For example, if you are upgrading a FortiGate; understand the impact on any FortiAP, FortiAuthenticator, FortiToken, FortiManager, or FortiAnalyzer you have on your environment. l Ensure you have a list of adjacent devices to the upgrading platform and have administrative access to them, just in case you need to do some troubleshooting. Are you upgrading FortiWeb? Make sure you can administratively access the Web Applications. Are you upgrading a FortiGate? Make sure you can administratively access the surrounding switches and routers.
• Have a step-by-step plan on how to perform and test the upgrade. You want to make sure you think of the worst situation before it happens, and have predefined courses of action, instead of thinking under pressure when something already went wrong.
• Define a set of tests (that include critical business applications that should be working) to make sure the upgrade went fine. If any test does not go well, define which ones mandate a rollback and which ones can be tolerated for further troubleshooting. This set of tests should be run before and after the upgrade to compare results, and they should be the same.
• Define a clear rollback plan. If something goes wrong with the upgrade or the tests, the rollback plan will help you get your environment back to a known and operational status. The plan must clearly state the conditions under which the rollback will be started.
• Declare configuration freezes. A little bit before and after the upgrade. The idea is to reduce the amount of variables to take into consideration if something goes wrong.
• Perform a “Quality Assurance” upgrade. Grab a copy of the production configuration, load it on a non-production box and execute the upgrade there to see if there are any issues on the process. Then adjust your plan according to the results you obtained.
• Have a list of information elements to be gathered if something goes wrong. This ensures that, even if the upgrade fails, you will collect enough information so you can troubleshoot the issue without needing to repeat the problem. Get help from Fortinet Support if you need to check what else could be missing on your list.
• Define a test monitoring period after the change was completed. Even if the upgrade went smoothly, something could still go wrong. Make sure you monitor the upgraded system for at least one business cycle. Business cycles may be a week, a month, or a quarter, depending on your organization’s business priorities.

Execute the upgrade plan

Execution of an upgrade is just as key as planning.

Once you are performing the upgrade, the pressure will rise and stress might peak. This is why you should stick to the plan you created with a cool head.

Resist the temptation to take decisions while performing the upgrade, as your judgment will be clouded by the stress of the moment, even if a new decision seems to be “obvious” at such time. If your plan says you should rollback, then execute the rollback despite the potential “We-can-fix-this-very-quickly” mentality.

While performing the upgrade, make sure all the involved components are permanently monitored before, during, and after the upgrade, either via monitoring systems, SNMP alerts, or at least with tools like a ping. Critical resources like CPU, memory, network, and/or disk utilization must also be constantly monitored.

To avoid misunderstandings, when performing the tests for each critical application defined on the planning, make sure there are formal notifications on the results for each user area, service, system, and/or application tested.

Regardless if you have to rollback or not, if a problem occurs, make sure you gather as much information about the problem as possible, so you can later place a Support ticket to find a solution.

Last but not least, document the upgrade:

• Enable your terminal emulation program to leave trace of all the commands executed and all the output generated. If you are performing steps via GUI, consider using a video capture tool to document it. l Document any command or change performed over the adjacent/interdependent systems. Make sure they are acknowledged by the relevant administrators
• Document any deviations performed over the upgrade plan. This is planned-versus-actual.

Learn more about change management

Change Management and Change Control are huge knowledge areas in the field of Information Systems and Computer/Network Security.

This document is by no means a comprehensive list on what you should do when performing an upgrade, with either

Fortinet or any other technology. It is merely a list of important things you should take into consideration when

performing upgrades which are the result of years of experience dealing with changes on critical environments, as it is common that security devices are protecting critical applications and processes.

There are vast resources on the topic: books, public white papers, blog entries, etc. If you search the Internet for the “Change Control Best Practices” or “Change Management Best Practices” you will get many interesting documents.

Performing a firmware upgrade

Upgrading a firewall is something that should be compared to upgrading the operating system on your computer. It’s not to be taken lightly! You want to make sure everything is backed up and you have some options available if things go awry. Assuming it all seems to work you also want a list of things to do in order to confirm everything is working properly. Finally, you need enough time to do it. All really simple stuff, but what does this mean in relation to upgrading your FortiGate? It means, you follow these simple steps:

1. Backup and store old configuration (full configuration backup from CLI).

Digging into this a little, step 1 is easy to understand. Do a full backup of your old configuration. This is all part of your disaster recovery plan. If the upgrade fails in some way you need to make sure you can get the Firewall back up and running. The best way to do this is to get it back to a state where you know what the behavior was. For more information, refer to Performing a configuration backup on page 18.

2. Have copy of old firmware available.

Step 2, is also part of your disaster recovery. If the upgrade fails you might be able to switch the active partition. But as a Professional, you need to be prepared for the worst case scenario where you can’t do that. Which means you’ll need your old firmware.

3. Have disaster recovery option on standby — especially if remote.

Step 3, is your plan for what to do in the event of a critical failure. As we’re talking FortiGate this means that your firewall doesn’t come back after the upgrade. What this means is that you need to be able to get to the console port in order to find out why. Maybe it’s DHCP and the IP changed, maybe the OS is corrupt, who knows? Get to the console and find out.

There could be a simple fix. If there’s not, then be prepared for a format and TFTP reload.

4. Read the release notes, including the upgrade path and bug information.

Step 4, READ THE RELEASE NOTES. They contain all kinds of information, known bugs, fixed bugs even upgrade issues like lost configuration settings. Not all upgrade information is ever contained in any products release notes. That does not mean they are devoid of good/useful information. Read them, digest them, then a few days later read them again.

5. Double check everything.

Step 5, do a double check of everything. Is your TFTP server working, does your console connection function, is there anything in the release notes that could impact your upgrade procedure, do you have your configuration backed up? Make sure you’ve done everything.

Step 6, do the upgrade. Doing an upgrade doesn’t take very long, a few minutes (less a lot of times) but make sure you schedule enough time for it. At the end of the day an upgrade can succeed or fail. If it succeeds you want some time to check/confirm that any important features you have are working (VPNs etc). If it fails you’ll need time to sort things out.

Performing a firmware downgrade

Just like upgrading, you need to make sure it’s done properly. While similar, the steps are somewhat different since there are other pitfalls in this case.

1. Locate pre-upgrade configuration file.

Step 1 is very important. This is why, when you upgrade you make a backup of your old configuration and save it. If you don’t, then you’ll need to rebuild manually.

2. Have copy of old firmware available.

Step 2 is fairly obvious. Even with devices that have multiple partitions and your downgrade process is simply going to be to switch the active partition, this could go wrong. In which case, you may be without Internet access. A professional has a plan for when things go wrong.

3. Have disaster recovery option on standby — especially if remote.

Step 3 is no different from before. Hopefully you don’t need to format the unit, but be prepared for that, just in case.

4. Read the release notes — is a downgrade possible, or necessary?

Step 4, once again, is to READ THE RELEASE NOTES. In this case, you will need to do this for the version you are on, and the version you are downgrading too, and everything in between (if you are going back multiple major releases or patches). Maybe the OS switched from 32 to 64 bits somewhere between the two firmware releases. In order to make sure you don’t get nailed by something like that you need to check the upgrade and downgrade information in every major release and patch, as it may have a direct impact on your options.

5. Double check everything.
6. Downgrade — all settings, except those needed for access, are lost.

Step 5 and 6 are the same as before. Double check everything, then downgrade.

7. Restore pre-upgrade configuration.

Step 7 is new. Obviously most settings are lost when you downgrade so in order to get back up and running you will need to restore your old configuration file.

Performing a configuration backup

Once you configure the FortiGate unit and it is working correctly, it is extremely important that you backup the configuration. In some cases, you may need to reset the FortiGate unit to factory defaults or perform a TFTP upload of the firmware, which will erase the existing configuration. In these instances, the configuration on the device will have to be recreated, unless a backup can be used to restore it.

It is also recommended that once any further changes are made that you backup the configuration immediately, to ensure you have the most current configuration available. Also, ensure you backup the configuration before upgrading the FortiGate unit’s firmware. Should anything happen during the upgrade that changes the configuration, you can easily restore the saved configuration.

Always backup the configuration and store it on the management computer or off-site. You have the option to save the configuration file to various locations including the local PC, USB key, FTP and TFTP site.The latter two are configurable through the CLI only.

If you have VDOMs, you can back up the configuration of the entire FortiGate unit or only a specific VDOM. Note that if you are using FortiManager or FortiCloud, full backups are performed and the option to backup individual VDOMs will not appear.

To back up the FortiGate configuration – GUI:

1. Go to Dashboard.
2. On the System Information widget, select Backup next to System Configuration.
3. Select to backup to your Local PC or to a USB Disk.

The USB Disk option will be grayed out if no USB drive is inserted in the USB port. You can also backup to the FortiManager using the CLI.

4. If VDOMs are enabled, select to backup the entire FortiGate configuration (Full Config) or only a specific VDOM configuration (VDOM Config).
5. If backing up a VDOM configuration, select the VDOM name from the list.
6. Select Encrypt configuration file.

Encryption must be enabled on the backup file to back up VPN certificates.

7. Enter a password and enter it again to confirm it. You will need this password to restore the file.
8. Select Backup.
9. The web browser will prompt you for a location to save the configuration file. The configuration file will have a .conf extension.

To back up the FortiGate configuration – CLI:

execute backup config management-station <comment>

… or …

execute backup config usb <backup_filename> [<backup_password>] … or for FTP (note that port number, username are optional depending on the FTP site)… execute backup config ftp <backup_filename> <ftp_server> [<port>] [<user_name>] [<password>]

… or for TFTP … execute backup config tftp <backup_filename> <tftp_servers> <password> Use the same commands to backup a VDOM configuration by first entering the commands:

config vdom edit <vdom_name>

Backing up a configuration file using SCP

You can use secure copy protocol (SCP) to download the configuration file from the FortiGate unit as an alternative method of backing up the configuration file or an individual VDOM configuration file. This is done by enabling SCP for and administrator account and enabling SSH on a port used by the SCP client application to connect to the FortiGate unit. SCP is enabled using the CLI commands:

config system global set admin-scp enable end

Use the same commands to backup a VDOM configuration by first entering the commands:

config global

set admin-scp enable

end config vdom

edit <vdom_name>