Log View and Log Quota Management

You can view log information by device or by log group.

When ADOMs are enabled, each ADOM has its own information displayed in Log View.

Log View can display the real-time log or historical (Analytics) logs.

Log Browse can display logs from both the current, active log file and any compressed log files.

Types of logs collected for each device

FortiAnalyzer can collect logs from managed FortiGate, FortiCarrier, FortiCache, FortiMail, FortiManager, FortiSandbox, FortiWeb, FortiClient, and syslog servers. Following is a description of the types of logs FortiAnalyzer collects from each type of device:

Device Type              Log Type

FortiAnalyzer             Event

FortiAuthenticator      Event

FortiGate                   Traffic Security: Antivirus, Intrusion Prevention, Application Control, Web Filter, File Filter, DNS, Data Leak Prevention, Email Filter, Web Application Firewall, Vulnerability Scan, VoIP, FortiClient Event: Endpoint, HA, Compliance, System, Router, VPN, User, WAN Opt. & Cache, WiFi File Filter logs are sent when the File Filter sensor is enabled in the FortiOS Web Filter profile. You can enable the File Filter sensor in FortiOS at Security Profiles > Web Filters.

FortiCarrier                Traffic, Event, GTP

FortiCache                 Traffic, Event, Antivirus, Web Filter

FortiClient                 Traffic, Event, Vulnerability Scan

FortiDDoS                 Event, Intrusion Prevention

FortiMail                    History, Event, Antivirus, Email Filter.

Device Type              Log Type

FortiMail logs support cross-log functionality. When viewing History, Event, Antivirus, or Email Filter logs from FortiMail, you can click on the Session ID to see correlated logs.

FortiManager             Event

FortiSandbox            Malware, Network Alerts

FortiWeb                   Event, Intrusion Prevention, Traffic

Syslog                       Generic

Traffic logs

Traffic logs record the traffic flowing through your FortiGate unit. Since traffic needs firewall policies to properly flow through FortiGate, this type of logging is also called firewall policy logging. Firewall policies control all traffic attempting to pass through the FortiGate unit, between FortiGate interfaces, zones, and VLAN sub-interfaces.

Security logs

Security logs (FortiGate) record all antivirus, web filtering, file filtering, application control, intrusion prevention, email filtering, data leak prevention, vulnerability scan, and VoIP activity on your managed devices.

DNS logs

DNS logs (FortiGate) record the DNS activity on your managed devices.

Event logs

Event logs record administration management and Fortinet device system activity, such as when a configuration changes, or admin login or HA events occur. Event logs are important because they record Fortinet device system activity which provides valuable information about how your Fortinet unit is performing. FortiGate event logs includes System, Router, VPN, User, and WiFi menu objects to provide you with more granularity when viewing and searching log data.

The logs displayed on your FortiAnalyzer depends on the device type logging to it and the enabled features. FortiGate, FortiCarrier, FortiCache, FortiMail, FortiManager, FortiWeb, FortiSandbox, FortiClient, and Syslog logging is supported. ADOMs must be enabled to support non-FortiGate logging.

In a Security Fabric ADOM, all device logs are displayed.

Log messages

You can view log information by device or by log group.

Viewing the log message list of a specific log type

You can find FortiMail and FortiWeb logs in their default ADOMs.

To view the log message list:

1. If using ADOMs, ensure that you are in the correct ADOM.
2. Go to Log View, and select a log type from the tree menu.

The corresponding log messages list is displayed.

Viewing message details

To view message details:

1. Double-click a message in the message list.

The details pane is displayed to the right of the message list, with the fields categorized in tree view.

You can display the log details pane below the message list by clicking the Bottom icon in the log details pane. When the log details pane is displayed below the message list, you can move it to the right of the log message list by clicking the Right icon. This is sometimes referred to as docking the pane to the bottom or right of the screen.

The log details pane provides shortcuts for adding filters and for showing or hiding a column. Right-click a log field to select an option.

Customizing displayed columns

The columns displayed in the log message list can be customized and reordered as needed.

To customize what columns to display:

1. In the toolbar of the log message list view, click Column Settings and select a column to hide or display. The available columns vary depending on the device and log type.
2. To add other columns, click More Columns. In the Column Settings dialog box, select the columns to show or hide.
3. To reset to the default columns, click Reset to Default.
4. Click OK.

To change the order of the displayed columns:

Place the cursor in the column title and move a column by drag and drop.

Customizing default columns

In Log View, you can select the columns that are displayed as the default by clicking Save as Default in the Column Settings menu when customizing columns. See Customizing displayed columns on page 45.

Customizing the default column view can only be done on a Super_User administrator profile.

Default column customization is applied per devtype/logtype across all ADOMs.

The GUI displays columns based on the following order of priority:

1. Displays the user’s column customizations (if defined).
2. Displays the default columns set by the Super_User administrator (if defined).
3. Displays the system default columns.

Customized default column configuration is preserved during upgrades.

Filtering messages

You can apply filters to the message list. Filters are not case-sensitive by default. If available, select Tools > Case Sensitive Search to create case-sensitive filters.

Filtering messages using filters in the toolbar

1. Go to the view you want.

Regular search	Click Add Filter and select a filter from the dropdown list, then type a value. Only displayed columns are available in the dropdown list. You can use search operators in regular search.
Switching between regular search and advanced search	At the right end of the Add Filter box, click the Switch to Advanced Search icon  or click the Switch to RegularSearch icon .
Advanced search	In Advanced Search mode, enter the search criteria (log field names and values).
Search operators and syntax	If available, click  at the right end of the Add Filter box to view search operators and syntax. See also Filter search operators and syntax on page 48.
CLI string “freestyle” search	Searches the string within the indexed fields configured using the CLI command: config ts-index-field. For example, if the indexed fields have been configured using these CLI commands: config system sql config ts-index-field edit “FGT-traffic” set value “app,dstip,proto,service,srcip,user,utmaction” next end end Then if you type “Skype” in the Add Filter box, FortiAnalyzer searches for “Skype” within these indexed fields: app,dstip,proto,service,srcip,user and utmaction. You can combine freestyle search with other search methods, for example: Skype user=David.

2. In the toolbar, make other selections such as devices, time period, which columns to display, etc.

Filtering messages using the right-click menu

In a log message list, right-click an entry and select a filter criterion. The search criterion with a  icon returns entries matching the filter values, while the search criterion with a  icon returns entries that do not match the filter values.

Depending on the column in which your cursor is placed when you right-click, Log View uses the column value as the filter criteria. This context-sensitive filter is only available for certain columns.

To see log field name of a filter/column, right-click the column of a log entry and select a context-sensitive filter. The Add Filter box shows log field name.

Context-sensitive filters are available for each log field in the log details pane. See Viewing message details on page 44.

Filtering messages using smart action filters

For Log View windows that have an Action column, the Action column displays smart information according to policy (log field action) and utmaction (UTM profile action).

The Action column displays a green checkmark Accept icon when both policy and UTM profile allow the traffic to pass through, that is, both the log field action and UTM profile action specify allow to this traffic.

The Action column displays a red X Deny icon and the reason when either the log field action or UTM profile action deny the traffic.

If the traffic is denied due to policy, the deny reason is based on the policy log field action.

If the traffic is denied due to UTM profile, the deny reason is based on the FortiView threattype from craction. craction shows which type of threat triggered the UTM action. The threattype, craction, and crscore fields are configured in FortiGate in Log & Report. For more information, see the FortiOS -Log Message Reference in the Fortinet Document Library.

A filter applied to the Action column is always a smart action filter.

The smart action filter uses the FortiGate UTM profile to determine what the Action column displays. If the FortiGate UTM profile has set an action to allow, then the Action column will display that line with a green Accept icon, even if the craction field defines that traffic as a threat. The green Accept icon does not display any explanation.

In the scenario where the craction field defines the traffic as a threat but the FortiGate UTM profile has set an action to allow, that line in the Log View Action column displays a green Accept icon. The green Accept icon does not display any explanation.

Filter search operators and syntax

Operators or symbols	Syntax
And	Find log entries containing all the search terms. Connect the terms with a space character, or “and”. Examples: 1.    user=henry group=sales 2.    user=henry and group=sales
Or	Find log entries containing any of the search terms. Separate the terms with “or” or a comma “,”. Examples: 1.    user=henry or srcip=10.1.0.15 2.    user=henry,linda
Not	Find log entries that do NOT contain the search terms. Add “-” before the field name. Example: -user=henry
>, <	Find log entries greater than or less than a value, or within a range. This operator only applies to integer fields. Example: policyid>1 and policyid<10
IP subnet/range search	Find log entries within a certain IP subnet or range. Examples:
Operators or symbols		Syntax
		1.    srcip=192.168.1.0/24 2.    srcip=10.1.0.1-10.1.0.254
Wildcard search		You can use wildcard searches for all field types. Examples: 1.    srcip=192.168.1.* 2.    policyid=1* 3.    user=*

Filtering FortiClient log messages in FortiGate traffic logs

For FortiClient endpoints registered to FortiGate devices, you can filter log messages in FortiGate traffic log files that are triggered by FortiClient.

To Filter FortiClient log messages:

1. Go to Log View > Traffic.
2. In the Add Filter box, type fct_devid=*. A list of FortiGate traffic logs triggered by FortiClient is displayed.
3. In the message log list, select a FortiGate traffic log to view the details in the bottom pane.
4. Click the FortiClient tab, and double-click a FortiClient traffic log to see details.

The FortiClient tab is available only when the FortiGate traffic logs reference FortiClient traffic logs.

Viewing historical and real-time logs

By default, Log View displays historical logs. Custom View and Chart Builder are only available in historical log view.

To view real-time logs, in the log message list view toolbar, click Tools > Real-time Log.

To switch back to historical log view, click Tools > Historical Log.

Viewing raw and formatted logs

By default, Log View displays formatted logs. The log view you select affects available view options. You cannot customize columns when viewing raw logs.

To view raw logs, in the log message list view toolbar, click Tools > Display Raw.

To switch back to formatted log view, click Tools > Formatted Log.

For more information about FortiGate raw logs, see the FortiGate Log Message Reference in the Fortinet Document Library. For more information about raw logs of other devices, see the Log Message Reference for the platform type.

Custom views

Use Custom View to save the filter setting, device selection, and the time period you have specified.

To create a new custom view:

1. If using ADOMs, ensure that you are in the correct ADOM.
2. Go to Log View, and select a log type.
3. In the content pane, customize the log view as needed by adding filters, specifying devices, and/or specifying a time period.
4. In the toolbar, click Custom View.
5. In the Name field, type a name for the new custom view.
6. Click OK. The custom view is now displayed under Log View > Custom View.

To edit a custom view:

1. If using ADOMs, ensure that you are in the correct ADOM.
2. Go to the Log View > Custom View.
3. In the toolbar, edit the filter settings, and click GO.
4. In the toolbar, click Custom View.
5. Click Save to save the changes to the existing custom view or click Save as to save the changes to a new custom view.
6. Click OK.

To view the traffic log of a custom view:

1. If using ADOMs, ensure that you are in the correct ADOM.
2. Go to the Log View > Custom View.
3. Right-click the name of a custom view and select View Traffic.

Downloading log messages

You can download historical log messages to the management computer as a text or CSV file. You cannot download real-time log messages.

To download log messages:

1. If using ADOMs, ensure that you are in the correct ADOM.
2. Go to Log View, and select a log type.
3. In the toolbar, click Tools > Download.
4. In the Download Logs dialog box, configure download options: l In the Log file format dropdown list, select Text or CSV. l To compress the downloaded file, select Compress with gzip.

l To download only the current log message page, select Current Page. To download all the pages in the log message list, select All Pages.

5. Click Download.

Creating charts

Log View includes a Chart Builder for you to build custom charts for each type of log messages.

To create charts with Chart Builder:

1. If using ADOMs, ensure that you are in the correct ADOM.
2. Go to Log View, and select a log type.
3. In the toolbar, click Tools > Chart Builder.
4. In the Chart Builder dialog box, configure the chart and click Save.

Name	Type a name for the chart.
Columns	Select which columns of data to include in the chart based on the log messages that are displayed on the Log View page.
Group By	Select how to group data in the chart.
Order By	Select how to order data in the chart.
Sort	Select a sort order for data in the chart.
Show Limit	Show Limit
Device	Displays the device(s) selected on the Log View page.
Time Frame	Displays the time frame selected on the Log View page.
Query	Displays the query being built.
Preview	Displays a preview of the chart.

Log groups

You can group devices into log groups. You can view FortiView summaries, display logs, generate reports, or create handlers for a log group. Log groups are virtual so they do not have SQL databases or occupy additional disk space.

When you add a device with VDOMs to a log group, all VDOMs are automatically added.

To create a new log group:

1. Go to Log View > Log Group.
2. In the content pane toolbar, click Create New.
3. In the Create New Log Group dialog box, type a log group name and add devices to the log group.
4. Click OK.

Log browse

When a log file reaches its maximum size or a scheduled time, FortiAnalyzer rolls the active log file by renaming the file. The file name is in the form of xlog.N.log, where x is a letter indicating the log type, and N is a unique number corresponding to the time the first log entry was received. For information about setting the maximum file size and log rolling options, see Device logs on page 216.

Log Browse displays log files stored for both devices and the FortiAnalyzer itself, and you can log in the compressed phase of the log workflow.

To view log files:

1. Go to Log View > Log Browse
2. Select a log file, and click Display to open the log file and display the log messages in formatted view.

You can perform all the same actions as with the log message list. See Viewing message details on page 44.

Importing a log file

Imported log files can be useful when restoring data or loading log data for temporary use. For example, if you have older log files from a device, you can import these logs to the FortiAnalyzer unit so that you can generate reports containing older data.

Log files can also be imported into a different FortiAnalyzer unit. Before importing the log file you must add all devices included in the log file to the importing FortiAnalyzer.

To insert imported logs into the SQL database, the config system sqlstart-time and rebuild-eventstart-time must be older than the date of the logs that are imported and the storage policy for analytic data (the Keep Logs forAnalytics field) must also extend back far enough.

To set the SQL start time and rebuild event start time using CLI commands:

config system sql set start-time <start-time-and-date>

set rebuild-event-start-time <start-time-and-date>

end

Where <start-time-and-date> is in the format hh:mm yyyy/mm/dd.

To import a log file:

1. If using ADOMs, ensure that you are in the correct ADOM.
2. Go to Log View > Log Browse and click Import in the toolbar.
3. In the Device dropdown list, select the device the imported log file belongs to or select [Take From Imported File] to read the device ID from the log file.

If you select [Take From Imported File], the log file must contain a device_id field in its log messages.

4. Drag and drop the log file onto the dialog box, or click Add Files and locate the file to be imported on your local computer.
5. Click OK. A message appears, stating that the upload is beginning, but will be canceled if you leave the page.
6. Click OK. The upload time varies depending on the size of the file and the speed of the connection. After the log file is successfully uploaded, FortiAnalyzer inspects the file:
   • If the device_id field in the uploaded log file does not match the device, the import fails. Click Return to try again.
   • If you selected [Take From Imported File] and the FortiAnalyzer unit’s device list does not currently contain that device, an error is displayed stating Invalid Device ID.

Downloading a log file

You can download a log file to save it as a backup or to use outside the FortiAnalyzer unit. The download consists of either the entire log file, or a partial log file, as selected by your current log view filter settings and, if downloading a raw file, the time span specified.

To download a log file:

1. Go to Log View > Log Browse and select the log file that you want to download.
2. In the toolbar, click Download.
3. In the Download Log File(s) dialog box, configure download options:
   • In the Log file format dropdown list, select Native, Text, or CSV.
   • If you want to compress the downloaded file, select Compress with gzip.
4. Click Download.

Deleting log files

To delete log files:

1. Go to Log View > Log Browse.
2. Select one or more files and click Delete.
3. Click OK to confirm.

Log and file storage

Logs and files are stored on the FortiAnalyzer hard disks. Logs are also temporarily stored in the SQL database.

When ADOMs are enabled, settings can be specified for each ADOM that apply only to the devices in it. When ADOMs are disabled, the settings apply to all managed devices.

Data policy and disk utilization settings for devices are collectively called log storage settings. Global log and file storage settings apply to all logs and files, regardless of log storage settings (see File Management on page 220). Both the global and log storage settings are always active.

Disk space allocation

On the FortiAnalyzer, the system reserves 5% to 20% of the disk space for system usage and unexpected quota overflow. The remaining 80% to 95% of the disk space is available for allocation to devices.

Reports are stored in the reserved space.

Total Available Disk Size	Reserved Disk Quota
Small Disk (up to 500GB)	The system reserves either 20% or 50GB of disk space, whichever is smaller.
Medium Disk (up to 1TB)	The system reserves either 15% or 100GB of disk space, whichever is smaller.
Large Disk (up to 3TB)	The system reserves either 10% or 200GB of disk space, whichever is smaller.
Very Large Disk (5TB and higher)	The system reserves either 5% or 300GB of disk space, whichever is smaller.

The RAID level you select determines the disk size and the reserved disk quota level. For example, a FortiAnalyzer 1000C with four 1TB disks configured in RAID 10 is considered a large disk, so 10%, or 100GB, of disk space is reserved.

Log and file workflow

When devices send logs to a FortiAnalyzer unit, the logs enter the following workflow automatically:

1. Compressed logs are received and saved in a log file on the FortiAnalyzer disks.

When a log file reaches a specified size, FortiAnalyzer rolls it over and archives it, and creates a new log file to receive incoming logs. You can specify the size at which the log file rolls over. See Device logs on page 216.

2. Logs are indexed in the SQL database to support analysis.

You can specify how long to keep logs indexed using a data policy. See Log storage information on page 57.

3. Logs are purged from the SQL database, but remain compressed in a log file on the FortiAnalyzer disks.
4. Logs are deleted from the FortiAnalyzer disks.

You can specify how long to keep logs using a data policy. See Log storage information on page 57.

In the indexed phase, logs are indexed in the SQL database for a specified length of time so they can be used for analysis. Indexed, or Analytics, logs are considered online, and details about them can be used viewed in the SOC, Log View, and Incidents & Events panes. You can also generate reports about the logs in the Reports pane.

In the compressed phase, logs are compressed and archived in FortiAnalyzer disks for a specified length of time for the purpose of retention. Compressed, or Archived, logs are considered offline, and their details cannot be immediately viewed or used to generate reports.

The following table summarizes the differences between indexed and compressed log phases:

Log Phase	Location	Immediate Analytic Support
Indexed	Compressed in log file and indexed in SQL database	Yes. Logs are available for analytic use in SOC, Incidents & Events, and Reports.
Compressed	Compressed in log file	No.

Automatic deletion

Logs and files are automatically deleted from the FortiAnalyzer unit according to the following settings:

• Global automatic file deletion

File management settings specify when to delete the oldest Archive logs, quarantined files, reports, and archived files from disks, regardless of the log storage settings. For more information, see File Management on page 220. l Data policy

Data policies specify how long to store Analytics and Archive logs for each device. When the specified length of time expires, Archive logs for the device are automatically deleted from the FortiAnalyzer device’s disks.

• Disk utilization

Disk utilization settings delete the oldest Archive logs for each device when the allotted disk space is filled. The allotted disk space is defined by the log storage settings. Alerts warn you when the disk space usage reaches a configured percentage.

All deletion policies are active on the FortiAnalyzer unit at all times, and you should carefully configure each policy. For example, if the disk fullness policy for a device hits its threshold before the global automatic file deletion policy for the FortiAnalyzer unit, Archive logs for the affected device are automatically deleted. Conversely, if the global automatic file deletion policy hits its threshold first, the oldest Archive logs on the FortiAnalyzer unit are automatically deleted regardless of the log storage settings associated with the device.

The following table summarizes the automatic deletion polices:

Policy	Scope	Trigger
Global automatic file deletion	All logs, files, and reports on the system	When the specified length of time expires, old files are automatically deleted. This policy applies to all files in the system regardless of the data policy settings associated with devices.
Data policy	Logs for the device with which the data policy is associated	When the specified length of retention time expires, old Archive logs for the device are deleted. This policy affects only Archive logs for the device with which the data policy is associated.
Disk utilization	Logs for the device with which the log storage settings are associated	When the specified threshold is reached for the allotted amount of disk space for the device, the oldest Archive logs are deleted for the device. This policy affects only Archive logs for the device with which the log storage settings are associated.

Logs for deleted devices

When you delete one or more devices from FortiAnalyzer, the raw log files and archive packets are deleted, and the action is recorded in the local event log. However, the logs that have been inserted into the SQL database are not deleted from the SQL database. As a result, logs for the deleted devices might display in the Log View and SOC > FortiView panes, and any reports based on the logs might include results.

The following are ways you can remove logs from the SQL database for deleted devices.

• Rebuild the SQL database for the ADOM to which deleted devices belonged or rebuild the entire SQL database.
• Configure the log storage policy. When the deleted device logs are older than the Keep Logs forAnalytics setting, they are deleted. Also, when analytic logs exceed their disk quota, the SQL database is trimmed starting with the oldest database tables. For more information, see Configuring log storage policy on page 59.
• Configure global automatic file deletion settings in System Settings > Advanced > File Management. When the deleted device logs are older than the configured setting, they are deleted. For more information, see File Management on page 220.

Log storage information

To view log storage information and to configure log storage policies, go to System Settings > Storage Info.

If ADOMs are enabled, you can view and configure the data policies and disk usage for each ADOM.

The log storage policy affects only the logs and SQL database of the devices associated with the log storage policy. Reports are not affected. See Disk space allocation on page 54.

The following information and options are available:

Edit	Edit the selected ADOM’s log storage policy.
Refresh	Refresh the page.
Search	Enter a search term to search the list.
Name	The name of the ADOM. ADOMs are listed in two groups: FortiGates and OtherDevice Types.
Analytics (Actual/Config Days)	The age, in days, of the oldest Analytics logs (Actual Days), and the number of days Analytics logs will be kept according to the data policy (Config Days).
Archive (Actual/Config Days)		The age, in days, of the oldest Archive logs (Actual Days) and the number of days Archive logs will be kept according to the data policy (Config Days).
Max Storage		The maximum disk space allotted to the ADOM (for both Analytics and Archive logs). See Disk space allocation on page 54 for more information.
Analytics Usage (Used/Max)		How much disk space Analytics logs have used, and the maximum disk space allotted for them.
Archive Usage (Used/Max)		How much disk space Archive logs have used and the maximum disk space allotted for them.

Storage information

To view log storage policy and statistics, go to System Settings > Storage Info.

The top part of Storage Info shows visualizations of disk space usage for Analytic and Archive logs where the policy diagrams show an overview and the graphs show disk space usage details. The bottom part shows the log storage policy.

The policy diagram shows the percentage of the disk space quota that is used. Hover your cursor over the diagram to view the used, free, and total allotted disk space. The configured length of time that logs are stored is also shown.

The graphs show the amount disk space used over time. Click Max Line to show a line on the graph for the total space allotted. Hover over a spot in the graph to view the used and available disk space at that specific date and time. Click the graph to view a breakdown of the disk space usage by device.

When the used quota approaches 100 percent, a warning message displays when accessing the Storage Statistics pane.

Click Configure Now to open the Edit Log Storage Policy dialog box where you can adjust log storage policies to prevent running out of allocated space (see Configuring log storage policy on page 59), or click Remind Me Later to resolve the issue another time.

Configuring log storage policy

The log storage policy affects the logs and SQL database of the device associated with the log storage policy.

If you change log storage settings, the new date ranges affect Analytics and Archive logs currently in the FortiAnalyzer device. Depending on the date change, Analytics logs might be purged from the database, Archive logs might be added back to the database, and Archive logs outside the date range might be deleted.

To configure log storage settings:

1. Go to System Settings > Storage Info.
2. Double-click on an ADOM, right-click on an ADOM and then select Edit from the menu, or select the ADOM then click Edit in the toolbar. Scroll to the log storage policy sections at the bottom of the Edit Log Storage Policy
3. Configure the following settings, then click OK.

Data Policy
Keep Logs for Analytics	Specify how long to keep Analytics logs.
Keep Logs for Archive	Specify how long to keep Archive logs. Make sure your setting meets your organization’s regulatory requirements.

 

Disk Utilization
Maximum Allowed	Specify the amount of disk space allotted. See also Disk space allocation on page 54.
Analytics : Archive	Specify the disk space ratio between Analytics and Archive logs. Analytics logs require more space than Archive logs. Click the Modify checkbox to change the setting.
Alert and Delete When Usage Reaches	Specify the percentage of allotted disk space usage that will trigger an alert messages and start automatically deleting logs. The oldest Archive log files or Analytics database tables are deleted first.