Troubleshooting
This section offers troubleshooting options for Carrier-related issues.
This section includes:
FortiOS Carrier diagnose commands
Applying IPS signatures to IP packets within GTP-U tunnels
GTP packets are not moving along your network
FortiOS Carrier diagnose commands
This section includes diagnose commands specific to FortiOS Carrier features such as GTP.
GTP related diagnose commands
This CLI command allows you to gain information on GTP packets, logs, statistics, and other information.
diag firewall gtp <command>
| apn list <gtp_profile> | The APN list entries in the specified GTP profile |
| auth-ggsns show <gtp_profile> | The authorized GGSNs entries for the specified GTP profile. Any GGSNs not on this list will not be recognized. |
| auth-sgsns show <gtp_profile> | The authorized SGSNs list entries for the specified GTP profile. Any SGSNs not on this list will not be recognized. |
| handover-grp show <gtp_
profile> |
The handover group showing the range of allowed handover group IP addresses. The handover group acts like a white list of allowed GTP addresses with a default deny at the end — if the GTP address is not on the list, it is denied. |
| ie-remove-policy list <gtp_ profile> | List of IE policies in the IE removal policy for this GTP profile. The information displayed includes the message count for this policy, the length of the SGSN, the list of IEs, and list of SGSN IP addresses. |
| imsi list <gtp_profile> | IMSI filter entries for this GTP profile. The information displayed includes the message count for this filter, length of the IMSI, the length of the APN and IMSI, and of course the IMSI and APN values. |
| invalid-sgsns-to-long list <gtp_ profile> | List of SGSNs that do not match the filter criteria. These SGSNs will be logged. |
| ip-policy list <gtp_profile> | List the IP policies including message count for each policy, the action to take, the source and destination IP addresses or ranges, and masks. |
Applying IPS signatures to IP packets within GTP-U tunnels
| noip-policy <gtp_profile> | List the non-IP policies including the message count, which mode, the action to take, and the start and end protocols to be used by decimal number. |
| path {list | flush} | Select list or flush.
List the GTP related paths in FortiOS Carrier memory. Flush the GTP related paths from memory. |
| policy list <gtp_policy> | The GTP advanced filter policy information for this GTP profile. The information displayed for each entry includes a count for messages matching this filter, a hexidecimal mask of which message types to match, the associated flags, action to take on a match, APN selection mode, MSISDN, RAT types, RAI, ULI, and IMEI. |
| profile list | Displays information about the configured GTP profiles.
You will not be able to see the bulk of the information if you do not log the output to a file. |
| runtime-stat flush | Select to flush the GTP runtime statistics from memory. |
| stat | Display the GTP runtime statistics — details on current GTP activity. This information includes how many tunnels are active, how many GTP profiles exist, how many IMSI filter entries, how many APN filter entries, advanced policy filter entries, IE remove policy filter entries, IP policy filter entries, clashes, and dropped packets. |
| tunnel {list | flush} | Select one of list or flush.
List lists all the GTP tunnels currently active. Flush clears the list of active GTP tunnels. This does not clear the clash counter displayed in the stat command. |
Applying IPS signatures to IP packets within GTP-U tunnels
GTP-U (GTP user data tunnelling) tunnels carry user data packets, signaling messages and error information. GTP-U uses UDP port 2152. Carrier-enabled FortiGate units can apply IPS intrusion protection and detection to GTP-U user data sessions.
To apply IPS to GTP-U user data sessions, add an IPS Sensor to a profile and add the profile to a security policy that accepts GTP-U tunnels. The security policy Service field must be set to GTP or ANY to accept GTP-U packets.
The Carrier-enabled FortiGate unit intercepts packets with destination port 2152, removes the GTP header and handles the packets as regular IP packets. Applying an IPS sensor to the IP packets, the Carrier-enabled FortiGate unit can log attacks and pass or drop packets depending on the configuration of the sensor.
If the packet is GTP-in-GTP, or a nested tunnel, the packets are passed or blocked without being inspected.
To apply an IPS sensor to GTP-U tunnels
- Go to Security Profiles > Intrusion Protection and select Create New (+) to add an IPS Sensor.
- Configure the IPS Sensor to detect attacks and log, drop, or pass attack packets. See the Intrusion Protection section of the FortiOS UTM Guide.
- Go to Policy & Objects > IPv4 Policy and apply the IPS sensor to the security policy.
- Go to Policy & Objects > IPv4 Policy and select Create New to add a security policy or select a security policy.
- Configure the security policy to accept GTP traffic.
In the security policy configure the source and destination settings to match the GTP traffic. Service to GTP or ANY so that the security policy accepts GTP traffic.
- Select the GTP profile within the security policy.
- Configure any other required security policy settings.
- Select OK to save the security policy.
GTP packets are not moving along your network
When GTP packets are not getting to their destination, this could be caused by any one of a number of issues. General troubleshooting principals apply here.
The following sections provide some suggestions on how to troubleshoot this issue:
- Attempt to identify the section of your network with the problem l Ensure you have an APN configured l Check the logs and adjust their settings if required l Check the routing table l Perform a sniffer trace
- Generate specific packets to test the network
Attempt to identify the section of your network with the problem
The first step is to determine how widespread this problem is. Does it affect the whole GPRS network, or just one or two devices?
If the entire network is has this problem, the solution is likely a more general one such as ensuring the security policies allow GTP traffic to pass, the GTP profile specifies SSGNs and GSGNs, or ensuring the GTP general settings are not overly limiting.
If one part of the network is affected, the problem is more likely centered around configurations with those network devices specified such as the handover group, or authorized SGSNs/GGSNs. It is also possible that small portions of the network may have hardware related issues such as cabling or faulty hardware. This section does not address those issues, and assumes hardware is not the problem.
The handover group is a white list of GTP addresses allowed to handle GTP messages. If a device’s address is not on this list, it will be denied.
GTP packets are not moving along your network
Ensure you have an APN configured
When you configure your GTP profile, ensure you first configure the APN. Without it, there will be no flow of traffic. The APN is used in nearly all GTP communications and without it, the Carrier-enabled FortiGate unit doesn’t have the information it needs.
Check the logs and adjust their settings if required
During normal operation, the log settings will show any problems on the network but may not provide the level of details required to fully troubleshoot the problem. The reason for this is that the level of detail required for troubleshooting would quickly overwhelm the daily logs without any real benefit.
GTP related events in the event log will have message IDs in the range 41216 to 41222. For more information on GTP log messages, see the Log Message Reference. For more information on logging in general, see the Logging and Reporting guide.
Once there is a problem to troubleshoot, check the logs to trace the traffic patterns and narrow down the possible sources of the problem. There may be enough detail for you to locate and fix the problem without changing the log settings.
Remember to set any changes you made to the log settings back to their original values when you are done troubleshooting. Otherwise, the amount of detail will overwhelm your logging.
However, if more detail is required you can change settings such as:
- Lower the Log Frequency number in GTP Profiles so fewer or no log messages are dropped. This will allow a more accurate picture of everything happening on the network, where you may have had only a partial picture before.
- Ensure all the GTP log events are enabled to provide you with a complete picture. l Ensure that all relevant event types are enabled under Log & Report > Log Config > Log Settings.
For more information on GTP related logging, see Logging events on the Carrier-enabled FortiGate unit.
General information to look for in the logs includes:
- Are all packets having problems or just certain types? l Are all devices on the network having problem, or just certain devices? l Is it just GTP traffic that is having problems or are all types of traffic having the same problem?
Check the routing table
On any network, the routing table determines how packets reach their destination. This is also true on a carrier network.
If the Carrier-enabled FortiGate unit is running in NAT mode, verify that all desired routes are in the routing table — local subnets, default routes, specific static routes, and dynamic routing protocols. For complete information, it is best to check the routing table in the CLI. This method provides more complete information.
To check the routing table using the CLI
# get router info routing-table all
Codes: K – kernel, C – connected, S – static, R – RIP, B – BGP
O – OSPF, IA – OSPF inter area
N1 – OSPF NSSA external type 1, N2 – OSPF NSSA external type 2 E1 – OSPF external type 1, E2 – OSPF external type 2
i – IS-IS, L1 – IS-IS level-1, L2 – IS-IS level-2, ia – IS-IS inter area * – candidate default
S* 0.0.0.0/0 [10/0] via 192.168.183.254, port2
S 1.0.0.0/8 [10/0] via 192.168.183.254, port2
S 2.0.0.0/8 [10/0] via 192.168.183.254, port2
C 10.142.0.0/23 is directly connected, port3
B 10.160.0.0/23 [20/0] via 10.142.0.74, port3, 2d18h02m C 192.168.182.0/23 is directly connected, port2
Examining an entry from the routing table above:
B 10.160.0.0/23 [20/0] via 10.142.0.74, port3, 2d18h02m
| B | BGP. The routing protocol used. |
| 10.160.0.0/23 | The destination of this route including netmask. |
| [20/0] | 20 indicates and administrative distance of 20 out of a range of 0 to 255.
0 is an additional metric associated with this route, such as in OSPF |
| 10.142.0.74 | The gateway, or next hop. |
| port3 | The interface used by this route. |
| 2d18h02m | How old this route is, in this case almost three days old. |
Perform a sniffer trace
When troubleshooting network traffic, it helps to look inside the headers of packets to determine if they are traveling along the route you expect. Packet sniffing can also be called a network tap, packet capture, or logic analyzing.
If your Carrier-enabled FortiGate unit has NP interfaces that are offloading traffic, this will change the sniffer trace. Before performing a trace on any NP interfaces, disable offloading on those interfaces.
What can sniffing packets tell you
If you are running a constant traffic application such as ping, packet sniffing can tell you if the traffic is reaching the destination, what the port of entry is on the Carrier-enabled FortiGate unit, if the ARP resolution is correct, and if the traffic is being sent back to the source as expected.
GTP packets are not moving along your network
Sniffing packets can also tell you if the Carrier-enabled FortiGate unit is silently dropping packets for reasons such as RPF (Reverse Path Forwarding), also called Anti Spoofing. This prevents an IP packet from being forwarded if its source IP address either does not belong to a locally attached subnet (local interface), or be a hop on the routing between the FortiOS Carrier and another source (static route, RIP, OSPF, BGP). Note that RPF can be disabled by turning on asymmetric routing in the CLI (config system setting, set asymmetric enable), however this will disable stateful inspection on the Carrier-enabled FortiGate unit and consequently cause many features to be turned off.
If you configure virtual IP addresses on your Carrier-enabled FortiGate unit, the unit will use those addresses in preference to the physical IP addresses. If not configured properly, secondary IP addresses can cause a broadcast storm. You will notice the secondary address being preferred when you are sniffing packets because all the traffic will be using the virtual IP addresses. This is due to the ARP update that is sent out when the VIP address is configured.
How to sniff packets
The general form of the internal FortiOS packet sniffer command is: diag sniffer packet <interface_name> <‘filter’> <verbose> <count>
To stop the sniffer, type CTRL+C.
| <interface_name> | The name of the interface to sniff, such as port1 or internal. This can also be any to sniff all interfaces. |
| <‘filter’> | What to look for in the information the sniffer reads. none indicates no filtering, and all packets will be displayed as the other arguments indicate.
The filter must be inside single quotes (‘). |
| <verbose> | The level of verbosity as one of:
1 – print header of packets 2 – print header and data from IP of packets 3 – print header and data from Ethernet of packets |
| <count> | The number of packets the sniffer reads before stopping. If you don’t put a number here, the sniffer will run forever unit you stop it with <CTRL C>. |
For a simple sniffing example, enter the CLI command diag sniffer packet port1 none 1 3. This
will display the next 3 packets on the port1 interface using no filtering, and using verbose level 1. At this verbosity level you can see the source IP and port, the destination IP and port, action (such as ack), and sequence numbers.
In the output below, port 443 indicates these are HTTPS packets, and 172.20.120.17 is both sending and receiving traffic.
Head_Office_620b # diag sniffer packet port1 none 1 3 interfaces=[port1] filters=[none]
0.545306 172.20.120.17.52989 -> 172.20.120.141.443: psh 3177924955 ack 1854307757
0.545963 172.20.120.141.443 -> 172.20.120.17.52989: psh 1854307757 ack 3177925808
0.562409 172.20.120.17.52988 -> 172.20.120.141.443: psh 4225311614 ack 3314279933
Generate specific packets to test the network
If some packets are being delivered as expected while others are not, or after you believe you have fixed the problem, it is a good idea to generate specific traffic to test your network.
For example if you discover through log messages and packet sniffing that Create PDP Context Request messages are not being delivered between two SGSNs, you can generate those specific messages on your network to confirm they are the problem, and later that you have solved the problem and they are now being delivered as expected.
This step requires a third party traffic generation tool, either hardware or software. This is not be supported by Fortinet.