How to diagnose HA out of sync messages
This section describes how to use the commands diagnose sys ha showcsum and diagnose debug to diagnose the cause of HA out of sync messages.
If HA synchronization is not successful, use the following procedures on each cluster unit to find the cause.
To determine why HA synchronization does not occur
1. Connect to each cluster unit CLI by connected to the console port.
2. Enter the following commands to enable debugging and display HA out of sync messages.
diagnose debug enable
diagnose debug console timestamp enable diagnose debug application hatalk -1 diagnose debug application hasync -1
Collect the console output and compare the out of sync messages with the information on page 203.
3. Enter the following commands to turn off debugging.
diagnose debug disable diagnose debug reset
To determine what part of the configuration is causing the problem
If the previous procedure displays messages that include sync object 0x30 (for example, HA_SYNC_SETTING_ CONFIGURATION = 0x03) there is a synchronization problem with the configuration. Use the following steps to determine the part of the configuration that is causing the problem.
If your cluster consists of two cluster units, use this procedure to capture the configuration checksums for each unit. If your cluster consists of more that two cluster units, repeat this procedure for all cluster units that returned messages that include 0x30 sync object messages.
1. Connect to each cluster unit CLI by connected to the console port.
2. Enter the following command to turn on terminal capture
diagnose debug enable
3. Enter the following command to stop HA synchronization.
execute ha sync stop
4. Enter the following command to display configuration checksums.
diagnose sys ha showcsum 1
5. Copy the output to a text file.
6. Repeat for all affected units.
7. Compare the text file from the primary unit with the text file from each cluster unit to find the checksums that do not match.
You can use a diff function to compare text files.
8. Repeat steps 4 to 7 for each checksum level:
diagnose sys ha showcsum 2 diagnose sys ha showcsum 3 diagnose sys ha showcsum 4 diagnose sys ha showcsum 5 diagnose sys ha showcsum 6 diagnose sys ha showcsum 7 diagnose sys ha showcsum 8
9. When the non-matching checksum is found, attempt to drill down further. This is possible for objects that have sub-components.
For example you can enter the following commands:
diagnose sys ha showcsum system.global diagnose sys ha showcsum system.interface
Generally it is the first non-matching checksum in one of the levels that is the cause of the synchronization problem.
10. Attempt to can remove/change the part of the configuration that is causing the problem. You can do this by making configuration changes from the primary unit or subordinate unit CLI.
11. Enter the following commands to start HA configuration and stop debugging:
execute ha sync start diagnose debug disable diagnose debug reset
Recalculating the checksums to resolve out of sync messages
Sometimes an error can occur when checksums are being calculated by the cluster. As a result of this calculation error the CLI console could display out of sync error messages even though the cluster is otherwise operating normally. You can also sometimes see checksum calculation errors in diagnose sys ha showcsum command output when the checksums listed in the debugzone output don’t match the checksums in the checksum part of the output.
One solution to this problem could be to re-calculate the checksums. The re-calculated checksums should match and the out of sync error messages should stop appearing.
You can use the following command to re-calculate HA checksums:
diagnose sys ha csum-recalculate [<vdom-name> | global]
Just entering the command without options recalculates all checksums. You can specify a VDOM name to just recalculate the checksums for that VDOM. You can also enter global to recalculate the global checksum.