Troubleshooting Oracle RAC Node Eviction

The RAC Node Eviction Troubleshooting Tool is designed to help diagnose and resolve RAC node eviction issues by analyzing uploaded log files. When a known solution exists, the tool provides detailed symptoms, causes, and recommended solutions.

Benefits of Using the RAC Node Eviction Troubleshooting Tool

• Automated Analysis: The tool examines uploaded logs and suggests fixes when applicable.
• Troubleshooting Reports: Reports can be saved for future reference.
• Diagnostic Guide: Offers step-by-step troubleshooting guidance.
• Simplified SR Creation: Automatically populates fields when creating a Service Request (SR).

Required Information for Troubleshooting

To effectively diagnose a RAC node eviction, gather the following details:

1. Identify the Evicted Node

Two methods can be used:

• On the Evicted Node: Log in and execute:

  # uname -n
  

• Using Clusterware Logs: Check the alert.log for a message containing “Reconfiguration Complete” to find active nodes.

  Example:

  CRS-1601: CSSD Reconfiguration complete. Active nodes are node3 node4.
  

2. Determine the Exact Time of Eviction

• For a node that rebooted:

  # last reboot
  

  Example Output:

  reboot   system boot   2.6.14-235.0.0.0   Mon Oct 21 12:26  
  reboot   system boot   2.6.14-235.0.0.0   Tue Oct 22 16:10  
  

• For eviction without reboot: Check alert.log for when Clusterware restarted.

  Example Log Entry:

  [cssd(2254)] CRS-1601: CSSD Reconfiguration complete. Active nodes are rac3 rac4.
  

  The above log entry shows Clusterware restarted at 2024-12-28 09:49:45.219.

Data Collection for Analysis

For Linux/UNIX (11gR2, 12.1.0.1, 19c), execute the following as the root user:

# script /tmp/diag.log
# id
# env
# cd <temp-directory-with-plenty-of-free-space>
# $GRID_HOME/bin/diagcollection.sh
# exit

Uploading Diagnostic Files

The following .gz files will be generated in the current directory. Upload these files along with /tmp/diag.log for analysis:

• crsData_<hostname>.tar.gz
• ocrData_<hostname>.tar.gz
• oraData_<hostname>.tar.gz
• coreData_<hostname>.tar.gz (if --core option is specified)
• os_<hostname>.tar.gz

Using the above steps, you can effectively troubleshoot and resolve Oracle RAC node eviction issues.

If needed, create a Service Request (SR) with the collected logs for further assistance.