sitemap

RSS地图

收藏本站

设为首页

Oracle研究中心

当前位置:Oracle研究中心 > 故障案例 >

【案例】Oracle RAC IPC send timeout error导致RAC的节点挂起解决办法

时间:2016-11-22 22:11   来源:Oracle研究中心   作者:网络   点击:

天萃荷净 Oracle研究中心案例分析:运维DBA反映Oracle RAC环境数据库节点挂起,分享日志发现是由于IPC send timeout error导致RAC的节点挂起。

本站文章除注明转载外,均为本站原创: 转载自love wife & love life —Roger 的Oracle技术博客
本文链接地址: “IPC send timeout error” 导致RAC的节点挂起

一同事管理的一套3 nodes rac,上周出现故障,由于其中一个节点出现IPC send timeout,导致另外2个node挂住,应用无法访问,最后只能重启node。

根据 alert 整理了一个简单的时间列表:
###### sndb3 ######

Fri Nov 25 09:26:08 2011   IPC Send timeout detected. Receiver ospid 340396   -- sndb3
..........  
Fri Nov 25 09:28:54 2011   IPC Send timeout detected. Receiver ospid 340396   -- sndb3

Fri Nov 25 10:04:19 2011   Trace dumping is performing id=[cdmp_20111125100426]

Fri Nov 25 10:10:07 2011   Warning: cluster_database_instances (2) of node 0 is <= my node id (2)

Fri Nov 25 10:12:20 2011   Reconfiguration started  -- 开始  reconfig 此时 List of nodes: 0 2

Fri Nov 25 10:28:05 2011   Reconfiguration started  -- 第2次 reconfig List of nodes: 0 1 2

Fri Nov 25 10:49:03 2011   Reconfiguration started  -- 第3次 reconfig List of nodes: 2

Fri Nov 25 10:49:50 2011   Reconfiguration started  -- 第4次 reconfig List of nodes: 1 2

Fri Nov 25 10:55:17 2011   Reconfiguration started  -- 第5次 reconfig List of nodes: 2

Fri Nov 25 10:55:18 2011   Reconfiguration complete

Fri Nov 25 10:55:19 2011   Shutting down instance (abort)  -- abort 数据库

Fri Nov 25 10:56:47 2011   Starting ORACLE instance -- 正常启动数据库
###### sndb1 ######

Fri Nov 25 09:26:29 2011   IPC Send timeout detected.Sender: ospid 352952
                           Receiver: inst 3 binc -981855203 ospid 340396

Fri Nov 25 09:26:32 2011   ORA-27506: IPC error connecting to a port

Fri Nov 25 09:26:33 2011   ORA-07445: exception encountered: core dump [] [] [] [] [] []

Fri Nov 25 09:26:38 2011   IPC Send timeout detected.Sender: ospid 258258
                           Receiver: inst 3 binc -981855203 ospid 340396

Fri Nov 25 09:27:05 2011   ORA-27506: IPC error connecting to a port

                           ORA-07445: exception encountered: core dump [] [] [] [] [] []

Fri Nov 25 09:42:09 2011   ORA-07445: exception encountered: core dump [] [] [] [] [] []
                           ORA-03135: connection lost contact
                           ORA-03135: connection lost contact

Fri Nov 25 10:12:41 2011   Reconfiguration started  -- 第1次 reconfig List of nodes: 0 2

Fri Nov 25 10:22:55 2011   IPC Send timeout detected.Sender: ospid 127746
                           Receiver: inst 3 binc -981855203 ospid 340396
.............
.............

Fri Nov 25 10:22:58 2011   IPC Send timeout detected.Sender: ospid 299594
                           Receiver: inst 3 binc -981855203 ospid 340396

Fri Nov 25 10:22:58 2011   ORA-27506: IPC error connecting to a port

Fri Nov 25 10:22:59 2011   ORA-07445: exception encountered: core dump [] [] [] [] [] []
.............

Fri Nov 25 10:23:25 2011   ORA-07445: exception encountered: core dump [] [] [] [] [] []

Fri Nov 25 10:28:26 2011   Reconfiguration started -- 第2次 reconfig List of nodes: 0 1 2

Fri Nov 25 10:33:55 2011   Reconfiguration complete

Fri Nov 25 10:39:14 2011   IPC Send timeout detected.Sender: ospid 377580
                           Receiver: inst 3 binc -981855203 ospid 340396

Fri Nov 25 10:39:18 2011   ORA-27506: IPC error connecting to a port
                           ORA-07445: exception encountered: core dump [] [] [] [] [] []

Fri Nov 25 10:49:19 2011   ORA-29746: Cluster Synchronization Service is being shut down.

Fri Nov 25 10:49:20 2011   ORA-27300: OS system dependent operation:invalid_process failed with status: 0
                           ORA-27301: OS failure message: Error 0
                           ORA-27302: failure occurred at: skgpstime3
                           ORA-27144: attempt to kill process failed
                           ORA-29746: Cluster Synchronization Service is being shut down.
                           Instance terminated by LMON, pid = 365348

Fri Nov 25 11:02:12 2011   Starting ORACLE instance (normal) -- 正常启动数据库实例 sndb1
###### sndb2 ######

Fri Nov 25 09:26:20 2011   Trace dumping is performing id=[cdmp_20111125092634]

Fri Nov 25 09:26:24 2011   IPC Send timeout detected.Sender: ospid 352604
                           Receiver: inst 3 binc -981855203 ospid 340396

Fri Nov 25 09:26:50 2011   ORA-27506: IPC error connecting to a port
                           ORA-07445: exception encountered: core dump [] [] [] [] [] []

Fri Nov 25 10:04:24 2011   ORA-07445: exception encountered: core dump [] [] [] [] [] []

Fri Nov 25 10:04:25 2011   ORA-07445: exception encountered: core dump [] [] [] [] [] []
                           ORA-03113: end-of-file on communication channel

Fri Nov 25 10:10:13 2011   ORA-07445: exception encountered: core dump [] [] [] [] [] []
                           ORA-03135: connection lost contact
                           ORA-03135: connection lost contact

Fri Nov 25 10:12:23 2011   Shutting down instance (abort)  -- abort 数据库

Fri Nov 25 10:12:23 2011   ORA-27300: OS system dependent operation:invalid_process failed with status: 0
                           ORA-27301: OS failure message: Error 0
                           ORA-27302: failure occurred at: skgpstime3
                           ORA-27144: attempt to kill process failed

Fri Nov 25 10:12:58 2011   Instance terminated by USER, pid = 304124

Fri Nov 25 10:27:52 2011   Starting ORACLE instance (normal) -- 正常启动数据库

Fri Nov 25 10:28:10 2011   lmon registered with NM - instance id 2 (internal mem no 1)

Fri Nov 25 10:28:12 2011   Reconfiguration started  -- 第1次 reconfig  List of nodes:0 1 2

Fri Nov 25 10:33:23 2011   IPC Send timeout detected.Sender: ospid 529138

Fri Nov 25 10:33:26 2011   ORA-27506: IPC error connecting to a port

Fri Nov 25 10:33:34 2011   ORA-00482: LMD* process terminated with error

Fri Nov 25 10:33:36 2011   Shutting down instance (abort)  -- 第2次 abort 数据库
...............
...............
Fri Nov 25 10:34:47 2011   ORA-27300: OS system dependent operation:invalid_process failed with status: 0
                           ORA-27301: OS failure message: Error 0
                           ORA-27302: failure occurred at: skgpstime3
                           ORA-27144: attempt to kill process failed

Fri Nov 25 10:34:53 2011   Instance terminated by USER, pid = 446862

Fri Nov 25 10:49:39 2011   Starting ORACLE instance (normal) -- 第2次正常启动数据库

Fri Nov 25 10:49:57 2011   Reconfiguration started -- 第2次 reconfig List of nodes: 1 2

Fri Nov 25 10:56:39 2011   Instance terminated by USER, pid = 205414

Fri Nov 25 11:03:59 2011   Starting ORACLE instance (normal) -- 第3次正常启动数据库

Fri Nov 25 11:04:17 2011   Reconfiguration started -- 第3次 reconfig List of nodes: 0 1 2

整体的梳理情况如上,据同事反映,该rac在25号9~10点之间,由于报错ipc send out,导致1,2两个node hang住,他的直接描述是应用都无法连接1,2 node了,最后他通过srvctl强制停掉1,2 节点。

我们知道,出现ipc超时,而已引起系统系统挂住的情况,可能有如下几种:

1. cpu负载很高
2. 内存不足
3. 网络问题,如心跳异常
4. 系统参数设置,比如udp,tcp相关参数设置过小
5. crs和db版本不一致(这点我不确定,是因为这里的情况是crs 10.2.0.4, db 10.2.0.1), 这点是我临时加上的。


下面我们来一一排查进行分析:

sndb3 节点在故障之间的 osw 监控信息如下:
zzz ***Fri Nov 25 09:25:46 EAT 2011

System configuration: lcpu=16 mem=63936MB

kthr    memory              page              faults        cpu   
----- ----------- ------------------------ ------------ -----------
r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa
0  0 4002889 11233515   0   0   0   0    0   0 6238 240716 20451 11  6 81  2
2  0 4002865 11233537   0   0   0   0    0   0 6215 240551 21467 10  6 82  2
2  0 4002895 11233502   0   0   0   0    0   0 5594 239027 19584 11  5 82  2

zzz ***Fri Nov 25 09:26:16 EAT 2011

System configuration: lcpu=16 mem=63936MB

kthr    memory              page              faults        cpu   
----- ----------- ------------------------ ------------ -----------
r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa
0  1 4002579 11229653   0   0   0   0    0   0 1950 31763 8699  4  2 88  5
0  1 4002657 11229073   0   0   0   0    0   0 2325 27718 9141  3  2 90  5
2  2 4002661 11228588   0   0   0   0    0   0 1891 22869 8332  4  1 90  5

zzz ***Fri Nov 25 09:26:46 EAT 2011

System configuration: lcpu=16 mem=63936MB

kthr    memory              page              faults        cpu   
----- ----------- ------------------------ ------------ -----------
r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa
1  0 4009439 11223429   0   0   0   0    0   0 820 21974 4532  4  2 95  0
1  0 4009445 11223419   0   0   0   0    0   0 797 13064 4497  3  1 96  0
1  0 4009445 11223416   0   0   0   0    0   0 621 11103 3988  3  1 96  0

zzz ***Fri Nov 25 09:27:16 EAT 2011

System configuration: lcpu=16 mem=63936MB

kthr    memory              page              faults        cpu   
----- ----------- ------------------------ ------------ -----------
r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa
3  1 4012753 11217337   0   0   0   0    0   0 303 229232 3042  6 20 73  0
2  0 4012780 11216595   0   0   0   0    0   0 612 107627 3761  5  9 86  0
2  0 4012780 11216592   0   0   0   0    0   0 770 12774 4312  3  1 96  0

从上可以看到,sndb3 在ipc send超时出现时,系统cpu和内存均非常空闲,下面来检查网络:
zzzFri Nov 25 09:25:46 EAT 2011

Name  Mtu   Network     Address            Ipkts Ierrs    Opkts Oerrs  Coll
en3   1500  link#2      0.21.5e.6.8.37    56004124     0   948458    10     0
                        01:00:5e:00:00:01
en3   1500  192.168.0   192.168.0.3       56004124     0   948458    10     0
                        224.0.0.1         
en0   1500  link#3      0.21.5e.6.8.f2    623180180     0 574030148     3     0
                        01:00:5e:00:00:01
en0   1500  10.10.10    10.10.10.12       623180180     0 574030148     3     0
                        224.0.0.1         
en1   1500  link#4      0.21.5e.6.8.f3           0     0  1401015 1401015     0
                        01:00:5e:00:00:01
en1   1500  10.76.200   10.76.200.132            0     0  1401015 1401015     0
                        224.0.0.1         
en2   1500  link#5      0.21.5e.6.8.36    435553601     0 119387543     3     0
                        01:00:5e:00:00:01
en2   1500  10.76.1     10.76.1.143       435553601     0 119387543     3     0
                        224.0.0.1         
en2   1500  10.76.1     10.76.1.144       435553601     0 119387543     3     0
                        224.0.0.1         
lo0   16896 link#1                        14167657     0 14181993     0     0
lo0   16896 127         127.0.0.1         14167657     0 14181993     0     0
                        224.0.0.1         
lo0   16896 ::1                           14167657     0 14181993     0     0
                        ff01::1         
                        ff02::1:ff00:1  
                        ff02::1         
icmp:
95502471 calls to icmp_error
0 errors not generated because old message was icmp
Output histogram:
  echo reply: 4295876
  destination unreachable: 226000
872 messages with bad code fields
0 messages < minimum length
12 bad checksums
0 messages with bad length
Input histogram:
  echo reply: 1458209
  destination unreachable: 56666
  echo: 4295876
  time exceeded: 191
4295876 message responses generated
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
8 membership reports sent
tcp:
122743924 packets sent
  116431735 data packets (2302053184 bytes)
  25761 data packets (10003811 bytes) retransmitted
  3553016 ack-only packets (3361721 delayed)
  0 URG only packets
  13 window probe packets
  2423518 window update packets
  309881 control packets
  28566873 large sends
  1403066833 bytes sent using largesend
  65160 bytes is the biggest largesend
386118499 packets received
  369946371 acks (for 2302274662 bytes)
  74335 duplicate acks
  0 acks for unsent data
  98730125 packets (1777353999 bytes) received in-sequence
  23016 completely duplicate packets (3194142 bytes)
  0 old duplicate packets
  20 packets with some dup. data (8594 bytes duped)
  35986 out-of-order packets (2602936 bytes)
  0 packets (0 bytes) of data after window
  0 window probes
  49947 window update packets
  7 packets received after close
  0 packets with bad hardware assisted checksum
  0 discarded for bad checksums
  0 discarded for bad header offset fields
  0 discarded because packet too short
  13 discarded by listeners
  0 discarded due to listener's queue full
  283911590 ack packet headers correctly predicted
  15339170 data packet headers correctly predicted
59734 connection requests
59629 connection accepts
83810 connections established (including accepts)
119948 connections closed (including 1831 drops)
0 connections with ECN capability
0 times responded to ECN
35446 embryonic connections dropped
351698548 segments updated rtt (of 112846079 attempts)
0 segments with congestion window reduced bit set
0 segments with congestion experienced bit set
0 resends due to path MTU discovery
36306 path MTU discovery terminations due to retransmits
203836 retransmit timeouts
  439 connections dropped by rexmit timeout
189 fast retransmits
  26 when congestion window less than 4 segments
134 newreno retransmits
27 times avoided false fast retransmits
30 persist timeouts
  0 connections dropped due to persist timeout
46287 keepalive timeouts
  9999 keepalive probes sent
  36288 connections dropped by keepalive
0 times SACK blocks array is extended
0 times SACK holes array is extended
0 packets dropped due to memory allocation failure
0 connections in timewait reused
0 delayed ACKs for SYN
0 delayed ACKs for FIN
0 send_and_disconnects
0 spliced connections
0 spliced connections closed
0 spliced connections reset
0 spliced connections timeout
0 spliced connections persist timeout
0 spliced connections keepalive timeout
udp:
702526878 datagrams received
0 incomplete headers
3 bad data length fields
5 bad checksums
95502471 dropped due to no socket
47780240 broadcast/multicast datagrams dropped due to no socket
100987 socket buffer overflows <== Note here.
559143172 delivered
559613903 datagrams output
ip:
1106726708 total pachttp://www.oracleplus.netkets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
13553929 fragments received
9 fragments dropped (dup or out of space)
841 fragments dropped after timeout
3122140 packets reassembled ok
1094399939 packets for this host
56393 packets for unknown/unsupported protocol
0 packets forwarded
207190 packets not forwardable
0 redirects sent
693606490 packets sent from this host
991 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
4245736 output datagrams fragmented
20081318 fragments created
396 datagrams that can't be fragmented
1631395 IP Multicast packets dropped due to no receiver
0 successful path MTU discovery cycles
0 path MTU rediscovery cycles attempted
0 path MTU discovery no-response estimates
0 path MTU discovery response timeouts
0 path MTU discovery decreases detected
0 path MTU discovery packets sent
0 path MTU discovery memory allocation failures
0 ipintrq overflows
2 with illegal source
0 packets processed by threads
0 packets dropped by threads
14925 packets dropped due to the full socket receive buffer
0 dead gateway detection packets sent
0 dead gateway detection packet allocation failures
0 dead gateway detection gateway allocation failures

ipv6:
0 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with incorrect version number
0 with illegal source
0 input packets without enough memory
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets reassembled ok
0 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded
0 packets not forwardable
0 too big packets not forwarded
0 packets sent from this host
0 packets sent with fabricated ipv6 header
0 output packets dropped due to no bufs, etc.
0 output packets without enough memory
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 packets dropped due to the full socket receive buffer
0 packets not delivered due to bad raw IPv6 checksum
icmpv6:
0 calls to icmp6_error
0 errors not generated because old message was icmpv6
Output histogram:
  unreachable: 0
  packets too big: 0
  time exceeded: 0
  parameter problems: 0
  redirects: 0
  echo requests: 0
  echo replies: 0
  group queries: 0
  group reports: 0
  group terminations: 0
  router solicitations: 0
  router advertisements: 0
  neighbor solicitations: 0
  neighbor advertisements: 0
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Input histogram:
  unreachable: 0
  packets too big: 0
  time exceeded: 0
  parameter problems: 0
  echo requests: 0
  echo replies: 0
  group queries: 0
   bad group queries: 0
  group reports: 0
   bad group reports: 0
   our groups' reports: 0
  group terminations: 0
  bad group terminations: 0
  router solicitations: 0
  bad router solicitations: 0
  router advertisements: 0
  bad router advertisements: 0
  neighbor solicitations: 0
  bad neighbor solicitations: 0
  neighbor advertisements: 0
  bad neighbor advertisements: 0
  redirects: 0
  bad redirects: 0
  mobility calls when not started: 0
  home agent address discovery requests: 0
  bad home agent address discovery requests: 0
  home agent address discovery replies: 0
  bad home agent address discovery replies: 0
  prefix solicitations: 0
  bad prefix solicitations: 0
  prefix advertisements: 0
  bad prefix advertisements: 0
0 message responses generated
从上面的 100987 socket buffer overflows 信息来看,似乎有包溢出,从这里看可能是udp相关参数设置有问题。

下面再来分析下crs的日志以及lmd,lmon等关键性进程的trace:
*** 2011-11-25 09:26:38.408
IPC Send timeout detected.Sender: ospid 258258
Receiver: inst 3 binc -981855203 ospid 340396
SKGXPCTX: 0x11053d158 ctx
wait delta 0 sec (14 msec) ctx ts 0x4b421 last ts 0x4b422
user cpu time since last wait 0 sec 0 ticks
system cpu time since last wait 0 sec 0 ticks
locked 1
blocked 1
timed wait receives 0
admno 0x3c67bf54 admport:
SSKGXPT 0x1053d4a4 flags  active network 0
info for network 0
socket no 11  IP 10.10.10.10  UDP 36952
HACMP network_id 0 sflags SSKGXPT_UP
context timestamp 0x4b421
buffers queued on port 11057a888
    sconno     accono   ertt  state   seq#   sent  async   sync rtrans   acks
       ach     accono     sconno      admno  state   seq#    rcv rtrans   acks
0x1105d9050 0x2c61e946 0x48f33837 0x1368ad14     40  32764      1      0      1
Submitting synchronized dump request [268435460]
*** 2011-11-25 09:27:05.144
ORA-27506: IPC error connecting to a port
Assertion failed: 0, file  skgxpu.c, line 8568
*** 2011-11-25 09:27:05.144
NOTICE: Process received signal SIGLOST. Therefore, the call
        stack does not point to the location of the problem.
        Take a look at /var/adm/messages to find why this
        process got SIGLOST
Exception signal: 6 0x9000000002832b4 (pthread_kill+0080) e8410028
Registers:

#if defined (SIGLOST) /* resource lost (eg, record-lock lost) */
  signal_names[SIGLOST] = "SIGLOST";
#endif
###### sndb3 的lmon trace如下 ######

*** 2011-11-25 10:12:20.202
kjfcrfg: DRM window size = 2048->2048 (min lognb = 14)
*** 2011-11-25 10:12:20.210
Reconfiguration started (old inc 6, new inc 8)
Synchronization timeout interval: 1020 sec
List of nodes:
0 2
*** 2011-11-25 10:12:20.213
Global Resource Directory frozen
node 0
node 2
asby init, 0/0/x2
asby returns, 0/0/x2/false
* Domain maps before reconfiguration:
*   DOMAIN 0 (valid 1): 0 1 2
* End of domain mappings
* kjbdomrcfg2: domain 0 invalid = TRUE
* Domain maps after recomputation:
*   DOMAIN 0 (valid 0): 0 2
* End of domain mappings
Dead  inst 1
Join  inst
Exist inst 0 2
KSXPLOOKUP: warning cnh incarnation number may be stale 8 8 6
KSXPLOOKUP: warning cnh incarnation number may be stale 8 8 6
KSXPLOOKUP: warning cnh incarnation number may be stale 8 8 6
KSXPLOOKUP: warning cnh incarnation number may be stale 8 8 6
KSXPLOOKUP: warning cnh incarnation number may be stale 8 8 6
Active Sendback Threshold = 50 %
Communication channels reestablished
sent syncr inc 8 lvl 1 to 0 (8,5/0/0)
synca inc 8 lvl 1 rcvd (8.5)
replayed dom info (8.6)
received master 0 (8.6)
sent syncr inc 8 lvl 2 to 0 (8,7/0/0)
synca inc 8 lvl 2 rcvd (8.7)
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Set master node info
sent syncr inc 8 lvl 3 to 0 (8,13/0/0)
synca inc 8 lvl 3 rcvd (8.13)
Submitted all remote-enqueue requests
kjfcrfg: Number of mesgs sent to node 0 = 2204
sent syncr inc 8 lvl 4 to 0 (8,15/0/0)
synca inc 8 lvl 4 rcvd (8.15)
Dwn-cvts replayed, VALBLKs dubious
sent syncr inc 8 lvl 5 to 0 (8,18/0/0)
synca inc 8 lvl 5 rcvd (8.18)
All grantable enqueues granted
sent syncr inc 8 lvl 6 to 0 (8,20/0/0)
synca inc 8 lvl 6 rcvd (8.20)
*** 2011-11-25 10:12:22.061
Submitted all GCS cache requests
sent syncr inc 8 lvl 7 to 0 (8,22/0/0)
synca inc 8 lvl 7 rcvd (8.22)
Fix write in gcs resources
sent syncr inc 8 lvl 8 to 0 (8,24/0/0)
synca inc 8 lvl 8 rcvd (8.24)
*** 2011-11-25 10:12:23.459

其他2个node的crsd ocssd 以及lmon lmd等trace也没有发现有价值的信息,而且在9:26~10:12之间都没任何信息。

只有lmd trace中有点有价值的信息:
SKGXPDOCON:connection refused: out of  memory
SSKGXPID 0xffffb640 network 0 Internet address 10.10.10.11  UDP port number 61800
SKGXPDOCON:connection refused: out of  memory
SSKGXPID 0xffffb640 network 0 Internet address 10.10.10.11  UDP port number 61800
*** 2011-11-25 09:26:20.449
SKGXPDOCON:connection refused: out of  memory
SSKGXPID 0xffffb640 network 0 Internet address 10.10.10.10  UDP port number 36992
SKGXPDOCON:connection refused: out of  memory
SSKGXPID 0xffffb640 network 0 Internet address 10.10.10.10  UDP port number 33548
*** 2011-11-25 09:27:15.598
Received ORADEBUG command 'dump errorstack 1' from process Unix process pid: 275164, image:
*** 2011-11-25 09:27:15.598

另外通过oraadebug ipc,我们也能看到rac心跳确实是使用的udp协议,如下:
Shared Memory:
ID  KEY
10485768 0xe1892820
Maximum processes:               = 750
ksxpdmp:  facility 0 ( ) (0x1, 0x0) counts 0, 0
ksxpdmp:  Dumping the osd context
SKGXPCTX: 0x1105a4c78 ctx
wait delta 13 sec (13149 msec) ctx ts 0xb last ts 0xb
user cpu time since last wait -171798692 sec -1 ticks
system cpu time since last wait 0 sec 0 ticks
locked 1
blocked 0
timed wait receives 0
admno 0x104b9dec admport:
SSKGXPT 0x105a4fc4 flags SSKGXPT_READPENDING active network 0
info for network 0
socket no 7  IP 10.10.10.12  UDP 55895 <== Note here.
HACMP network_id 0 sflags SSKGXPT_UP
context timestamp 0xb
no ports
    sconno     accono   ertt  state   seq#   sent  async   sync rtrans   acks
0x3115c6c7 0x6e0d721f     32      3  32765      2      2      0      0      2
另外os方面也没有发现有价值的信息,故障期间的 awr top5 如下:


关于ges/gcs enter server mode 这两个等待事件表现正常,表现为该实例在进行recover,smaon进程表现出的等待为该event。

下面是google到的解释:
In a RAC instance the during the instance recovery, first pass scan can be delayed by 300ms-1.5s waiting
on GRD (Global resource directory). During that time SMAON enters the wait event which is called "ges enter server mode" .
另外有点值得关注是,SQ相关等待较高。


通过下面的Global Cache and Enqueue Services - Messaging Statistics 也能看出cluster interconnect 通信消耗还是挺大的。

另外mos上找到一个相关的文档,不过跟这个情况看上去不符合,该文档如下:

IPC Send timeout detected and Instance eviction on RDBMS and ASM 10.2.0.1 RAC [ID 370182.1]


最后给出的邮件回复如下:

该参数设置过小可能会导致rac node挂起, 故认为第2种可能性更大。

当前系统该值为:

            udp_recvspace = 655360
            udp_sendspace = 65536

根据官方文档,建议修改为如下值:

            udp_recvspace = 2621440(该参数可以不调整)
            udp_sendspace = 262144

udp_sendspace 官方文档推荐值是(DB_BLOCK_SIZE * DB_MULTIBLOCK_READ_COUNT) + 4 KB


参数调整以后,要监控一段时间。

补充:从awr来看,top 5中 enq: SQ - contention 等待较高,建议调大sequence的cache值(默认值20较小)
TOYS_SEQ_ZF 下存在SEQUENCE,通过如下方式进行修改:

ALTER SEQUENCE owner.sequence_name cache 1000;

检查SEQUENCE cache值:

SET LINES 200
SET pagesize 1000
col owner FOR a20
col SEQUENCE_NAME FOR a30

SELECT owner,sequence_name,MAX_VALUE,INCREMENT_BY,CYCLE_FLAG,ORDER_FLAG,CACHE_SIZE
  FROM dba_sequence;

--------------------------------------ORACLE-DBA----------------------------------------

最权威、专业的Oracle案例资源汇总之【案例】Oracle RAC IPC send timeout error导致RAC的节点挂起解决办法

本文由大师惜分飞原创分享,网址:http://www.oracleplus.net/arch/1312.html

Oracle研究中心

关键词:

Oracle RAC IPC send timeout error解决笔记

IPC send timeout error导致RAC的节点挂起解决办法