sitemap

RSS地图

收藏本站

设为首页

Oracle研究中心

当前位置:Oracle研究中心 > 故障案例 >

【案例】Oracle服务器diag进程占据了12g的磁盘空间分析解决办法

时间:2016-11-13 20:10   来源:Oracle研究中心   作者:网络   点击:

天萃荷净 Oracle研究中心案例分析:巡检时发现某套rac的awr居然没有生成,而且手工创建快照也没反应。发现某个节点的/oracle 一共20g,oracle本身才8G左右,另外的12g不知道被什么东西占据了,ls -ltr居然无法看见,经主机工程师检查,发现是diag进程占据了12g的磁盘空间。

在操作之前,我做了一个systemstate dump,对该dump我简单的分析了一下,发现了如下有用信息:

System global information:
     processes: base c0000000ad649960, size 500, cleanup c0000000ad68c020
     allocation: free sessions c0000000ad78d620, free calls 0000000000000000
     control alloc errors: 0 (process), 0 (session), 0 (call)
     PMON latch cleanup depth: 0
     seconds since PMON's last scan for dead processes: 19  ###### 有19个死进程

###### 我们首先来搜索diag进程:######
PROCESS 3:
  —————————————-
  SO: c0000000ad68c810, type: 2, owner: 0000000000000000, flag: INIT/-/-/0x00
  (process) Oracle pid=3, calls cur/top: c0000000adab8358/c0000000adab8358, flag: (2) SYSTEM
            int error: 0, call error: 0, sess error: 0, txn error 0
  (post info) last post received: 0 0 24
              last post received-location: ksasnd
              last process to post me: c0000000ad6956f0 235 0
              last post sent: 0 0 48
              last post sent-location: ksoreq_reply
              last process posted by me: c0000000ad782910 1 0
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: c0000000ad784f20
    O/S info: user: oracle, term: UNKNOWN, ospid: 27628
    OSD pid info: Unix process pid: 27628, image: oracle@uamdb2 (DIAG)

###### 然后搜索c0000000ad68c810 字符串,发现如下内容:######
    SO: c0000000ab2bac00, type: 19, owner: c0000000ad68c810, flag: INIT/-/-/0x00
     GES MSG BUFFERS: st=emp chunk=0x0000000000000000 hdr=0x0000000000000000 lnk=0x0000000000000000 flags=0x0 inc=0
      outq=0 sndq=0 opid=0 prmb=0x0
      mbg[i]=(0 0) mbg[b]=(0 0) mbg[r]=(0 0)
      fmq[i]=(0 0) fmq[b]=(0 0) fmq[r]=(0 0)
      mop[s]=0 mop[q]=0 pendq=0 zmbq=0
      nonksxp_recvs=0
    ————process 0xc0000000ab2bac00——————–
    proc version      : 0
    Local node        : 1
    pid               : 27628
    lkp_node          : 1
    svr_mode          : 0
    proc state        : KJP_UNFREEZE
    Last drm hb acked : 0
    Total accesses    : 5
    Imm.  accesses    : 0
    Locks on ASTQ     : 0
    Locks Pending AST : 0
    Granted locks     : 0
    AST_Q:
    PENDING_Q:
    GRANTED_Q:
    —————————————-
    SO: c0000000ada72528, type: 4, owner: c0000000ad68c810, flag: INIT/-/-/0x00
    (session) sid: 554 trans: 0000000000000000, creator: c0000000ad68c810, flag: (51) USR/- BSY/-/-/-/-/-
              DID: 0000-0000-00000000, short-term DID: 0000-0000-00000000
              txn branch: 0000000000000000
              oct: 0, prv: 0, sql: 0000000000000000, psql: 0000000000000000, user: 0/SYS
    service name: SYS$BACKGROUND
    waiting for 'DIAG idle wait' blocking sess=0x0000000000000000 seq=331 wait_time=0 seconds since wait started=62193
                component=1, where=1, wait time(millisec)=c8
    Dumping Session Wait History
     for 'DIAG idle wait' count=1 wait_time=195274
                component=1, where=1, wait time(millisec)=c8
     for 'DIAG idle wait' count=1 wait_time=195286
                component=1, where=1, wait time(millisec)=c8
     for 'DIAG idle wait' count=1 wait_time=214760
                component=1, where=1, wait time(millisec)=c8
     for 'DIAG idle wait' count=1 wait_time=195267
                component=1, where=1, wait time(millisec)=c8
     for 'DIAG idle wait' count=1 wait_time=195282
                component=1, wherhttp://www.oracleplus.nete=1, wait time(millisec)=c8
     for 'DIAG idle wait' count=1 wait_time=195282
                component=1, where=1, wait time(millisec)=c8
     for 'DIAG idle wait' count=1 wait_time=195281
                component=1, where=1, wait time(millisec)=c8
     for 'DIAG idle wait' count=1 wait_time=195278
                component=1, where=1, wait time(millisec)=c8
     for 'DIAG idle wait' count=1 wait_time=195280
                component=1, where=1, wait time(millisec)=c8
     for 'DIAG idle wait' count=1 wait_time=195270
                component=1, where=1, wait time(millisec)=c8
    temporary object counter: 0
      —————————————-
      UOL used : 0 locks(used=0, free=0)
      KGX Atomic Operation Log c0000000acb7a5a0
       Mutex 0000000000000000(0, 0) idn 0 oper NONE
       Library Cache uid 554 efd 0 whr 0 slp 0
      KGX Atomic Operation Log c0000000acb7a5e8
       Mutex 0000000000000000(0, 0) idn 0 oper NONE
       Library Cache uid 554 efd 0 whr 0 slp 0
      KGX Atomic Operation Log c0000000acb7a630
       Mutex 0000000000000000(0, 0) idn 0 oper NONE
       Library Cache uid 554 efd 0 whr 0 slp 0
      —————————————-
      SO: c0000000a1a7fa18, type: 41, owner: c0000000ada72528, flag: INIT/-/-/0x00
      (dummy) nxc=0, nlb=0 
    —————————————-
    SO: c0000000adab8358, type: 3, owner: c0000000ad68c810, flag: INIT/-/-/0x00
    (call) sess: cur c0000000ada72528, rec 0, usr c0000000ada72528; depth: 0
    —————————————-
    SO: c0000000ab2adec8, type: 16, owner: c0000000ad68c810, flag: INIT/-/-/0x00
    (osp req holder)

###### 然后来看mmon进程:######
   SO: c0000000add59f60, type: 11, owner: c0000000ad692f40, flag: INIT/-/-/0x00
    (broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: c0000000ad692f40,
                       event: 1, last message event: 1381841,
                       last message waited event: 1381841, messages read: 1376494
                       channel: (c0000000add81260) MMON remote action broadcast channel
                                scope: 27, event: 1381841, last mesage event: 1381841,
                                publishers/subscribers: 2/2,
                                messages published: 1376494
    —————————————-
    SO: c0000000ada60ee0, type: 4, owner: c0000000ad692f40, flag: INIT/-/-/0x00
    (session) sid: 541 trans: 0000000000000000, creator: c0000000ad692f40, flag: (51) USR/- BSY/-/-/-/-/-
              DID: 0002-0010-00000003, short-term DID: 0002-0010-00000004
              txn branch: 0000000000000000
              oct: 0, prv: 0, sql: c0000000ac7f3760, psql: 0000000000000000, user: 0/SYS
    service name: SYS$BACKGROUND
    waiting for 'rdbms ipc message' blocking sess=0x0000000000000000 seq=49622 wait_time=0 seconds since wait started=2
                timeout=12c, =0, =0
    Dumping Session Wait History
     for 'rdbms ipc message' count=1 wait_time=1503849
                timeout=99, =0, =0
     for 'rdbms ipc message' count=1 wait_time=1435494
                timeout=12c, =0, =0
     for 'rdbms ipc message' count=1 wait_time=2939422
                timeout=12c, =0, =0
     for 'rdbms ipc message' count=1 wait_time=517538
                timeout=34, =0, =0
     for 'rdbms ipc message' count=1 wait_time=2421822
                timeout=12c, =0, =0
     for 'rdbms ipc message' count=1 wait_time=2470581
                timeout=fc, =0, =0
     for 'rdbms ipc message' count=1 wait_time=1
                timeout=fc, =0, =0
     for 'rdbms ipc message' count=1 wait_time=468661
                timeout=12c, =0, =0
     for 'rdbms ipc message' count=1 wait_time=2933950
                timeout=12c, =0, =0
     for 'os thread startup' count=1 wait_time=41005
                =0, =0, =0

另外看了几个其他进程的信息,发现都是在等待 rdbms ipc message。
记得当时还做了一个oradebug hanganalyze 3,发现lck0进程居然阻塞了451个objects。
由于当时的产生的trace我忘记取下来了, 所以就只能用这个trace中去寻找信息了。

###### 搜索lck字符串:######

   PROCESS 20:
  —————————————-
  SO: c0000000ad694f00, type: 2, owner: 0000000000000000, flag: INIT/-/-/0x00
  (process) Oracle pid=20, calls cur/top: c0000000adabb810/c0000000adabb810, flag: (6) SYSTEM
            int error: 0, call error: 0, sess error: 0, txn error 0
  (post info) last post received: 0 0 33
              last post received-location: ksrpublish
              last process to post me: c0000000ad694f00 1 6
              last post sent: 0 0 33
              last post sent-location: ksrpublish
              last process posted by me: c0000000ad694f00 1 6
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: c0000000ad784f20
    O/S info: user: oracle, term: UNKNOWN, ospid: 27684
    OSD pid info: Unix process pid: 27684, image: oracle@uamdb2 (LCK0)

###### 然后搜索字符串c0000000ad694f00 ######

      SO: c0000000adbb9fd0, type: 5, owner: c0000000ad694f00, flag: INIT/-/-/0x00
    (enqueue) XR-00000000-00000000 DID: 0000-0014-00000003
    lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x3
    res: 0xc0000000adcbf510, mode: NULL, lock_flag: 0x0
    own: 0xc0000000ada5b940, sess: 0xc0000000ada5b940, proc: 0xc0000000ad694f00, prv: 0xc0000000adcbf520
    slk: 0xc0000000ab40db98

这段内容是enqueue state object信息,发现该进程持有一个enqueue。
从这里的 res_flag: 0x3 可以判断:这个enqueue是global类型的,而且正在被持有,还没被free。

继续往下搜索,发现了如下类似的大量信息:

    SO: c0000000add5e1e8, type: 11, owner: c0000000ad694f00, flag: INIT/-/-/0x00
    (broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: c0000000ad694f00,
                       event: 3, last message event: 3,
                       last message waited event: 3, messages read: 0
                       channel: (c0000000add81108) kea interrupt signal channel
                                scope: 5, event: 3, last mesage event: 0,
                                publishers/subscribers: 1/2,
                                messages published: 0
    —————————————-
    SO: c0000000add5e0d8, type: 11, owner: c0000000ad694f00, flag: INIT/-/-/0x00
    (broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: c0000000ad694f00,
                       event: 3, last message event: 3326313,
                       last message waited event: 3326313, messages read: 3324562
                       channel: (c0000000add7ee18) kxfp control signal channel
                                scope: 5, event: 3326313, last mesage event: 3326313,
                                publishers/subscribers: 4/2,
                                messages published: 3324562

这里的type值为11,代表的意思是:Broadcast Handle(即广播消息句柄)。难怪做oradebug的时候,发现这个lck进程阻塞了451个objects,而且等待居然都是rdbms ipc message。

继续往下搜索,发现了如下关键信息:

    —————————————-
    SO: c0000000ada5b940, type: 4, owner: c0000000ad694f00, flag: INIT/-/-/0x00
    (session) sid: 537 trans: 0000000000000000, creator: c0000000ad694f00, flag: (51) USR/- BSY/-/-/-/-/-
              DID: 0000-0014-00000003, short-term DID: 0000-0000-00000000
              txn branch: 0000000000000000
              oct: 0, prv: 0, sql: 0000000000000000, psql: 0000000000000000, user: 0/SYS
    service name: SYS$BACKGROUND
    waiting for 'rdbms ipc message' blocking sess=0x0000000000000000 seq=43421 wait_time=0 seconds since wait started=1
                timeout=12c, =0, =0
    Dumping Session Wait History
     for 'ksxr poll remote instances' count=1 wait_time=7
                =0, =0, =0
     for 'ksxr poll remote instances' count=1 wait_time=7
                =0, =0, =0
     for 'rdbms ipc message' count=1 wait_time=1906819
                timeout=c2, =0, =0
     for 'ksxr poll remote instances' count=1 wait_time=9
                =0, =0, =0
     for 'rdbms ipc message' count=1 wait_time=12
                timeout=c2, =0, =0
     for 'rdbms ipc message' count=1 wait_time=1041870
                timeout=12c, =0, =0
     for 'ksxr poll remote instances' count=1 wait_time=7
                =0, =0, =0
     for 'ksxr poll remote instances' count=1 wait_time=10
                =0, =0, =0
     for 'rdbms ipc message' count=1 wait_time=807692
                timeout=52, =0, =0
     for 'rdbms ipc message' count=1 wait_time=27
                timeout=52, =0, =0
    temporary object counter: 0
      —————————————-
      UOL used : 0 locks(used=0, free=0)
      KGX Atomic Operation Log c0000000aca2e6e8
       Mutex 0000000000000000(0, 0) idn 0 oper NONE
       Cursor Pin uid 537 efd 3 whr 11 slp 0
      KGX Atomic Operation Log c0000000aca2e730
       Mutex 0000000000000000(0, 0) idn 0 oper NONE
       Library Cache uid 537 efd 0 whr 0 slp 0
      KGX Atomic Operation Log c0000000aca2e778
       Mutex 0000000000000000(0, 0) idn 0 oper NONE
       Library Cache uid 537 efd 0 whr 0 slp 0
      —————————————-
      SO: c0000000a1a7fe78, type: 41, owner: c0000000ada5b940, flag: INIT/-/-/0x00
      (dummy) nxc=0, nlb=0 
    —————————————-
    SO: c0000000add5ab28, type: 11, owner: c0000000ad694f00, flag: INIT/-/-/0x00
    (broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: c0000000ad694f00,
                       event: 20, last message event: 26,
                       last message waited event: 26, messages read: 1
                       channel: (c0000000add7a2d8) system events broadcast channel
                                scope: 2, event: 204545, last mesage event: 26,
                                publishers/subscribers: 0/75,
                                messages published: 1
    —————————————-
    SO: c0000000ab2ba708, type: 19, owner: c0000000ad694f00, flag: INIT/-/-/0x00
     GES MSG BUFFERS: st=emp chunk=0x0000000000000000 hdr=0x0000000000000000 lnk=0x0000000000000000 flags=0x0 inc=36
      outq=0 sndq=0 opid=20 prmb=0x0
      mbg[i]=(3277535 648561) mbg[b]=(0 0) mbg[r]=(0 0)
      fmq[i]=(4 0) fmq[b]=(0 0) fmq[r]=(0 0)
      mop[s]=648561 mop[q]=3277535 pendq=0 zmbq=0
      nonksxp_recvs=0
    ————process 0xc0000000ab2ba708——————–
    proc version      : 0
    Local node        : 1
    pid               : 27684
    lkp_node          : 1
    svr_mode          : 0
    proc state        : KJP_NORMAL
    Last drm hb acked : 0
    Total accesses    : 221605743
    Imm.  accesses    : 196696635
    Locks on ASTQ     : 0
    Locks Pending AST : 113
    Granted locks     : 56081
    AST_Q:
    PENDING_Q:
    lp 0xc0000000acb505c0 gl KJUSERNL rl KJUSERPR rp 0xc000000094eeda88 [0x2345][0xffffffff],[IV]
      master 1 pid 27684 bast 0 rseq 1 mseq 0 history 0x9565a
      convert opt KJUSERGETVALUE KJUSERNODEADLOCKWAIT KJUSERNODEADLOCKBLOCK
    lp 0xc00000007a220300 gl KJUSERNL rl KJUSERPR rp 0xc000000094eeda88 [0x2345][0xffffffff],[IV]
      master 1 pid 27684 bast 0 rseq 1 mseq 0 history 0x565a565a
      convert opt KJUSERGETVALUE KJUSERNODEADLOCKWAIT KJUSERNODEADLOCKBLOCK
    lp 0xc000000074ac4c38 gl KJUSERNL rl KJUSERPR rp 0xc000000094eeda88 [0x2345][0xffffffff],[IV]
      master 1 pid 27684 bast 0 rseq 1 mseq 0 history 0x565a565a
      convert opt KJUSERGETVALUE KJUSERNODEADLOCKWAIT KJUSERNODEADLOCKBLOCK
    lp 0xc0000000748ca940 gl KJUSERNL rl KJUSERPR rp 0xc000000094eeda88 [0x2345][0xffffffff],[IV]
      master 1 pid 27684 bast 0 rseq 1 mseq 0 history 0x565a565a

这里的rp 0xc000000094eeda88 我猜应该是enqueue等待的资源名,于是搜索resouce 0xc000000094eeda88字符串,发现了如下信息:

       ———-enqueue 0xc00000007a3cb2e0————————
      lock version     : 1
      Owner node       : 1
      grant_level      : KJUSERNL
      req_level        : KJUSERPR
      bast_level       : KJUSERNL
      notify_func      : 0x0000000000000000
      resp             : 0xc000000094eeda88
      procp            : 0xc0000000ab2ba708
      pid              : 27684
      proc version     : 0
      oprocp           : 0x0000000000000000
      opid             : 0
      group lock owner : 0x0000000000000000
      xid              : 0000-0000-00000000
      dd_time          : 0.0 secs
      dd_count         : 0
      timeout          : 0.0 secs
      On_timer_q       : N
      On_dd_q          : N
      lock_state       : CONVERTING
      Open Options     :  KJUSERPROCESS_OWNED KJUSERCLIENTLOCK
      Convert options  : KJUSERGETVALUE KJUSERNODEADLOCKWAIT KJUSERNODEADLOCKBLOCK
      History          : 0x565a565a
      Msg_Seq          : 0x0
      res_seq          : 1
      valblk           : 0x000000020001fcc00000000000000000 .
      ———-resource 0xc000000094eeda88———————-
      resname       : [0x2345][0xffffffff],[IV]
      Local node    : 1
      dir_node      : 1
      master_node   : 1
      hv idx        : 35
      hv last r.inc : 6
      current inc   : 36
      hv status     : 0
      hv master     : 1
      open options  :
      grant_bits    : KJUSERNL KJUSERPR
      grant mode    : KJUSERNL  KJUSERCR  KJUSERCW  KJUSERPR  KJUSERPW  KJUSEREX
      count         : 114         0         0         333         0         0
      val_state     : KJUSERVS_VALUE
      valblk        : 0x000000020001fcc00000000000000000 .
      access_node   : 1
      vbreq_state   : 0
      state         : x8
      resp          : 0xc000000094eeda88
      On Scan_q     : N
      Total accesses: 4524902
      Imm.  accesses: 1948489
      Granted_locks : 333
      Cvting_locks  : 114

总的来说,给人的感觉似乎lck是罪魁祸首,其实不竟然。
用awk脚本格式化了这个systemstate dump后,得到下面的结果:

  System State 1
~~~~~~~~~~~~~~~~
1:                                    
2:  waiting for 'pmon timer'            wait
3:  waiting for 'DIAG idle wait'        wait
4:  waiting for 'rdbms ipc message'     wait
5:  waiting for 'rdbms ipc message'     wait
6:  waiting for 'ges remote message'    wait
7:  waiting for 'gcs remote message'    wait
8:  waiting for 'gcs remote message'    wait
9:  waiting for 'rdbms ipc message'     wait
10: waiting for 'rdbms ipc message'     wait
11: waiting for 'rdbms ipc message'     wait
12: waiting for 'rdbms ipc message'     wait
13: waiting for 'smon timer'            wait
14: waiting for 'rdbms ipc message'     wait
15: waiting for 'rdbms ipc message'     wait
16: waiting for 'rdbms ipc message'     wait
17: waiting for 'rdbms ipc message'     wait
18:                                   
19:                                   
20: waiting for 'rdbms ipc message'     wait
21: waiting for 'SQL*Net message from client' wait
22: waiting for 'rdbms ipc message'     wait
23: waiting for 'rdbms ipc message'     wait
24: waiting for 'SQL*Net message from client' wait
25: waiting for 'Streams AQ: qmn coordinator idle wait' wait
26: waiting for 'SQL*Net message from client' wait
27: waiting for 'SQL*Net message from client' wait
     Cmd: PL/SQL Execute
28: for 'Streams AQ: waiting for time management or cleanup tasks' wait
29:                                   
30: waiting for 'SQL*Net message from client' wait
31: waiting for 'SQL*Net message from client' wait
32: waiting for 'SQL*Net message from client' wait
33: waiting for 'SQL*Net message from client' wait
34: waiting for 'SQL*Net message from client' wait
35: waiting for 'SQL*Net message from client' wait
36: waiting for 'SQL*Net message from client' wait
37: waiting for 'SQL*Net message from client' wait
38: waiting for 'SQL*Net message from client' wait
39: waiting for 'SQL*Net message from client' wait
     Cmd: Insert
40: waiting for 'SQL*Net message from client' wait
41: waiting for 'SQL*Net message from client' wait
42: last wait for 'SQL*Net message from client'
43: waiting for 'SQL*Net message from client' wait
45: waiting for 'Streams AQ: qmn slave idle wait' wait
46: waiting for 'SQL*Net message from client' wait
     Cmd: Insert
47: waiting for 'SQL*Net message from client' wait
48: waiting for 'SQL*Net message from client' wait
49: waiting for 'SQL*Net message from client' wait
50: waiting for 'SQL*Net message from client' wait
52: waiting for 'SQL*Net message from client' wait
53: waiting for 'SQL*Net message from client' wait
54: waiting for 'SQL*Net message from client' wait
55: waiting for 'SQL*Net message from client' wait
56: waiting for 'SQL*Net message from client' wait
58: waiting for 'SQL*Net message from client' wait
59: waiting for 'SQL*Net message from client' wait
60: waiting for 'SQL*Net message from client' wait
61: waiting for 'SQL*Net message from client' wait
65: waiting for 'SQL*Net message from client' wait
66: waiting for 'SQL*Net message from client' wait
90: waiting for 'SQL*Net message from client' wait
91: waiting for 'SQL*Net message from client' wait
92: waiting for 'SQL*Net message from client' wait
93: waiting for 'SQL*Net message from client' wait
94: waiting for 'SQL*Net message from client' wait
95: waiting for 'SQL*Net message from client' wait
96: waiting for 'SQL*Net message from client' wait
106:waiting for 'SQL*Net message from client' wait
107:waiting for 'SQL*Net message from client' wait
108:waiting for 'SQL*Net message from client' wait
109:waiting for 'SQL*Net message from client' wait
110:waiting for 'SQL*Net message from client' wait
111:waiting for 'SQL*Net message from client' wait
112:waiting for 'SQL*Net message from client' wait
113:waiting for 'SQL*Net message from client' wait
114:waiting for 'SQL*Net message from client' wait
115:waiting for 'SQL*Net message from client' wait
116:waiting for 'SQL*Net message from client' wait
117:waiting for 'SQL*Net message from client' wait
120:waiting for 'SQL*Net message from client' wait

NO BLOCKING PROCESSES FOUND

881983 Lines Processed.

从分析结果来看,断定pmon进程才是真正的罪魁祸首,搜索pmon关键字:

PROCESS 2:
  —————————————-
  SO: c0000000ad68c020, type: 2, owner: 0000000000000000, flag: INIT/-/-/0x00
  (process) Oracle pid=2, calls cur/top: c0000000adab8098/c0000000adab8098, flag: (e) SYSTEM
            int error: 0, call error: 0, sess error: 0, txn error 0
  (post info) last post received: 0 0 48
              last post received-location: ksoreq_reply
              last process to post me: c0000000ad692750 4 2
              last post sent: 0 0 24
              last post sent-location: ksasnd
              last process posted by me: c0000000ad68d000 1 6
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: c0000000ad784f20
    O/S info: user: oracle, term: UNKNOWN, ospid: 27626
    OSD pid info: Unix process pid: 27626, image: oracle@uamdb2 (PMON)
###### 省略部分信息 ######
  —————————————-
    SO: c0000000add586a8, type: 11, owner: c0000000ad68c020, flag: INIT/-/-/0x00
    (broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: c0000000ad68c020,
                       event: 1, last message event: 1,
                       last message waited event: 1, messages read: 0
                       channel: (c0000000add7e4b0) scumnt mount lock
                                scope: 1, event: 15, last mesage event: 0,
                                publishers/subscribers: 0/14,
                                messages published: 0
    —————————————-
    SO: c0000000ada73a90, type: 4, owner: c0000000ad68c020, flag: INIT/-/-/0x00
    (session) sid: 555 trans: 0000000000000000, creator: c0000000ad68c020, flag: (51) USR/- BSY/-/-/-/-/-
              DID: 0002-0002-00000004, short-term DID: 0000-0000-00000000
              txn branch: 0000000000000000
              oct: 0, prv: 0, sql: 0000000000000000, psql: 0000000000000000, user: 0/SYS
    service name: SYS$BACKGROUND
    waiting for 'pmon timer' blocking sess=0x0000000000000000 seq=50 wait_time=0 seconds since wait started=75789
                duration=12c, =0, =0
    Dumping Session Wait History
     for 'pmon timer' count=1 wait_time=2929152
                duration=12c, =0, =0
     for 'pmon timer' count=1 wait_time=2929462
                duration=12c, =0, =0
     for 'pmon timer' count=1 wait_time=2929351
                duration=12c, =0, =0
     for 'pmon timer' count=1 wait_time=2929175
                duration=12c, =0, =0
     for 'pmon timer' count=1 wait_time=2928083
                duration=12c, =0, =0
     for 'pmon timer' count=1 wait_time=2929290
                duration=12c, =0, =0
     for 'pmon timer' count=1 wait_time=2929509
                duration=12c, =0, =0
     for 'pmon timer' count=1 wait_time=2929262
                duration=12c, =0, =0
     for 'pmon timer' count=1 wait_time=2929283
                duration=12c, =0, =0
     for 'pmon timer' count=1 wait_time=2929525
                duration=12c, =0, =0
    temporary object counter: 0

2929152/3600/24=33天左右。换句话说,pmon进程在1个月前就出现异常了,同事也说5月26号的时候重启过,
后来似乎就有问题了,掐指一算,时间似乎刚刚对上。

最后我在中午申请了一点停机时间,我们进行了处理,首先我kill了diag进程,12g的磁盘空间得到了释放,
最开始我断定是lck进程导致的,于是kill了该进程,结果直接导致该节点重启了,不过还好的是,23s就重启完成了。

--------------------------------------ORACLE-DBA----------------------------------------

最权威、专业的Oracle案例资源汇总之【案例】Oracle服务器diag进程占据了12g的磁盘空间分析解决办法

本文由大师惜分飞原创分享,网址:http://www.oracleplus.net/arch/1256.html

Oracle研究中心

关键词:

diag进程占据了12g的磁盘空间

lck进程导致pmon无法释放内存