[스크랩] Crash dump 분석하기 및 자동분석 스크립트

출처 : 아이네트호스팅


솔라리스 시스템에서 갑작스런 패닉 현상이 나타나게 되면 보통 crash 디렉터리를 지정을
하지 않았다면 /var/crash/[hostname]/ 밑에 unix 파일과 core 파일을 남깁니다..

그 파일을 이용하여 시스템 패닉전에 무슨일이 있었는지 분석이 가능 합니다.

아래는 분석에 사용하는 명령어들과 보기입니다.

우선 core 파일과 unix파일이 있는 디렉터리로 이동 합니다. ( /var/crash/[hostname] )

1. strings

# strings vmcore.0 | grep SunOS ---> OS 의 종류를 보여준다.

# strings vmcore.0 | grep macine ---> 어떤 호스트인지 열거 하여 준다.

# strings vmcore.0 | more ---> 다 보여준다.

2. netstat

# netstat -d unix.0 vmcore.0 --> 네트웍 통계보기

# netstat -n unix.0 vmcore.0 --> NFS통계보기

3. arp

# arp -a unix.0 vmcore.0 --> arp 테이블보기

4. ipcs

# ipcs -a -N unix.0 -C vmcore.0 --> ipc 보기


5. crash

사용법 : crash [ -d dumpfile ] [ -n namelist ] [ -w output-file ]

예)

# crash -d unix.0 vmcore.0
dumpfile = vmcore.0, namelist = /dev/ksyms, outfile = stdout
> status
system name: SunOS
release: 5.7
node name: ns
version: Generic_106541-15
machine name: sun4u
time of crash: 수 11월 7 11:23:10 2001
age of system: 1 hr., 6 min.
panicstr: [AFT1] errID 0x000003a4.70f365cc %sError(s)
See previous message(s) for details
panic registers:
pc: 10010104 sp: 400275d8
> help
help [-w filename] function[s]
help function
alias:
acceptable aliases are uniquely identifiable initial substrings
> help p
p [-e] [-f] [-l] [-w filename] [([-p] [-a] tbl_entry | #procid)... | -r]
tbl_entry = slot number | address | symbol | expression | range
process table
alias: proc
acceptable aliases are uniquely identifiable initial substrings
p -e
PROC TABLE SIZE = 2922
SLOT ST PID PPID PGID SID UID PRI NAME FLAGS
0 t 0 0 0 0 0 96 sched load sys lock
1 s 1 0 0 0 0 58 init load
2 s 2 0 0 0 0 98 pageout load sys lock nowait
3 s 3 0 0 0 0 60 fsflush load sys lock nowait
4 s 347 1 347 347 0 58 sac load jctl
5 s 350 1 350 350 0 37 dmispd load
6 s 138 1 138 138 0 41 keyserv load
7 s 53 1 53 53 0 43 devfseventd load
8 s 57 1 57 57 0 50 devfsadm load
9 s 136 1 136 136 0 58 rpcbind load
10 s 205 1 205 205 0 53 nscd load
11 s 130 1 130 130 60001 58 tocsin load
12 s 188 1 188 188 0 52 syslogd load
13 s 178 1 178 178 0 58 automountd load
14 s 170 1 170 170 0 58 in.named load jctl
15 s 163 1 163 163 0 0 inetd load
16 s 233 228 228 228 1006 23 mshttpd load
17 s 300 267 0 0 0 58 mysqld load
18 s 193 1 193 193 0 49 cron load
19 s 232 228 228 228 1006 33 mshttpd load
20 s 228 1 228 228 0 59 mshttpd load
21 s 234 228 228 228 1006 13 mshttpd load
22 s 235 228 228 228 1006 3 mshttpd load
23 s 236 228 228 228 1006 0 mshttpd load
24 s 249 1 249 249 0 58 utmpd load
25 s 267 1 0 0 0 10 safe_mysqld load
26 s 253 1 253 253 0 58 sendmail load jctl
27 s 265 1 265 265 0 58 httpd load
28 s 10139 265 265 265 60001 58 httpd load
29 s 10146 265 265 265 60001 58 httpd load
30 s 10141 265 265 265 60001 58 httpd load
31 s 10151 265 265 265 60001 58 httpd load
32 s 10140 265 265 265 60001 58 httpd load
33 s 281 1 279 279 0 60 auditd load
36 s 331 1 331 331 0 48 dtlogin load jctl
37 s 325 1 325 325 0 23 mountd load
38 s 327 1 327 327 0 33 nfsd load
40 s 351 347 347 347 0 58 listen load nowait jctl
41 s 348 1 348 348 0 55 ttymon load
42 s 14566 331 14566 14566 0 59 Xsun load
43 s 352 347 347 347 0 58 ttymon load jctl
44 s 10137 265 265 265 60001 53 httpd load
50 s 14585 1 14574 14574 0 59 fbconsole load
55 s 14605 14574 14605 14605 0 59 dtgreet load
60 s 14574 331 14574 14574 0 50 dtlogin load
71 s 10150 265 265 265 60001 58 httpd load
76 s 10138 265 265 265 60001 58 httpd load
82 s 11077 265 265 265 60001 58 httpd load
90 s 11085 265 265 265 60001 58 httpd load
> p -l
p_lock: owner 0 waiters 0
cr_lock: owner 0 waiters 0
Condition variable p_cv: 0
Condition variable p_flag_cv: 0
Condition variable p_lwpexit: 0
Condition variable p_holdlwps: 0
Condition variable p_srwchan_cv: 0
82 s 11077 265 265 265 60001 58 httpd load

p_lock: owner 0 waiters 0
cr_lock: owner 0 waiters 0
Condition variable p_cv: 0
Condition variable p_flag_cv: 0
Condition variable p_lwpexit: 0
Condition variable p_holdlwps: 0
Condition variable p_srwchan_cv: 0
90 s 11085 265 265 265 60001 58 httpd load
.
.
.
. 이하생략..

> q

6. adb 사용하기

# adb -k unix.0 vmcore.0 ( unix파일이나 core파일 대신 커널쪽을 볼수도 있다 ex> adb -k /dev/ksyms /dev/mem )
adb -k unix.0 vmcore.0
physmem 5b01
*panicstr/s ---> 패닉 메세지가 무엇이었나 보여준다.
0x40027768: [AFT1] errID 0x000003a4.70f365cc %sError(s)
See previous message(s) for details
$<utsname ---> 호스트 이름과 장비에 관해 열거해준다.

utsname:
utsname: sys SunOS
utsname+0x101: node ns
utsname+0x202: release 5.7
utsname+0x303: version Generic_106541-15
utsname+0x404: machine sun4u

srpc_domain/s ---> 도메인
srpc_domain:
srpc_domain:

hw_provider/s ---> 제조업체
hw_provider:
hw_provider: Sun_Microsystems

*time-(lbolt%0t100)=Y --->부트시간/날짜
2001 Oct 6 21:26:01

$<msgbuf ---> 최근 메세지 버펄를 열거 하여 준다
SunOS Release 5.7 Version Generic_106541-15 [UNIX(R) System V R
elease 4.0]
0x700b57e0: Copyright (c) 1983-1999, Sun Microsystems, Inc.
0x700b53c3: Ethernet address = 8:0:20:86:23:9e
0x700b4fa0: mem = 196608K (0xc000000)
0x700b4b80: avail mem = 188989440
0x700b4763: root nexus = Sun Ultra 1 SBus (UltraSPARC 167MHz)
0x700b4343: sbus0 at root: UPA 0x1f 0x0 ...
0x70155ee3: sbus0 is /sbus@1f,0
0x70155ac0: dma0 at sbus0: SBus0 slot 0xe offset 0x8400000
0x701556a3: dma0 is /sbus@1f,0/espdma@e,8400000
0x70155280: /sbus@1f,0/espdma@e,8400000/esp@e,8800000 (esp0):
esp-options=0x46
0x70154e60: esp0 at dma0: SBus0 slot 0xe offset 0x8800000 Onboard device spa
rc9 ipl 4
0x70154a43: esp0 is /sbus@1f,0/espdma@e,8400000/esp@e,8800000
0x70154620: NOTICE: Memory scrubber exiting
0x70154200: sd0 at esp0:
0x70177d40: target 0 lun 0
0x70177923: sd0 is /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0
0x70177500: <FUJITSU-MAB3091SC-2703 cyl 7185 alt 2 hd 10 sec 248>
0x701770e0: sd1 at esp0:
0x70176cc0: target 1 lun 0
0x701768a3: sd1 is /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@1,0
0x70176480: <SUN4.2G cyl 3880 alt 2 hd 16 sec 135>
0x70176060: sd4 at esp0:
0x70179c00: target 4 lun 0
0x701797e3: sd4 is /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@4,0
0x701793c0: <SUN4.2G cyl 3880 alt 2 hd 16 sec 135>
0x70178fa0: sd6 at esp0:
0x70178b80: target 6 lun 0
0x70178763: sd6 is /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@6,0
0x70178347: root on /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a fstyp
e ufs
0x700efee0: zs0 at sbus0: SBus0 slot 0xf offset 0x1100000 Onboard device spa
rc9 ipl 12
0x700efac3: zs0 is /sbus@1f,0/zs@f,1100000
0x700ef6a0: zs1 at sbus0: SBus0 slot 0xf offset 0x1000000 Onboard device spa
rc9 ipl 12
0x700ef283: zs1 is /sbus@1f,0/zs@f,1000000
0x700eee66: keyboard is </sbus@1f,0/zs@f,1000000> major <29> minor <2>
0x700eea46: mouse is </sbus@1f,0/zs@f,1000000:b> major <29> minor <3>
0x700ee626: stdin is </sbus@1f,0/zs@f,1000000> major <29> minor <2>
0x700ee200: cgsix0 at sbus0: SBus0 slot 0x1 offset 0x0 SBus level 5 sparc9 i
pl 9
0x7015fea3: cgsix0 is /sbus@1f,0/cgsix@1,0
0x7015fbe1: cgsix0: screen 1152x900, single buffered, 1M mappable, rev 11
0x7015f926: stdout is </sbus@1f,0/cgsix@1,0> major <39> minor <0>
0x7015f660: cpu0: SUNW,UltraSPARC (upaid 0 impl 0x10 ver 0x22 clock 167 MHz)
0x7015f3a0: ledma0 at sbus0: SBus0 slot 0xe offset 0x8400010
0x7015f0e0: le0 at ledma0: SBus0 slot 0xe offset 0x8c00000 Onboard device sp
arc9 ipl 6
0x7015ee23: le0 is /sbus@1f,0/ledma@e,8400010/le@e,8c00000
0x7015eb63: dump on /dev/dsk/c0t0d0s1 size 1000 MB
0x7015e8a2: pseudo-device: devinfo0
0x7015e5e3: devinfo0 is /pseudo/devinfo@0
0x7015e320: fd0 at sbus0: SBus0 slot 0xf offset 0x1400000 Onboard device spa
rc9 ipl 11
0x7015e063: fd0 is /sbus@1f,0/SUNW,fdtwo@f,1400000
0x70559d60: sbusmem0 at sbus0: SBus0 slot 0x0 offset 0x0
0x70559aa3: sbusmem0 is /sbus@1f,0/sbusmem@0,0
0x705597e0: sbusmem1 at sbus0: SBus0 slot 0x1 offset 0x0
0x70559523: sbusmem1 is /sbus@1f,0/sbusmem@1,0
0x70559260: sbusmem2 at sbus0: SBus0 slot 0x2 offset 0x0
0x70558fa3: sbusmem2 is /sbus@1f,0/sbusmem@2,0
0x70558ce0: sbusmem3 at sbus0: SBus0 slot 0x3 offset 0x0
0x70558a23: sbusmem3 is /sbus@1f,0/sbusmem@3,0
0x7042b70b: WARNING: [AFT1] EDP event on CPU0 Data access at TL=0, errID 0x0
00003a4.70f365cc
AFSR 0x00000000.80400002<PRIV,EDP> AFAR 0xffffffff.ffffffff
AFSR.PSYND 0x0002(Score 95) AFSR.ETS 0x00 Fault_PC 0x1018af18
UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000 UDBL.ESYND 0x00
0x7055834b: [AFT2] errID 0x000003a4.70f365cc No error found in ecache (No fa
ult PA available)
0x70558080: panic[cpu0]/thread=40027e60:
0x70558b80: [AFT1] errID 0x000003a4.70f365cc EDP Error(s)
See previous message(s) for details
0x70154fc0:
0x7015f7c3: syncing file systems...
0x7015fd43: [4]
0x7015f243: [4]
0x7015f500: panic[cpu0]/thread=4003fe60:
0x700ef800: panic sync timeout
0x7015ea00:
0x700efc23: dumping to /dev/dsk/c0t0d0s1, offset 209780736

$c ---> C 스택 역추적
complete_panic(0xf,0x10437c00,0x40027e60,0x7011478a,0x0,0x10107880) + 24
do_panic(0x1,0x40027954,0x10107f70,0x0,0x0,0x400277bb) + 174
vcmn_err(0x3,0x40027768,0x40027954,0x7efefeff,0x81010100,0xff00) + 14
cpu_aflt_log(0x400278f8,0x400277bb,0x40027978,0x3,0x1010785c,0x10107880) + 4c0

rootfs$<bootobj ---> 루투장치는 어떤것인가

rootfs:
rootfs: fstype ufs^@^@^@^@^@^@^@^@^@^@^@^@^@
rootfs+0x10: name /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@
rootfs+0x90: flags size vp
0 0 0

swapfile$<bootobj ---> 스왑장치는 어떤것인지 열거

swapfile:
swapfile: fstype ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
swapfile+0x10: name ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@
swapfile+0x90: flags size vp
0 0 0

$q ---> 종료

=========================================================================================================================================

자동분석 스크립트입니다.

iscda.sh

사용방법은

#/var/crash/[hostname]/
안에 위의 파일을 넣어놓습니다.  만약 갑자기 reboot 되거나 하면 dumpadm.conf 및 coreadm.conf 를 건드리지 않는한 위의 경로에 unix.0 vmcore.0 등의 파일이 생기게 됩니다.
"hostname" 이 Server name 이라는건 아시겠죠?

그다음  #/var/crash/[hostname]/iscda unix.0 vmcore.0 >> /log/iscda.txt

이렇게 하시고

#vi /log/iscda.txt 파일을 보시면 됩니다.