CPU100%调试解析

gdb

1. 用top命令查看哪个进程占用CPU高
2. 用top -H命令查看哪个进程占用CPU高(top -H -p pid查看进程内各个线程占用的CPU百分比)

shell[root@localhost ~]# top -H -f 24640
    top: unknown argument 'f'
usage:  top -hv | -bcisSHM -d delay -n iterations [-u user | -U user] -p pid [,pid ...]
[root@localhost ~]# top -H -p 24640
top - 11:10:06 up 37 days, 15 min,  5 users,  load average: 1.00, 0.74, 0.39
Tasks:   3 total,   1 running,   2 sleeping,   0 stopped,   0 zombie
Cpu(s): 25.5%us,  0.7%sy,  0.0%ni, 73.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2073292k total,  1484976k used,   588316k free,   449328k buffers
Swap:  4128760k total,    43484k used,  4085276k free,   678048k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                          
24642 root      25   0 23372  864  736 R 99.9  0.0   3:55.46 a.out                                                                             
24640 root      15   0 23372  864  736 S  0.0  0.0   0:00.00 a.out                                                                             
24641 root      15   0 23372  864  736 S  0.0  0.0   0:00.00 a.out   

有图可以看出该进程中 线程24642线程占用的CPU高

利用
3. 利用gstack -p pid >gstack.log /查看进程中各线程的函数调用栈

 1 Thread 3 (Thread 0xb7fa8b90 (LWP 24641)):
  2 #0  0x00236402 in __kernel_vsyscall ()
  3 #1  0x00b652b6 in nanosleep () from /lib/libc.so.6
  4 #2  0x00b650df in sleep () from /lib/libc.so.6
  5 #3  0x08048734 in threadFun1 ()
  6 #4  0x00c4e49b in start_thread () from /lib/libpthread.so.0
  7 #5  0x00ba542e in clone () from /lib/libc.so.6
  8 Thread 2 (Thread 0xb75a7b90 (LWP 24642)):
  9 #0  0x00b65101 in sleep () from /lib/libc.so.6
 10 #1  0x00c4e49b in start_thread () from /lib/libpthread.so.0
 11 #2  0x00ba542e in clone () from /lib/libc.so.6
 12 Thread 1 (Thread 0xb7fa96d0 (LWP 24640)):
 13 #0  0x00236402 in __kernel_vsyscall ()
 14 #1  0x00c4f5a7 in pthread_join () from /lib/libpthread.so.0
 15 #2  0x080486e0 in main ()
~                                 

1. 使用gcore命令转存进程映像及内存上下文

#gcore 14094

该命令生成core文件core.14094

1. 用starce 命令查看系统调用和花费的时间

#strace -T -r -c -p 14094  
-c 参数显示统计消息,出掉此参数可以查看每个系统调用的时间及返回值
% time     seconds  usecs/call     calls    errors        syscall
------ ----------- ----------- --------- --------- ----------------------------
99.99   22.683879        3385      6702                     poll
  0.00    0.001132           0      6702                     gettimeofday
  0.00    0.000127           1       208       208          accept
  0.00    0.000022          22         1                    read
  0.00    0.000000           0         1                    write
  0.00    0.000000           0         1                    close
  0.00    0.000000           0        14                    time
  0.00    0.000000           0         2                    stat64
  0.00    0.000000           0         4                    clock_gettime
  0.00    0.000000           0         7                    send
  0.00    0.000000           0        10        10          recvfrom
------ ----------- ----------- --------- --------- ------------------------------
100.00   22.685160                 13652       218 total

6.用gdb调试core文件,并切换到 特定线程

可以根据详细的函数栈进行gdb调试,打印一些变量值,并结合源代码分析为何会占用很高的CPU

流程为:

进程ID—>线程ID—>线程函数调用栈—>函数耗时和调用统计—>源代码分析

定位系统性能瓶颈 #strace&ltrace