澳门新萄京strace追踪线程调用

#more pmp.log
TS 845703637.1530408066 2018-07-01 03:21:06
[New LWP 3981]
[New LWP 3875]
[New LWP 3874]
[New LWP 3846]
[New LWP 3845]
[New LWP 3844]
[New LWP 3843]
[New LWP 3842]
[New LWP 3841]
[New LWP 3840]
[New LWP 3839]
[New LWP 3838]
[New LWP 3837]
[New LWP 3836]
[New LWP 3835]
[New LWP 3833]
[New LWP 3832]
[New LWP 3831]
[New LWP 3830]
[New LWP 3829]
[New LWP 3828]
[New LWP 3827]
[New LWP 3826]
[New LWP 3825]
[New LWP 3824]
[New LWP 3823]
[New LWP 3822] //We got 28 new LWP from 3822 to 3981.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f50efe74b7d in poll () from /lib64/libc.so.6

Thread 28 (Thread 0x7f50e7ec1700 (LWP 3822)): //It's the last thread in pmp.log file here.
#0  0x00007f50efdbf7da in sigwaitinfo () from /lib64/libc.so.6
#1  0x0000000000f5171b in timer_notify_thread_func (arg=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/mysys/posix_timers.c:77
#2  0x0000000000f707b4 in pfs_spawn_thread (arg=0x30cb000) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/perfschema/pfs.cc:2190
#3  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f50efe7f1ad in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f50ded07700 (LWP 3823)):
#0  0x00007f50f11b8644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000000000106a514 in LinuxAIOHandler::collect (this=0x7f50ded06e10) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:2500
#2  0x000000000106bc34 in LinuxAIOHandler::poll (this=0x7f50ded06e10, m1=0x7f50ded06ec8, m2=0x7f50ded06ec0, request=0x7f50ded06e90) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:2646
#3  0x000000000106d65c in os_aio_linux_handler (request=0x7f50ded06e90, m2=0x7f50ded06ec0, m1=0x7f50ded06ec8, global_segment=0) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:2702
#4  os_aio_handler (segment=0, m1=0x7f50ded06ec8, m2=0x7f50ded06ec0, request=0x7f50ded06e90) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:6259
#5  0x000000000120196d in fil_aio_wait (segment=0) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/fil/fil0fil.cc:5835
#6  0x000000000110e980 in io_handler_thread (arg=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/srv/srv0start.cc:311
#7  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f50efe7f1ad in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f50de506700 (LWP 3824)):
#0  0x00007f50f11b8644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000000000106a514 in LinuxAIOHandler::collect (this=0x7f50de505e10) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:2500
#2  0x000000000106bc34 in LinuxAIOHandler::poll (this=0x7f50de505e10, m1=0x7f50de505ec8, m2=0x7f50de505ec0, request=0x7f50de505e90) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:2646
#3  0x000000000106d65c in os_aio_linux_handler (request=0x7f50de505e90, m2=0x7f50de505ec0, m1=0x7f50de505ec8, global_segment=1) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:2702
#4  os_aio_handler (segment=1, m1=0x7f50de505ec8, m2=0x7f50de505ec0, request=0x7f50de505e90) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:6259
#5  0x000000000120196d in fil_aio_wait (segment=1) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/fil/fil0fil.cc:5835
#6  0x000000000110e980 in io_handler_thread (arg=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/srv/srv0start.cc:311
#7  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f50efe7f1ad in clone () from /lib64/libc.so.6

//Omitted.

Thread 4 (Thread 0x7f50deeb2700 (LWP 3874)):
#0  0x00007f50f13c8ec1 in sigwait () from /lib64/libpthread.so.0
#1  0x00000000007c357b in signal_hand (arg=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/mysqld.cc:2120
#2  0x0000000000f707b4 in pfs_spawn_thread (arg=0x386a740) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/perfschema/pfs.cc:2190
#3  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f50efe7f1ad in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f50be7fc700 (LWP 3875)):
#0  0x00007f50f13c5705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000ebe6f5 in native_cond_wait (mutex=<optimized out>, cond=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/include/thr_cond.h:140
#2  my_cond_wait (mp=<optimized out>, cond=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/include/thr_cond.h:195
#3  inline_mysql_cond_wait (src_line=874, src_file=0x15d92e0 "/export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/rpl_gtid_persist.cc", mutex=<optimized out>, that=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/include/mysql/psi/mysql_thread.h:1184
#4  compress_gtid_table (p_thd=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/rpl_gtid_persist.cc:874
#5  0x0000000000f707b4 in pfs_spawn_thread (arg=0x386a740) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/perfschema/pfs.cc:2190
#6  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f50efe7f1ad in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f50dee81700 (LWP 3981)):
#0  0x00007f50efe74b7d in poll () from /lib64/libc.so.6
#1  0x00000000012bae8f in vio_io_wait (vio=<optimized out>, event=<optimized out>, timeout=300000) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/vio/viosocket.c:786
#2  0x00000000012baf83 in vio_socket_io_wait (vio=<optimized out>, event=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/vio/viosocket.c:77
#3  0x00000000012bb5b8 in vio_read (vio=0x7f50b0000bb0, buf=0x7f50b000a3c0 "\001", size=4) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/vio/viosocket.c:132
#4  0x0000000000c6de03 in net_read_raw_loop (net=0x7f50b0002570, count=4) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/net_serv.cc:672
#5  0x0000000000c6e69b in net_read_packet_header (net=0x7f50b0002570) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/net_serv.cc:756
#6  net_read_packet (net=0x7f50b0002570, complen=0x7f50dee80d38) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/net_serv.cc:822
#7  0x0000000000c6e94c in my_net_read (net=0x7f50b0002570) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/net_serv.cc:899
#8  0x0000000000c7c1dc in Protocol_classic::read_packet (this=0x7f50b0001e18) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/protocol_classic.cc:808
#9  0x0000000000c7af52 in Protocol_classic::get_command (this=0x7f50b0001e18, com_data=0x7f50dee80de0, cmd=0x7f50dee80e0c) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/protocol_classic.cc:965
#10 0x0000000000d1cf47 in do_command (thd=0x7f50b0000dc0) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/sql_parse.cc:938
#11 0x0000000000ded7ac in handle_connection (arg=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/conn_handler/connection_handler_per_thread.cc:300
#12 0x0000000000f707b4 in pfs_spawn_thread (arg=0x382fa60) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/perfschema/pfs.cc:2190
#13 0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f50efe7f1ad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f50f17e8740 (LWP 3817)):
#0  0x00007f50efe74b7d in poll () from /lib64/libc.so.6
#1  0x0000000000deeaf9 in Mysqld_socket_listener::listen_for_connection_event (this=0x30d96f0) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/conn_handler/socket_connection.cc:852
#2  0x00000000007c89c4 in connection_event_loop (this=0x373e050) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/conn_handler/connection_acceptor.h:66
#3  mysqld_main (argc=79, argv=0x2f897b8) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/mysqld.cc:5122
#4  0x00007f50efdaaaf5 in __libc_start_main () from /lib64/libc.so.6
#5  0x00000000007be479 in _start ()
TS 788084395.1530408210 2018-07-01 03:23:30

[root@zlm1 03:37:50 ~]
#
  • pt-stalk
    is another tool in diagnosing system performance and similar with
    pt-pmp but not the same.
  • pt-stalk
    will generate many statistic files which can really help you
    collecting information in almost every aspect.
  • pt-stalk
    has four main triggers:gdb,oprofile,strace,tcpdump.It’s flexible to
    collect data with diffrent dimensionality.

最后
man pstack
pstack – print a stack trace of a running process
一定于gdb内的bt,打出栈音信。

Pre-compiling BTrace scripts

It is possible to precompile BTrace program using btracec script.
btracec is a javac-like program that takes a BTrace program and produces
a .class file.

    btracec [-I <include-path>] [-cp <classpath>] [-d <directory>] <one-or-more-BTrace-.java-files>

where

  • include-path is a set of include directories that are searched for
    header files. BTrace includes a simple preprocess with support for
    #define, #include and conditional compilation. It is not like
    a complete C/C++ preprocessor – but a useful subset. See the sample
    “ThreadBean.java”. If -I is not specified, BTrace skips the
    preprocessor invocation step.
  • classpath is the classpath used for compiling BTrace program(s).
    Default is “.”
  • directory is the output directory where compiled .class files are
    stored. Default is “.”.

This script uses BTrace compiler class – rather than regular javac and
therefore will validate your BTrace program at compile time [so that
you can avoid BTrace verify error at runtime].

 

 

  1. [root@xxxx]strace -p 24091
  2. Process xxx attached – interrupt to quit
  3. [ Process PID=24091 runs in 32 bit mode. ]
  4. futex(0xc5bbbd8, FUTEX_WAIT, 24215 NULL

Field Annotations

  • @com.sun.btrace.annotations.Export annotation can be used with
    BTrace fields (static fields) to specify that the field has to be
    mapped to a jvmstat counter. Using this, a BTrace program can expose
    tracing counters to external jvmstat clients (such as jstat). Refer
    to the
    sample ThreadCounter.java
  • @com.sun.btrace.annotations.Property annotation can be used to
    flag a specific (static) field as a MBean attribute. If a BTrace
    class has atleast one static field with @Property attribute, then a
    MBean is created and registered with platform MBean server. JMX
    clients such as VisualVM, jconsole can be used to view such BTrace
    MBeans. After attaching BTrace to the target program, you can attach
    VisualVM or jconsole to the same program and view the newly created
    BTrace MBean. With VisualVM and jconsole, you can use MBeans tab to
    view the BTrace domain and check out it’s attributes. Refer to the
    samples ThreadCounterBean and HistogramBean.java.
  • @com.sun.btrace.annotations.TLS annotation can be used with
    BTrace fields (static fields) to specify that the field is a thread
    local field. Each Java thread gets a separate “copy” of the field.
    These thread local fields may be used by BTrace programs to identify
    whether we reached multiple probe actions from the same thread or
    not. Refer to the
    samples OnThrow.java andWebServiceTracker.java

Trace gdb stack of mysqld into a
file(start mysqld and then execute pt-pmp).

 

方法一:
strace -fp pid , 能够追踪全体线程, 进度的连串调用。

Class Annotations

  • @com.sun.btrace.annotations.DTrace annotation can be used to
    associate a simple one-liner D-script (inlined in BTrace Java class)
    with the BTrace program. Refer to the
    sample DTraceInline.java.
  • @com.sun.btrace.annotations.DTraceRef annotation can be used
    to associate a D-script (stored in a separate file) with the BTrace
    program. Refer to the
    sample DTraceRefDemo.java.
  • @com.sun.btrace.annotations.BTrace annotation must be used to
    designate a given Java class as a BTrace program. This annotation is
    enforced by the BTrace compiler as well as by the BTrace agent.

 

 

措施二(本质同方法一,但更明了):
pstree -p pid 可以查阅102线程程序的历程树。
在针对各类线程,每个的strace -p pid

optional

  • port is the server socket port at which BTrace agent listens for
    clients. Default is 2020.
  • path is the classpath used for compiling BTrace program. Default
    is “.”.
  • args is command line arguments passed to BTrace program. BTrace
    program can access these using the built-in functions “$” and
    “$length”.

Preface

 

复制代码

Method Annotations

  • @com.sun.btrace.annotations.OnMethod annotation
    can be used to specify target class(es), target method(s) and
    “location(s)” within the method(s). An action method annotated by
    this annotation is called when the matching method(s) reaches
    specified the location. In OnMethod annotation, traced class name is
    specified by “clazz” property and traced method is specified by
    “method” property. “clazz” may be a fully qualified class name
    (like java.awt.Component or a regular expression specified
    within two forward slashes. Refer to the
    samples NewComponent.java and Classload.java.
    The regular expression can match zero or more classes in which case
    all matching classes are instrumented. For
    example /java\\.awt\\..+/ matches all classes in java.awt
    package. Also, method name can be a regular expression as well –
    matching zero or more methods. Refer to the
    sample MultiClass.java.
    There is another way to abstractly specify traced class(es) and
    method(s). Traced classes and methods may be specified by
    annotation. For example, if the “clazz” attribute is specified
    as @javax.jws.WebService BTrace will instrument all classes
    that are annotated by the WebService annotation. Similarly, method
    level annotations may be used to specify methods abstractly. Refer
    to the
    sampleWebServiceTracker.java.
    It is also possible to combine regular expressions with
    annotations – like @/com\\.acme\\..+/ matches any class that
    is annotated by any annotation that matches the given regular
    expression. It is possible to match multiple classes by specifying
    super type. i.e., match all classes that are subtypes of a given
    super type. +java.lang.Runnable matches all classes
    implementing java.lang.Runnable interface. Refer to the
    sample SubtypeTracer.java.
  • @com.sun.btrace.annotations.OnTimer annotation
    can be used to specify tracing actions that have to run periodically
    once every N milliseconds. Time period is specified as long “value”
    property of this annotation. Refer to the
    sample Histogram.java
  • @com.sun.btrace.annotations.OnError annotation
    can be used to specify actions that are run whenever any exception
    is thrown by tracing actions of some other probe. BTrace method
    annotated by this annotation is called when any exception is thrown
    by any of the other BTrace action methods in the same BTrace class.
  • @com.sun.btrace.annotations.OnExit annotation
    can be used to specify actions that are run when BTrace code calls
    “exit(int)” built-in function to finish the tracing “session”. Refer
    to the
    sample ProbeExit.java.
  • @com.sun.btrace.annotations.OnEvent annotation
    is used to associate tracing methods with “external” events send by
    BTrace client. BTrace methods annotated by this annotation are
    called when BTrace client sends an “event”. Client may send an event
    based on some form of user request to send (like pressing Ctrl-C or
    a GUI menu). String value may used as the name of the event. This
    way certain tracing actions can be executed whenever an external
    event “fires”. As of now, the command line BTrace client sends
    “events” whenever use presses Ctrl-C (SIGINT). On SIGINT, a console
    menu is shown to send an event or exit the client [which is the
    default for SIGINT]. Refer to the
    sample HistoOnEvent.java
  • @com.sun.btrace.annotations.OnLowMemory annotation
    can be used to trace memory threshold exceed event. See
    sampleMemAlerter.java
  • @com.sun.btrace.annotations.OnProbe annotation
    can be used to specify to avoid using implementation internal
    classes in BTrace scripts. @OnProbe probe specifications are mapped
    to one or more @OnMethod specifications by the BTrace VM agent.
    Currently, this mapping is done using a XML probe descriptor file
    [accessed by the BTrace agent]. Refer to the
    sample SocketTracker1.java and
    associated probe descriptor
    file java.net.socket.xml.
    When running this sample, this xml file needs to be copied in the
    directory where the target JVM runs (or fix probeDescPath option in
    btracer.bat to point to whereever the .xml file is).

Exapmle

 1 [root@zlm2 08:31:41 /var/lib/pt-stalk]
 2 #ls -lrt
 3 total 3928
 4 -rw-r--r-- 1 root root     383 Jul  9 08:19 2018_07_09_08_19_39-trigger
 5 -rw-r--r-- 1 root root    9845 Jul  9 08:19 2018_07_09_08_19_39-pmap
 6 -rw-r--r-- 1 root root   16554 Jul  9 08:19 2018_07_09_08_19_39-variables
 7 -rw-r--r-- 1 root root    4352 Jul  9 08:19 2018_07_09_08_19_39-innodbstatus1
 8 -rw-r--r-- 1 root root     384 Jul  9 08:19 2018_07_09_08_19_39-mutex-status1
 9 -rw-r--r-- 1 root root    8507 Jul  9 08:19 2018_07_09_08_19_39-ps
10 -rw-r--r-- 1 root root   10356 Jul  9 08:19 2018_07_09_08_19_39-lsof
11 -rw-r--r-- 1 root root    5280 Jul  9 08:19 2018_07_09_08_19_39-opentables1
12 -rw-r--r-- 1 root root    8204 Jul  9 08:19 2018_07_09_08_19_39-top
13 -rw-r--r-- 1 root root   28343 Jul  9 08:19 2018_07_09_08_19_39-sysctl
14 -rw-r--r-- 1 root root     139 Jul  9 08:20 2018_07_09_08_19_39-disk-space
15 -rw-r--r-- 1 root root 1084769 Jul  9 08:20 2018_07_09_08_19_39-mysqladmin
16 -rw-r--r-- 1 root root    2782 Jul  9 08:20 2018_07_09_08_19_39-vmstat
17 -rw-r--r-- 1 root root   24750 Jul  9 08:20 2018_07_09_08_19_39-procstat
18 -rw-r--r-- 1 root root   14403 Jul  9 08:20 2018_07_09_08_19_39-diskstats
19 -rw-r--r-- 1 root root   72469 Jul  9 08:20 2018_07_09_08_19_39-procvmstat
20 -rw-r--r-- 1 root root   51960 Jul  9 08:20 2018_07_09_08_19_39-netstat_s
21 -rw-r--r-- 1 root root  296850 Jul  9 08:20 2018_07_09_08_19_39-slabinfo
22 -rw-r--r-- 1 root root   33210 Jul  9 08:20 2018_07_09_08_19_39-interrupts
23 -rw-r--r-- 1 root root   48390 Jul  9 08:20 2018_07_09_08_19_39-netstat
24 -rw-r--r-- 1 root root   37260 Jul  9 08:20 2018_07_09_08_19_39-meminfo
25 -rw-r--r-- 1 root root   17100 Jul  9 08:20 2018_07_09_08_19_39-df
26 -rw-r--r-- 1 root root   40753 Jul  9 08:20 2018_07_09_08_19_39-processlist
27 -rw-r--r-- 1 root root    4555 Jul  9 08:20 2018_07_09_08_19_39-innodbstatus2
28 -rw-r--r-- 1 root root   74476 Jul  9 08:20 2018_07_09_08_19_39-transactions
29 -rw-r--r-- 1 root root       5 Jul  9 08:20 2018_07_09_08_19_39-hostname
30 -rw-r--r-- 1 root root     385 Jul  9 08:20 2018_07_09_08_19_39-mutex-status2
31 -rw-r--r-- 1 root root    5280 Jul  9 08:20 2018_07_09_08_19_39-opentables2
32 -rw-r--r-- 1 root root     325 Jul  9 08:20 2018_07_09_08_19_39-vmstat-overall
33 -rw-r--r-- 1 root root   24763 Jul  9 08:20 2018_07_09_08_19_39-output
34 
35 //There're a series of files(starts with "tigger" and ends with "output") each time we collect the diagnostic data.
36 
37 -rw-r--r-- 1 root root     383 Jul  9 08:24 2018_07_09_08_24_39-trigger
38 -rw-r--r-- 1 root root   10251 Jul  9 08:24 2018_07_09_08_24_39-pmap
39 -rw-r--r-- 1 root root   16554 Jul  9 08:24 2018_07_09_08_24_39-variables
40 -rw-r--r-- 1 root root    4353 Jul  9 08:24 2018_07_09_08_24_39-innodbstatus1
41 -rw-r--r-- 1 root root    8583 Jul  9 08:24 2018_07_09_08_24_39-ps
42 -rw-r--r-- 1 root root     386 Jul  9 08:24 2018_07_09_08_24_39-mutex-status1
43 -rw-r--r-- 1 root root   28343 Jul  9 08:24 2018_07_09_08_24_39-sysctl
44 -rw-r--r-- 1 root root   10534 Jul  9 08:24 2018_07_09_08_24_39-lsof
45 -rw-r--r-- 1 root root    5280 Jul  9 08:24 2018_07_09_08_24_39-opentables1
46 -rw-r--r-- 1 root root    8053 Jul  9 08:24 2018_07_09_08_24_39-top
47 -rw-r--r-- 1 root root     139 Jul  9 08:25 2018_07_09_08_24_39-disk-space
48 -rw-r--r-- 1 root root    2783 Jul  9 08:25 2018_07_09_08_24_39-vmstat
49 -rw-r--r-- 1 root root 1084769 Jul  9 08:25 2018_07_09_08_24_39-mysqladmin
50 -rw-r--r-- 1 root root   24899 Jul  9 08:25 2018_07_09_08_24_39-procstat
51 -rw-r--r-- 1 root root   48390 Jul  9 08:25 2018_07_09_08_24_39-netstat
52 -rw-r--r-- 1 root root  296850 Jul  9 08:25 2018_07_09_08_24_39-slabinfo
53 -rw-r--r-- 1 root root   52230 Jul  9 08:25 2018_07_09_08_24_39-netstat_s
54 -rw-r--r-- 1 root root   33210 Jul  9 08:25 2018_07_09_08_24_39-interrupts
55 -rw-r--r-- 1 root root   37260 Jul  9 08:25 2018_07_09_08_24_39-meminfo
56 -rw-r--r-- 1 root root   14460 Jul  9 08:25 2018_07_09_08_24_39-diskstats
57 -rw-r--r-- 1 root root   17100 Jul  9 08:25 2018_07_09_08_24_39-df
58 -rw-r--r-- 1 root root   72914 Jul  9 08:25 2018_07_09_08_24_39-procvmstat
59 -rw-r--r-- 1 root root   44370 Jul  9 08:25 2018_07_09_08_24_39-processlist
60 -rw-r--r-- 1 root root    4445 Jul  9 08:25 2018_07_09_08_24_39-innodbstatus2
61 -rw-r--r-- 1 root root   76282 Jul  9 08:25 2018_07_09_08_24_39-transactions
62 -rw-r--r-- 1 root root       5 Jul  9 08:25 2018_07_09_08_24_39-hostname
63 -rw-r--r-- 1 root root     387 Jul  9 08:25 2018_07_09_08_24_39-mutex-status2
64 -rw-r--r-- 1 root root    5280 Jul  9 08:25 2018_07_09_08_24_39-opentables2
65 -rw-r--r-- 1 root root     325 Jul  9 08:25 2018_07_09_08_24_39-vmstat-overall
66 -rw-r--r-- 1 root root   24763 Jul  9 08:25 2018_07_09_08_24_39-output
67 
68 [root@zlm2 08:33:37 /var/lib/pt-stalk]
69 #cat 2018_07_09_08_24_39-trigger
70 2018_07_09_08_24_39 Check results: status(Threads_connected)=6, matched=yes, cycles_true=1
71 2018_07_09_08_24_39 pt-stalk ran with --function=status --variable=Threads_connected --threshold=2 --match= --cycles=1 --interval=1 --iterations= --run-time=30 --sleep=300 --dest=/var/lib/pt-stalk --prefix= --notify-by-email= --log=/var/log/pt-stalk.log --pid=/var/run/pt-stalk.pid --plugin=
72 
73 //The trigger file records the options we have used.
74 
75 [root@zlm2 08:34:17 /var/lib/pt-stalk]
76 #cat 2018_07_09_08_24_39-vmstat-overall
77 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
78  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
79 18  2   9692  60716      0 583316    0    0    11   139   92  597  4  1 96  0  0
80  4  0  10440  78828      0 569672    0   25    89  1965 1455 12046 80 20  0  0  0
81  
82  //The vmstat-overall records the vmstat output.
83  //each file contains the relevant diagnostic data about what their name called.I'm not going to demonstrate all of them.

您的难题不是strace不能够跟踪,而是你的线程锁住了。
futex应该是内核里对pthread_mutex_lock完结的四个函数。

Argument Annotations

  • @com.sun.btrace.annotations.Self annotation
    can be used to mark an argument to hold *this* (or *self*)
    value. Refer to the
    samples AWTEventTracer.java or AllCalls1.java
  • @com.sun.btrace.annotations.Return annotation
    can be used to mark an argument to hold the return value. Refer to
    the
    sampleClassload.java
  • @com.sun.btrace.annotations.CalledInstance annotation
    can be used to mark an argument to hold the called instance value.
    Refer to the
    sample AllCalls2.java
  • @com.sun.btrace.annotations.CalledMethod can
    be used to mark an argument to hold the called method name. Refer to
    the
    samples澳门新萄京,AllCalls1.java or AllCalls2.java
 1 1.What's gdb?
 2 gdb is tool of linux,a debugger of GNU project.It can help you to see what's going on inside the program while it executes.For example,the usual case is the program running slowly or almost to be crashed.
 3 
 4 2.What can gdb do?
 5 It can catch details in the act belowl:
 6 1).Start the program and specify anything the might affect its behavior.
 7 2).Let the program stop on specific conditions.
 8 3).Check the details when the program stops.
 9 4).Change things in the program and try to correct effect of potential problems.
10 
11 3.What's LWP?
12 LWP is abbreviation of Light Weight Process.
13 Process:You'll find a process in background after you've executed a linux command(which can be observed by "ps -ef").
14 Thread:It's nothing but a flow of execution of the linux process.
15 Light Weight Process:It's almost the same with "thread" but a term at kernal level.
16 
17 Here's some detail of it:
18 What are Linux Processes, Threads, Light Weight Processes, and Process State
19 https://www.thegeekstuff.com/2013/11/linux-process-and-threads/ 

 

BTrace Program Structure

A BTrace program is a plain Java class that has one or
more public static void methods that are annotated with BTrace
annotations.
The annotations are used to specify traced program “locations” (also
known as “probe points”). The tracing actions are specified inside the
static method bodies. These static methods are referred as “action”
methods.

 

Introduce

BTrace Command Line

BTrace is run using the command line tool btrace as shown below:

    btrace [-I <include-path>] [-p <port>] [-cp <classpath>] <pid> <btrace-script> [<args>]

where

  • include-path is a set of include directories that are searched for
    header files. BTrace includes a simple preprocess with support for
    #define, #include and conditional compilation. It is not like
    a complete C/C++ preprocessor – but a useful subset. See the sample
    “ThreadBean.java”. If -I is not specified, BTrace skips the
    preprocessor invocation step.
  • port is the port in which BTrace agent listens. This is optional
    argument.
  • classpath is set of directories, jar files where BTrace searches
    for classes during compilation. Default is “.”.
  • pid is the process id of the traced Java program
  • btrace-script is the trace program. If it is a “.java”, then it is
    compiled before submission. Or else, it is assumed to be
    pre-compiled [i.e., it has to be a .class] and submitted.

**Check the content of output
file.**

 1 (root@localhost mysql3306.sock)[sysbench]>show global status like '%Threads%';
 2 +------------------------+-------+
 3 | Variable_name          | Value |
 4 +------------------------+-------+
 5 | Delayed_insert_threads | 0     |
 6 | Slow_launch_threads    | 0     |
 7 | Threads_cached         | 1     |
 8 | Threads_connected      | 5     | //Threads_connected has been up to 5.
 9 | Threads_created        | 5     |
10 | Threads_running        | 1     |
11 +------------------------+-------+
12 6 rows in set (0.04 sec)
13 
14 
15 (root@localhost mysql3306.sock)[sysbench]>show global status like '%Threads%';
16 +------------------------+-------+
17 | Variable_name          | Value |
18 +------------------------+-------+
19 | Delayed_insert_threads | 0     |
20 | Slow_launch_threads    | 0     |
21 | Threads_cached         | 3     |
22 | Threads_connected      | 6     | //Second time,Threads_connected has been up to 6.
23 | Threads_created        | 9     |
24 | Threads_running        | 1     |
25 +------------------------+-------+
26 6 rows in set (0.01 sec)

BTrace Terminology

Probe Point
“location” or “event” at which a set of tracing statements are executed.
Probe point is “place” or “event” of interest where we want to execute
some tracing statements.

Trace Actions or Actions
Trace statements that are executed whenever a probe “fires”.

Action Methods
BTrace trace statements that are executed when a probe fires are defined
inside a static method a class. Such methods are called “action”
methods.

**Parameters(few parameters in this
tool)**

    We have a lot of methods to diagnose
problems in our system such as strace,pstack,gstack,gdb,pt-pmp,etc.But
sometimes there will be some fitful performance issues which are not so
easy to trace.Thus,pt-stalk may help us in diagnosing these kind of
problems.

Starting an application with BTrace agent

So far, we saw how to trace a running Java program. It is also possible
to start an application with BTrace agent in it. If you want to start
tracing the application from the very “beginning”, you may want to start
the app with BTrace agent and specify a trace script along with it
[i.e., BTrace agent is attach-on-demand loadable as well as
pre-loadable agent] You can use the following command to start an app
and specify BTrace script file. But, you need to precompile your BTrace
script for
this kind of usage.

    java -javaagent:btrace-agent.jar=script=<pre-compiled-btrace-script> <MainClass> <AppArguments>

When starting the application this way, the trace output goes to a file
named <btrace-class-file-name>.btrace in the current directory.
Also, you can avoid starting server for other remote BTrace clients by
specifying noServer=true as an argument to the BTrace agent.

There is a convenient script called btracer to do the above:

    btracer <pre-compiled-btrace.class> <application-main-class> <application-args>

发表评论

电子邮件地址不会被公开。 必填项已用*标注

网站地图xml地图