你的位置:首页 > 操作系统

[操作系统]Linux 性能优化工具 perf top


1. perf

perf 是一个调查 Linux 中各种性能问题的有力工具。

NAME    perf - Performance analysis tools for LinuxSYNOPSIS    perf [--version] [--help] COMMAND [ARGS]DESCRIPTION    Performance counters for Linux are a new kernel-based subsystem that provide a framework for all things performance    analysis. It covers hardware level (CPU/PMU, Performance Monitoring Unit) features and software features (software    counters, tracepoints) as well.SEE ALSO    perf-stat(1), perf-top(1), perf-record(1), perf-report(1), perf-list(1)

性能 counter 是 Linux 新内核中提供了专门的子系统 , 该子系统提供了一个专门的框架用于各种性能分析。包括各种硬件性能指标和软件性能指标。

perf 包含了:perf-stat (perf stat), perf-top (perf top), perf-record (perf record), perf-list (perf list)

注意:在查看manual时需要使用 man perf-top, 但是执行命令时使用:perf top

2. perf list

perf list 主要是用于列出有哪些可用的event,可以供 perf top -e eventname 来分析。

perf list [hw|sw|cache|tracepoint|pmu|event_glob]

perf list hw 可以列出 hard ware 的event, sw 是software的event, 其它类似

[root@localhost ~]# perf list hw[root@localhost ~]# perf list sw cpu-clock                     [Software event] task-clock                     [Software event] page-faults OR faults               [Software event] context-switches OR cs               [Software event] cpu-migrations OR migrations            [Software event] minor-faults                    [Software event] major-faults                    [Software event] alignment-faults                  [Software event] emulation-faults                  [Software event] dummy                       [Software event]
[root@localhost ~]# perf list pmu
[root@localhost ~]# perf list event_glob

List of pre-defined events (to be used in -e):
[root@localhost ~]#

perf list tracepoint 有搞定700多个(tracepoint events):

 ext4:ext4_free_inode                [Tracepoint event] ext4:ext4_request_inode              [Tracepoint event] ext4:ext4_allocate_inode              [Tracepoint event] ext4:ext4_write_begin               [Tracepoint event] ext4:ext4_da_write_begin              [Tracepoint event] ext4:ext4_ordered_write_end            [Tracepoint event] ext4:ext4_writeback_write_end           [Tracepoint event] ext4:ext4_journalled_write_end           [Tracepoint event] ext4:ext4_da_write_end               [Tracepoint event] ext4:ext4_writepage                [Tracepoint event] ext4:ext4_da_writepages              [Tracepoint event] ext4:ext4_da_write_pages              [Tracepoint event] ext4:ext4_da_writepages_result           [Tracepoint event] ext4:ext4_discard_blocks              [Tracepoint event] ext4:ext4_mb_new_inode_pa             [Tracepoint event] ext4:ext4_mb_new_group_pa             [Tracepoint event] ext4:ext4_mb_release_inode_pa           [Tracepoint event] ext4:ext4_mb_release_group_pa           [Tracepoint event] ext4:ext4_discard_preallocations          [Tracepoint event] ext4:ext4_mb_discard_preallocations        [Tracepoint event] ext4:ext4_request_blocks              [Tracepoint event] ext4:ext4_allocate_blocks             [Tracepoint event] ext4:ext4_free_blocks               [Tracepoint event] ext4:ext4_sync_file                [Tracepoint event] ext4:ext4_sync_fs                 [Tracepoint event] ext4:ext4_alloc_da_blocks             [Tracepoint event] ext4:ext4_mballoc_alloc              [Tracepoint event] ext4:ext4_mballoc_prealloc             [Tracepoint event] ext4:ext4_mballoc_discard             [Tracepoint event] ext4:ext4_mballoc_free               [Tracepoint event] ext4:ext4_trim_extent               [Tracepoint event] ext4:ext4_trim_all_free              [Tracepoint event] ext4:ext4_update_reserve_space           [Tracepoint event] jbd2:jbd2_checkpoint                [Tracepoint event] jbd2:jbd2_start_commit               [Tracepoint event] jbd2:jbd2_commit_locking              [Tracepoint event] jbd2:jbd2_commit_flushing             [Tracepoint event] jbd2:jbd2_commit_logging              [Tracepoint event] jbd2:jbd2_drop_transaction             [Tracepoint event] jbd2:jbd2_end_commit                [Tracepoint event] jbd2:jbd2_submit_inode_data            [Tracepoint event] jbd2:jbd2_run_stats                [Tracepoint event] jbd2:jbd2_checkpoint_stats             [Tracepoint event] jbd2:jbd2_update_superblock_end          [Tracepoint event] skb:kfree_skb                   [Tracepoint event] skb:consume_skb                  [Tracepoint event] skb:skb_copy_datagram_iovec            [Tracepoint event] net:net_dev_xmit                  [Tracepoint event] net:net_dev_queue                 [Tracepoint event] net:netif_receive_skb               [Tracepoint event] net:netif_rx                    [Tracepoint event] napi:napi_poll                   [Tracepoint event] sock:sock_rcvqueue_full              [Tracepoint event] sock:sock_exceed_buf_limit             [Tracepoint event] udp:udp_fail_queue_rcv_skb             [Tracepoint event] scsi:scsi_dispatch_cmd_start            [Tracepoint event] scsi:scsi_dispatch_cmd_error            [Tracepoint event] scsi:scsi_dispatch_cmd_done            [Tracepoint event] scsi:scsi_dispatch_cmd_timeout           [Tracepoint event] scsi:scsi_eh_wakeup                [Tracepoint event] ras:aer_event                   [Tracepoint event] block:block_rq_abort                [Tracepoint event] block:block_rq_requeue               [Tracepoint event] block:block_rq_complete              [Tracepoint event] block:block_rq_insert               [Tracepoint event] block:block_rq_issue                [Tracepoint event] block:block_bio_bounce               [Tracepoint event] block:block_bio_complete              [Tracepoint event] block:block_bio_backmerge             [Tracepoint event] block:block_bio_frontmerge             [Tracepoint event] block:block_bio_queue               [Tracepoint event] block:block_getrq                 [Tracepoint event] block:block_sleeprq                [Tracepoint event] block:block_plug                  [Tracepoint event] block:block_unplug_timer              [Tracepoint event] block:block_unplug_io               [Tracepoint event] block:block_split                 [Tracepoint event] block:block_remap                 [Tracepoint event] block:block_rq_remap                [Tracepoint event] writeback:writeback_nothread            [Tracepoint event] writeback:writeback_queue             [Tracepoint event] writeback:writeback_exec              [Tracepoint event] writeback:writeback_pages_written         [Tracepoint event] writeback:writeback_nowork             [Tracepoint event] writeback:writeback_bdi_register          [Tracepoint event] writeback:writeback_bdi_unregister         [Tracepoint event] writeback:writeback_task_start           [Tracepoint event] writeback:writeback_task_stop           [Tracepoint event] writeback:wbc_writeback_start           [Tracepoint event] writeback:wbc_writeback_written          [Tracepoint event] writeback:wbc_writeback_wait            [Tracepoint event] writeback:wbc_balance_dirty_start         [Tracepoint event] writeback:wbc_balance_dirty_written        [Tracepoint event] writeback:wbc_balance_dirty_wait          [Tracepoint event] writeback:wbc_writepage              [Tracepoint event] kmem:kmalloc                    [Tracepoint event] kmem:kmem_cache_alloc               [Tracepoint event] kmem:kmalloc_node                 [Tracepoint event] kmem:kmem_cache_alloc_node             [Tracepoint event] kmem:kfree                     [Tracepoint event] kmem:kmem_cache_free                [Tracepoint event] kmem:mm_page_free_direct              [Tracepoint event] kmem:mm_pagevec_free                [Tracepoint event] kmem:mm_page_alloc                 [Tracepoint event] kmem:mm_page_alloc_zone_locked           [Tracepoint event] kmem:mm_page_pcpu_drain              [Tracepoint event] kmem:mm_page_alloc_extfrag             [Tracepoint event] kmem:mm_anon_fault                 [Tracepoint event] kmem:mm_anon_pgin                 [Tracepoint event] kmem:mm_anon_cow                  [Tracepoint event] kmem:mm_anon_userfree               [Tracepoint event] kmem:mm_anon_unmap                 [Tracepoint event] kmem:mm_filemap_fault               [Tracepoint event] kmem:mm_filemap_cow                [Tracepoint event] kmem:mm_filemap_unmap               [Tracepoint event] kmem:mm_filemap_userunmap             [Tracepoint event] kmem:mm_pagereclaim_pgout             [Tracepoint event] kmem:mm_pagereclaim_free              [Tracepoint event] kmem:mm_background_writeout            [Tracepoint event] kmem:mm_olddata_writeout              [Tracepoint event] kmem:mm_balancedirty_writeout           [Tracepoint event] kmem:mm_kswapd_ran                 [Tracepoint event] kmem:mm_directreclaim_reclaimall          [Tracepoint event] kmem:mm_directreclaim_reclaimzone         [Tracepoint event] kmem:mm_pagereclaim_shrinkzone           [Tracepoint event] kmem:mm_pagereclaim_shrinkactive          [Tracepoint event] kmem:mm_pagereclaim_shrinkinactive         [Tracepoint event] kmem:mm_kernel_pagefault              [Tracepoint event] kmem:mm_vmscan_kswapd_sleep            [Tracepoint event] kmem:mm_vmscan_kswapd_wake             [Tracepoint event] kmem:mm_vmscan_wakeup_kswapd            [Tracepoint event] kmem:mm_vmscan_direct_reclaim_begin        [Tracepoint event] kmem:mm_vmscan_direct_reclaim_end         [Tracepoint event] kmem:mm_vmscan_lru_isolate             [Tracepoint event] kmem:mm_vmscan_writepage              [Tracepoint event] power:power_start                 [Tracepoint event] power:power_frequency               [Tracepoint event] power:power_end                  [Tracepoint event] power:pstate_sample                [Tracepoint event] module:module_load                 [Tracepoint event] module:module_free                 [Tracepoint event] module:module_get                 [Tracepoint event] module:module_put                 [Tracepoint event] module:module_request               [Tracepoint event] workqueue:workqueue_insertion           [Tracepoint event] workqueue:workqueue_execution           [Tracepoint event] workqueue:workqueue_creation            [Tracepoint event] workqueue:workqueue_destruction          [Tracepoint event] signal:signal_generate               [Tracepoint event] signal:signal_deliver               [Tracepoint event] timer:timer_init                  [Tracepoint event] timer:timer_start                 [Tracepoint event] timer:timer_expire_entry              [Tracepoint event] timer:timer_expire_exit              [Tracepoint event] timer:timer_cancel                 [Tracepoint event] timer:hrtimer_init                 [Tracepoint event] timer:hrtimer_start                [Tracepoint event] timer:hrtimer_expire_entry             [Tracepoint event] timer:hrtimer_expire_exit             [Tracepoint event] timer:hrtimer_cancel                [Tracepoint event] timer:itimer_state                 [Tracepoint event] timer:itimer_expire                [Tracepoint event] irq:irq_handler_entry               [Tracepoint event] irq:irq_handler_exit                [Tracepoint event] irq:softirq_entry                 [Tracepoint event] irq:softirq_exit                  [Tracepoint event] irq:softirq_raise                 [Tracepoint event] sched:sched_kthread_stop              [Tracepoint event] sched:sched_kthread_stop_ret            [Tracepoint event] sched:sched_wait_task               [Tracepoint event] sched:sched_wakeup                 [Tracepoint event] sched:sched_wakeup_new               [Tracepoint event] sched:sched_switch                 [Tracepoint event] sched:sched_migrate_task              [Tracepoint event] sched:sched_process_free              [Tracepoint event] sched:sched_process_exit              [Tracepoint event] sched:sched_process_wait              [Tracepoint event] sched:sched_process_fork              [Tracepoint event] sched:sched_stat_wait               [Tracepoint event] sched:sched_stat_sleep               [Tracepoint event] sched:sched_stat_iowait              [Tracepoint event] sched:sched_stat_blocked              [Tracepoint event] sched:sched_stat_runtime              [Tracepoint event] mce:mce_record                   [Tracepoint event] syscalls:sys_enter_sendmmsg            [Tracepoint event] syscalls:sys_exit_sendmmsg             [Tracepoint event] syscalls:sys_enter_recvmmsg            [Tracepoint event] syscalls:sys_exit_recvmmsg             [Tracepoint event] syscalls:sys_enter_socketcall           [Tracepoint event] syscalls:sys_exit_socketcall            [Tracepoint event] syscalls:sys_enter_add_key             [Tracepoint event] syscalls:sys_exit_add_key             [Tracepoint event] syscalls:sys_enter_request_key           [Tracepoint event] syscalls:sys_exit_request_key           [Tracepoint event] syscalls:sys_enter_keyctl             [Tracepoint event] syscalls:sys_exit_keyctl              [Tracepoint event] syscalls:sys_enter_mq_open             [Tracepoint event] syscalls:sys_exit_mq_open             [Tracepoint event] syscalls:sys_enter_mq_unlink            [Tracepoint event] syscalls:sys_exit_mq_unlink            [Tracepoint event] syscalls:sys_enter_mq_timedsend          [Tracepoint event] syscalls:sys_exit_mq_timedsend           [Tracepoint event] syscalls:sys_enter_mq_timedreceive         [Tracepoint event] syscalls:sys_exit_mq_timedreceive         [Tracepoint event] syscalls:sys_enter_mq_notify            [Tracepoint event] syscalls:sys_exit_mq_notify            [Tracepoint event] syscalls:sys_enter_mq_getsetattr          [Tracepoint event] syscalls:sys_exit_mq_getsetattr          [Tracepoint event] syscalls:sys_enter_quotactl            [Tracepoint event] syscalls:sys_exit_quotactl             [Tracepoint event] syscalls:sys_enter_nfsservctl           [Tracepoint event] syscalls:sys_exit_nfsservctl            [Tracepoint event] syscalls:sys_enter_flock              [Tracepoint event] syscalls:sys_exit_flock              [Tracepoint event] syscalls:sys_enter_io_setup            [Tracepoint event] syscalls:sys_exit_io_setup             [Tracepoint event] syscalls:sys_enter_io_destroy           [Tracepoint event] syscalls:sys_exit_io_destroy            [Tracepoint event] syscalls:sys_enter_io_submit            [Tracepoint event] syscalls:sys_exit_io_submit            [Tracepoint event] syscalls:sys_enter_io_cancel            [Tracepoint event] syscalls:sys_exit_io_cancel            [Tracepoint event] syscalls:sys_enter_io_getevents          [Tracepoint event] syscalls:sys_exit_io_getevents           [Tracepoint event] syscalls:sys_enter_eventfd2            [Tracepoint event] syscalls:sys_exit_eventfd2             [Tracepoint event] syscalls:sys_enter_eventfd             [Tracepoint event] syscalls:sys_exit_eventfd             [Tracepoint event] syscalls:sys_enter_timerfd_create         [Tracepoint event] syscalls:sys_exit_timerfd_create          [Tracepoint event] syscalls:sys_enter_timerfd_settime         [Tracepoint event] syscalls:sys_exit_timerfd_settime         [Tracepoint event] syscalls:sys_enter_timerfd_gettime         [Tracepoint event] syscalls:sys_exit_timerfd_gettime         [Tracepoint event] syscalls:sys_enter_signalfd4            [Tracepoint event] syscalls:sys_exit_signalfd4            [Tracepoint event] syscalls:sys_enter_signalfd            [Tracepoint event] syscalls:sys_exit_signalfd             [Tracepoint event] syscalls:sys_enter_epoll_create1          [Tracepoint event] syscalls:sys_exit_epoll_create1          [Tracepoint event] syscalls:sys_enter_epoll_create          [Tracepoint event] syscalls:sys_exit_epoll_create           [Tracepoint event] syscalls:sys_enter_epoll_ctl            [Tracepoint event] syscalls:sys_exit_epoll_ctl            [Tracepoint event] syscalls:sys_enter_epoll_wait           [Tracepoint event] syscalls:sys_exit_epoll_wait            [Tracepoint event] syscalls:sys_enter_epoll_pwait           [Tracepoint event] syscalls:sys_exit_epoll_pwait           [Tracepoint event] syscalls:sys_enter_inotify_init1          [Tracepoint event] syscalls:sys_exit_inotify_init1          [Tracepoint event] syscalls:sys_enter_inotify_init          [Tracepoint event] syscalls:sys_exit_inotify_init           [Tracepoint event] syscalls:sys_enter_inotify_add_watch        [Tracepoint event] syscalls:sys_exit_inotify_add_watch        [Tracepoint event] syscalls:sys_enter_inotify_rm_watch        [Tracepoint event] syscalls:sys_exit_inotify_rm_watch         [Tracepoint event] syscalls:sys_enter_ioprio_set           [Tracepoint event] syscalls:sys_exit_ioprio_set            [Tracepoint event] syscalls:sys_enter_ioprio_get           [Tracepoint event] syscalls:sys_exit_ioprio_get            [Tracepoint event] syscalls:sys_enter_bdflush             [Tracepoint event] syscalls:sys_exit_bdflush             [Tracepoint event] syscalls:sys_enter_statfs             [Tracepoint event] syscalls:sys_exit_statfs              [Tracepoint event] syscalls:sys_enter_statfs64            [Tracepoint event] syscalls:sys_exit_statfs64             [Tracepoint event] syscalls:sys_enter_fstatfs             [Tracepoint event] syscalls:sys_exit_fstatfs             [Tracepoint event] syscalls:sys_enter_fstatfs64            [Tracepoint event] syscalls:sys_exit_fstatfs64            [Tracepoint event] syscalls:sys_enter_ustat              [Tracepoint event] syscalls:sys_exit_ustat              [Tracepoint event] syscalls:sys_enter_utime              [Tracepoint event] syscalls:sys_exit_utime              [Tracepoint event] syscalls:sys_enter_utimensat            [Tracepoint event] syscalls:sys_exit_utimensat            [Tracepoint event] syscalls:sys_enter_futimesat            [Tracepoint event] syscalls:sys_exit_futimesat            [Tracepoint event] syscalls:sys_enter_utimes             [Tracepoint event] syscalls:sys_exit_utimes              [Tracepoint event] syscalls:sys_enter_sync              [Tracepoint event] syscalls:sys_exit_sync               [Tracepoint event] syscalls:sys_enter_syncfs             [Tracepoint event] syscalls:sys_exit_syncfs              [Tracepoint event] syscalls:sys_enter_fsync              [Tracepoint event] syscalls:sys_exit_fsync              [Tracepoint event] syscalls:sys_enter_fdatasync            [Tracepoint event] syscalls:sys_exit_fdatasync            [Tracepoint event] syscalls:sys_enter_vmsplice            [Tracepoint event] syscalls:sys_exit_vmsplice             [Tracepoint event] syscalls:sys_enter_splice             [Tracepoint event] syscalls:sys_exit_splice              [Tracepoint event] syscalls:sys_enter_tee               [Tracepoint event] syscalls:sys_exit_tee               [Tracepoint event] syscalls:sys_enter_setxattr            [Tracepoint event] syscalls:sys_exit_setxattr             [Tracepoint event] syscalls:sys_enter_lsetxattr            [Tracepoint event] syscalls:sys_exit_lsetxattr            [Tracepoint event] syscalls:sys_enter_fsetxattr            [Tracepoint event] syscalls:sys_exit_fsetxattr            [Tracepoint event] syscalls:sys_enter_getxattr            [Tracepoint event] syscalls:sys_exit_getxattr             [Tracepoint event] syscalls:sys_enter_lgetxattr            [Tracepoint event] syscalls:sys_exit_lgetxattr            [Tracepoint event] syscalls:sys_enter_fgetxattr            [Tracepoint event] syscalls:sys_exit_fgetxattr            [Tracepoint event] syscalls:sys_enter_listxattr            [Tracepoint event] syscalls:sys_exit_listxattr            [Tracepoint event] syscalls:sys_enter_llistxattr           [Tracepoint event] syscalls:sys_exit_llistxattr            [Tracepoint event] syscalls:sys_enter_flistxattr           [Tracepoint event] syscalls:sys_exit_flistxattr            [Tracepoint event] syscalls:sys_enter_removexattr           [Tracepoint event] syscalls:sys_exit_removexattr           [Tracepoint event] syscalls:sys_enter_lremovexattr          [Tracepoint event] syscalls:sys_exit_lremovexattr           [Tracepoint event] syscalls:sys_enter_fremovexattr          [Tracepoint event] syscalls:sys_exit_fremovexattr           [Tracepoint event] syscalls:sys_enter_umount             [Tracepoint event] syscalls:sys_exit_umount              [Tracepoint event] syscalls:sys_enter_oldumount            [Tracepoint event] syscalls:sys_exit_oldumount            [Tracepoint event] syscalls:sys_enter_mount              [Tracepoint event] syscalls:sys_exit_mount              [Tracepoint event] syscalls:sys_enter_pivot_root           [Tracepoint event] syscalls:sys_exit_pivot_root            [Tracepoint event] syscalls:sys_enter_sysfs              [Tracepoint event] syscalls:sys_exit_sysfs              [Tracepoint event] syscalls:sys_enter_getcwd             [Tracepoint event] syscalls:sys_exit_getcwd              [Tracepoint event] syscalls:sys_enter_select             [Tracepoint event] syscalls:sys_exit_select              [Tracepoint event] syscalls:sys_enter_pselect6            [Tracepoint event] syscalls:sys_exit_pselect6             [Tracepoint event] syscalls:sys_enter_poll              [Tracepoint event] syscalls:sys_exit_poll               [Tracepoint event] syscalls:sys_enter_ppoll              [Tracepoint event] syscalls:sys_exit_ppoll              [Tracepoint event] syscalls:sys_enter_old_readdir           [Tracepoint event] syscalls:sys_exit_old_readdir           [Tracepoint event] syscalls:sys_enter_getdents            [Tracepoint event] syscalls:sys_exit_getdents             [Tracepoint event] syscalls:sys_enter_getdents64           [Tracepoint event] syscalls:sys_exit_getdents64            [Tracepoint event] syscalls:sys_enter_ioctl              [Tracepoint event] syscalls:sys_exit_ioctl              [Tracepoint event] syscalls:sys_enter_dup3              [Tracepoint event] syscalls:sys_exit_dup3               [Tracepoint event] syscalls:sys_enter_dup2              [Tracepoint event] syscalls:sys_exit_dup2               [Tracepoint event] syscalls:sys_enter_dup               [Tracepoint event] syscalls:sys_exit_dup               [Tracepoint event] syscalls:sys_enter_fcntl              [Tracepoint event] syscalls:sys_exit_fcntl              [Tracepoint event] syscalls:sys_enter_fcntl64             [Tracepoint event] syscalls:sys_exit_fcntl64             [Tracepoint event] syscalls:sys_enter_mknodat             [Tracepoint event] syscalls:sys_exit_mknodat             [Tracepoint event] syscalls:sys_enter_mknod              [Tracepoint event] syscalls:sys_exit_mknod              [Tracepoint event] syscalls:sys_enter_mkdirat             [Tracepoint event] syscalls:sys_exit_mkdirat             [Tracepoint event] syscalls:sys_enter_mkdir              [Tracepoint event] syscalls:sys_exit_mkdir              [Tracepoint event] syscalls:sys_enter_rmdir              [Tracepoint event] syscalls:sys_exit_rmdir              [Tracepoint event] syscalls:sys_enter_unlinkat            [Tracepoint event] syscalls:sys_exit_unlinkat             [Tracepoint event] syscalls:sys_enter_unlink             [Tracepoint event] syscalls:sys_exit_unlink              [Tracepoint event] syscalls:sys_enter_symlinkat            [Tracepoint event] syscalls:sys_exit_symlinkat            [Tracepoint event] syscalls:sys_enter_symlink             [Tracepoint event] syscalls:sys_exit_symlink             [Tracepoint event] syscalls:sys_enter_linkat             [Tracepoint event] syscalls:sys_exit_linkat              [Tracepoint event] syscalls:sys_enter_link              [Tracepoint event] syscalls:sys_exit_link               [Tracepoint event] syscalls:sys_enter_renameat            [Tracepoint event] syscalls:sys_exit_renameat             [Tracepoint event] syscalls:sys_enter_rename             [Tracepoint event] syscalls:sys_exit_rename              [Tracepoint event] syscalls:sys_enter_pipe2              [Tracepoint event] syscalls:sys_exit_pipe2              [Tracepoint event] syscalls:sys_enter_pipe              [Tracepoint event] syscalls:sys_exit_pipe               [Tracepoint event] syscalls:sys_enter_uselib             [Tracepoint event] syscalls:sys_exit_uselib              [Tracepoint event] syscalls:sys_enter_stat              [Tracepoint event] syscalls:sys_exit_stat               [Tracepoint event] syscalls:sys_enter_lstat              [Tracepoint event] syscalls:sys_exit_lstat              [Tracepoint event] syscalls:sys_enter_fstat              [Tracepoint event] syscalls:sys_exit_fstat              [Tracepoint event] syscalls:sys_enter_newstat             [Tracepoint event] syscalls:sys_exit_newstat             [Tracepoint event] syscalls:sys_enter_newlstat            [Tracepoint event] syscalls:sys_exit_newlstat             [Tracepoint event] syscalls:sys_enter_newfstat            [Tracepoint event] syscalls:sys_exit_newfstat             [Tracepoint event] syscalls:sys_enter_readlinkat           [Tracepoint event] syscalls:sys_exit_readlinkat            [Tracepoint event] syscalls:sys_enter_readlink            [Tracepoint event] syscalls:sys_exit_readlink             [Tracepoint event] syscalls:sys_enter_stat64             [Tracepoint event] syscalls:sys_exit_stat64              [Tracepoint event] syscalls:sys_enter_lstat64             [Tracepoint event] syscalls:sys_exit_lstat64             [Tracepoint event] syscalls:sys_enter_fstat64             [Tracepoint event] syscalls:sys_exit_fstat64             [Tracepoint event] syscalls:sys_enter_fstatat64            [Tracepoint event] syscalls:sys_exit_fstatat64            [Tracepoint event] syscalls:sys_enter_lseek              [Tracepoint event] syscalls:sys_exit_lseek              [Tracepoint event] syscalls:sys_enter_llseek             [Tracepoint event] syscalls:sys_exit_llseek              [Tracepoint event] syscalls:sys_enter_read              [Tracepoint event] syscalls:sys_exit_read               [Tracepoint event] syscalls:sys_enter_write              [Tracepoint event] syscalls:sys_exit_write              [Tracepoint event] syscalls:sys_enter_readv              [Tracepoint event] syscalls:sys_exit_readv              [Tracepoint event] syscalls:sys_enter_writev             [Tracepoint event] syscalls:sys_exit_writev              [Tracepoint event] syscalls:sys_enter_preadv             [Tracepoint event] syscalls:sys_exit_preadv              [Tracepoint event] syscalls:sys_enter_pwritev             [Tracepoint event] syscalls:sys_exit_pwritev             [Tracepoint event] syscalls:sys_enter_sendfile            [Tracepoint event] syscalls:sys_exit_sendfile             [Tracepoint event] syscalls:sys_enter_sendfile64           [Tracepoint event] syscalls:sys_exit_sendfile64            [Tracepoint event] syscalls:sys_enter_truncate            [Tracepoint event] syscalls:sys_exit_truncate             [Tracepoint event] syscalls:sys_enter_ftruncate            [Tracepoint event] syscalls:sys_exit_ftruncate            [Tracepoint event] syscalls:sys_enter_faccessat            [Tracepoint event] syscalls:sys_exit_faccessat            [Tracepoint event] syscalls:sys_enter_access             [Tracepoint event] syscalls:sys_exit_access              [Tracepoint event] syscalls:sys_enter_chdir              [Tracepoint event] syscalls:sys_exit_chdir              [Tracepoint event] syscalls:sys_enter_fchdir             [Tracepoint event] syscalls:sys_exit_fchdir              [Tracepoint event] syscalls:sys_enter_chroot             [Tracepoint event] syscalls:sys_exit_chroot              [Tracepoint event] syscalls:sys_enter_fchmod             [Tracepoint event] syscalls:sys_exit_fchmod              [Tracepoint event] syscalls:sys_enter_fchmodat            [Tracepoint event] syscalls:sys_exit_fchmodat             [Tracepoint event] syscalls:sys_enter_chmod              [Tracepoint event] syscalls:sys_exit_chmod              [Tracepoint event] syscalls:sys_enter_fchownat            [Tracepoint event] syscalls:sys_exit_fchownat             [Tracepoint event] syscalls:sys_enter_chown              [Tracepoint event] syscalls:sys_exit_chown              [Tracepoint event] syscalls:sys_enter_lchown             [Tracepoint event] syscalls:sys_exit_lchown              [Tracepoint event] syscalls:sys_enter_fchown             [Tracepoint event] syscalls:sys_exit_fchown              [Tracepoint event] syscalls:sys_enter_open              [Tracepoint event] syscalls:sys_exit_open               [Tracepoint event] syscalls:sys_enter_openat             [Tracepoint event] syscalls:sys_exit_openat              [Tracepoint event] syscalls:sys_enter_creat              [Tracepoint event] syscalls:sys_exit_creat              [Tracepoint event] syscalls:sys_enter_close              [Tracepoint event] syscalls:sys_exit_close              [Tracepoint event] syscalls:sys_enter_vhangup             [Tracepoint event] syscalls:sys_exit_vhangup             [Tracepoint event] syscalls:sys_enter_swapoff             [Tracepoint event] syscalls:sys_exit_swapoff             [Tracepoint event] syscalls:sys_enter_swapon             [Tracepoint event] syscalls:sys_exit_swapon              [Tracepoint event] syscalls:sys_enter_process_vm_readv        [Tracepoint event] syscalls:sys_exit_process_vm_readv         [Tracepoint event] syscalls:sys_enter_process_vm_writev        [Tracepoint event] syscalls:sys_exit_process_vm_writev        [Tracepoint event] syscalls:sys_enter_msync              [Tracepoint event] syscalls:sys_exit_msync              [Tracepoint event] syscalls:sys_enter_mremap             [Tracepoint event] syscalls:sys_exit_mremap              [Tracepoint event] syscalls:sys_enter_mprotect            [Tracepoint event] syscalls:sys_exit_mprotect             [Tracepoint event] syscalls:sys_enter_brk               [Tracepoint event] syscalls:sys_exit_brk               [Tracepoint event] syscalls:sys_enter_munmap             [Tracepoint event] syscalls:sys_exit_munmap              [Tracepoint event] syscalls:sys_enter_mlock              [Tracepoint event] syscalls:sys_exit_mlock              [Tracepoint event] syscalls:sys_enter_munlock             [Tracepoint event] syscalls:sys_exit_munlock             [Tracepoint event] syscalls:sys_enter_mlockall            [Tracepoint event] syscalls:sys_exit_mlockall             [Tracepoint event] syscalls:sys_enter_munlockall           [Tracepoint event] syscalls:sys_exit_munlockall            [Tracepoint event] syscalls:sys_enter_mincore             [Tracepoint event] syscalls:sys_exit_mincore             [Tracepoint event] syscalls:sys_enter_madvise             [Tracepoint event] syscalls:sys_exit_madvise             [Tracepoint event] syscalls:sys_enter_remap_file_pages        [Tracepoint event] syscalls:sys_exit_remap_file_pages         [Tracepoint event] syscalls:sys_enter_mmap_pgoff           [Tracepoint event] syscalls:sys_exit_mmap_pgoff            [Tracepoint event] syscalls:sys_enter_perf_event_open         [Tracepoint event] syscalls:sys_exit_perf_event_open         [Tracepoint event] syscalls:sys_enter_kexec_load           [Tracepoint event] syscalls:sys_exit_kexec_load            [Tracepoint event] syscalls:sys_enter_acct              [Tracepoint event] syscalls:sys_exit_acct               [Tracepoint event] syscalls:sys_enter_delete_module          [Tracepoint event] syscalls:sys_exit_delete_module          [Tracepoint event] syscalls:sys_enter_init_module           [Tracepoint event] syscalls:sys_exit_init_module           [Tracepoint event] syscalls:sys_enter_chown16             [Tracepoint event] syscalls:sys_exit_chown16             [Tracepoint event] syscalls:sys_enter_lchown16            [Tracepoint event] syscalls:sys_exit_lchown16             [Tracepoint event] syscalls:sys_enter_fchown16            [Tracepoint event] syscalls:sys_exit_fchown16             [Tracepoint event] syscalls:sys_enter_setregid16           [Tracepoint event] syscalls:sys_exit_setregid16            [Tracepoint event] syscalls:sys_enter_setgid16            [Tracepoint event] syscalls:sys_exit_setgid16             [Tracepoint event] syscalls:sys_enter_setreuid16           [Tracepoint event] syscalls:sys_exit_setreuid16            [Tracepoint event] syscalls:sys_enter_setuid16            [Tracepoint event] syscalls:sys_exit_setuid16             [Tracepoint event] syscalls:sys_enter_setresuid16           [Tracepoint event] syscalls:sys_exit_setresuid16           [Tracepoint event] syscalls:sys_enter_getresuid16           [Tracepoint event] syscalls:sys_exit_getresuid16           [Tracepoint event] syscalls:sys_enter_setresgid16           [Tracepoint event] syscalls:sys_exit_setresgid16           [Tracepoint event] syscalls:sys_enter_getresgid16           [Tracepoint event] syscalls:sys_exit_getresgid16           [Tracepoint event] syscalls:sys_enter_setfsuid16           [Tracepoint event] syscalls:sys_exit_setfsuid16            [Tracepoint event] syscalls:sys_enter_setfsgid16           [Tracepoint event] syscalls:sys_exit_setfsgid16            [Tracepoint event] syscalls:sys_enter_getgroups16           [Tracepoint event] syscalls:sys_exit_getgroups16           [Tracepoint event] syscalls:sys_enter_setgroups16           [Tracepoint event] syscalls:sys_exit_setgroups16           [Tracepoint event] syscalls:sys_enter_getuid16            [Tracepoint event] syscalls:sys_exit_getuid16             [Tracepoint event] syscalls:sys_enter_geteuid16            [Tracepoint event] syscalls:sys_exit_geteuid16            [Tracepoint event] syscalls:sys_enter_getgid16            [Tracepoint event] syscalls:sys_exit_getgid16             [Tracepoint event] syscalls:sys_enter_getegid16            [Tracepoint event] syscalls:sys_exit_getegid16            [Tracepoint event] syscalls:sys_enter_set_robust_list         [Tracepoint event] syscalls:sys_exit_set_robust_list         [Tracepoint event] syscalls:sys_enter_get_robust_list         [Tracepoint event] syscalls:sys_exit_get_robust_list         [Tracepoint event] syscalls:sys_enter_futex              [Tracepoint event] syscalls:sys_exit_futex              [Tracepoint event] syscalls:sys_enter_getgroups            [Tracepoint event] syscalls:sys_exit_getgroups            [Tracepoint event] syscalls:sys_enter_setgroups            [Tracepoint event] syscalls:sys_exit_setgroups            [Tracepoint event] syscalls:sys_enter_setns              [Tracepoint event] syscalls:sys_exit_setns              [Tracepoint event] syscalls:sys_enter_nanosleep            [Tracepoint event] syscalls:sys_exit_nanosleep            [Tracepoint event] syscalls:sys_enter_timer_create          [Tracepoint event] syscalls:sys_exit_timer_create           [Tracepoint event] syscalls:sys_enter_timer_gettime          [Tracepoint event] syscalls:sys_exit_timer_gettime          [Tracepoint event] syscalls:sys_enter_timer_getoverrun        [Tracepoint event] syscalls:sys_exit_timer_getoverrun         [Tracepoint event] syscalls:sys_enter_timer_settime          [Tracepoint event] syscalls:sys_exit_timer_settime          [Tracepoint event] syscalls:sys_enter_timer_delete          [Tracepoint event] syscalls:sys_exit_timer_delete           [Tracepoint event] syscalls:sys_enter_clock_settime          [Tracepoint event] syscalls:sys_exit_clock_settime          [Tracepoint event] syscalls:sys_enter_clock_gettime          [Tracepoint event] syscalls:sys_exit_clock_gettime          [Tracepoint event] syscalls:sys_enter_clock_adjtime          [Tracepoint event] syscalls:sys_exit_clock_adjtime          [Tracepoint event] syscalls:sys_enter_clock_getres          [Tracepoint event] syscalls:sys_exit_clock_getres           [Tracepoint event] syscalls:sys_enter_clock_nanosleep         [Tracepoint event] syscalls:sys_exit_clock_nanosleep         [Tracepoint event] syscalls:sys_enter_setpriority           [Tracepoint event] syscalls:sys_exit_setpriority           [Tracepoint event] syscalls:sys_enter_getpriority           [Tracepoint event] syscalls:sys_exit_getpriority           [Tracepoint event] syscalls:sys_enter_reboot             [Tracepoint event] syscalls:sys_exit_reboot              [Tracepoint event] syscalls:sys_enter_setregid            [Tracepoint event] syscalls:sys_exit_setregid             [Tracepoint event] syscalls:sys_enter_setgid             [Tracepoint event] syscalls:sys_exit_setgid              [Tracepoint event] syscalls:sys_enter_setreuid            [Tracepoint event] syscalls:sys_exit_setreuid             [Tracepoint event] syscalls:sys_enter_setuid             [Tracepoint event] syscalls:sys_exit_setuid              [Tracepoint event] syscalls:sys_enter_setresuid            [Tracepoint event] syscalls:sys_exit_setresuid            [Tracepoint event] syscalls:sys_enter_getresuid            [Tracepoint event] syscalls:sys_exit_getresuid            [Tracepoint event] syscalls:sys_enter_setresgid            [Tracepoint event] syscalls:sys_exit_setresgid            [Tracepoint event] syscalls:sys_enter_getresgid            [Tracepoint event] syscalls:sys_exit_getresgid            [Tracepoint event] syscalls:sys_enter_setfsuid            [Tracepoint event] syscalls:sys_exit_setfsuid             [Tracepoint event] syscalls:sys_enter_setfsgid            [Tracepoint event] syscalls:sys_exit_setfsgid             [Tracepoint event] syscalls:sys_enter_times              [Tracepoint event] syscalls:sys_exit_times              [Tracepoint event] syscalls:sys_enter_setpgid             [Tracepoint event] syscalls:sys_exit_setpgid             [Tracepoint event] syscalls:sys_enter_getpgid             [Tracepoint event] syscalls:sys_exit_getpgid             [Tracepoint event] syscalls:sys_enter_getpgrp             [Tracepoint event] syscalls:sys_exit_getpgrp             [Tracepoint event] syscalls:sys_enter_getsid             [Tracepoint event] syscalls:sys_exit_getsid              [Tracepoint event] syscalls:sys_enter_setsid             [Tracepoint event] syscalls:sys_exit_setsid              [Tracepoint event] syscalls:sys_enter_newuname            [Tracepoint event] syscalls:sys_exit_newuname             [Tracepoint event] syscalls:sys_enter_sethostname           [Tracepoint event] syscalls:sys_exit_sethostname           [Tracepoint event] syscalls:sys_enter_setdomainname          [Tracepoint event] syscalls:sys_exit_setdomainname          [Tracepoint event] syscalls:sys_enter_getrlimit            [Tracepoint event] syscalls:sys_exit_getrlimit            [Tracepoint event] syscalls:sys_enter_old_getrlimit          [Tracepoint event] syscalls:sys_exit_old_getrlimit          [Tracepoint event] syscalls:sys_enter_setrlimit            [Tracepoint event] syscalls:sys_exit_setrlimit            [Tracepoint event] syscalls:sys_enter_getrusage            [Tracepoint event] syscalls:sys_exit_getrusage            [Tracepoint event] syscalls:sys_enter_umask              [Tracepoint event] syscalls:sys_exit_umask              [Tracepoint event] syscalls:sys_enter_prctl              [Tracepoint event] syscalls:sys_exit_prctl              [Tracepoint event] syscalls:sys_enter_getcpu             [Tracepoint event] syscalls:sys_exit_getcpu              [Tracepoint event] syscalls:sys_enter_restart_syscall         [Tracepoint event] syscalls:sys_exit_restart_syscall         [Tracepoint event] syscalls:sys_enter_rt_sigprocmask         [Tracepoint event] syscalls:sys_exit_rt_sigprocmask          [Tracepoint event] syscalls:sys_enter_rt_sigpending          [Tracepoint event] syscalls:sys_exit_rt_sigpending          [Tracepoint event] syscalls:sys_enter_rt_sigtimedwait         [Tracepoint event] syscalls:sys_exit_rt_sigtimedwait         [Tracepoint event] syscalls:sys_enter_kill              [Tracepoint event] syscalls:sys_exit_kill               [Tracepoint event] syscalls:sys_enter_tgkill             [Tracepoint event] syscalls:sys_exit_tgkill              [Tracepoint event] syscalls:sys_enter_tkill              [Tracepoint event] syscalls:sys_exit_tkill              [Tracepoint event] syscalls:sys_enter_rt_sigqueueinfo         [Tracepoint event] syscalls:sys_exit_rt_sigqueueinfo         [Tracepoint event] syscalls:sys_enter_rt_tgsigqueueinfo        [Tracepoint event] syscalls:sys_exit_rt_tgsigqueueinfo        [Tracepoint event] syscalls:sys_enter_sigpending           [Tracepoint event] syscalls:sys_exit_sigpending            [Tracepoint event] syscalls:sys_enter_sigprocmask           [Tracepoint event] syscalls:sys_exit_sigprocmask           [Tracepoint event] syscalls:sys_enter_rt_sigaction          [Tracepoint event] syscalls:sys_exit_rt_sigaction           [Tracepoint event] syscalls:sys_enter_sgetmask            [Tracepoint event] syscalls:sys_exit_sgetmask             [Tracepoint event] syscalls:sys_enter_ssetmask            [Tracepoint event] syscalls:sys_exit_ssetmask             [Tracepoint event] syscalls:sys_enter_signal             [Tracepoint event] syscalls:sys_exit_signal              [Tracepoint event] syscalls:sys_enter_pause              [Tracepoint event] syscalls:sys_exit_pause              [Tracepoint event] syscalls:sys_enter_rt_sigsuspend          [Tracepoint event] syscalls:sys_exit_rt_sigsuspend          [Tracepoint event] syscalls:sys_enter_alarm              [Tracepoint event] syscalls:sys_exit_alarm              [Tracepoint event] syscalls:sys_enter_getpid             [Tracepoint event] syscalls:sys_exit_getpid              [Tracepoint event] syscalls:sys_enter_getppid             [Tracepoint event] syscalls:sys_exit_getppid             [Tracepoint event] syscalls:sys_enter_getuid             [Tracepoint event] syscalls:sys_exit_getuid              [Tracepoint event] syscalls:sys_enter_geteuid             [Tracepoint event] syscalls:sys_exit_geteuid             [Tracepoint event] syscalls:sys_enter_getgid             [Tracepoint event] syscalls:sys_exit_getgid              [Tracepoint event] syscalls:sys_enter_getegid             [Tracepoint event] syscalls:sys_exit_getegid             [Tracepoint event] syscalls:sys_enter_gettid             [Tracepoint event] syscalls:sys_exit_gettid              [Tracepoint event] syscalls:sys_enter_sysinfo             [Tracepoint event] syscalls:sys_exit_sysinfo             [Tracepoint event] syscalls:sys_enter_ptrace             [Tracepoint event] syscalls:sys_exit_ptrace              [Tracepoint event] syscalls:sys_enter_capget             [Tracepoint event] syscalls:sys_exit_capget              [Tracepoint event] syscalls:sys_enter_capset             [Tracepoint event] syscalls:sys_exit_capset              [Tracepoint event] syscalls:sys_enter_sysctl             [Tracepoint event] syscalls:sys_exit_sysctl              [Tracepoint event] syscalls:sys_enter_time              [Tracepoint event] syscalls:sys_exit_time               [Tracepoint event] syscalls:sys_enter_stime              [Tracepoint event] syscalls:sys_exit_stime              [Tracepoint event] syscalls:sys_enter_gettimeofday          [Tracepoint event] syscalls:sys_exit_gettimeofday           [Tracepoint event] syscalls:sys_enter_settimeofday          [Tracepoint event] syscalls:sys_exit_settimeofday           [Tracepoint event] syscalls:sys_enter_adjtimex            [Tracepoint event] syscalls:sys_exit_adjtimex             [Tracepoint event] syscalls:sys_enter_getitimer            [Tracepoint event] syscalls:sys_exit_getitimer            [Tracepoint event] syscalls:sys_enter_setitimer            [Tracepoint event] syscalls:sys_exit_setitimer            [Tracepoint event] syscalls:sys_enter_exit              [Tracepoint event] syscalls:sys_exit_exit               [Tracepoint event] syscalls:sys_enter_exit_group           [Tracepoint event] syscalls:sys_exit_exit_group            [Tracepoint event] syscalls:sys_enter_waitid             [Tracepoint event] syscalls:sys_exit_waitid              [Tracepoint event] syscalls:sys_enter_wait4              [Tracepoint event] syscalls:sys_exit_wait4              [Tracepoint event] syscalls:sys_enter_waitpid             [Tracepoint event] syscalls:sys_exit_waitpid             [Tracepoint event] syscalls:sys_enter_syslog             [Tracepoint event] syscalls:sys_exit_syslog              [Tracepoint event] syscalls:sys_enter_personality           [Tracepoint event] syscalls:sys_exit_personality           [Tracepoint event] syscalls:sys_enter_set_tid_address         [Tracepoint event] syscalls:sys_exit_set_tid_address         [Tracepoint event] syscalls:sys_enter_unshare             [Tracepoint event] syscalls:sys_exit_unshare             [Tracepoint event] syscalls:sys_enter_nice              [Tracepoint event] syscalls:sys_exit_nice               [Tracepoint event] syscalls:sys_enter_sched_setscheduler       [Tracepoint event] syscalls:sys_exit_sched_setscheduler        [Tracepoint event] syscalls:sys_enter_sched_setparam         [Tracepoint event] syscalls:sys_exit_sched_setparam          [Tracepoint event] syscalls:sys_enter_sched_getscheduler       [Tracepoint event] syscalls:sys_exit_sched_getscheduler        [Tracepoint event] syscalls:sys_enter_sched_getparam         [Tracepoint event] syscalls:sys_exit_sched_getparam          [Tracepoint event] syscalls:sys_enter_sched_setaffinity        [Tracepoint event] syscalls:sys_exit_sched_setaffinity        [Tracepoint event] syscalls:sys_enter_sched_getaffinity        [Tracepoint event] syscalls:sys_exit_sched_getaffinity        [Tracepoint event] syscalls:sys_enter_sched_yield           [Tracepoint event] syscalls:sys_exit_sched_yield           [Tracepoint event] syscalls:sys_enter_sched_get_priority_max     [Tracepoint event] syscalls:sys_exit_sched_get_priority_max      [Tracepoint event] syscalls:sys_enter_sched_get_priority_min     [Tracepoint event] syscalls:sys_exit_sched_get_priority_min      [Tracepoint event] syscalls:sys_enter_sched_rr_get_interval      [Tracepoint event] syscalls:sys_exit_sched_rr_get_interval      [Tracepoint event] syscalls:sys_enter                 [Tracepoint event] syscalls:sys_exit                 [Tracepoint event] irq_vectors:local_timer_entry           [Tracepoint event] irq_vectors:local_timer_exit            [Tracepoint event] irq_vectors:reschedule_entry            [Tracepoint event] irq_vectors:reschedule_exit            [Tracepoint event] irq_vectors:spurious_apic_entry          [Tracepoint event] irq_vectors:spurious_apic_exit           [Tracepoint event] irq_vectors:error_apic_entry            [Tracepoint event] irq_vectors:error_apic_exit            [Tracepoint event] irq_vectors:x86_platform_ipi_entry         [Tracepoint event] irq_vectors:x86_platform_ipi_exit         [Tracepoint event] irq_vectors:irq_work_entry             [Tracepoint event] irq_vectors:irq_work_exit             [Tracepoint event] irq_vectors:call_function_entry          [Tracepoint event] irq_vectors:call_function_exit           [Tracepoint event] irq_vectors:call_function_single_entry       [Tracepoint event] irq_vectors:call_function_single_exit       [Tracepoint event] irq_vectors:threshold_apic_entry          [Tracepoint event] irq_vectors:threshold_apic_exit          [Tracepoint event] irq_vectors:thermal_apic_entry           [Tracepoint event] irq_vectors:thermal_apic_exit           [Tracepoint event]

tracepoint

分成了几大类:

ext4 文件系统的tracepoint events,如果是其它文件系统,比如XFS,也有对应的tracepoint event;

jbd2 文件日志的tracepoint events;

skb 内存的tracepoint events;

net,napi,sock,udp:网络的tracepoint events;

scsi, block, writeback 磁盘IO

kmem 内存

sched 调度

syscalls 系统调用

...

3. perf top -e xxx

上面的 perf list [hw|sw...] 可以知道所有的 tracepoint events, 接下来我们就可以使用 perf top -e tracepoint-event 来专门获得指定的tracepoint的信息:

1)perf top -e cpu-clock: 查看CPU的使用

2)perf top -e faults : 查看 page faults

3)perf top -e block:block_rq_issue : 查看系统IO的请求,比如可以在发现系统IO异常时,可以使用该命令进行调查,就能指定到底是什么原因导致的IO异常。 block_rq_issue 表示 block_request_issue 就是IO请求数。其实从这些可以看出,分析和调查Linux上的各种性能问题,需要我们对Linux内核有比较多的了解,不然恐怕是无从下手的。

4. 案例

参见:http://blog.chinaunix.net/uid-20785090-id-4282589.html