Linux服务器文件删除空间未释放的问题

一、问题起源

在Linux系统中，通过rm删除文件将会从文件系统的目录结构上解除链接(unlink)，如果文件是被打开的（有一个进程正在使用），那么进程将仍然可以读取该文件磁盘空间也一直被占用

这样就会导致我们明明删除了文件，但是磁盘空间却未被释放

二、问题分析

1、首先获得一个已经被删除但是仍然被应用程序占用的文件列表

root@instance-zo0fkfng:~# lsof|grep deleted init     1       root  10w   REG       253,1    283   155958 /var/log/upstart/network-manager.log.1 (deleted)init     1       root  12w   REG       253,1    314   155934 /var/log/upstart/mountall.log.1 (deleted)init     1       root  13w   REG       253,1    508   131203 /var/log/upstart/modemmanager.log.1 (deleted)init     1       root  17w   REG       253,1    881   155967 /var/log/upstart/mysql.log.1 (deleted)init     1       root  24w   REG       253,1     63   155949 /var/log/upstart/dbus.log.1 (deleted)mysqld   1159      mysql  4u   REG       253,1     0    374 /tmp/ibkjeh3Z (deleted)mysqld   1159      mysql  5u   REG       253,1     0    3458 /tmp/ibdbHetZ (deleted)mysqld   1159      mysql  6u   REG       253,1     0   29466 /tmp/ibQSxcTY (deleted)mysqld   1159      mysql  7u   REG       253,1     0    282 /tmp/ibxSgZNX (deleted)mysqld   1159      mysql  11u   REG       253,1     0   29469 /tmp/ibXTzqnX (deleted)mysqld   1159 1185   mysql  4u   REG       253,1     0    374 /tmp/ibkjeh3Z (deleted)mysqld   1159 17965   mysql  4u   REG       253,1     0    374 /tmp/ibkjeh3Z (deleted)mysqld   1159 17965   mysql  5u   REG       253,1     0    3458 /tmp/ibdbHetZ (deleted)mysqld   1159 17965   mysql  6u   REG       253,1     0   29466 /tmp/ibQSxcTY (deleted)mysqld   1159 17965   mysql  7u   REG       253,1     0    282 /tmp/ibxSgZNX (deleted)mysqld   1159 17965   mysql  11u   REG       253,1     0   29469 /tmp/ibXTzqnX (deleted)php5-fpm  3724     www-data  3ur   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm  3853     www-data  3ur   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm  7741       root  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm  9900     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm  9901     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm  9930     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm  9961     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm  9965     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm  9966     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10126     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10127     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10128     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10174     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10196     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10248     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10316     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10375     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10396     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10400     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10401     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10438     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10479     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10511     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 10531     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 11150     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 11638     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 11670     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 12210     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 14974     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 17101     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 17161     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 17261     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)php5-fpm 17470     www-data  3u   REG       253,1     0   29468 /tmp/.ZendSem.gfHFuR (deleted)

从输出结果可以看到哪些文件还被使用，未被释放空间

或者

root@instance-zo0fkfng:~# find /proc/*/fd -ls | grep '(deleted)' 71253  0 lrwx------  1 root   root      64 Jun 10 22:09 /proc/1159/fd/4 -> /tmp/ibkjeh3Z\ (deleted) 71254  0 lrwx------  1 root   root      64 Jun 10 22:09 /proc/1159/fd/5 -> /tmp/ibdbHetZ\ (deleted) 71255  0 lrwx------  1 root   root      64 Jun 10 22:09 /proc/1159/fd/6 -> /tmp/ibQSxcTY\ (deleted) 71256  0 lrwx------  1 root   root      64 Jun 10 22:09 /proc/1159/fd/7 -> /tmp/ibxSgZNX\ (deleted) 71260  0 lrwx------  1 root   root      64 Jun 10 22:09 /proc/1159/fd/11 -> /tmp/ibXTzqnX\ (deleted)221991074  0 lrwx------  1 root   root      64 Aug 27 19:43 /proc/18132/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)221991101  0 lrwx------  1 www-data www-data    64 Aug 27 19:43 /proc/18135/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)221991122  0 lrwx------  1 www-data www-data    64 Aug 27 19:43 /proc/18136/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)221991147  0 lrwx------  1 www-data www-data    64 Aug 27 19:43 /proc/18137/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)221991168  0 lrwx------  1 www-data www-data    64 Aug 27 19:43 /proc/18138/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)221991189  0 lrwx------  1 www-data www-data    64 Aug 27 19:43 /proc/18139/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)221991210  0 lrwx------  1 www-data www-data    64 Aug 27 19:43 /proc/18140/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)221991231  0 lrwx------  1 www-data www-data    64 Aug 27 19:43 /proc/18141/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)221991252  0 lrwx------  1 www-data www-data    64 Aug 27 19:43 /proc/18142/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)221991273  0 lrwx------  1 www-data www-data    64 Aug 27 19:43 /proc/18143/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)221991294  0 lrwx------  1 www-data www-data    64 Aug 27 19:43 /proc/18144/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)221991315  0 lrwx------  1 www-data www-data    64 Aug 27 19:43 /proc/18147/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)222001888  0 lrwx------  1 www-data www-data    64 Aug 27 19:44 /proc/18157/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)222001909  0 lrwx------  1 www-data www-data    64 Aug 27 19:44 /proc/18161/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)222001938  0 lrwx------  1 www-data www-data    64 Aug 27 19:44 /proc/18170/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)222001959  0 lrwx------  1 www-data www-data    64 Aug 27 19:44 /proc/18178/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)find: `/proc/18182/fd/5': No such file or directory 55130  0 l-wx------  1 root   root      64 Jun 10 22:09 /proc/1/fd/10 -> /var/log/upstart/network-manager.log.1\ (deleted) 55132  0 l-wx------  1 root   root      64 Jun 10 22:09 /proc/1/fd/12 -> /var/log/upstart/mountall.log.1\ (deleted) 55133  0 l-wx------  1 root   root      64 Jun 10 22:09 /proc/1/fd/13 -> /var/log/upstart/modemmanager.log.1\ (deleted) 55137  0 l-wx------  1 root   root      64 Jun 10 22:09 /proc/1/fd/17 -> /var/log/upstart/mysql.log.1\ (deleted) 55142  0 l-wx------  1 root   root      64 Jun 10 22:09 /proc/1/fd/24 -> /var/log/upstart/dbus.log.1\ (deleted)find: `/proc/self/fd/5': No such file or directory

2、如何让进程释放文件，进而释放磁盘空间

kill掉相应的进程或者重启该进程，让os自动回收磁盘空间

当linux打开一个文件的时候,Linux内核会为每一个进程在/proc/ 『/proc/nnnn/fd/目录（nnnn为pid）』建立一个以其pid为名的目录用来保存进程的相关信息，而其子目录fd保存的是该进程打开的所有文件的fd（fd：file descriptor）

kill进程是通过截断proc文件系统中的文件可以强制要求系统回收分配给正在使用的的文件

root@instance-zo0fkfng:~# service php5-fpm restartroot@instance-zo0fkfng:~# lsof|grep deleted init     1       root  10w   REG       253,1    283   155958 /var/log/upstart/network-manager.log.1 (deleted)init     1       root  12w   REG       253,1    314   155934 /var/log/upstart/mountall.log.1 (deleted)init     1       root  13w   REG       253,1    508   131203 /var/log/upstart/modemmanager.log.1 (deleted)init     1       root  17w   REG       253,1    881   155967 /var/log/upstart/mysql.log.1 (deleted)init     1       root  24w   REG       253,1     63   155949 /var/log/upstart/dbus.log.1 (deleted)mysqld   1159      mysql  4u   REG       253,1     0    374 /tmp/ibkjeh3Z (deleted)mysqld   1159      mysql  5u   REG       253,1     0    3458 /tmp/ibdbHetZ (deleted)mysqld   1159      mysql  6u   REG       253,1     0   29466 /tmp/ibQSxcTY (deleted)mysqld   1159      mysql  7u   REG       253,1     0    282 /tmp/ibxSgZNX (deleted)mysqld   1159      mysql  11u   REG       253,1     0   29469 /tmp/ibXTzqnX (deleted)mysqld   1159 1185   mysql  4u   REG       253,1     0    374 /tmp/ibkjeh3Z (deleted)mysqld   1159 1185   mysql  5u   REG       253,1     0    3458 /tmp/ibdbHetZ (deleted)mysqld   1159 1185   mysql  6u   REG       253,1     0   29466 /tmp/ibQSxcTY (deleted)mysqld   1159 1185   mysql  7u   REG       253,1     0    282 /tmp/ibxSgZNX (deleted)mysqld   1159 1185   mysql  11u   REG       253,1     0   29469 /tmp/ibXTzqnX (deleted)mysqld   1159 1186   mysql  4u   REG       253,1     0    374 /tmp/ibkjeh3Z (deleted)mysqld   1159 1186   mysql  5u   REG       253,1     0    3458 /tmp/ibdbHetZ (deleted)mysqld   1159 18149   mysql  4u   REG       253,1     0    374 /tmp/ibkjeh3Z (deleted)mysqld   1159 18149   mysql  5u   REG       253,1     0    3458 /tmp/ibdbHetZ (deleted)mysqld   1159 18149   mysql  6u   REG       253,1     0   29466 /tmp/ibQSxcTY (deleted)mysqld   1159 18149   mysql  7u   REG       253,1     0    282 /tmp/ibxSgZNX (deleted)mysqld   1159 18149   mysql  11u   REG       253,1     0   29469 /tmp/ibXTzqnX (deleted)mysqld   1159 18150   mysql  4u   REG       253,1     0    374 /tmp/ibkjeh3Z (deleted)mysqld   1159 18150   mysql  5u   REG       253,1     0    3458 /tmp/ibdbHetZ (deleted)mysqld   1159 18150   mysql  6u   REG       253,1     0   29466 /tmp/ibQSxcTY (deleted)mysqld   1159 18150   mysql  7u   REG       253,1     0    282 /tmp/ibxSgZNX (deleted)mysqld   1159 18150   mysql  11u   REG       253,1     0   29469 /tmp/ibXTzqnX (deleted)php5-fpm 18132       root  3u   REG       253,1     0   28407 /tmp/.ZendSem.hIpOFr (deleted)php5-fpm 18135     www-data  3u   REG       253,1     0   28407 /tmp/.ZendSem.hIpOFr (deleted)php5-fpm 18136     www-data  3u   REG       253,1     0   28407 /tmp/.ZendSem.hIpOFr (deleted)php5-fpm 18137     www-data  3u   REG       253,1     0   28407 /tmp/.ZendSem.hIpOFr (deleted)php5-fpm 18138     www-data  3u   REG       253,1     0   28407 /tmp/.ZendSem.hIpOFr (deleted)php5-fpm 18139     www-data  3u   REG       253,1     0   28407 /tmp/.ZendSem.hIpOFr (deleted)php5-fpm 18140     www-data  3u   REG       253,1     0   28407 /tmp/.ZendSem.hIpOFr (deleted)php5-fpm 18141     www-data  3u   REG       253,1     0   28407 /tmp/.ZendSem.hIpOFr (deleted)php5-fpm 18142     www-data  3u   REG       253,1     0   28407 /tmp/.ZendSem.hIpOFr (deleted)php5-fpm 18143     www-data  3u   REG       253,1     0   28407 /tmp/.ZendSem.hIpOFr (deleted)php5-fpm 18144     www-data  3u   REG       253,1     0   28407 /tmp/.ZendSem.hIpOFr (deleted)php5-fpm 18147     www-data  3u   REG       253,1     0   28407 /tmp/.ZendSem.hIpOFr (deleted)

可以看到php5-fpm变少了

还有一种方式，清除文件

: > /path/to/the/file.log: > "/proc/$pid/fd/$fd"

三、文件恢复

在说明问题之前，先介绍下一些文件的基本概念：

文件实际上是一个指向inode的链接, inode链接包含了文件的所有属性, 比如权限和所有者, 数据块地址(文件存储在磁盘的这些数据块中). 当你删除(rm)一个文件, 实际删除了指向inode的链接, 并没有删除inode的内容. 进程可能还在使用. 只有当inode的所有链接完全移去, 然后这些数据块将可以写入新的数据.
proc文件系统可以协助我们恢复数据. 每一个系统上的进程在/proc都有一个目录和自己的名字，里面包含了一个fd(文件描述符)子目录(进程需要打开文件的所有链接). 如果从文件系统中删除一个文件, 此处还有一个inode的引用：/proc/进程号/fd/文件描述符

你需要知道打开文件的进程号(pid)和文件描述符(fd). 这些都可以通过lsof工具方便获得, lsof的意思是”list open files, 列出(进程)打开的文件”. 然后你将可以从/proc拷贝出需要恢复的数据.

1.创建一个测试文件并且备份下，方面后续验证

touch testfilecp testfile testfile.backup.2014

2.查看文件的相关信息

stat testfileFile: 'testfile'Size: 343545 Blocks: 241 IO Block: 4096 regular fileDevice: fd00h/64768d Inode: 361579 Links: 1Access: (0664/-rw-rw-r–) Uid: ( 505/ zhaoke) Gid: ( 505/ zhaoke)Access: 2014-11-09 15:00:38.000000000 +0800Modify: 2014-11-09 15:00:34.000000000 +0800Change: 2014-04-09 15:00:34.000000000 +0800

3.删除文件

rm testfile

4.查看文件

ls -l testfilels: testfile: No such file or directorystat testfilestat: cannot stat 'testfile': No such file or directory

testfile文件删除了，但不要终止仍在使用文件的进程，因为一旦终止, 文件将很难恢复.

现在我们开始找回数据之旅，先使用lsof命令查看下

lsof | grep testfiletail 5317 root 4r REG 253,0 343545 361579 /root/testfile (deleted)

第一个纵行是进程的名称(命令名), 第二纵行是进程号(PID), 第四纵行是文件描述符
现在你知道5317进程仍有打开文件, 文件描述符是4. 那我们开始从/proc里面拷贝出数据.
你可能会考虑使用cp -a, 但实际上没有作用, 你将拷贝的是一个指向被删除文件的符号链接:

ls -l /proc/5317/fd/4lr-x—— 1 root root 64 09 15:00 /proc/5317/fd/4 -> /root/testfile (deleted)

使用cp -a命令测试恢复

 cp -a /proc/5317/fd/4 testfile.backup

使用ls命令来查看

ls -l testfile.backuplrwxrwxrwx 1 root root 29 09 15:02 testfile.backup -> /roor/testfile (deleted)

通过上面的命令我们发现，使用cp -a命令，其恢复的是一个指向被删除文件的符号链接

使用file命令分别查看文件和文件描述符

1.查看文件

file testfile.backuptestfile.backup: broken symbolic link to '/root/testfile (deleted)'

2.查看文件描述符

file /proc/5317/fd/4/proc/5317/fd/4: broken symbolic link to '/root/myfile (deleted)'

根据上面的file结果，可以使用cp拷贝出文件描述符数据到一个文件中，如下:

cp /proc/5317/fd/4 testfile.new

使用上面的命令恢复后，我们需要最终确认一下文件是否恢复，以及文件内容是否正确:

 ls -l testfile.new

然后把新旧的两个文件对比

diff testfile.new myfile.backup

4、lsof的一些其他用法

1、查看端口号

root@localhost:~# lsof -i :80COMMAND   PID   USER  FD  TYPE  DEVICE SIZE/OFF NODE NAMEAliYunDun 1996   root  11u IPv4 117919702   0t0 TCP 124.46.185.34:42372->140.205.140.205:http (ESTABLISHED)aegis_qua 2114   root  11u IPv4   14868   0t0 TCP 124.46.185.34:43077->42.156.166.25:http (CLOSE_WAIT)nginx   13706 www-data  5u IPv4 140102705   0t0 TCP 124.46.185.34:http->140.205.127.64:41851 (ESTABLISHED)nginx   13706 www-data  15u IPv4  6150826   0t0 TCP *:http (LISTEN)nginx   13706 www-data  35u IPv4 140096235   0t0 TCP 124.46.185.34:http->198.11.137.21:47450 (ESTABLISHED)nginx   13706 www-data  36u IPv4 140095680   0t0 TCP 124.46.185.34:http->140.205.127.52:39887 (ESTABLISHED)nginx   13706 www-data  39u IPv4 140102712   0t0 TCP 124.46.185.34:http->198.11.137.5:37626 (ESTABLISHED)nginx   13706 www-data  40u IPv4 140102699   0t0 TCP 124.46.185.34:http->140.205.127.11:36374 (ESTABLISHED)nginx   13706 www-data  41u IPv4 140102759   0t0 TCP 124.46.185.34:http->101.200.101.217:47550 (ESTABLISHED)nginx   13706 www-data  42u IPv4 140102722   0t0 TCP 124.46.185.34:http->101.200.101.201:38595 (ESTABLISHED)nginx   13706 www-data  43u IPv4 140102840   0t0 TCP 124.46.185.34:http->140.205.127.58:45859 (ESTABLISHED)nginx   13706 www-data  44u IPv4 140095709   0t0 TCP 124.46.185.34:http->101.200.101.216:44828 (ESTABLISHED)nginx   13706 www-data  46u IPv4 140070070   0t0 TCP 124.46.185.34:http->140.205.127.18:24113 (ESTABLISHED)nginx   13706 www-data  47u IPv4 140103294   0t0 TCP 124.46.185.34:http->198.11.137.61:43262 (ESTABLISHED)nginx   13706 www-data  48u IPv4 140102761   0t0 TCP 124.46.185.34:http->101.200.101.220:31912 (ESTABLISHED)nginx   13706 www-data  50u IPv4 140103297   0t0 TCP 124.46.185.34:http->101.200.101.203:42752 (ESTABLISHED)nginx   13706 www-data  52u IPv4 140095824   0t0 TCP 124.46.185.34:http->198.11.137.14:62968 (ESTABLISHED)nginx   13706 www-data  53u IPv4 140102730   0t0 TCP 124.46.185.34:http->101.200.101.205:27518 (ESTABLISHED)nginx   13706 www-data  54u IPv4 140102822   0t0 TCP 124.46.185.34:http->140.205.127.51:58945 (ESTABLISHED)nginx   13706 www-data  55u IPv4 140103313   0t0 TCP 124.46.185.34:http->140.205.127.15:20785 (ESTABLISHED)nginx   13706 www-data  56u IPv4 140102812   0t0 TCP 124.46.185.34:http->140.205.31.212:1963 (ESTABLISHED)nginx   13706 www-data  57u IPv4 140102929   0t0 TCP 124.46.185.34:http->101.200.101.213:42452 (ESTABLISHED)nginx   13706 www-data  58u IPv4 140095749   0t0 TCP 124.46.185.34:http->101.200.101.208:50444 (ESTABLISHED)nginx   13706 www-data  59u IPv4 140102769   0t0 TCP 124.46.185.34:http->140.205.127.60:59679 (ESTABLISHED)nginx   13706 www-data  60u IPv4 140102814   0t0 TCP 124.46.185.34:http->140.205.127.2:25156 (ESTABLISHED)nginx   13706 www-data  61u IPv4 140102771   0t0 TCP 124.46.185.34:http->198.11.137.53:16935 (ESTABLISHED)nginx   13706 www-data  62u IPv4 140102845   0t0 TCP 124.46.185.34:http->198.11.137.2:50535 (ESTABLISHED)nginx   13706 www-data  63u IPv4 140102773   0t0 TCP 124.46.185.34:http->198.11.137.52:8521 (ESTABLISHED)nginx   13706 www-data  64u IPv4 140102856   0t0 TCP 124.46.185.34:http->140.205.127.23:55360 (ESTABLISHED)nginx   13706 www-data  66u IPv4 140075271   0t0 TCP 124.46.185.34:http->140.205.127.48:64726 (ESTABLISHED)nginx   13706 www-data  67u IPv4 140102937   0t0 TCP 124.46.185.34:http->101.200.101.204:10400 (ESTABLISHED)nginx   13706 www-data  68u IPv4 140102932   0t0 TCP 124.46.185.34:http->101.200.101.213:42455 (ESTABLISHED)nginx   13706 www-data  69u IPv4 140102941   0t0 TCP 124.46.185.34:http->140.205.127.24:31585 (ESTABLISHED)nginx   13706 www-data  70u IPv4 140102934   0t0 TCP 124.46.185.34:http->198.11.137.50:11837 (ESTABLISHED)nginx   13706 www-data  71u IPv4 140103309   0t0 TCP 124.46.185.34:http->140.205.127.15:20777 (ESTABLISHED)nginx   13706 www-data  92u IPv4 140070742   0t0 TCP 124.46.185.34:http->140.205.127.31:65012 (ESTABLISHED)nginx   13707 www-data  5u IPv4 140092233   0t0 TCP 124.46.185.34:http->140.205.127.60:57698 (ESTABLISHED)nginx   13707 www-data  15u IPv4  6150826   0t0 TCP *:http (LISTEN)nginx   13707 www-data  17u IPv4 140092036   0t0 TCP 124.46.185.34:http->140.205.127.51:54339 (ESTABLISHED)nginx   13707 www-data  30u IPv4 140092031   0t0 TCP 124.46.185.34:http->198.11.137.59:64831 (ESTABLISHED)nginx   13707 www-data  35u IPv4 140092214   0t0 TCP 124.46.185.34:http->140.205.31.213:40510 (ESTABLISHED)nginx   13707 www-data  39u IPv4 140092202   0t0 TCP 124.46.185.34:http->140.205.127.9:19592 (ESTABLISHED)nginx   13707 www-data  44u IPv4 140092205   0t0 TCP 124.46.185.34:http->140.205.127.7:24059 (ESTABLISHED)nginx   13707 www-data  46u IPv4 140103741   0t0 TCP 124.46.185.34:http->198.11.137.22:61619 (ESTABLISHED)nginx   13707 www-data  51u IPv4 140072557   0t0 TCP 124.46.185.34:http->140.205.127.50:51697 (ESTABLISHED)nginx   13707 www-data  61u IPv4 140056450   0t0 TCP 124.46.185.34:http->140.205.127.37:27131 (ESTABLISHED)nginx   13707 www-data  69u IPv4 140072632   0t0 TCP 124.46.185.34:http->140.205.127.13:20352 (ESTABLISHED)nginx   13708 www-data  15u IPv4  6150826   0t0 TCP *:http (LISTEN)

2、查看进程

root@localhost:~# lsof -p 13721 COMMAND  PID   USER  FD  TYPE       DEVICE SIZE/OFF   NODE NAMEnginx  13721 www-data cwd  DIR       253,1   4096    2 /nginx  13721 www-data rtd  DIR       253,1   4096    2 /nginx  13721 www-data txt  REG       253,1  843688 1195996 /usr/sbin/nginxnginx  13721 www-data mem  REG       253,1   48024  269175 /lib/x86_64-linux-gnu/libnss_files-2.15.songinx  13721 www-data mem  REG       253,1   47680  262260 /lib/x86_64-linux-gnu/libnss_nis-2.15.songinx  13721 www-data mem  REG       253,1   97248  269169 /lib/x86_64-linux-gnu/libnsl-2.15.songinx  13721 www-data mem  REG       253,1   35680  269174 /lib/x86_64-linux-gnu/libnss_compat-2.15.songinx  13721 www-data mem  REG       253,1  1807032  269167 /lib/x86_64-linux-gnu/libc-2.15.songinx  13721 www-data mem  REG       253,1  215136 1186378 /usr/lib/libGeoIP.so.1.4.8nginx  13721 www-data mem  REG       253,1  270400 1181963 /usr/lib/x86_64-linux-gnu/libgd.so.2.0.0nginx  13721 www-data mem  REG       253,1   84928 1195989 /usr/lib/x86_64-linux-gnu/libexslt.so.0.8.15nginx  13721 www-data mem  REG       253,1   43288  269172 /lib/x86_64-linux-gnu/libcrypt-2.15.songinx  13721 www-data mem  REG       253,1  135366  269170 /lib/x86_64-linux-gnu/libpthread-2.15.songinx  13721 www-data mem  REG       253,1  149280  269173 /lib/x86_64-linux-gnu/ld-2.15.songinx  13721 www-data DEL  REG        0,4      6150828 /dev/zeronginx  13721 www-data  0u  CHR        1,3    0t0   1029 /dev/nullnginx  13721 www-data  1u  CHR        1,3    0t0   1029 /dev/nullnginx  13721 www-data  4w  REG       253,1   85704  918896 /var/log/nginx/access.lognginx  13721 www-data  13u unix 0xffff8807ff4ef500    0t0 59526717 socketnginx  13721 www-data  14w  REG       253,1  438012  918916 /var/log/nginx/admin.access.lognginx  13721 www-data  15u IPv4      6150826    0t0   TCP *:http (LISTEN)nginx  13721 www-data  16u IPv4      6150827    0t0   TCP *:http-alt (LISTEN)nginx  13721 www-data  19u unix 0xffff8807ff4ee900    0t0 59526719 socketnginx  13721 www-data  20u IPv4     140132456    0t0   TCP 124.46.185.34:http->140.205.127.60:16099 (ESTABLISHED)nginx  13721 www-data  21u unix 0xffff8807ff4ebf00    0t0 59526721 socketnginx  13721 www-data  22u IPv4     140132818    0t0   TCP 124.46.185.34:http->140.205.127.17:33358 (ESTABLISHED)nginx  13721 www-data  23u unix 0xffff8807ff4ee600    0t0 59526723 socketnginx  13721 www-data  24u IPv4     140132826    0t0   TCP 124.46.185.34:http->198.11.137.59:25415 (ESTABLISHED)nginx  13721 www-data  25u unix 0xffff8807ff4e9200    0t0 59526725 socketnginx  13721 www-data  26u IPv4     140106470    0t0   TCP 124.46.185.34:http->101.200.101.204:37659 (ESTABLISHED)nginx  13721 www-data  27u IPv4     140106678    0t0   TCP 124.46.185.34:http->140.205.127.34:53569 (ESTABLISHED)nginx  13721 www-data  30u IPv4     140132831    0t0   TCP 124.46.185.34:http->140.205.127.44:8165 (ESTABLISHED)nginx  13721 www-data  31u IPv4     140132828    0t0   TCP 124.46.185.34:http->47.88.135.31:64913 (ESTABLISHED)nginx  13721 www-data  33u IPv4     140132871    0t0   TCP 124.46.185.34:http->140.205.31.220:57845 (ESTABLISHED)nginx  13721 www-data  34u IPv4     140132842    0t0   TCP 124.46.185.34:http->195.27.31.56:55498 (ESTABLISHED)nginx  13721 www-data  35u IPv4     140132679    0t0   TCP 124.46.185.34:http->140.205.127.26:62043 (ESTABLISHED)nginx  13721 www-data  36u IPv4     140132838    0t0   TCP 124.46.185.34:http->140.205.127.26:64949 (ESTABLISHED)nginx  13721 www-data  37u unix 0xffff8807ff4ef200    0t0 59526727 socketnginx  13721 www-data  38u IPv4     140132862    0t0   TCP 124.46.185.34:http->140.205.127.25:23082 (ESTABLISHED)nginx  13721 www-data  39u unix 0xffff8807ff4ea700    0t0 59526729 socketnginx  13721 www-data  41u unix 0xffff8807ff4ec200    0t0 59526731 socketnginx  13721 www-data  42u IPv4     140132845    0t0   TCP 124.46.185.34:http->101.200.101.216:55441 (ESTABLISHED)nginx  13721 www-data  43u unix 0xffff8807ff4eb900    0t0 59526733 socketnginx  13721 www-data  44u IPv4     140106547    0t0   TCP 124.46.185.34:http->140.205.31.211:60180 (ESTABLISHED)nginx  13721 www-data  45u unix 0xffff8807ff4e9b00    0t0 59526735 socketnginx  13721 www-data  46u IPv4     140132879    0t0   TCP 124.46.185.34:http->140.205.31.203:42821 (ESTABLISHED)nginx  13721 www-data  47u unix 0xffff8807ff4e8300    0t0 59526737 socketnginx  13721 www-data  48u IPv4     140132847    0t0   TCP 124.46.185.34:http->140.205.127.45:53830 (ESTABLISHED)nginx  13721 www-data  49u unix 0xffff8807ff4ee000    0t0 59526739 socketnginx  13721 www-data  50u IPv4     140077698    0t0   TCP 124.46.185.34:http->140.205.127.43:43558 (ESTABLISHED)nginx  13721 www-data  51u unix 0xffff8807ff4ed400    0t0 59526741 socketnginx  13721 www-data  52u IPv4     140132857    0t0   TCP 124.46.185.34:http->198.11.137.19:22613 (ESTABLISHED)nginx  13721 www-data  53u unix 0xffff8807ff4ece00    0t0 59526743 socketnginx  13721 www-data  55u unix 0xffff8807ff4e8000    0t0 59526745 socketnginx  13721 www-data  56u IPv4     140132853    0t0   TCP 124.46.185.34:http->140.205.31.212:23679 (ESTABLISHED)nginx  13721 www-data  57u IPv4     140132859    0t0   TCP 124.46.185.34:http->140.205.127.61:7242 (ESTABLISHED)nginx  13721 www-data  58u unix 0xffff8807ff4e9800    0t0 59526748 socketnginx  13721 www-data  59u IPv4     140132864    0t0   TCP 124.46.185.34:http->101.200.101.201:35787 (ESTABLISHED)nginx  13721 www-data  70u IPv4     140093317    0t0   TCP 124.46.185.34:http->140.205.127.34:37276 (ESTABLISHED)nginx  13721 www-data  80u 0000        0,9     0   6874 anon_inodenginx  13721 www-data  87u IPv4     140105862    0t0   TCP 124.46.185.34:http->140.205.127.11:40316 (ESTABLISHED)nginx  13721 www-data  88u IPv4     140093346    0t0   TCP 124.46.185.34:http->140.205.127.36:29659 (ESTABLISHED)

3、lsof -c init

root@iZ25n2yx37sZ:~# lsof -c initCOMMAND PID USER  FD  TYPE       DEVICE SIZE/OFF  NODE NAMEinit   1 root cwd  DIR       253,1   4096   2 /init   1 root rtd  DIR       253,1   4096   2 /init   1 root txt  REG       253,1  167192 393221 /sbin/initinit   1 root mem  REG       253,1  48024 269175 /lib/x86_64-linux-gnu/libnss_files-2.15.soinit   1 root mem  REG       253,1  47680 262260 /lib/x86_64-linux-gnu/libnss_nis-2.15.soinit   1 root mem  REG       253,1  97248 269169 /lib/x86_64-linux-gnu/libnsl-2.15.soinit   1 root mem  REG       253,1  35680 269174 /lib/x86_64-linux-gnu/libnss_compat-2.15.soinit   1 root mem  REG       253,1 1807032 269167 /lib/x86_64-linux-gnu/libc-2.15.soinit   1 root mem  REG       253,1  31752 262372 /lib/x86_64-linux-gnu/librt-2.15.soinit   1 root mem  REG       253,1  135366 269170 /lib/x86_64-linux-gnu/libpthread-2.15.soinit   1 root mem  REG       253,1  276392 262185 /lib/x86_64-linux-gnu/libdbus-1.so.3.5.8init   1 root mem  REG       253,1  38888 262171 /lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0init   1 root mem  REG       253,1  96240 262174 /lib/x86_64-linux-gnu/libnih.so.1.0.0init   1 root mem  REG       253,1  149280 269173 /lib/x86_64-linux-gnu/ld-2.15.soinit   1 root  0u  CHR        1,3   0t0  1029 /dev/nullinit   1 root  1u  CHR        1,3   0t0  1029 /dev/nullinit   1 root  2u  CHR        1,3   0t0  1029 /dev/nullinit   1 root  3r FIFO        0,8   0t0  7012 pipeinit   1 root  4w FIFO        0,8   0t0  7012 pipeinit   1 root  5r 0000        0,9    0  6874 anon_inodeinit   1 root  6r 0000        0,9    0  6874 anon_inodeinit   1 root  7u unix 0xffff88080b948000   0t0  7013 socketinit   1 root  8u unix 0xffff88080b918600   0t0 12351 socketinit   1 root  9u unix 0xffff880807670c00   0t0  1238 socketinit   1 root  10u unix 0xffff8808076a9500   0t0  7100 socketinit   1 root  11w  REG       253,1   633 919710 /var/log/upstart/mysql.log.1 (deleted)init   1 root  23u  CHR        5,2   0t0  1111 /dev/ptmx

　　。。。。