阿里内部员工,排查Java问题常用的工具单
极客挖掘机
共 7560字,需浏览 16分钟
·
2020-09-11 22:26
链接:yq.aliyun.com/articles/69520?utm_content=m_10360
Linux命令类
tail -300f shopbase.log #倒数300行并进入实时监听文件写入模式
grep
grep forest f.txt #文件查找
grep forest f.txt cpf.txt #多文件查找
grep 'log' /home/admin -r -n #目录下查找所有符合关键字的文件
cat f.txt | grep -i shopbase
grep 'shopbase' /home/admin -r -n --include *.{vm,java} #指定文件后缀
grep 'shopbase' /home/admin -r -n --exclude *.{vm,java} #反匹配
seq 10 | grep 5 -A 3 #上匹配
seq 10 | grep 5 -B 3 #下匹配
seq 10 | grep 5 -C 3 #上下匹配,平时用这个就妥了
cat f.txt | grep -c 'SHOPBASE'
awk
awk '{print $4,$6}' f.txt
awk '{print NR,$0}' f.txt cpf.txt
awk '{print FNR,$0}' f.txt cpf.txt
awk '{print FNR,FILENAME,$0}' f.txt cpf.txt
awk '{print FILENAME,"NR="NR,"FNR="FNR,"$"NF"="$NF}' f.txt cpf.txt
echo 1:2:3:4 | awk -F: '{print $1,$2,$3,$4}'
awk '/ldb/ {print}' f.txt #匹配ldb
awk '!/ldb/ {print}' f.txt #不匹配ldb
awk '/ldb/ && /LISTEN/ {print}' f.txt #匹配ldb和LISTEN
awk '$5 ~ /ldb/ {print}' f.txt #第五列匹配ldb
find
sudo -u admin find /home/admin /tmp /usr -name \*.log(多个目录去找)
find . -iname \*.txt(大小写都匹配)
find . -type d(当前目录下的所有子目录)
find /usr -type l(当前目录下所有的符号链接)
find /usr -type l -name "z*" -ls(符号链接的详细信息 eg:inode,目录)
find /home/admin -size +250000k(超过250000k的文件,当然+改成-就是小于了)
find /home/admin f -perm 777 -exec ls -l {} \; (按照权限查询文件)
find /home/admin -atime -1 1天内访问过的文件
find /home/admin -ctime -1 1天内状态改变过的文件
find /home/admin -mtime -1 1天内修改过的文件
find /home/admin -amin -1 1分钟内访问过的文件
find /home/admin -cmin -1 1分钟内状态改变过的文件
find /home/admin -mmin -1 1分钟内修改过的文件
pgm
pgm -A -f vm-shopbase 'cat /home/admin/shopbase/logs/shopbase.log.2017-01-17|grep 2069861630'
tsar
tsar ###可以查看最近一天的各项指标
![screenshot.png](https://filescdn.proginn.com/4087697c0aa7888b963c2340a4c1e8aa/eff648cc7713740d8107392dc64c0c03.webp)
tsar --live ###可以查看实时指标,默认五秒一刷
![screenshot.png](https://filescdn.proginn.com/5f28e807600450e44f92e2873a75a687/06439bfc841d3eebe79d7599f47cdad7.webp)
tsar -d 20161218 ###指定查看某天的数据,貌似最多只能看四个月的数据
![screenshot.png](https://filescdn.proginn.com/9c5e092a4ca814675de6653c5126d274/da1b9386a244fd7752efed71048d1297.webp)
tsar --mem
tsar --load
tsar --cpu
###当然这个也可以和-d参数配合来查询某天的单个指标的情况
![screenshot.png](https://filescdn.proginn.com/1112f067bdc6659e3c1fd73ff374e414/d869ac662478160c0998b0148599181b.webp)
![screenshot.png](https://filescdn.proginn.com/2d8114e0ecfa305629635fbbdf650bf0/22db438f35cd2c8edb298a898e317071.webp)
![screenshot.png](https://filescdn.proginn.com/6f1f87116be74b9b71beaab32dd3fa7f/1058b73efbb1cb2756afada8e99fb544.webp)
top
ps -ef | grep java
top -H -p pid
其他
netstat -nat|awk '{print $6}'|sort|uniq -c|sort -rn
#查看当前连接,注意close_wait偏高的情况,比如如下
![screenshot.png](https://filescdn.proginn.com/6be7ab10b805bb6ac7ba3b35cd5773e4/8b447b853a6174e386cf451d3d53256e.webp)
![screenshot.png](https://filescdn.proginn.com/5faffaacecc50113ce0ec5c4163f4b3f/367a94a544e8e988e07d8e7b318f43ee.webp)
排查利器
@OnMethod(clazz = "java.util.ArrayList", method="add", location = @Location(value = Kind.CALL, clazz = "/.*/", method = "/.*/"))
public static void m(@ProbeClassName String probeClass, @ProbeMethodName String probeMethod, @TargetInstance Object instance, @TargetMethodOrField String method) {
if(getInt(field("java.util.ArrayList", "size"), instance) > 479){
println("check who ArrayList.add method:" + probeClass + "#" + probeMethod + ", method:" + method + ", size:" + getInt(field("java.util.ArrayList", "size"), instance));
jstack();
println();
println("===========================");
println();
}
}
@OnMethod(clazz = "com.taobao.sellerhome.transfer.biz.impl.C2CApplyerServiceImpl", method="nav", location = @Location(value = Kind.RETURN))
public static void mt(long userId, int current, int relation, String check, String redirectUrl, @Return AnyType result) {
println("parameter# userId:" + userId + ", current:" + current + ", relation:" + relation + ", check:" + check + ", redirectUrl:" + redirectUrl + ", result:" + result);
}
经过观察,1.3.9的release输出不稳定,要多触发几次才能看到正确的结果
正则表达式匹配trace类时范围一定要控制,否则极有可能出现跑满CPU导致应用卡死的情况
由于是字节码注入的原理,想要应用恢复到正常情况,需要重启应用。
Greys
sc -df xxx
: 输出当前类的详情,包括源码位置和classloader结构trace class method
: 相当喜欢这个功能! 很早前可以早JProfiler看到这个功能。打印出当前方法调用的耗时情况,细分到每个方法。javOSize
classes
:通过修改了字节码,改变了类的内容,即时生效。所以可以做到快速的在某个地方打个日志看看输出,缺点是对代码的侵入性太大。但是如果自己知道自己在干嘛,的确是不错的玩意儿。JProfiler
官网请移步https://www.ej-technologies.com/products/jprofiler/overview.html
大杀器
详情请移步http://www.eclipse.org/mat/
sudo -u admin /opt/taobao/java/bin/jps -mlvV
![screenshot.png](https://filescdn.proginn.com/524740ae74ee4fe389ce17b5b8b1b9b5/ea97e500cf25e14a16b200d848817291.webp)
jstack
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstack 2815
![screenshot.png](https://filescdn.proginn.com/a38f4f769eec12056f04bb31b31ca3b9/612e706e8d61a7907253093b957cd286.webp)
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstack -m 2815
![screenshot.png](https://filescdn.proginn.com/019d380db355f74cdb87ed989b486bd9/57ec2cc158d0e85e9948dd01e9c7f183.webp)
jinfo
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jinfo -flags 2815
![screenshot.png](https://filescdn.proginn.com/eadf4c17520146314dcf2fdaf769b1c8/2723278f8d11ac20ab4807ea27a36c7f.webp)
jmap
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -heap 2815
![screenshot.png](https://filescdn.proginn.com/38ec3e7981b465c83c54788c44fd523c/5e5bdaed439847047b33aaccb2f395fd.webp)
![screenshot.png](https://filescdn.proginn.com/0a7c5b80da8a0c1bad2960b788671039/abcd59380cee612abcb8e0da52256071.webp)
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -dump:live,format=b,file=/tmp/heap2.bin 2815
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -dump:format=b,file=/tmp/heap3.bin 2815
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -histo 2815 | head -10
![screenshot.png](https://filescdn.proginn.com/cc5f99169b4d3de9a0b04c960615b02f/e43b1c34cc8dc111b331f41db1dd766b.webp)
jstat
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstat -gcutil 2815 1000
![screenshot.png](https://filescdn.proginn.com/597ed6bfae1cb16648cfde6afd3c5af6/007f3116c5e83432fbab24531b003ed0.webp)
jdb
jdb可以用来预发debug,假设你预发的java_home是/opt/taobao/java/,远程调试端口是8000.那么
sudo -u admin /opt/taobao/java/bin/jdb -attach 8000
.![screenshot.png](https://filescdn.proginn.com/27c4959db8f1c8ca459320bb85eeac9c/b9616f494cd50d0f347d0bb385bb21dd.webp)
具体参数可见oracle官方说明http://docs.oracle.com/javase/7/docs/technotes/tools/windows/jdb.html
CHLSDB
sudo -u admin /opt/taobao/java/bin/java -classpath /opt/taobao/java/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB
http://rednaxelafx.iteye.com/blog/1847971
![screenshot.png](https://filescdn.proginn.com/090e0db65751ffe12b2327020987d0d7/025fa7175470160f2d646371e379f69b.webp)
maven helper
VM options
-XX:+TraceClassLoading
结果形如[Loaded java.lang.invoke.MethodHandleImpl$Lazy from D:\programme\jdk\jdk8U74\jre\lib\rt.jar]
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/admin/logs/java.hprof
jar包冲突
mvn dependency:tree > ~/dependency.txt
mvn dependency:tree -Dverbose -Dincludes=groupId:artifactId
-XX:+TraceClassLoading
-verbose
greys:sc
tomcat-classloader-locate
curl http://localhost:8006/classloader/locate?class=org.apache.xerces.xs.XSObjec
其他
sudo dmesg|grep -i kill|less
[6710782.021013] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_scoe_adj=0
[6710782.070639] [] ? oom_kill_process+0x68/0x140
[6710782.257588] Task in /LXC011175068174 killed as a result of limit of /LXC011175068174
[6710784.698347] Memory cgroup out of memory: Kill process 215701 (java) score 854 or sacrifice child
[6710784.707978] Killed process 215701, UID 679, (java) total-vm:11017300kB, anon-rss:7152432kB, file-rss:1232kB
解释一下OOM killer(Out-Of-Memory killer),该机制会监控机器的内存资源消耗。当机器内存耗尽前,该机制会扫描所有的进程(按照一定规则计算,内存占用,时间等),挑选出得分最高的进程,然后杀死,从而保护机器。
log实际时间=格林威治1970-01-01+(当前时间秒数-系统启动至今的秒数+dmesg打印的log时间)秒数:
date -d "1970-01-01 UTC `echo "$(date +%s)-$(cat /proc/uptime|cut -f 1 -d' ')+12288812.926194"|bc ` seconds"
来源:yq.aliyun.com/articles/69520?utm_content=m_10360
版权申明:内容来源网络,版权归原创者所有。除非无法确认,我们都会标明作者及出处,如有侵权烦请告知,我们会立即删除并表示歉意。谢谢!
![](https://filescdn.proginn.com/126d8ff4694978c5999864dee5ca0884/ac1970fca464d4aea96154646c03ebd1.webp)
评论