English 中文(简体)
Oliver OOM Killer and Java Process
原标题:Linux OOM Killer and Java Process

我经常在我们的生产环境中面临由欧林特·索罗斯杀害的Tomcat进程的问题。

它对var/var/log/mes的检查表明, j没有被tain, j被夺走。

-Xms20480m -Xmx20480m on a 32 GB Box.

页: 1

Is the OOM causing this crash ? or the crash happened because of OOM ? How can I debug this issue ?

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f4c3230aad7, pid=16248, tid=139964439296320
#
# JRE version: Java(TM) SE Runtime Environment (7.0_45-b18) (build 1.7.0_45-b18)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x674ad7]  JVM_Clone+0x97
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x0000000001313800):  JavaThread "http-bio-14080-exec-17" daemon [_thread_in_native, id=18943, stack(0x00007f4c029f7000,0x00007f4c02af8000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (), si_addr=0x0000000000000000

Registers:
RAX=0x0000000000010000, RBX=0x80000001000004db, RCX=0x0000000302bd6fb0, RDX=0x00007f4c32acda50
RSP=0x00007f4c02af2c88, RBP=0x00007f4c02af2d70, RSI=0x00007f4c02af2d80, RDI=0x00000000013139e8
R8 =0x0000000000000001, R9 =0x00000002f8191228, R10=0x00007f4c2e231658, R11=0x00007f4c2e231638
R12=0x0000000302bd6fb0, R13=0x00000007e0002420, R14=0x0000000000000000, R15=0x0000000001313800
RIP=0x00007f4c3230aad7, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000000
  TRAPNO=0x000000000000000d

Top of Stack: (sp=0x00007f4c02af2c88)
0x00007f4c02af2c88:   00007f4c3230aa63 00000000129196d1
0x00007f4c02af2c98:   00000002f817b620 00000000013139e8
0x00007f4c02af2ca8:   00007f4c02af2d18 00000002f8191080
0x00007f4c02af2cb8:   0000000000000009 0000000001313800
0x00007f4c02af2cc8:   0000000000000000 0000000001313800
0x00007f4c02af2cd8:   00000000138f5509 0000000000000000
0x00007f4c02af2ce8:   00007f4c2e382ae8 0000000000000000
0x00007f4c02af2cf8:   00007f4c02af2cf8 00000002f817b620
0x00007f4c02af2d08:   00000002f8190fd8 00000007e01ab278
0x00007f4c02af2d18:   00000002f8190ec0 00000007e0002420
0x00007f4c02af2d28:   0000000000000000 00000007e0002420
0x00007f4c02af2d38:   0000000000000000 0000000000000000
0x00007f4c02af2d48:   00000007e0002420 0000000000000000
0x00007f4c02af2d58:   00007f4c02af2de0 0000000000000000
0x00007f4c02af2d68:   0000000001313800 00007f4c02af2dc0
0x00007f4c02af2d78:   00007f4c2e2316c6 0000000302bd6fb0
0x00007f4c02af2d88:   0000000000000000 00007f4c02af2e08
0x00007f4c02af2d98:   00007f4c2e1bc8e1 00007f4c02af2db8
0x00007f4c02af2da8:   00007f4c2e1bc8e1 00000002f8190fd8
0x00007f4c02af2db8:   00000002f817b620 00007f4c02af2e28
0x00007f4c02af2dc8:   00007f4c2e1bc233 00000007e1ec4e9f
0x00007f4c02af2dd8:   00007f4c2e1bc233 0000000302bd6fb0
0x00007f4c02af2de8:   00007f4c02af2de8 00000007e1b7a5db
0x00007f4c02af2df8:   00007f4c02af2e30 00000007e37575a8
0x00007f4c02af2e08:   0000000000000000 00000007e1b7a5e8
0x00007f4c02af2e18:   00007f4c02af2de0 00007f4c02af2e40
0x00007f4c02af2e28:   00007f4c02af2ed0 00007f4c2edaaf34
0x00007f4c02af2e38:   00007f4c2edaaf34 00007f4c0000000a
0x00007f4c02af2e48:   00000007e416ec20 0000000000000000
0x00007f4c02af2e58:   00000007e416e2b8 00007f4c02af2ed0
0x00007f4c02af2e68:   00007f4c2e1bc233 00007f4c02af2ed0
0x00007f4c02af2e78:   00007f4c2e1bc233 000000000000000a 

Instructions: (pc=0x00007f4c3230aad7)
0x00007f4c3230aab7:   85 60 02 00 00 06 00 00 00 4c 89 6d b0 4c 8b 23
0x00007f4c3230aac7:   4d 85 e4 0f 84 68 02 00 00 49 8b 9d 20 01 00 00
0x00007f4c3230aad7:   48 83 7b 10 f7 0f 87 6e 02 00 00 48 8b 43 10 48
0x00007f4c3230aae7:   8d 50 08 48 3b 53 18 0f 87 9c 02 00 00 48 89 53 

Register to memory mapping:

RAX=0x0000000000010000 is an unknown value
RBX=0x80000001000004db is an unknown value
RCX=0x0000000302bd6fb0 is an oop
[Lcom.mycompany.MyClass$Type; 
 - klass:  com/mycompany/MyClass$Type []
 - length: 16
RDX=0x00007f4c32acda50: <offset 0xe37a50> in /usr/java/jre64-1.7.0_45/jre/lib/amd64/server/libjvm.so at 0x00007f4c31c96000
RSP=0x00007f4c02af2c88 is pointing into the stack for thread: 0x0000000001313800
RBP=0x00007f4c02af2d70 is pointing into the stack for thread: 0x0000000001313800
RSI=0x00007f4c02af2d80 is pointing into the stack for thread: 0x0000000001313800
RDI=0x00000000013139e8 is an unknown value
R8 =0x0000000000000001 is an unknown value
R9 =
[error occurred during error reporting (printing register info), id 0xb]

Stack: [0x00007f4c029f7000,0x00007f4c02af8000],  sp=0x00007f4c02af2c88,  free space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x674ad7]  JVM_Clone+0x97
J  java.lang.Object.clone()Ljava/lang/Object;
j  com.mycompany.MyClass$Type.values()[Lcom/mycompany/MyClass$Type;+3

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J  java.lang.Object.clone()Ljava/lang/Object;
j  com.mycompany.MyClass$Type.values()[Lcom/mycompany/MyClass$Type;+3

<>/>产出-

myhostname kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
myhostname kernel: java cpuset=/ mems_allowed=0
myhostname kernel: Pid: 32307, comm: java Not tainted 2.6.39-400.209.1.el5uek #1
myhostname kernel: Call Trace:
myhostname kernel:  [<ffffffff811136b4>] dump_header+0x94/0xe0
myhostname kernel:  [<ffffffff811137fd>] oom_kill_process+0x6d/0x160
myhostname kernel:  [<ffffffff811139ec>] out_of_memory+0xfc/0x210
myhostname kernel:  [<ffffffff811187ec>] __alloc_pages_slowpath+0x64c/0x660
myhostname kernel:  [<ffffffff811189b4>] __alloc_pages_nodemask+0x1b4/0x200
myhostname kernel:  [<ffffffff8111b140>] ? __do_page_cache_readahead+0xe0/0x170
myhostname kernel:  [<ffffffff81150893>] alloc_pages_current+0xb3/0x120
myhostname kernel:  [<ffffffff811100da>] __page_cache_alloc+0x9a/0xb0
myhostname kernel:  [<ffffffff8111097f>] page_cache_read+0x4f/0xb0
myhostname kernel:  [<ffffffff81111a54>] filemap_fault+0x174/0x270
myhostname kernel:  [<ffffffff81137a2c>] __do_fault+0x5c/0x550
myhostname kernel:  [<ffffffff81137fc6>] do_linear_fault+0x36/0x40
myhostname kernel:  [<ffffffff81510b6e>] ? call_function_interrupt+0xe/0x20
myhostname kernel:  [<ffffffff81138044>] handle_pte_fault+0x74/0x190
myhostname kernel:  [<ffffffff815106ae>] ? apic_timer_interrupt+0xe/0x20
myhostname kernel:  [<ffffffff8113828f>] handle_mm_fault+0x12f/0x1b0
myhostname kernel:  [<ffffffff8150b1cd>] do_page_fault+0x17d/0x4b0
myhostname kernel:  [<ffffffff8117b821>] ? user_path_at+0x11/0x20
myhostname kernel:  [<ffffffff81170516>] ? vfs_fstatat+0x56/0x90
myhostname kernel:  [<ffffffff8117067b>] ? vfs_stat+0x1b/0x20
myhostname kernel:  [<ffffffff81507cd5>] page_fault+0x25/0x30
myhostname kernel: Mem-Info:
myhostname kernel: Node 0 DMA per-cpu:
myhostname kernel: CPU    0: hi:    0, btch:   1 usd:   0
myhostname kernel: CPU    1: hi:    0, btch:   1 usd:   0
myhostname kernel: CPU    2: hi:    0, btch:   1 usd:   0
myhostname kernel: CPU    3: hi:    0, btch:   1 usd:   0
myhostname kernel: CPU    4: hi:    0, btch:   1 usd:   0
myhostname kernel: CPU    5: hi:    0, btch:   1 usd:   0
myhostname kernel: Node 0 DMA32 per-cpu:
myhostname kernel: CPU    0: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    1: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    2: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    3: hi:  186, btch:  31 usd:  11
myhostname kernel: CPU    4: hi:  186, btch:  31 usd:  30
myhostname kernel: CPU    5: hi:  186, btch:  31 usd:   0
myhostname kernel: Node 0 Normal per-cpu:
myhostname kernel: CPU    0: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    1: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    2: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    3: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    4: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    5: hi:  186, btch:  31 usd:   0
myhostname kernel: active_anon:4372468 inactive_anon:275213 isolated_anon:0
myhostname kernel:  active_file:17 inactive_file:21 isolated_file:0
myhostname kernel:  unevictable:6002 dirty:24 writeback:4 unstable:0
myhostname kernel:  free:38369 slab_reclaimable:3708 slab_unreclaimable:9265
myhostname kernel:  mapped:1253 shmem:67 pagetables:10880 bounce:0
myhostname kernel: Node 0 DMA free:15880kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
myhostname kernel: lowmem_reserve[]: 0 3000 32290 32290
myhostname kernel: Node 0 DMA32 free:119288kB min:2136kB low:2668kB high:3204kB active_anon:30224kB inactive_anon:9152kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:16kB writeback:4kB mapped:36kB shmem:0kB slab_reclaimable:40kB slab_unreclaimable:232kB kernel_stack:24kB pagetables:48kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:67 all_unreclaimable? yes
myhostname kernel: lowmem_reserve[]: 0 0 29290 29290
myhostname kernel: Node 0 Normal free:17564kB min:20856kB low:26068kB high:31284kB active_anon:17459648kB inactive_anon:1091700kB active_file:120kB inactive_file:88kB unevictable:24008kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:24008kB dirty:80kB writeback:12kB mapped:4976kB shmem:268kB slab_reclaimable:14792kB slab_unreclaimable:36828kB kernel_stack:2768kB pagetables:43472kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:14972 all_unreclaimable? yes
myhostname kernel: lowmem_reserve[]: 0 0 0 0
myhostname kernel: Node 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15880kB
myhostname kernel: Node 0 DMA32: 458*4kB 494*8kB 284*16kB 55*32kB 7*64kB 4*128kB 5*256kB 7*512kB 7*1024kB 6*2048kB 20*4096kB = 119288kB
myhostname kernel: Node 0 Normal: 3333*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 17428kB
myhostname kernel: 115783 total pagecache pages
myhostname kernel: 114466 pages in swap cache
myhostname kernel: Swap cache stats: add 2183060, delete 2068594, find 1860109/1985849
myhostname kernel: Free swap  = 0kB
myhostname kernel: Total swap = 2097148kB
myhostname kernel: 8388592 pages RAM
myhostname kernel: 134806 pages reserved
myhostname kernel: 6845 pages shared
myhostname kernel: 8210059 pages non-shared
问题回答

更有可能是,OMM问题正在造成坠毁。 使用比Xmx理论更多的记忆的一个共同原因是,如果你使用本土记忆。 之所以能够做到这一点,是因为你重新使用一个分配大量物体的日本国家数据库,或者如果你重新记录地图档案等。

你们应尝试在你们的java法典中增加一些标志,以印出Java的记忆,认为它使用Rumontime。 共计 记忆等。 然后,将这些价值观与你通过顶部看到的价值观相比较,看看是否有其他东西需要记忆。

Is the OOM deather causing 恐慌吗?

我认为不是。 根据https://www.kernel.org/doc/gorman/html/understand/understand016.html,“rel=”第13章,由Mel Gorman介绍“低温含水层虚拟记忆管理员”。

13.4 D. 杀害选定过程

Once a task is selected, the list is walked again and each process that shares the same mm_struct as the selected process (i.e. they are threads) is sent a signal. If the process has CAP_SYS_RAWIO capabilities, a SIGTERM is sent to give the process a chance of exiting cleanly, otherwise a SIGKILL is sent.

如果证书程序得到<代码>SIGKILL的信号,它就获得了生成坠毁报告的机会。 因此,在OOM杀手事件之前,发生了汽车事故。

我看到的所有其他来源(即信号)都说了同样的事情。


但是,关于<代码>CAP_SYS_RAWIO的内容是什么?

  • 这种能力只能由特权用户(或程序)确定。 它允许通过“dev/kmem”获取实物I/O港口和掩饰记忆。 海事组织不可能确定你的核查和核查程序具有这种能力。

  • 如果联合认证程序确实具备这一能力,它本会发出信号。 但是,坠毁报告说,这是对信号的反应。


因此,科索沃事故赔偿公司坠毁和OMO杀手标志的信息显然有相关性?

我不知道,但一个可能的顺序如下:

  1. Something corrupts the memory in the JVM; e.g. some badly written native library.
  2. The JVM (eventually) falls over with a segmentation fault (reported via a SIGSEGV signal) due to the previous corruption.
  3. Something (before or after the SIGSEGV that crashed the JVM) triggers an excessive amount of virtual memory paging.
  4. The OOM killer is alerted ... and does its thing.

另一种可能性是,这只是一个巧合,即坠毁和OM杀手的标志是用于不同的联合机器人。


总之,根源是造成分裂性错的原因。 (我可以告诉你......)

this is linux limit thread number.

二级违约数限制1024,最高为65535,

页: 1





相关问题
Spring Properties File

Hi have this j2ee web application developed using spring framework. I have a problem with rendering mnessages in nihongo characters from the properties file. I tried converting the file to ascii using ...

Logging a global ID in multiple components

I have a system which contains multiple applications connected together using JMS and Spring Integration. Messages get sent along a chain of applications. [App A] -> [App B] -> [App C] We set a ...

Java Library Size

If I m given two Java Libraries in Jar format, 1 having no bells and whistles, and the other having lots of them that will mostly go unused.... my question is: How will the larger, mostly unused ...

How to get the Array Class for a given Class in Java?

I have a Class variable that holds a certain type and I need to get a variable that holds the corresponding array class. The best I could come up with is this: Class arrayOfFooClass = java.lang....

SQLite , Derby vs file system

I m working on a Java desktop application that reads and writes from/to different files. I think a better solution would be to replace the file system by a SQLite database. How hard is it to migrate ...

热门标签