SIP Router Project
FS#236 - Kamailio crash when about 300,000 tls connections opened.
Attached to Project:
sip-router
Opened by Shaobin.Feng (saxon_leo) - Thursday, 14 June 2012, 06:44 GMT
Last edited by Daniel-Constantin Mierla (miconda) - Thursday, 25 October 2012, 07:59 GMT
Opened by Shaobin.Feng (saxon_leo) - Thursday, 14 June 2012, 06:44 GMT
Last edited by Daniel-Constantin Mierla (miconda) - Thursday, 25 October 2012, 07:59 GMT
|
DetailsKamailio crash when about 300,000 tls connections opened.
=========================================================version info version: kamailio 3.2.3 (x86_64/linux) 59f87e flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 32MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 59f87e compiled on 10:51:38 Jun 13 2012 with gcc 4.5.2 openssl version 1.0.0e ==========================================================config info relate config enable_tls=yes open_files_limit = 500000 tcp_children = 80 tcp_connection_lifetime=3600 tcp_max_connections = 500000 tcp_no_connect = yes tcp_poll_method = epoll_et modparam("tls", "ssl_release_buffers", 1) modparam("tls", "ssl_freelist_max_len", 0) modparam("tls", "ssl_max_send_fragment", 2048) modparam("usrloc", "db_mode", 0) ============================================================operate info Once a connection was created success,while send OPTIONS to keep the connection active before lifetime achieve. ==============================================================log info 426 Jun 14 13:58:40 B2-Ubuntu-01 /usr/local/sbin/kamailio[31516]: ALERT: <core> [main.c:751]: child process 31610 exited by a signal 9 427 Jun 14 13:58:40 B2-Ubuntu-01 /usr/local/sbin/kamailio[31516]: ALERT: <core> [main.c:754]: core was not generated 428 Jun 14 13:58:40 B2-Ubuntu-01 /usr/local/sbin/kamailio[31516]: INFO: <core> [main.c:766]: INFO: terminating due to SIGCHLD 429 Jun 14 13:58:40 B2-Ubuntu-01 /usr/local/sbin/kamailio[31600]: INFO: <core> [main.c:817]: INFO: signal 15 received 430 Jun 14 13:58:40 B2-Ubuntu-01 /usr/local/sbin/kamailio[31520]: INFO: <core> [main.c:817]: INFO: signal 15 received ... 522 Jun 14 13:59:39 B2-Ubuntu-01 /usr/local/sbin/kamailio[31516]: : <core> [main.c:660]: BUG: shutdown timeout triggered, dying... ==============================================================process info sercmd core.ps ..... 31610 tcp main process ===============================================================core file info casue the size limit, attach the gdb screenshot 
QQ截图20120614143846.jpg
(87.8 KiB)
|
This task depends upon
Closed by Daniel-Constantin Mierla (miconda)
Thursday, 25 October 2012, 07:59 GMT
Reason for closing: Fixed
Additional comments about closing: Reported issue is fixed in 3.3.x. Create a new issue if you still face slow processing with latest 3.3.x, adding details from benchmark as suggested.
Thursday, 25 October 2012, 07:59 GMT
Reason for closing: Fixed
Additional comments about closing: Reported issue is fixed in 3.3.x. Create a new issue if you still face slow processing with latest 3.3.x, adding details from benchmark as suggested.
How much memory do you have on this server and how much is allocated to kamailio (private and shared).
Kamailio itself should not eat more than allocated, but other libraries may need own extra memory for internal stuff and can cause such a case. Can you grab also the info about used memory in the system (e.g., via top) just before the crash?
root@B2-Ubuntu-01:/usr/local/src/kamailio-3.2.3# cat /proc/meminfo
MemTotal: 8195876 kB
MemFree: 7881204 kB
Buffers: 18284 kB
Cached: 44848 kB
SwapCached: 25660 kB
Active: 46864 kB
Inactive: 64928 kB
Active(anon): 22488 kB
Inactive(anon): 26448 kB
Active(file): 24376 kB
Inactive(file): 38480 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 2928636 kB
SwapFree: 2384952 kB
Dirty: 32 kB
Writeback: 0 kB
AnonPages: 23608 kB
Mapped: 5860 kB
Shmem: 272 kB
Slab: 30632 kB
SReclaimable: 18280 kB
SUnreclaim: 12352 kB
KernelStack: 1704 kB
PageTables: 5292 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 7026572 kB
Committed_AS: 1715016 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 290428 kB
VmallocChunk: 34359445308 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 45056 kB
DirectMap2M: 8343552 kB
=========================================================kamailio mem info
kamctlrc STARTOPTIONS="-m 6144"
config.h #define PKG_MEM_SIZE 32
=========================================================system info before crash
root@B2-Ubuntu-01: free -m
total used free shared buffers cached
Mem: 8003 7931 72 0 1 2939
-/+ buffers/cache: 4990 3013
Swap: 2859 2850 9
Well,didn't save top info.
Try to reproduce and watch the top about used memory, if kamailio is increasing, it will be killed by OS. Look at the syslog messages, iirc, there should be a message about it.
=================================================================syslog info
35944 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542670] msms invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
35945 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542676] msms cpuset=/ mems_allowed=0
35946 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542681] Pid: 1245, comm: msms Not tainted 2.6.38-8-server #42-Ubuntu
35947 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542683] Call Trace:
35948 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542693] [<ffffffff810b9bed>] ? cpuset_print_task_mems_allowed+0x9d/0xb0
35949 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542702] [<ffffffff8110e501>] ? dump_header+0x91/0x1e0
35950 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542708] [<ffffffff8127924c>] ? security_real_capable_noaudit+0x3c/0x70
35951 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542713] [<ffffffff8110eb7d>] ? oom_kill_process+0x8d/0x190
35952 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542716] [<ffffffff8110efc2>] ? out_of_memory+0x102/0x240
35953 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542721] [<ffffffff81113ff9>] ? __alloc_pages_nodemask+0x7b9/0x830
35954 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542728] [<ffffffff81149475>] ? alloc_pages_current+0xa5/0x110
35955 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542733] [<ffffffff8110b6cf>] ? __page_cache_alloc+0x8f/0xa0
35956 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542737] [<ffffffff81116a25>] ? __do_page_cache_readahead+0xd5/0x220
35957 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542742] [<ffffffff81116ed1>] ? ra_submit+0x21/0x30
35958 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542746] [<ffffffff8110d84e>] ? filemap_fault+0x46e/0x490
35959 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542750] [<ffffffff8112c744>] ? __do_fault+0x54/0x520
35960 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542754] [<ffffffff8112ffda>] ? handle_pte_fault+0xfa/0x210
35961 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542757] [<ffffffff81114c8d>] ? __free_pages+0x2d/0x40
35962 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542763] [<ffffffff8112d44f>] ? __pte_alloc+0xdf/0x100
35963 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542767] [<ffffffff8113135d>] ? handle_mm_fault+0x16d/0x250
35964 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542773] [<ffffffff815db502>] ? do_page_fault+0x1a2/0x540
35965 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542778] [<ffffffff8105f6f0>] ? default_wake_function+0x0/0x20
35966 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542783] [<ffffffff815d7f95>] ? page_fault+0x25/0x30
35967 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542785] Mem-Info:
35968 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542787] Node 0 DMA per-cpu:
35969 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542791] CPU 0: hi: 0, btch: 1 usd: 0
35970 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542793] CPU 1: hi: 0, btch: 1 usd: 0
35971 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542795] Node 0 DMA32 per-cpu:
35972 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542798] CPU 0: hi: 186, btch: 31 usd: 31
35973 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542801] CPU 1: hi: 186, btch: 31 usd: 39
35974 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542802] Node 0 Normal per-cpu:
35975 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542805] CPU 0: hi: 186, btch: 31 usd: 58
35976 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542808] CPU 1: hi: 186, btch: 31 usd: 52
35977 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542813] active_anon:1006237 inactive_anon:194761 isolated_anon:0
35978 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542815] active_file:50 inactive_file:251 isolated_file:0
35979 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542816] unevictable:0 dirty:0 writeback:1 unstable:0
35980 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542817] free:11853 slab_reclaimable:75300 slab_unreclaimable:172394
35981 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542818] mapped:589298 shmem:834308 pagetables:228372 bounce:0
35982 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542821] Node 0 DMA free:15868kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:15684kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
35983 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542832] lowmem_reserve[]: 0 3000 8050 8050
35984 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542836] Node 0 DMA32 free:24436kB min:4272kB low:5340kB high:6408kB active_anon:1295636kB inactive_anon:324148kB active_file:8kB inactive_file:212kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:1007288kB shmem:1465792kB slab_reclaimable:146328kB slab_unreclaimable:325888kB kernel_stack:32kB pagetables:384964kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:363 all_unreclaimable? yes
35985 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542849] lowmem_reserve[]: 0 0 5050 5050
35986 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542853] Node 0 Normal free:7108kB min:7192kB low:8988kB high:10788kB active_anon:2729312kB inactive_anon:454896kB active_file:192kB inactive_file:792kB unevictable:0kB isolated(anon):
0kB isolated(file):0kB present:5171200kB mlocked:0kB dirty:0kB writeback:4kB mapped:1349904kB shmem:1871440kB slab_reclaimable:154872kB slab_unreclaimable:363680kB kernel_stack:2448kB pagetables:528524kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:1621 all_unreclaimable? yes
35987 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542865] lowmem_reserve[]: 0 0 0 0
35988 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542869] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15868kB
35989 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542880] Node 0 DMA32: 2147*4kB 883*8kB 263*16kB 99*32kB 20*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24436kB
35990 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542890] Node 0 Normal: 21*4kB 396*8kB 39*16kB 47*32kB 23*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7108kB
35991 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542901] 906514 total pagecache pages
35992 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542902] 71787 pages in swap cache
35993 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542905] Swap cache stats: add 38296398, delete 38224611, find 12978309/16500356
35994 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542907] Free swap = 0kB
35995 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542909] Total swap = 2928636kB
35996 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573173] 2097136 pages RAM
35997 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573176] 48231 pages reserved
35998 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573178] 4894633 pages shared
35999 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573179] 1447063 pages non-shared
36000 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573181] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
36001 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573193] [ 277] 0 277 4263 0 1 0 0 upstart-udev-br
36002 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573197] [ 280] 0 280 5330 1 1 -17 -1000 udevd
36003 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573202] [ 354] 0 354 5319 1 0 -17 -1000 udevd
36004 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573206] [ 359] 0 359 5319 1 0 -17 -1000 udevd
36005 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573211] [ 521] 0 521 3751 0 1 0 0 upstart-socket-
36006 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573220] [ 568] 0 568 12366 0 0 -17 -1000 sshd
36007 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573224] [ 574] 101 574 46547 62 0 0 0 rsyslogd
36008 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573228] [ 634] 0 634 1549 1 0 0 0 getty
36009 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573232] [ 638] 0 638 1549 1 0 0 0 getty
36010 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573236] [ 643] 0 643 1549 1 1 0 0 getty
36011 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573240] [ 644] 0 644 1549 1 0 0 0 getty
36012 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573244] [ 651] 0 651 1549 1 1 0 0 getty
36013 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573248] [ 659] 0 659 3945 22 1 0 0 irqbalance
36014 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573252] [ 674] 999 674 116646 331 1 0 0 mysqld
36015 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573256] [ 679] 0 679 4732 24 1 0 0 cron
36016 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573260] [ 680] 0 680 4182 0 0 0 0 atd
36017 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573264] [ 1185] 0 1185 1549 1 0 0 0 getty
36018 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573269] [ 1226] 0 1226 58508 0 1 0 0 alias_cpno
36019 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573272] [ 1245] 0 1245 55258 71 1 0 0 msms
36020 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573276] [ 1268] 0 1268 3002 21 1 0 0 mpusher_srv_mgr
35989 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542880] Node 0 DMA32: 2147*4kB 883*8kB 263*16kB 99*32kB 20*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24436kB
35990 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542890] Node 0 Normal: 21*4kB 396*8kB 39*16kB 47*32kB 23*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7108kB
35991 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542901] 906514 total pagecache pages
35992 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542902] 71787 pages in swap cache
35993 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542905] Swap cache stats: add 38296398, delete 38224611, find 12978309/16500356
35994 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542907] Free swap = 0kB
35995 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542909] Total swap = 2928636kB
35996 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573173] 2097136 pages RAM
35997 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573176] 48231 pages reserved
35998 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573178] 4894633 pages shared
35999 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573179] 1447063 pages non-shared
36000 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573181] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
36001 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573193] [ 277] 0 277 4263 0 1 0 0 upstart-udev-br
36002 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573197] [ 280] 0 280 5330 1 1 -17 -1000 udevd
36003 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573202] [ 354] 0 354 5319 1 0 -17 -1000 udevd
36004 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573206] [ 359] 0 359 5319 1 0 -17 -1000 udevd
36005 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573211] [ 521] 0 521 3751 0 1 0 0 upstart-socket-
36006 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573220] [ 568] 0 568 12366 0 0 -17 -1000 sshd
36007 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573224] [ 574] 101 574 46547 62 0 0 0 rsyslogd
36008 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573228] [ 634] 0 634 1549 1 0 0 0 getty
36009 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573232] [ 638] 0 638 1549 1 0 0 0 getty
36010 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573236] [ 643] 0 643 1549 1 1 0 0 getty
36011 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573240] [ 644] 0 644 1549 1 0 0 0 getty
36012 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573244] [ 651] 0 651 1549 1 1 0 0 getty
36013 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573248] [ 659] 0 659 3945 22 1 0 0 irqbalance
36014 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573252] [ 674] 999 674 116646 331 1 0 0 mysqld
36015 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573256] [ 679] 0 679 4732 24 1 0 0 cron
36016 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573260] [ 680] 0 680 4182 0 0 0 0 atd
36017 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573264] [ 1185] 0 1185 1549 1 0 0 0 getty
36018 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573269] [ 1226] 0 1226 58508 0 1 0 0 alias_cpno
36019 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573272] [ 1245] 0 1245 55258 71 1 0 0 msms
36020 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573276] [ 1268] 0 1268 3002 21 1 0 0 mpusher_srv_mgr
36021 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573281] [ 1293] 0 1293 126461 4 0 0 0 mpusher
36022 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573285] [ 1312] 0 1312 22301 39 1 0 0 Redirect
36023 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573289] [ 1345] 0 1345 67494 24 1 0 0 Location
36024 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573313] [29245] 0 29245 144258 0 1 0 0 aliasServer
36025 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573317] [ 3255] 0 3255 17701 0 1 0 0 sshd
36026 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573321] [ 3283] 1000 3283 17701 0 0 0 0 sshd
36027 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573325] [ 3285] 1000 3285 3150 1 0 0 0 sftp-server
36028 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573330] [ 661] 0 661 17701 4 0 0 0 sshd
36029 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573334] [ 676] 1000 676 17780 47 0 0 0 sshd
36030 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573338] [ 677] 1000 677 7012 2 1 0 0 bash
36031 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573343] [ 2017] 0 2017 5414 2 1 0 0 sudo
36032 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573347] [ 2018] 0 2018 5966 122 1 0 0 bash
36033 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573351] [ 2372] 1000 2372 6993 2 0 0 0 bash
36034 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573356] [ 2481] 1000 2481 7004 1 1 0 0 bash
36035 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573360] [19064] 1000 19064 7022 188 0 0 0 bash
36036 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573364] [10929] 0 10929 11890 4 1 0 0 searchonline
36037 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573368] [19784] 0 19784 8607 2 1 0 0 sudo
36038 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573372] [19785] 0 19785 5893 46 1 0 0 bash
36039 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573377] [31167] 0 31167 17701 0 0 0 0 sshd
36040 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573381] [31181] 1000 31181 17701 0 0 0 0 sshd
36041 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573385] [31182] 1000 31182 3150 0 1 0 0 sftp-server
36042 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573389] [ 1108] 0 1108 17701 2 0 0 0 sshd
36043 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573393] [ 1122] 1000 1122 17701 1 0 0 0 sshd
36044 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573397] [ 1123] 1000 1123 6984 0 0 0 0 bash
36045 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573401] [ 2896] 0 2896 17701 0 0 0 0 sshd
36046 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573405] [ 2910] 1000 2910 17743 0 0 0 0 sshd
36047 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573409] [ 2911] 1000 2911 3150 1 0 0 0 sftp-server
36048 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573413] [ 3061] 0 3061 17701 0 0 0 0 sshd
36049 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573418] [ 3075] 1000 3075 17701 0 0 0 0 sshd
36050 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573421] [ 3076] 1000 3076 3150 1 0 0 0 sftp-server
36051 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573428] [ 4937] 0 4937 1599502 498 1 0 0 kamailio
36052 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573433] [ 4938] 0 4938 1599502 118 1 0 0 kamailio
36053 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573437] [ 4939] 0 4939 1599502 118 1 0 0 kamailio
36054 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573441] [ 4940] 0 4940 1599502 134 0 0 0 kamailio
36055 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573445] [ 4941] 0 4941 1599502 127 1 0 0 kamailio
36056 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573449] [ 4942] 0 4942 1599502 77 0 0 0 kamailio
36057 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573453] [ 4943] 0 4943 1599502 77 1 0 0 kamailio
36058 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573457] [ 4944] 0 4944 1599502 77 1 0 0 kamailio
36059 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573461] [ 4945] 0 4945 1599502 77 0 0 0 kamailio
36060 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573465] [ 4946] 0 4946 1599502 162941 0 0 0 kamailio
36061 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573468] [ 4947] 0 4947 1599502 338 0 0 0 kamailio
36062 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573472] [ 4948] 0 4948 1599503 80 1 0 0 kamailio
36063 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573476] [ 4949] 0 4949 1599502 79 1 0 0 kamailio
36064 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573480] [ 4950] 0 4950 1599502 226423 0 0 0 kamailio
36065 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573484] [ 4951] 0 4951 1599539 55375 1 0 0 kamailio
36066 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573488] [ 4952] 0 4952 1599539 56286 1 0 0 kamailio
36067 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573492] [ 4953] 0 4953 1599539 54212 1 0 0 kamailio
36068 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573496] [ 4954] 0 4954 1599539 55651 1 0 0 kamailio
36069 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573500] [ 4955] 0 4955 1599539 55924 1 0 0 kamailio
36070 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573504] [ 4956] 0 4956 1599539 53863 1 0 0 kamailio
36071 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573508] [ 4957] 0 4957 1599539 55428 1 0 0 kamailio
36072 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573512] [ 4958] 0 4958 1599539 54994 1 0 0 kamailio
36073 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573516] [ 4959] 0 4959 1599539 55698 1 0 0 kamailio
36074 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573520] [ 4960] 0 4960 1599539 57541 1 0 0 kamailio
36075 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573524] [ 4961] 0 4961 1599539 55636 1 0 0 kamailio
36076 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573528] [ 4962] 0 4962 1599539 54362 1 0 0 kamailio
36077 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573532] [ 4963] 0 4963 1599539 55880 1 0 0 kamailio
36078 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573536] [ 4964] 0 4964 1599539 56076 1 0 0 kamailio
36079 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573540] [ 4965] 0 4965 1599539 54815 0 0 0 kamailio
36080 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573544] [ 4966] 0 4966 1599539 55154 0 0 0 kamailio
36081 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573547] [ 4967] 0 4967 1599539 55667 1 0 0 kamailio
36082 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573551] [ 4968] 0 4968 1599539 56113 0 0 0 kamailio
36083 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573555] [ 4969] 0 4969 1599539 55237 1 0 0 kamailio
36084 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573559] [ 4970] 0 4970 1599539 54841 0 0 0 kamailio
36085 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573563] [ 4971] 0 4971 1599539 55301 0 0 0 kamailio
36086 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573567] [ 4972] 0 4972 1599539 56195 0 0 0 kamailio
36087 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573571] [ 4973] 0 4973 1599539 55995 0 0 0 kamailio
36088 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573575] [ 4974] 0 4974 1599539 54559 1 0 0 kamailio
36089 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573579] [ 4975] 0 4975 1599539 54271 1 0 0 kamailio
36090 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573583] [ 4976] 0 4976 1599539 56839 0 0 0 kamailio
36091 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573587] [ 4977] 0 4977 1599539 55354 1 0 0 kamailio
36092 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573591] [ 4978] 0 4978 1599539 56289 0 0 0 kamailio
36093 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573595] [ 4979] 0 4979 1599539 54465 1 0 0 kamailio
36094 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573599] [ 4980] 0 4980 1599539 54083 1 0 0 kamailio
36095 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573603] [ 4981] 0 4981 1599539 54506 1 0 0 kamailio
36096 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573610] [ 4982] 0 4982 1599539 56366 1 0 0 kamailio
36097 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573613] [ 4983] 0 4983 1599539 56146 0 0 0 kamailio
36098 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573618] [ 4984] 0 4984 1599539 55425 0 0 0 kamailio
36099 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573621] [ 4985] 0 4985 1599539 55158 1 0 0 kamailio
36100 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573625] [ 4986] 0 4986 1599539 56390 0 0 0 kamailio
36101 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573629] [ 4987] 0 4987 1599539 56196 0 0 0 kamailio
36102 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573633] [ 4988] 0 4988 1599539 56342 1 0 0 kamailio
36103 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573637] [ 4989] 0 4989 1599539 54488 0 0 0 kamailio
36104 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573641] [ 4990] 0 4990 1599539 56236 0 0 0 kamailio
36105 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573645] [ 4991] 0 4991 1599539 56636 1 0 0 kamailio
36106 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573649] [ 4992] 0 4992 1599539 55957 1 0 0 kamailio
36107 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573653] [ 4993] 0 4993 1599539 55887 1 0 0 kamailio
36108 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573657] [ 4994] 0 4994 1599539 56170 1 0 0 kamailio
36109 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573661] [ 4995] 0 4995 1599539 55313 1 0 0 kamailio
36110 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573664] [ 4996] 0 4996 1599539 56071 1 0 0 kamailio
36111 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573668] [ 4997] 0 4997 1599539 54480 1 0 0 kamailio
36112 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573672] [ 4998] 0 4998 1599539 55038 1 0 0 kamailio
36113 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573676] [ 4999] 0 4999 1599539 54860 1 0 0 kamailio
36114 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573680] [ 5000] 0 5000 1599539 55906 1 0 0 kamailio
36115 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573684] [ 5001] 0 5001 1599539 54281 0 0 0 kamailio
36116 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573688] [ 5002] 0 5002 1599539 56046 1 0 0 kamailio
36117 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573692] [ 5003] 0 5003 1599539 53809 1 0 0 kamailio
36118 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573696] [ 5004] 0 5004 1599539 55271 1 0 0 kamailio
36119 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573700] [ 5005] 0 5005 1599539 55130 1 0 0 kamailio
36120 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573704] [ 5006] 0 5006 1599539 55354 1 0 0 kamailio
36121 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573708] [ 5007] 0 5007 1599539 54812 1 0 0 kamailio
36122 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573712] [ 5008] 0 5008 1599539 56222 1 0 0 kamailio
36123 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573716] [ 5009] 0 5009 1599539 55686 1 0 0 kamailio
36124 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573720] [ 5010] 0 5010 1599539 56041 0 0 0 kamailio
36125 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573724] [ 5011] 0 5011 1599539 54781 1 0 0 kamailio
36126 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573728] [ 5012] 0 5012 1599539 55973 0 0 0 kamailio
36127 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573732] [ 5013] 0 5013 1599539 55810 1 0 0 kamailio
36128 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573736] [ 5014] 0 5014 1599539 55330 0 0 0 kamailio
36129 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573739] [ 5015] 0 5015 1599539 54529 1 0 0 kamailio
36130 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573744] [ 5016] 0 5016 1599539 54964 0 0 0 kamailio
36131 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573748] [ 5017] 0 5017 1599539 55423 1 0 0 kamailio
36132 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573751] [ 5018] 0 5018 1599539 55416 0 0 0 kamailio
36133 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573755] [ 5019] 0 5019 1599539 56218 1 0 0 kamailio
36134 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573759] [ 5020] 0 5020 1599539 57193 1 0 0 kamailio
36135 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573763] [ 5021] 0 5021 1599539 56217 1 0 0 kamailio
36136 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573767] [ 5022] 0 5022 1599539 56432 1 0 0 kamailio
36137 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573771] [ 5023] 0 5023 1599539 56450 1 0 0 kamailio
36138 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573775] [ 5024] 0 5024 1599539 55694 0 0 0 kamailio
36139 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573779] [ 5025] 0 5025 1599539 56016 0 0 0 kamailio
36140 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573783] [ 5026] 0 5026 1599539 55858 1 0 0 kamailio
36141 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573789] [ 5027] 0 5027 1599539 55406 1 0 0 kamailio
36142 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573793] [ 5028] 0 5028 1599539 54413 1 0 0 kamailio
36143 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573797] [ 5029] 0 5029 1599539 55047 1 0 0 kamailio
36144 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573801] [ 5030] 0 5030 1599539 55776 1 0 0 kamailio
36145 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573805] [ 5031] 0 5031 1599502 352312 0 0 0 kamailio
36146 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573809] [ 5583] 0 5583 4868 107 0 0 0 top
36147 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573813] [ 5682] 0 5682 1717 15 0 0 0 tail
36148 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573817] [ 6931] 1000 6931 1733 31 0 0 0 netstat
36149 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573821] [ 6932] 1000 6932 2251 27 0 0 0 grep
36150 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573825] [ 6933] 1000 6933 1704 33 0 0 0 awk
36151 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573828] Out of memory: Kill process 5031 (kamailio) score 96 or sacrifice child
36152 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.577256] Killed process 5031 (kamailio) total-vm:6398008kB, anon-rss:17448kB, file-rss:1391792kB
36153 Jun 14 15:46:37 B2-Ubuntu-01 /usr/local/sbin/kamailio[4937]: ALERT: <core> [main.c:751]: child process 5031 exited by a signal 9
36154 Jun 14 15:46:37 B2-Ubuntu-01 /usr/local/sbin/kamailio[4937]: ALERT: <core> [main.c:754]: core was not generated
36155 Jun 14 15:46:37 B2-Ubuntu-01 /usr/local/sbin/kamailio[4937]: INFO: <core> [main.c:766]: INFO: terminating due to SIGCHLD
36156 Jun 14 15:46:37 B2-Ubuntu-01 /usr/local/sbin/kamailio[5030]: INFO: <core> [main.c:817]: INFO: signal 15 received
================================================Memory Overcommit info
root@B2-Ubuntu-01:~# cat /proc/sys/vm/overcommit_memory
0
Also read the TLS README carefully. There are various memory aspects documented, e.g.:
http://kamailio.org/docs/modules/3.2.x/modules/tls.html#tls.low_memory
See also the low memory thresholds:
http://kamailio.org/docs/modules/3.2.x/modules/tls.html#low_mem_threshold1
http://kamailio.org/docs/modules/3.2.x/modules/tls.html#low_mem_threshold2
Something really confuse me.
According to top info while connections achieve 310,000, there are almost 3G cached,why OS didn't use them?
And the server overcommit_memory=0,why someone can invoke oom-killer?
Since 2.5.30 the values are: 0 (default): as before: guess about how much overcommitment is reasonable, 1: never refuse any malloc(), 2: be precise about the overcommit - never commit a virtual address space larger than swap space plus a fraction overcommit_ratio of the physical memory. Here /proc/sys/vm/overcommit_ratio (by default 50) is another user-settable parameter. It is possible to set overcommit_ratio to values larger than 100. (See also Documentation/vm/overcommit-accounting.)
Thus, you have to set it to "2", any maybe also tweak /proc/sys/vm/overcommit_ratio
first of all ,thank you guys for your attention.
1,for suggestion 1(Klaus Darilion)
===========================================================result of {echo "2" > /proc/sys/vm/overcommit_memory}
kamailio couldn't start as normal
===============================log info
/usr/local/sbin/kamailio[1018]: ERROR: <core> [tcp_main.c:4822]: ERROR: tcp_main: fork failed: Cannot allocate memory
===============================info
the config just as above
2,for suggestion 2(Andrei Pelinescu-Onciul)
===========================================================result of {shm_force_alloc = yes and mlock_pages = yes in the .cfg}
a,the connection accept very slow, almost 10~30/s at the very beginning
b,the server could not be sshed
c,also crash as above when connection achieve almost 90,000 connections
3,I use kamailio as sip-server with almost 100,000 users(via tls) per machine.
And the num of users grow up every day.
So,any idea of this?
Well, when machine handled almost 30,000 connections, kamailio process sip message very slowly(in local net,almost delay 10s).
Anyway,kamailio not crash,nice!