sip-router

SIP Router Project

Tasklist

FS#236 - Kamailio crash when about 300,000 tls connections opened.

Attached to Project: sip-router
Opened by Shaobin.Feng (saxon_leo) - Thursday, 14 June 2012, 06:44 GMT
Last edited by Daniel-Constantin Mierla (miconda) - Thursday, 25 October 2012, 07:59 GMT
Task Type Bug Report
Category Core
Status Closed
Assigned To No-one
Operating System Linux
Severity Critical
Priority Normal
Reported Version 3.2
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Kamailio crash when about 300,000 tls connections opened.

=========================================================version info
version: kamailio 3.2.3 (x86_64/linux) 59f87e
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 32MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 59f87e
compiled on 10:51:38 Jun 13 2012 with gcc 4.5.2

openssl version 1.0.0e
==========================================================config info
relate config

enable_tls=yes
open_files_limit = 500000
tcp_children = 80
tcp_connection_lifetime=3600
tcp_max_connections = 500000
tcp_no_connect = yes
tcp_poll_method = epoll_et

modparam(”tls”, “ssl_release_buffers”, 1)
modparam(”tls”, “ssl_freelist_max_len”, 0)
modparam(”tls”, “ssl_max_send_fragment”, 2048)

modparam(”usrloc”, “db_mode”, 0)
============================================================operate info
Once a connection was created success,while send OPTIONS to keep the connection active before lifetime achieve.

==============================================================log info 426 Jun 14 13:58:40 B2-Ubuntu-01 /usr/local/sbin/kamailio[31516]: ALERT: <core> [main.c:751]: child process 31610 exited by a signal 9 427 Jun 14 13:58:40 B2-Ubuntu-01 /usr/local/sbin/kamailio[31516]: ALERT: <core> [main.c:754]: core was not generated 428 Jun 14 13:58:40 B2-Ubuntu-01 /usr/local/sbin/kamailio[31516]: INFO: <core> [main.c:766]: INFO: terminating due to SIGCHLD 429 Jun 14 13:58:40 B2-Ubuntu-01 /usr/local/sbin/kamailio[31600]: INFO: <core> [main.c:817]: INFO: signal 15 received 430 Jun 14 13:58:40 B2-Ubuntu-01 /usr/local/sbin/kamailio[31520]: INFO: <core> [main.c:817]: INFO: signal 15 received ... 522 Jun 14 13:59:39 B2-Ubuntu-01 /usr/local/sbin/kamailio[31516]: : <core> [main.c:660]: BUG: shutdown timeout triggered, dying... ==============================================================process info sercmd core.ps ..... 31610 tcp main process ===============================================================core file info casue the size limit, attach the gdb screenshot

This task depends upon

Closed by  Daniel-Constantin Mierla (miconda)
Thursday, 25 October 2012, 07:59 GMT
Reason for closing:  Fixed
Additional comments about closing:  Reported issue is fixed in 3.3.x. Create a new issue if you still face slow processing with latest 3.3.x, adding details from benchmark as suggested.
Comment by Daniel-Constantin Mierla (miconda) - Thursday, 14 June 2012, 06:54 GMT

Signal 9 is sent by the kernel when the process is eating almost entire memory.

How much memory do you have on this server and how much is allocated to kamailio (private and shared).

Kamailio itself should not eat more than allocated, but other libraries may need own extra memory for internal stuff and can cause such a case. Can you grab also the info about used memory in the system (e.g., via top) just before the crash?

Comment by Shaobin.Feng (saxon_leo) - Thursday, 14 June 2012, 07:05 GMT

============================================================================mem info root@B2-Ubuntu-01:/usr/local/src/kamailio-3.2.3# cat /proc/meminfo MemTotal: 8195876 kB MemFree: 7881204 kB Buffers: 18284 kB Cached: 44848 kB SwapCached: 25660 kB Active: 46864 kB Inactive: 64928 kB Active(anon): 22488 kB Inactive(anon): 26448 kB Active(file): 24376 kB Inactive(file): 38480 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 2928636 kB SwapFree: 2384952 kB Dirty: 32 kB Writeback: 0 kB AnonPages: 23608 kB Mapped: 5860 kB Shmem: 272 kB Slab: 30632 kB SReclaimable: 18280 kB SUnreclaim: 12352 kB KernelStack: 1704 kB PageTables: 5292 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 7026572 kB Committed_AS: 1715016 kB VmallocTotal: 34359738367 kB VmallocUsed: 290428 kB VmallocChunk: 34359445308 kB HardwareCorrupted: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 45056 kB DirectMap2M: 8343552 kB =========================================================kamailio mem info kamctlrc STARTOPTIONS="-m 6144" config.h #define PKG_MEM_SIZE 32 =========================================================system info before crash root@B2-Ubuntu-01: free -m total used free shared buffers cached Mem: 8003 7931 72 0 1 2939 -/+ buffers/cache: 4990 3013 Swap: 2859 2850 9 Well,didn't save top info.

Comment by Daniel-Constantin Mierla (miconda) - Thursday, 14 June 2012, 07:22 GMT

FYI, in v3.2.x, you can give -M parameter for specifying private memory, no need to change in config.h and recompile.

Try to reproduce and watch the top about used memory, if kamailio is increasing, it will be killed by OS. Look at the syslog messages, iirc, there should be a message about it.

Comment by Klaus Darilion (klaus3000) - Thursday, 14 June 2012, 07:25 GMT Comment by Shaobin.Feng (saxon_leo) - Thursday, 14 June 2012, 08:00 GMT

-M is nice,thanks.

=================================================================syslog info 35944 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542670] msms invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 35945 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542676] msms cpuset=/ mems_allowed=0 35946 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542681] Pid: 1245, comm: msms Not tainted 2.6.38-8-server #42-Ubuntu 35947 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542683] Call Trace: 35948 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542693] [<ffffffff810b9bed>] ? cpuset_print_task_mems_allowed+0x9d/0xb0 35949 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542702] [<ffffffff8110e501>] ? dump_header+0x91/0x1e0 35950 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542708] [<ffffffff8127924c>] ? security_real_capable_noaudit+0x3c/0x70 35951 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542713] [<ffffffff8110eb7d>] ? oom_kill_process+0x8d/0x190 35952 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542716] [<ffffffff8110efc2>] ? out_of_memory+0x102/0x240 35953 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542721] [<ffffffff81113ff9>] ? __alloc_pages_nodemask+0x7b9/0x830 35954 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542728] [<ffffffff81149475>] ? alloc_pages_current+0xa5/0x110 35955 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542733] [<ffffffff8110b6cf>] ? __page_cache_alloc+0x8f/0xa0 35956 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542737] [<ffffffff81116a25>] ? __do_page_cache_readahead+0xd5/0x220 35957 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542742] [<ffffffff81116ed1>] ? ra_submit+0x21/0x30 35958 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542746] [<ffffffff8110d84e>] ? filemap_fault+0x46e/0x490 35959 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542750] [<ffffffff8112c744>] ? __do_fault+0x54/0x520 35960 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542754] [<ffffffff8112ffda>] ? handle_pte_fault+0xfa/0x210 35961 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542757] [<ffffffff81114c8d>] ? __free_pages+0x2d/0x40 35962 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542763] [<ffffffff8112d44f>] ? __pte_alloc+0xdf/0x100 35963 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542767] [<ffffffff8113135d>] ? handle_mm_fault+0x16d/0x250 35964 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542773] [<ffffffff815db502>] ? do_page_fault+0x1a2/0x540 35965 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542778] [<ffffffff8105f6f0>] ? default_wake_function+0x0/0x20 35966 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542783] [<ffffffff815d7f95>] ? page_fault+0x25/0x30 35967 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542785] Mem-Info: 35968 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542787] Node 0 DMA per-cpu: 35969 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542791] CPU 0: hi: 0, btch: 1 usd: 0 35970 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542793] CPU 1: hi: 0, btch: 1 usd: 0 35971 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542795] Node 0 DMA32 per-cpu: 35972 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542798] CPU 0: hi: 186, btch: 31 usd: 31 35973 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542801] CPU 1: hi: 186, btch: 31 usd: 39 35974 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542802] Node 0 Normal per-cpu: 35975 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542805] CPU 0: hi: 186, btch: 31 usd: 58 35976 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542808] CPU 1: hi: 186, btch: 31 usd: 52 35977 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542813] active_anon:1006237 inactive_anon:194761 isolated_anon:0 35978 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542815] active_file:50 inactive_file:251 isolated_file:0 35979 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542816] unevictable:0 dirty:0 writeback:1 unstable:0 35980 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542817] free:11853 slab_reclaimable:75300 slab_unreclaimable:172394 35981 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542818] mapped:589298 shmem:834308 pagetables:228372 bounce:0 35982 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542821] Node 0 DMA free:15868kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15684kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 35983 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542832] lowmem_reserve[]: 0 3000 8050 8050 35984 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542836] Node 0 DMA32 free:24436kB min:4272kB low:5340kB high:6408kB active_anon:1295636kB inactive_anon:324148kB active_file:8kB inactive_file:212kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:1007288kB shmem:1465792kB slab_reclaimable:146328kB slab_unreclaimable:325888kB kernel_stack:32kB pagetables:384964kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:363 all_unreclaimable? yes 35985 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542849] lowmem_reserve[]: 0 0 5050 5050 35986 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542853] Node 0 Normal free:7108kB min:7192kB low:8988kB high:10788kB active_anon:2729312kB inactive_anon:454896kB active_file:192kB inactive_file:792kB unevictable:0kB isolated(anon): 0kB isolated(file):0kB present:5171200kB mlocked:0kB dirty:0kB writeback:4kB mapped:1349904kB shmem:1871440kB slab_reclaimable:154872kB slab_unreclaimable:363680kB kernel_stack:2448kB pagetables:528524kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1621 all_unreclaimable? yes 35987 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542865] lowmem_reserve[]: 0 0 0 0 35988 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542869] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15868kB 35989 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542880] Node 0 DMA32: 2147*4kB 883*8kB 263*16kB 99*32kB 20*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24436kB 35990 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542890] Node 0 Normal: 21*4kB 396*8kB 39*16kB 47*32kB 23*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7108kB 35991 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542901] 906514 total pagecache pages 35992 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542902] 71787 pages in swap cache 35993 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542905] Swap cache stats: add 38296398, delete 38224611, find 12978309/16500356 35994 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542907] Free swap = 0kB 35995 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542909] Total swap = 2928636kB 35996 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573173] 2097136 pages RAM 35997 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573176] 48231 pages reserved 35998 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573178] 4894633 pages shared 35999 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573179] 1447063 pages non-shared 36000 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573181] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name 36001 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573193] [ 277] 0 277 4263 0 1 0 0 upstart-udev-br 36002 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573197] [ 280] 0 280 5330 1 1 -17 -1000 udevd 36003 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573202] [ 354] 0 354 5319 1 0 -17 -1000 udevd 36004 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573206] [ 359] 0 359 5319 1 0 -17 -1000 udevd 36005 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573211] [ 521] 0 521 3751 0 1 0 0 upstart-socket- 36006 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573220] [ 568] 0 568 12366 0 0 -17 -1000 sshd 36007 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573224] [ 574] 101 574 46547 62 0 0 0 rsyslogd 36008 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573228] [ 634] 0 634 1549 1 0 0 0 getty 36009 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573232] [ 638] 0 638 1549 1 0 0 0 getty 36010 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573236] [ 643] 0 643 1549 1 1 0 0 getty 36011 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573240] [ 644] 0 644 1549 1 0 0 0 getty 36012 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573244] [ 651] 0 651 1549 1 1 0 0 getty 36013 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573248] [ 659] 0 659 3945 22 1 0 0 irqbalance 36014 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573252] [ 674] 999 674 116646 331 1 0 0 mysqld 36015 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573256] [ 679] 0 679 4732 24 1 0 0 cron 36016 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573260] [ 680] 0 680 4182 0 0 0 0 atd 36017 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573264] [ 1185] 0 1185 1549 1 0 0 0 getty 36018 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573269] [ 1226] 0 1226 58508 0 1 0 0 alias_cpno 36019 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573272] [ 1245] 0 1245 55258 71 1 0 0 msms 36020 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573276] [ 1268] 0 1268 3002 21 1 0 0 mpusher_srv_mgr 35989 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542880] Node 0 DMA32: 2147*4kB 883*8kB 263*16kB 99*32kB 20*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24436kB 35990 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542890] Node 0 Normal: 21*4kB 396*8kB 39*16kB 47*32kB 23*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7108kB 35991 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542901] 906514 total pagecache pages 35992 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542902] 71787 pages in swap cache 35993 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542905] Swap cache stats: add 38296398, delete 38224611, find 12978309/16500356 35994 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542907] Free swap = 0kB 35995 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.542909] Total swap = 2928636kB 35996 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573173] 2097136 pages RAM 35997 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573176] 48231 pages reserved 35998 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573178] 4894633 pages shared 35999 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573179] 1447063 pages non-shared 36000 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573181] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name 36001 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573193] [ 277] 0 277 4263 0 1 0 0 upstart-udev-br 36002 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573197] [ 280] 0 280 5330 1 1 -17 -1000 udevd 36003 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573202] [ 354] 0 354 5319 1 0 -17 -1000 udevd 36004 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573206] [ 359] 0 359 5319 1 0 -17 -1000 udevd 36005 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573211] [ 521] 0 521 3751 0 1 0 0 upstart-socket- 36006 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573220] [ 568] 0 568 12366 0 0 -17 -1000 sshd 36007 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573224] [ 574] 101 574 46547 62 0 0 0 rsyslogd 36008 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573228] [ 634] 0 634 1549 1 0 0 0 getty 36009 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573232] [ 638] 0 638 1549 1 0 0 0 getty 36010 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573236] [ 643] 0 643 1549 1 1 0 0 getty 36011 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573240] [ 644] 0 644 1549 1 0 0 0 getty 36012 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573244] [ 651] 0 651 1549 1 1 0 0 getty 36013 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573248] [ 659] 0 659 3945 22 1 0 0 irqbalance 36014 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573252] [ 674] 999 674 116646 331 1 0 0 mysqld 36015 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573256] [ 679] 0 679 4732 24 1 0 0 cron 36016 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573260] [ 680] 0 680 4182 0 0 0 0 atd 36017 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573264] [ 1185] 0 1185 1549 1 0 0 0 getty 36018 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573269] [ 1226] 0 1226 58508 0 1 0 0 alias_cpno 36019 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573272] [ 1245] 0 1245 55258 71 1 0 0 msms 36020 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573276] [ 1268] 0 1268 3002 21 1 0 0 mpusher_srv_mgr 36021 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573281] [ 1293] 0 1293 126461 4 0 0 0 mpusher 36022 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573285] [ 1312] 0 1312 22301 39 1 0 0 Redirect 36023 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573289] [ 1345] 0 1345 67494 24 1 0 0 Location 36024 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573313] [29245] 0 29245 144258 0 1 0 0 aliasServer 36025 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573317] [ 3255] 0 3255 17701 0 1 0 0 sshd 36026 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573321] [ 3283] 1000 3283 17701 0 0 0 0 sshd 36027 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573325] [ 3285] 1000 3285 3150 1 0 0 0 sftp-server 36028 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573330] [ 661] 0 661 17701 4 0 0 0 sshd 36029 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573334] [ 676] 1000 676 17780 47 0 0 0 sshd 36030 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573338] [ 677] 1000 677 7012 2 1 0 0 bash 36031 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573343] [ 2017] 0 2017 5414 2 1 0 0 sudo 36032 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573347] [ 2018] 0 2018 5966 122 1 0 0 bash 36033 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573351] [ 2372] 1000 2372 6993 2 0 0 0 bash 36034 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573356] [ 2481] 1000 2481 7004 1 1 0 0 bash 36035 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573360] [19064] 1000 19064 7022 188 0 0 0 bash 36036 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573364] [10929] 0 10929 11890 4 1 0 0 searchonline 36037 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573368] [19784] 0 19784 8607 2 1 0 0 sudo 36038 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573372] [19785] 0 19785 5893 46 1 0 0 bash 36039 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573377] [31167] 0 31167 17701 0 0 0 0 sshd 36040 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573381] [31181] 1000 31181 17701 0 0 0 0 sshd 36041 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573385] [31182] 1000 31182 3150 0 1 0 0 sftp-server 36042 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573389] [ 1108] 0 1108 17701 2 0 0 0 sshd 36043 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573393] [ 1122] 1000 1122 17701 1 0 0 0 sshd 36044 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573397] [ 1123] 1000 1123 6984 0 0 0 0 bash 36045 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573401] [ 2896] 0 2896 17701 0 0 0 0 sshd 36046 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573405] [ 2910] 1000 2910 17743 0 0 0 0 sshd 36047 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573409] [ 2911] 1000 2911 3150 1 0 0 0 sftp-server 36048 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573413] [ 3061] 0 3061 17701 0 0 0 0 sshd 36049 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573418] [ 3075] 1000 3075 17701 0 0 0 0 sshd 36050 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573421] [ 3076] 1000 3076 3150 1 0 0 0 sftp-server 36051 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573428] [ 4937] 0 4937 1599502 498 1 0 0 kamailio 36052 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573433] [ 4938] 0 4938 1599502 118 1 0 0 kamailio 36053 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573437] [ 4939] 0 4939 1599502 118 1 0 0 kamailio 36054 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573441] [ 4940] 0 4940 1599502 134 0 0 0 kamailio 36055 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573445] [ 4941] 0 4941 1599502 127 1 0 0 kamailio 36056 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573449] [ 4942] 0 4942 1599502 77 0 0 0 kamailio 36057 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573453] [ 4943] 0 4943 1599502 77 1 0 0 kamailio 36058 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573457] [ 4944] 0 4944 1599502 77 1 0 0 kamailio 36059 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573461] [ 4945] 0 4945 1599502 77 0 0 0 kamailio 36060 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573465] [ 4946] 0 4946 1599502 162941 0 0 0 kamailio 36061 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573468] [ 4947] 0 4947 1599502 338 0 0 0 kamailio 36062 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573472] [ 4948] 0 4948 1599503 80 1 0 0 kamailio 36063 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573476] [ 4949] 0 4949 1599502 79 1 0 0 kamailio 36064 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573480] [ 4950] 0 4950 1599502 226423 0 0 0 kamailio 36065 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573484] [ 4951] 0 4951 1599539 55375 1 0 0 kamailio 36066 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573488] [ 4952] 0 4952 1599539 56286 1 0 0 kamailio 36067 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573492] [ 4953] 0 4953 1599539 54212 1 0 0 kamailio 36068 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573496] [ 4954] 0 4954 1599539 55651 1 0 0 kamailio 36069 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573500] [ 4955] 0 4955 1599539 55924 1 0 0 kamailio 36070 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573504] [ 4956] 0 4956 1599539 53863 1 0 0 kamailio 36071 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573508] [ 4957] 0 4957 1599539 55428 1 0 0 kamailio 36072 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573512] [ 4958] 0 4958 1599539 54994 1 0 0 kamailio 36073 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573516] [ 4959] 0 4959 1599539 55698 1 0 0 kamailio 36074 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573520] [ 4960] 0 4960 1599539 57541 1 0 0 kamailio 36075 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573524] [ 4961] 0 4961 1599539 55636 1 0 0 kamailio 36076 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573528] [ 4962] 0 4962 1599539 54362 1 0 0 kamailio 36077 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573532] [ 4963] 0 4963 1599539 55880 1 0 0 kamailio 36078 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573536] [ 4964] 0 4964 1599539 56076 1 0 0 kamailio 36079 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573540] [ 4965] 0 4965 1599539 54815 0 0 0 kamailio 36080 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573544] [ 4966] 0 4966 1599539 55154 0 0 0 kamailio 36081 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573547] [ 4967] 0 4967 1599539 55667 1 0 0 kamailio 36082 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573551] [ 4968] 0 4968 1599539 56113 0 0 0 kamailio 36083 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573555] [ 4969] 0 4969 1599539 55237 1 0 0 kamailio 36084 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573559] [ 4970] 0 4970 1599539 54841 0 0 0 kamailio 36085 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573563] [ 4971] 0 4971 1599539 55301 0 0 0 kamailio 36086 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573567] [ 4972] 0 4972 1599539 56195 0 0 0 kamailio 36087 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573571] [ 4973] 0 4973 1599539 55995 0 0 0 kamailio 36088 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573575] [ 4974] 0 4974 1599539 54559 1 0 0 kamailio 36089 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573579] [ 4975] 0 4975 1599539 54271 1 0 0 kamailio 36090 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573583] [ 4976] 0 4976 1599539 56839 0 0 0 kamailio 36091 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573587] [ 4977] 0 4977 1599539 55354 1 0 0 kamailio 36092 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573591] [ 4978] 0 4978 1599539 56289 0 0 0 kamailio 36093 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573595] [ 4979] 0 4979 1599539 54465 1 0 0 kamailio 36094 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573599] [ 4980] 0 4980 1599539 54083 1 0 0 kamailio 36095 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573603] [ 4981] 0 4981 1599539 54506 1 0 0 kamailio 36096 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573610] [ 4982] 0 4982 1599539 56366 1 0 0 kamailio 36097 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573613] [ 4983] 0 4983 1599539 56146 0 0 0 kamailio 36098 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573618] [ 4984] 0 4984 1599539 55425 0 0 0 kamailio 36099 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573621] [ 4985] 0 4985 1599539 55158 1 0 0 kamailio 36100 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573625] [ 4986] 0 4986 1599539 56390 0 0 0 kamailio 36101 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573629] [ 4987] 0 4987 1599539 56196 0 0 0 kamailio 36102 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573633] [ 4988] 0 4988 1599539 56342 1 0 0 kamailio 36103 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573637] [ 4989] 0 4989 1599539 54488 0 0 0 kamailio 36104 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573641] [ 4990] 0 4990 1599539 56236 0 0 0 kamailio 36105 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573645] [ 4991] 0 4991 1599539 56636 1 0 0 kamailio 36106 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573649] [ 4992] 0 4992 1599539 55957 1 0 0 kamailio 36107 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573653] [ 4993] 0 4993 1599539 55887 1 0 0 kamailio 36108 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573657] [ 4994] 0 4994 1599539 56170 1 0 0 kamailio 36109 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573661] [ 4995] 0 4995 1599539 55313 1 0 0 kamailio 36110 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573664] [ 4996] 0 4996 1599539 56071 1 0 0 kamailio 36111 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573668] [ 4997] 0 4997 1599539 54480 1 0 0 kamailio 36112 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573672] [ 4998] 0 4998 1599539 55038 1 0 0 kamailio 36113 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573676] [ 4999] 0 4999 1599539 54860 1 0 0 kamailio 36114 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573680] [ 5000] 0 5000 1599539 55906 1 0 0 kamailio 36115 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573684] [ 5001] 0 5001 1599539 54281 0 0 0 kamailio 36116 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573688] [ 5002] 0 5002 1599539 56046 1 0 0 kamailio 36117 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573692] [ 5003] 0 5003 1599539 53809 1 0 0 kamailio 36118 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573696] [ 5004] 0 5004 1599539 55271 1 0 0 kamailio 36119 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573700] [ 5005] 0 5005 1599539 55130 1 0 0 kamailio 36120 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573704] [ 5006] 0 5006 1599539 55354 1 0 0 kamailio 36121 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573708] [ 5007] 0 5007 1599539 54812 1 0 0 kamailio 36122 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573712] [ 5008] 0 5008 1599539 56222 1 0 0 kamailio 36123 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573716] [ 5009] 0 5009 1599539 55686 1 0 0 kamailio 36124 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573720] [ 5010] 0 5010 1599539 56041 0 0 0 kamailio 36125 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573724] [ 5011] 0 5011 1599539 54781 1 0 0 kamailio 36126 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573728] [ 5012] 0 5012 1599539 55973 0 0 0 kamailio 36127 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573732] [ 5013] 0 5013 1599539 55810 1 0 0 kamailio 36128 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573736] [ 5014] 0 5014 1599539 55330 0 0 0 kamailio 36129 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573739] [ 5015] 0 5015 1599539 54529 1 0 0 kamailio 36130 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573744] [ 5016] 0 5016 1599539 54964 0 0 0 kamailio 36131 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573748] [ 5017] 0 5017 1599539 55423 1 0 0 kamailio 36132 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573751] [ 5018] 0 5018 1599539 55416 0 0 0 kamailio 36133 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573755] [ 5019] 0 5019 1599539 56218 1 0 0 kamailio 36134 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573759] [ 5020] 0 5020 1599539 57193 1 0 0 kamailio 36135 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573763] [ 5021] 0 5021 1599539 56217 1 0 0 kamailio 36136 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573767] [ 5022] 0 5022 1599539 56432 1 0 0 kamailio 36137 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573771] [ 5023] 0 5023 1599539 56450 1 0 0 kamailio 36138 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573775] [ 5024] 0 5024 1599539 55694 0 0 0 kamailio 36139 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573779] [ 5025] 0 5025 1599539 56016 0 0 0 kamailio 36140 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573783] [ 5026] 0 5026 1599539 55858 1 0 0 kamailio 36141 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573789] [ 5027] 0 5027 1599539 55406 1 0 0 kamailio 36142 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573793] [ 5028] 0 5028 1599539 54413 1 0 0 kamailio 36143 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573797] [ 5029] 0 5029 1599539 55047 1 0 0 kamailio 36144 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573801] [ 5030] 0 5030 1599539 55776 1 0 0 kamailio 36145 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573805] [ 5031] 0 5031 1599502 352312 0 0 0 kamailio 36146 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573809] [ 5583] 0 5583 4868 107 0 0 0 top 36147 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573813] [ 5682] 0 5682 1717 15 0 0 0 tail 36148 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573817] [ 6931] 1000 6931 1733 31 0 0 0 netstat 36149 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573821] [ 6932] 1000 6932 2251 27 0 0 0 grep 36150 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573825] [ 6933] 1000 6933 1704 33 0 0 0 awk 36151 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.573828] Out of memory: Kill process 5031 (kamailio) score 96 or sacrifice child 36152 Jun 14 15:46:31 B2-Ubuntu-01 kernel: [280774.577256] Killed process 5031 (kamailio) total-vm:6398008kB, anon-rss:17448kB, file-rss:1391792kB 36153 Jun 14 15:46:37 B2-Ubuntu-01 /usr/local/sbin/kamailio[4937]: ALERT: <core> [main.c:751]: child process 5031 exited by a signal 9 36154 Jun 14 15:46:37 B2-Ubuntu-01 /usr/local/sbin/kamailio[4937]: ALERT: <core> [main.c:754]: core was not generated 36155 Jun 14 15:46:37 B2-Ubuntu-01 /usr/local/sbin/kamailio[4937]: INFO: <core> [main.c:766]: INFO: terminating due to SIGCHLD 36156 Jun 14 15:46:37 B2-Ubuntu-01 /usr/local/sbin/kamailio[5030]: INFO: <core> [main.c:817]: INFO: signal 15 received

Comment by Shaobin.Feng (saxon_leo) - Thursday, 14 June 2012, 08:10 GMT

according to http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:signal9

================================================Memory Overcommit info
root@B2-Ubuntu-01:~# cat /proc/sys/vm/overcommit_memory
0

Comment by Klaus Darilion (klaus3000) - Thursday, 14 June 2012, 08:55 GMT

You can try setting /proc/sys/vm/overcommit_memory to “2”. Then system malloc should fail when running out of memory and there shouldn’t be a reason to kill Kamailio.

Also read the TLS README carefully. There are various memory aspects documented, e.g.:
http://kamailio.org/docs/modules/3.2.x/modules/tls.html#tls.low_memory

See also the low memory thresholds:
http://kamailio.org/docs/modules/3.2.x/modules/tls.html#low_mem_threshold1 http://kamailio.org/docs/modules/3.2.x/modules/tls.html#low_mem_threshold2

Comment by Shaobin.Feng (saxon_leo) - Thursday, 14 June 2012, 09:08 GMT

thanks for quick reply.

Something really confuse me.
According to top info while connections achieve 310,000, there are almost 3G cached,why OS didn’t use them?
And the server overcommit_memory=0,why someone can invoke oom-killer?

Comment by Klaus Darilion (klaus3000) - Thursday, 14 June 2012, 10:17 GMT

See http://www.win.tue.nl/~aeb/linux/lk/lk-9.html section “9.6 Overcommit and OOM”

Since 2.5.30 the values are: 0 (default): as before: guess about how much overcommitment is reasonable, 1: never refuse any malloc(), 2: be precise about the overcommit - never commit a virtual address space larger than swap space plus a fraction overcommit_ratio of the physical memory. Here /proc/sys/vm/overcommit_ratio (by default 50) is another user-settable parameter. It is possible to set overcommit_ratio to values larger than 100. (See also Documentation/vm/overcommit-accounting.)

Thus, you have to set it to “2”, any maybe also tweak /proc/sys/vm/overcommit_ratio

Comment by Andrei Pelinescu-Onciul (andrei) - Thursday, 14 June 2012, 10:46 GMT

You could try also forcing kamailio/ser to pre-fault all the shm memory pages on startup and to lock the pages in memory (shm_force_alloc = yes and mlock_pages = yes in the .cfg). This way you would see on startup if there is enough memory for your shared memory requirements and you will make also all the shared memory unswapable (better performance if it’s a dedicated k machine).

Comment by Shaobin.Feng (saxon_leo) - Monday, 18 June 2012, 01:17 GMT

Well,come again.

first of all ,thank you guys for your attention.

1,for suggestion 1(Klaus Darilion)
===========================================================result of {echo “2” > /proc/sys/vm/overcommit_memory}
kamailio couldn’t start as normal
===============================log info
/usr/local/sbin/kamailio[1018]: ERROR: <core> [tcp_main.c:4822]: ERROR: tcp_main: fork failed: Cannot allocate memory
===============================info
the config just as above

2,for suggestion 2(Andrei Pelinescu-Onciul)
===========================================================result of {shm_force_alloc = yes and mlock_pages = yes in the .cfg}
a,the connection accept very slow, almost 10~30/s at the very beginning
b,the server could not be sshed
c,also crash as above when connection achieve almost 90,000 connections

3,I use kamailio as sip-server with almost 100,000 users(via tls) per machine.
And the num of users grow up every day.
So,any idea of this?

Comment by Shaobin.Feng (saxon_leo) - Thursday, 28 June 2012, 03:04 GMT

kamailio-3.3.0 could handle this.
Well, when machine handled almost 30,000 connections, kamailio process sip message very slowly(in local net,almost delay 10s).
Anyway,kamailio not crash,nice!

Comment by Daniel-Constantin Mierla (miconda) - Tuesday, 02 October 2012, 18:40 GMT

Can you use benchmark module to detect what parts are slow? Is the config processing or before it?

Loading...