segfault in res_rtp_asterisk.so

[rustynail]

ViciBox v.7.0.4-170113
SVN 2796
Asterisk 11.25.1-vici

Hi there,

We are using PBXWebphone and its working great. But there is an annoying issue that I assume only (or mostly) occurs on servers which use webphone. I know this error might have been posted already but there seems no solution in old posts or those didn't worked for me. Here is the complete error message:

Aug 21 07:26:51 server1 kernel: asterisk[17451]: segfault at 3c ip 00007f97b87c86ed sp 00007f9765ae4f20 error 4 in res_rtp_asterisk.so[7f97b87a7000+51000]

I have found below debug info by using gdb with asterisk's core dump file. Can you please guide on it or this issue doesn't belong here and should be sent to "asterisk issue tracker" ?

Thanks.

GDB DEBUG:

server1:/home # gdb asterisk core
GNU gdb (GDB; openSUSE Leap 42.1) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from asterisk...(no debugging symbols found)...done.
[New LWP 17451]
[New LWP 1278]
[New LWP 1281]
[New LWP 1337]
[New LWP 1324]
[New LWP 1338]
[New LWP 1341]
[New LWP 2131]
[New LWP 1350]
[New LWP 2137]
[New LWP 1386]
[New LWP 2199]
[New LWP 1359]
[New LWP 1394]
[New LWP 2201]
[New LWP 1284]
[New LWP 1401]
[New LWP 1363]
[New LWP 1406]
[New LWP 1367]
[New LWP 1412]
[New LWP 1373]
[New LWP 1285]
[New LWP 1422]
[New LWP 1374]
[New LWP 1327]
[New LWP 1432]
[New LWP 1331]
[New LWP 1378]
[New LWP 1435]
[New LWP 1349]
[New LWP 1417]
[New LWP 1461]
[New LWP 1352]
[New LWP 1437]
[New LWP 1463]
[New LWP 1358]
[New LWP 1446]
[New LWP 1572]
[New LWP 1385]
[New LWP 1578]
[New LWP 1447]
[New LWP 27293]
[New LWP 1392]
[New LWP 1580]
[New LWP 1563]
[New LWP 1395]
[New LWP 1593]
[New LWP 2200]
[New LWP 1565]
[New LWP 1400]
[New LWP 1614]
[New LWP 1582]
[New LWP 2190]
[New LWP 1427]
[New LWP 1628]
[New LWP 1594]
[New LWP 1436]
[New LWP 1638]
[New LWP 2189]
[New LWP 1445]
[New LWP 1595]
[New LWP 1645]
[New LWP 2188]
[New LWP 1451]
[New LWP 1607]
[New LWP 1671]
[New LWP 2187]
[New LWP 1466]
[New LWP 1629]
[New LWP 1672]
[New LWP 1558]
[New LWP 1684]
[New LWP 1648]
[New LWP 1586]
[New LWP 2185]
[New LWP 1656]
[New LWP 1706]
[New LWP 1598]
[New LWP 1661]
[New LWP 1736]
[New LWP 2178]
[New LWP 1603]
[New LWP 1667]
[New LWP 1750]
[New LWP 1610]
[New LWP 2175]
[New LWP 1678]
[New LWP 1619]
[New LWP 2121]
[New LWP 1679]
[New LWP 2174]
[New LWP 1634]
[New LWP 2126]
[New LWP 1687]
[New LWP 2168]
[New LWP 1635]
[New LWP 1692]
[New LWP 2130]
[New LWP 1646]
[New LWP 2160]
[New LWP 1693]
[New LWP 2136]
[New LWP 1659]
[New LWP 1699]
[New LWP 2145]
[New LWP 2158]
[New LWP 1660]
[New LWP 1700]
[New LWP 2146]
[New LWP 1705]
[New LWP 2157]
[New LWP 1708]
[New LWP 2150]
[New LWP 1735]
[New LWP 2156]
[New LWP 1713]
[New LWP 2155]
[New LWP 1743]
[New LWP 1714]
[New LWP 2159]
[New LWP 1744]
[New LWP 2154]
[New LWP 1721]
[New LWP 2162]
[New LWP 1751]
[New LWP 1722]
[New LWP 2163]
[New LWP 2149]
[New LWP 2143]
[New LWP 1729]
[New LWP 2165]
[New LWP 2151]
[New LWP 2148]
[New LWP 1730]
[New LWP 2166]
[New LWP 2152]
[New LWP 1738]
[New LWP 2169]
[New LWP 2153]
[New LWP 2147]
[New LWP 1755]
[New LWP 2172]
[New LWP 2161]
[New LWP 2144]
[New LWP 1756]
[New LWP 2173]
[New LWP 2177]
[New LWP 2164]
[New LWP 2142]
[New LWP 2114]
[New LWP 2181]
[New LWP 2167]
[New LWP 2125]
[New LWP 2182]
[New LWP 2170]
[New LWP 2141]
[New LWP 2132]
[New LWP 2184]
[New LWP 2171]
[New LWP 2140]
[New LWP 2138]
[New LWP 2176]
[New LWP 2186]
[New LWP 2139]
[New LWP 2179]
[New LWP 2191]
[New LWP 1203]
[New LWP 2180]
[New LWP 2193]
[New LWP 1269]
[New LWP 2183]
[New LWP 1048]
[New LWP 2195]
[New LWP 1762]
[New LWP 2192]
[New LWP 2356]
[New LWP 2194]
[New LWP 2420]
[New LWP 2197]
[New LWP 27170]
[New LWP 17433]
[New LWP 2198]
[New LWP 1316]
[New LWP 1205]
[New LWP 19339]
[New LWP 2202]
[New LWP 2196]
[New LWP 2343]
[New LWP 2120]
[New LWP 1280]
[New LWP 7439]
[New LWP 1268]
[New LWP 1260]
[New LWP 1277]
[New LWP 17490]
[New LWP 1049]
[New LWP 2122]
[New LWP 2135]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/asterisk -vvvvvvvvvvvvvvvvvvvvvgcT'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f97b87c86ed in pj_stun_session_on_rx_pkt () from /usr/lib64/asterisk/modules/res_rtp_asterisk.so
[Current thread is 1 (Thread 0x7f9765aec700 (LWP 17451))]
Missing separate debuginfos, use: zypper install asterisk-debuginfo-11.25.1-80.4.x86_64
(gdb)
(gdb)
(gdb)
(gdb)
(gdb) bt full
#0 0x00007f97b87c86ed in pj_stun_session_on_rx_pkt () from /usr/lib64/asterisk/modules/res_rtp_asterisk.so
No symbol table info available.
#1 0x00007f97b87c3da9 in pj_ice_sess_on_rx_pkt () from /usr/lib64/asterisk/modules/res_rtp_asterisk.so
No symbol table info available.
#2 0x00007f97b87b93c9 in __rtp_recvfrom.constprop.28 () from /usr/lib64/asterisk/modules/res_rtp_asterisk.so
No symbol table info available.
#3 0x00007f97b87bc8dc in ast_rtp_read () from /usr/lib64/asterisk/modules/res_rtp_asterisk.so
No symbol table info available.
#4 0x00007f97afca6ef6 in sip_read () from /usr/lib64/asterisk/modules/chan_sip.so
No symbol table info available.
#5 0x0000000000492276 in __ast_read.part.12 ()
No symbol table info available.
#6 0x00007f97a850b9d5 in conf_run () from /usr/lib64/asterisk/modules/app_meetme.so
No symbol table info available.
#7 0x00007f97a8511f23 in conf_exec () from /usr/lib64/asterisk/modules/app_meetme.so
No symbol table info available.
#8 0x000000000053c40a in pbx_exec ()
No symbol table info available.
#9 0x0000000000547428 in pbx_extension_helper.constprop ()
No symbol table info available.
#10 0x000000000054d26a in __ast_pbx_run ()
No symbol table info available.
#11 0x000000000054e88b in pbx_thread ()
No symbol table info available.
#12 0x0000000000596a4a in dummy_start ()
No symbol table info available.
#13 0x00007f97fd21c0a4 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#14 0x00007f97fe6e402d in clone () from /lib64/libc.so.6
No symbol table info available.
(gdb)

[ambiorixg12]

I suggest you 2 things create this tread on the Asterisk forum and verify the Asterisk version you have because if you re using version 13, as it is a Long Term Support (LTS) release will be fully supported for 4 years, with one additional year of maintenance for security fixes, so based on the release time of Asterisk 13 I guess it is only receiving security fixes only

[williamconley]

Common problems which often do not manifest themselves in an easily recognized manner:

Out of RAM
Out of sockets

If the failure is occurring in RTP, it could just be that you've somehow run out of network ports in the range available for this application. Or even out of memory when trying to grab a socket. Check your syslogs for any errors in the same time-span.

[rustynail]

I suggest you 2 things create this tread on the Asterisk forum and verify the Asterisk version you have because if you re using version 13, as it is a Long Term Support (LTS) release will be fully supported for 4 years, with one additional year of maintenance for security fixes, so based on the release time of Asterisk 13 I guess it is only receiving security fixes only

Thanks ambiorixg12. I am using vicibox 7 with asterisk-11.25.1-vici. Sure I will also post on asterisk forum but posted here to see if the postfix "-vici" means this version of asterisk was compiled/added in ISO with some customization from vici team.

[rustynail]

Common problems which often do not manifest themselves in an easily recognized manner:

Out of RAM
Out of sockets

If the failure is occurring in RTP, it could just be that you've somehow run out of network ports in the range available for this application. Or even out of memory when trying to grab a socket. Check your syslogs for any errors in the same time-span.

Thanks for your reply William, always being helpful, I will specially pay attention to these 2 things in future, however

1. I doubt if RAM was consumed as it has 8GM of RAM and there was no sign of swap space being utilized.
2. For socket/network ports I will have to dig bit more into it.

I have set ulimit to 65535 for root, mysql and asterisk in "/etc/security/limits.conf" and opened udp ports 1025 to 65535 as our switch provider is sending traffic in this range.

journalctl doesn't show any other specific error except the segmentation fault stating in res_rtp_asterisk.so.

Cheers!

[williamconley]

Is your system whitelist locked? If not, you could very well be experiencing an attack. It's a very common vector to try to crash "anything" on a server if possible. Any time a crash occurs, rules are broken, and an exploit is born (i think a little bell goes off somewhere, too).

[rustynail]

Hi William,

Sorry for late reply as have been too busy with some stuff. Yes you are right, the servers are IP whitelisted hence setting default INPUT policy set to DROP and explicitly adding rule to whitelist custom ports for client IPs. Usually we whitelist 80, 443, 8088, 8089 for TCP and 1025 - 65535 for UDP.

Thanks.

[williamconley]

Usually we whitelist 80, 443, 8088, 8089 for TCP and 1025 - 65535 for UDP

I'm not sure what you just said there. But if you mean that you ACCEPT the above ports, that is the opposite of whitelist. To whitelist: You close ALL ports and only ACCEPT traffic from allowed IPs. (Sending/Source IPs, not Receiving/Target ports)

If you open the aforementioned ports, anyone on the internet can (and will) continually attack your server to try to gain control of your server or register a SIP account and make phone calls using your carriers.

Whitelist blocks those attacks as nobody has access to guess a password unless you have added their IP to your firewall as an Authorized IP.

[rustynail]

Hi William,

Sorry if I was unable to explain it properly but I mean I only allow those ports from white listed IPs. This was the white listed IPs are only able to connect through those ports and they can't use ssh and other ports. And due to DROP default policy all other IPs are totally denied of any traffic.

Below is ulimit

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 64093
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 64093
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


and below is setting of limits.conf

root soft nofile 65535
root hard nofile 65535
mysql soft nofile 65535
mysql hard nofile 65535
asterisk soft nofile 65535
asterisk hard nofile 65535


Today saw a different error for segfault:

Sep 18 11:55:14 kernel: traps: asterisk[14548] general protection ip:7f2f20c2a6ed sp:7f2ece0dec60 error:0 in res_rtp_asterisk.so[7f2f20c09000+51000]

Cheers!

[rustynail]

We are facing this issue many times on daily basis and by looking at system resources overall all looks withing limits, no swap used, no iw wait, no cpu usage and no socket errors etc... Clients are really frustrated and so do we but there seems no clue. All threads/posts here and on asterisk forums do not seem to help.

I am at the stage where I will soon start pulling my hair

Can you please point me to right direction for further troubleshooting. Main issue is that there is no source available as its ISO so I can't look into the source files or recompile with flags modified. Or can I get same version of asterisk and try to recompile its module on same machine without breaking its functionality?

Thanks.

[rustynail]

Wow, no one to reply here

[mflorell]

Vicibox 7 is currently unsupported and unmaintainable. I would suggest installing Vicibox 9 instead if you are having problems.

[rustynail]

Hi Matt,

Thanks for replying. Actually what I have noticed is that we have faced same issue on vici 8 and likely will be facing on vici 9 because this issue appears only with webphone (we haven't yet faced it with soft phones).
As from the logs we can see the issue starts once there is something happening with received packet in stun which is called when ICE is enabled and res_rtp_asterisk.so calls the stun functions.

We are currently in the phase of testing vici 9 and will report if we face same issue there.

Cheers!

[chornyi_taras]

Check this
viewtopic.php?f=8&t=37706#p130243

[rustynail]

Check this
http://www.vicidial.org/VICIDIALforum/v ... 06#p130243

Thanks for the link chornyi taras, Can you please further guide how can I apply this patch safely as the servers are in production and we are using vicibox image to build servers so no source is available in this case..
any way If I can compile and build .so and then can replace it in asterisk modules

[rustynail]

Hi Guys,

We still experienced this issue on one of our TEL server in cluster installed with vicibox 8.0.1 ISO. Do we still need to look for any patch or was it made part of the installer?

Vicibox-8.0.1
asterisk: 11.25.1-vici
svn: 3727
VERSION: 2.14-885a
BUILD: 230522-0903

Calls were almost 200 at that time and there was no other load observed on server.

Below is the error:

Jun 12 12:36:13 Server-T6 kernel: asterisk[1352]: segfault at 3c ip 00007f7aef06a9ad sp 00007f7911e72c70 error 4 in res_rtp_asterisk.so[7f7aef049000+51000]