Vicibox 5.03 server failing to start

[geoff3dmg]

I have 3 telephony servers in my cluster, one is fine, one will occasionally and the other will reliably fail to start up from boot. They appear to be running the vicidial init script before the network is fully up and fail attempting to contact the database server. This obviously leaves the system in an unusable state. Here's an example from the syslog during bootup for my #2 telephony server.

Code: Select all

2014-01-14T09:34:48.641465+00:00 vicidial-2 vicidial[1086]: will skip /usr/sbin/wanrouter (not installed)
2014-01-14T09:34:48.671758+00:00 vicidial-2 vicidial[1086]: Loading DAHDI drivers
2014-01-14T09:34:48.674846+00:00 vicidial-2 vicidial[1086]: Starting kernel module dahdi_dummy
2014-01-14T09:34:51.686021+00:00 vicidial-2 vicidial[1086]: Configuring DAHDI Hardware
2014-01-14T09:34:51.687650+00:00 vicidial-2 vicidial[1086]: DAHDI Tools Version - 2.8.0
2014-01-14T09:34:51.692476+00:00 vicidial-2 vicidial[1086]: DAHDI Version: 2.8.0.1
2014-01-14T09:34:51.693473+00:00 vicidial-2 vicidial[1086]: Echo Canceller(s):
2014-01-14T09:34:51.693963+00:00 vicidial-2 vicidial[1086]: Configuration
2014-01-14T09:34:51.694437+00:00 vicidial-2 vicidial[1086]: ======================
2014-01-14T09:34:51.694896+00:00 vicidial-2 vicidial[1086]: Channel map:
2014-01-14T09:34:51.695426+00:00 vicidial-2 vicidial[1086]: 0 channels to configure.
2014-01-14T09:35:01.698548+00:00 vicidial-2 vicidial[1086]: rolling Asterisk messages log...
2014-01-14T09:35:01.701383+00:00 vicidial-2 vicidial[1086]: rolling Asterisk event log...
2014-01-14T09:35:01.702036+00:00 vicidial-2 vicidial[1086]: rolling Asterisk cdr logs...
2014-01-14T09:35:01.702756+00:00 vicidial-2 vicidial[1086]: rolling Asterisk screen log...
2014-01-14T09:35:01.705161+00:00 vicidial-2 vicidial[1086]: FINISHED... EXITING
2014-01-14T09:35:04.563853+00:00 vicidial-2 vicidial[1086]: DBI connect('asterisk:192.168.0.11:3306','cron',...) failed: Can't connect to MySQL server on '192.168.0.11' (113) at /usr/share/astguiclient/AST_reset_mysql_vars.pl line 65.
2014-01-14T09:35:04.563927+00:00 vicidial-2 vicidial[1086]: Couldn't connect to database: Can't connect to MySQL server on '192.168.0.11' (113) at /usr/share/astguiclient/AST_reset_mysql_vars.pl line 65.
2014-01-14T09:35:06.574736+00:00 vicidial-2 vicidial[1086]: Starting Asterisk Unable to connect to remote asterisk (does /run/asterisk/asterisk.ctl exist?)
2014-01-14T09:35:07.539726+00:00 vicidial-2 vicidial[1086]: DBI connect('asterisk:192.168.0.11:3306','cron',...) failed: Can't connect to MySQL server on '192.168.0.11' (113) at /usr/share/astguiclient/start_asterisk_boot.pl line 105.
2014-01-14T09:35:07.539776+00:00 vicidial-2 vicidial[1086]: Couldn't connect to database: Can't connect to MySQL server on '192.168.0.11' (113) at /usr/share/astguiclient/start_asterisk_boot.pl line 105.
2014-01-14T09:35:07.542111+00:00 vicidial-2 vicidial[1086]: ..failed


I'm assuming there's something I can do to the systemd init scripts to make it wait/depend on the networking. I haven't looked too deeply into a solution yet though. I can manually work round the issue by running 'service vicidial start' after logging in over SSH of course. The servers operate normally once this intervention is made.

[williamconley]

drive full. network cable disconnected. no route to the db server ...?

try to log in to the db server via command line from the dialer. if you can't, neither can vicidial.

Code: Select all

mysql -u cron -p1234 -h 192.168.0.11 asterisk

[geoff3dmg]

drive full.

Code: Select all

vicidial-2:~ # df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G   40K  3.9G   1% /dev
tmpfs           4.0G     0  4.0G   0% /dev/shm
tmpfs           4.0G   27M  3.9G   1% /run
/dev/sda3        60G  2.3G   57G   4% /
tmpfs           4.0G     0  4.0G   0% /sys/fs/cgroup
tmpfs           6.0G     0  6.0G   0% /var/spool/asterisk/monitor
tmpfs           4.0G   27M  3.9G   1% /var/lock
tmpfs           4.0G   27M  3.9G   1% /var/run

network cable disconnected. no route to the db server ...?

Code: Select all

vicidial-2:~ # ping -c3 192.168.0.11
PING 192.168.0.11 (192.168.0.11) 56(84) bytes of data.
64 bytes from 192.168.0.11: icmp_seq=1 ttl=64 time=0.198 ms
64 bytes from 192.168.0.11: icmp_seq=2 ttl=64 time=0.135 ms
64 bytes from 192.168.0.11: icmp_seq=3 ttl=64 time=0.123 ms

--- 192.168.0.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.123/0.152/0.198/0.032 ms

try to log in to the db server via command line from the dialer. if you can't, neither can vicidial.

Code: Select all

mysql -u cron -p1234 -h 192.168.0.11 asterisk

Code: Select all

vicidial-2:~ # mysql -u cron -p1234 -h 192.168.0.11 asterisk
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 39360991
Server version: 5.5.33-MariaDB-log openSUSE package

Copyright (c) 2000, 2013, Oracle, Monty Program Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [asterisk]>

As I said, this only occurs during the boot process, hence blaming systemd dependancy issues.

[williamconley]

if your DB server is still booting when your Dialer tries to start ... it will fail. if you daily reboot (good idea), make sure the dialers reboot 5 minutes AFTER the DB server.

[geoff3dmg]

I do not reboot my DB server other than after SVN upgrades. It was online the whole time during the bootup process of the telephony server. I rebooted the telephoney server manually during working hours (as it fails on scheduled reboots via cron) due to the manual interventions requried to get it working.

[DomeDan]

Sounds like your network ain't up when vicidial starts up.
it should be because of the service settings:
/etc/rc.d/vicidial

Code: Select all

### BEGIN INIT INFO
# Provides:          Vicidial
# Required-Start:    $syslog $local_fs $network
# Should-Start:      $time mysql
# Required-Stop:     $syslog $local_fs $network
# Should-Stop:       mysql
# Default-Start:     3 4 5
# Default-Stop:      0 1 2 6
# Short-Description: Vicidial Call Center Suit


things to try:
ping the problem-server from the database server and boot up the problem-server,
watch when you start to get replies and also look where in the boot process the problem-server is.
does it begin to reply to ping after the warnings then you got a slow starting network interface.

then you can try to change the vicidial service settings /etc/rc.d/vicidial
make a backup of the file first just in case then try to set:
# Required-Start: $ALL
this should make the vicidial service start last, after all other services has started.

maybe you got a bad nic, maybe some bad capacitors, I've had a bad capacitor in a DELL server once

[williamconley]

I do not reboot my DB server other than after SVN upgrades. It was online the whole time during the bootup process of the telephony server. I rebooted the telephoney server manually during working hours (as it fails on scheduled reboots via cron) due to the manual interventions requried to get it working.

It sounds to me like you should wipe the Dialers clean and start over. If you are following the rules and NOT storing any data on the dialers (recordings pushed to the archive server and db on the DB server ...), then the dialers are actually disposable. Get a good clean install of the exact same Vicidial version using the Vicibox installer and you could suddenly have very little or no more worries.

Personally, I'd rebuild the entire system to be sure it was clean (but we install Vicidial every day and have No Fear of the install thanks to Kumba's most excellent installation media!)

[geoff3dmg]

ping the problem-server from the database server and boot up the problem-server,
watch when you start to get replies and also look where in the boot process the problem-server is.
does it begin to reply to ping after the warnings then you got a slow starting network interface.

This would appear to be the case, the systems networking takes a few seconds after the local console is avalible to be used, this is obviously long past when the network 'started' and the init scripts finished running.

then you can try to change the vicidial service settings /etc/rc.d/vicidial
make a backup of the file first just in case then try to set:
# Required-Start: $ALL
this should make the vicidial service start last, after all other services has started.

This made no difference, what did fix it was putting a 'sleep 10' in after line 103 of the init script.

It sounds to me like you should wipe the Dialers clean and start over. If you are following the rules and NOT storing any data on the dialers (recordings pushed to the archive server and db on the DB server ...), then the dialers are actually disposable. Get a good clean install of the exact same Vicidial version using the Vicibox installer and you could suddenly have very little or no more worries.

Personally, I'd rebuild the entire system to be sure it was clean (but we install Vicidial every day and have No Fear of the install thanks to Kumba's most excellent installation media!)

The #2 was formatted and rebuilt with the vicibox 5.03 AMD64 iso on monday, #3 was formatted and built the same way on the 2nd of Jan. The #2 machine was previously running Vicibox 4 as a standalone system for 12 months without incident. #3 was running Vicibox 3 standalone for over 18months without inicident.

I don't do much to a telephony server after it's built. change the NTP upstream servers to UK ones, install and configure fail2ban and alter the asterisk manager settings to be more secure.

[DomeDan]

Good to hear you found the issue!

What would be better is to put a sleep in the service that enables the interface, I think its this one: /etc/rc.d/network
So all other services that depends on network will work from start like NTP.

But the best would be to find the source of the problem, pop the hood of the server and look for bad capacitors
if this is the case then consider replacing the hardware.
But if this problem happen to two different servers and only applies to vicibox 5.0.3 then there's probably a bug located deeper into OpenSuSE