Delays loading the leads to hopper

All installation and configuration problems and questions

Moderators: gerski, enjay, williamconley, Op3r, Staydog, gardo, mflorell, MJCoate, mcargile, Kumba, Michael_N

Delays loading the leads to hopper

Postby jessiekidfernando » Thu Apr 25, 2019 12:14 am

Hello Everyone,

I've been seeing delays loading the leads to hopper every time this script "/usr/share/astguiclient/AST_VDhopper.pl" runs via cron every minute. When I tried to use --debug, it appears that there are a lot of "Doing DNC Check: XXXXXXXX -Y" and took almost more than 5 minutes to finish the script.

Do you know the reason and cause of this?

Thanks.
OS: Linux version 4.4.155-68-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) )
VERSION: 2.14-717a
BUILD: 190724-1603
Asterisk: 13.21.1-vici
Dahdi: 2.11.1
jessiekidfernando
 
Posts: 152
Joined: Fri Feb 08, 2019 5:49 pm

Re: Delays loading the leads to hopper

Postby alo » Thu Apr 25, 2019 12:14 pm

How Many records to you have in your vicidial_dnc table?
alo
 
Posts: 187
Joined: Wed Jun 20, 2012 10:21 am

Re: Delays loading the leads to hopper

Postby jessiekidfernando » Fri Apr 26, 2019 9:42 am

5million records. The weird thing here is it is happening only on a specific time.
OS: Linux version 4.4.155-68-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) )
VERSION: 2.14-717a
BUILD: 190724-1603
Asterisk: 13.21.1-vici
Dahdi: 2.11.1
jessiekidfernando
 
Posts: 152
Joined: Fri Feb 08, 2019 5:49 pm

Re: Delays loading the leads to hopper

Postby williamconley » Fri Apr 26, 2019 10:44 am

DNC should never be that large. These are all records of people you've called and they have said "do not call me!"? or did you download a DNC record from somewhere and toss it in your dNC table?
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20018
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: Delays loading the leads to hopper

Postby jessiekidfernando » Fri Apr 26, 2019 10:45 am

did you download a DNC record from somewhere and toss it in your dNC table? - Yes
OS: Linux version 4.4.155-68-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) )
VERSION: 2.14-717a
BUILD: 190724-1603
Asterisk: 13.21.1-vici
Dahdi: 2.11.1
jessiekidfernando
 
Posts: 152
Joined: Fri Feb 08, 2019 5:49 pm

Re: Delays loading the leads to hopper

Postby williamconley » Fri Apr 26, 2019 12:05 pm

That's why it is overloading. DNC lists from a government should be used to remove leads without loading them into Vicidial at all. Or to (nightly, weekly or monthly) compare to the lists in Vicidial and remove them.

The Vicidial "Company" or "Internal" DNC list is meant ONLY for people who have told your company to never call them again. It needs to be up to the second. This way if someone tells you at 2PM to NEVER F***ing call them again, at 4PM your dialer will NOT load them into the hopper and call them again.

This is not the same relationship as someone who went to a government site and put their name no a list for all call centers to not call them. The government gives you some leeway to enable that list and stop calling those members of the public. Usually 30 days.

If you have 30 days to remove someone from your lists, you can certainly perform a nightly check to get rid of them, not nearly as "intense" as "Yes, ma'am, we'll stop calling you ... yes, ma'am we understand that if we call you again you'll sue us, no ma'am we will definitly not be calling you again ... sorry again, ma'am." This relationship is special and has significant danger to your livelihood if not handled To The Second, which is why Vicidial has an Internal DNC list.

To meet this need, and stop call centers from being shut down due to this ongoing danger, the Vicidial Internal DNC system checks leads when are about to be dialed and REFUSES to dial them if the campaign is configured to adhere to the list. This means that each time Vicidial is about to dial a number, it checks against the internal DNC list. Every Time It Dials. The entire reason Vicidial has a "hopper" is to avoid the "entire" vicidial list each time it has to dial. The hopper is rarely more than a couple thousand numbers, and often has a system max capacity near 10k. Comparing 10k numbers to 5k DNS entries is a bit intense, but not too horrible. But when you must compare even 2K numbers against 5M ... that's gonna take some time. Every Time It Dials.

IMHO: You are trying to use a moped on the autobahn to carry several hundred passengers from A to B, when you SHOULD be putting those passengers on a Train instead.

Trying to load 5M leads into the DNC is an overload situation. If you MUST have 5M records in the Internal DNC, you will need a Very Powerful Database Server just to dial normally. The cost of that server will be much more than paying for a script to be built to remove National Do Not Call numbers from your system nightly.
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20018
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: Delays loading the leads to hopper

Postby nikolaos » Thu May 09, 2019 7:15 am

And what is a good (i.e. automated) way to delete entries from a list already uploaded to vicidial? I could query and update the database directly but I would like to know what table(s) to work on and if there are precautions I should take (e.g. stop some services first do my job then start them)
nikolaos
 
Posts: 7
Joined: Tue Apr 09, 2019 6:37 am

Re: Delays loading the leads to hopper

Postby jessiekidfernando » Mon May 13, 2019 10:31 am

Any idea why it is taking too long to finish the hopper script? This is the last line that I got when I tried to execute it using the argument --debug.

Please see image below.

https://snag.gy/xGm9TK.jpg

We are executing this script every minute.

/usr/share/astguiclient/AST_VDhopper.pl -q --run-check
OS: Linux version 4.4.155-68-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) )
VERSION: 2.14-717a
BUILD: 190724-1603
Asterisk: 13.21.1-vici
Dahdi: 2.11.1
jessiekidfernando
 
Posts: 152
Joined: Fri Feb 08, 2019 5:49 pm

Re: Delays loading the leads to hopper

Postby williamconley » Thu May 16, 2019 1:11 pm

run each of the queries yourself to see how long they take. they should take no more than one or two seconds.

note that a lot of callbacks and seriously slow down the system. call 'em or delete 'em.
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20018
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: Delays loading the leads to hopper

Postby nikolaos » Sun May 19, 2019 2:17 pm

williamconley wrote:DNC lists from a government should be used to remove leads without loading them into Vicidial at all. Or to (nightly, weekly or monthly) compare to the lists in Vicidial and remove them.


I would like to do both but I'm not sure how to do the later (remove entries from list already loaded to Vicidial).

As far as I can tell from a quick check all phone numbers end up in mysql's table vicidial_list. My first thought is to delete entries from vicidial_list and reboot. Is this OK or am I going to mess up vicidials internals?
nikolaos
 
Posts: 7
Joined: Tue Apr 09, 2019 6:37 am

Re: Delays loading the leads to hopper

Postby williamconley » Sun May 19, 2019 2:35 pm

nikolaos wrote:
williamconley wrote:DNC lists from a government should be used to remove leads without loading them into Vicidial at all. Or to (nightly, weekly or monthly) compare to the lists in Vicidial and remove them.


I would like to do both but I'm not sure how to do the later (remove entries from list already loaded to Vicidial).

As far as I can tell from a quick check all phone numbers end up in mysql's table vicidial_list. My first thought is to delete entries from vicidial_list and reboot. Is this OK or am I going to mess up vicidials internals?


reboot is not needed. but remove them from the hopper table as well.

poundteam has a nightly ndnc checker, and that's what it does. but no deletion: merely changes the status from whatever it was to NDNC. Don't put NDNC in your dial status list and you're good to go. Also, it will not allow a fresh copy of the same lead to be re-uploaded if you're using dupe check on upload.
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20018
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: Delays loading the leads to hopper

Postby nikolaos » Mon May 20, 2019 3:08 am

no deletion [is required]: merely change the status from whatever it was to NDNC. Don't put NDNC in your dial status list and you're good to go. Also, it will not allow a fresh copy of the same lead to be re-uploaded if you're using dupe check on upload.


Thanks for the guidance williamconley! Just to make sure I'm following you:

NDNC is a custom status and by default leads with this status will not be called. If one is foolish enough to go to Campaigns > {some campaign} > Add A Dial Status to Call: > [select NDNC] and click ADD it would override this sane default.

So basically all I have to do to clean up all my lists from a number that is in the National DNC list is a

Code: Select all
UPDATE vicidial_list SET status='NDNC' WHERE status<>'NDNC' AND phone_number="1234567890"


Am I correct?
nikolaos
 
Posts: 7
Joined: Tue Apr 09, 2019 6:37 am

Re: Delays loading the leads to hopper

Postby williamconley » Mon May 20, 2019 10:11 am

Code: Select all
UPDATE vicidial_list SET status='NDNC' WHERE phone_number="1234567890"


Code: Select all
delete from vicidial_hopper where lead_id IN (select lead_id from vicidial_list where status='NDNC')
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20018
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: Delays loading the leads to hopper

Postby nikolaos » Tue May 21, 2019 9:22 am

Thanks williamconley!
nikolaos
 
Posts: 7
Joined: Tue Apr 09, 2019 6:37 am


Return to Support

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 94 guests