CoreBOSBB
Batch Import issue - Printable Version

+- CoreBOSBB (https://discussions.corebos.org)
+-- Forum: Support (https://discussions.corebos.org/forumdisplay.php?fid=17)
+--- Forum: Administrator Support (https://discussions.corebos.org/forumdisplay.php?fid=8)
+--- Thread: Batch Import issue (/showthread.php?tid=2060)



Batch Import issue - Svemar - 01-04-2021

Hello,
Tried to import more than 400 products. Batch Import Threshold we have set to 250.
So first 250 records were imported and then process is stopped. Suppose next records should be imported as scheduled import, but even after 20 min number of records are the same.
Tried to do the samo on corebos demo and have same results - only 250records are added.
Any ideas?

Thanks


RE: Batch Import issue - Svemar - 01-04-2021

Looks like other (above 250 treshlold) are still pending (after more than 2 hours)

Import - Error:
There are still some unimported records in your import queue, blocking you from importing more data.
Clear data to clean it up and start with fresh import


RE: Batch Import issue - radu - 01-04-2021

Do you have the crontab configured to execute the corebos workflow script?


RE: Batch Import issue - Svemar - 01-04-2021

On the local machine crontab is configured, other sheduled coebos jobs are running as expected. I suppose crontab is also configured on the Corebos demo site (I have tried to import the same product list here too)


RE: Batch Import issue - radu - 01-04-2021

I see. Then is scheduled import task active in the scheduler - and with a 15-20 min interval?I'm not well versed with importing maybe Joe can help. However I raised the limit in our installation to 1000 and did not feel too much pressure(it executes a import in of ~800-900 products) about 2-3 minutes so not a lot of impact... specially to avoid the fragmentation of import. This would be a workaround, not necessarily recommended approach but depending on what you import and it's impact it may be feasible to just raise the limit. I would also test two batches of import(one of <=250 and the other one with the rest) to be sure you don't have any wrong data on some lines(it happened to me with errors in the csv). Lastly, it might be something else or something buggy in the app. enabling and examining corebos's log and apache's(or whatever webserver you have) log plus system logs referring to what crontab is spitting out might help to track down the issue.


RE: Batch Import issue - Svemar - 01-04-2021

Scheduled import task is active with the period of 15 min and it runs, but it does not imports any pending podoucts.
I can raise the batch import limit, but what to do if I want to import (actualy only update) 15k of products? For this I suppose we need to have batch import working.
Will wait for Joe's time window :-)


RE: Batch Import issue - tebajanga - 01-05-2021

(01-04-2021, 12:20 PM)Svemar Wrote: Hello,
Tried to import more than 400 products. Batch Import Threshold we have set to 250.
So first 250 records were imported and then process is stopped. Suppose next records should be imported as scheduled import, but even after 20 min number of records are the same.
Tried to do the samo on corebos demo and have same results - only 250records are added.
Any ideas?

Thanks

Hello,
I tried to import 527 products records and it worked. (See the attached images below)
I had similar issue like this a few months ago and I solved it by increasing server memory_limit and maximum_execution_time.
You can give it a try

[Image: r6NOWGQ]


RE: Batch Import issue - Svemar - 01-05-2021

Hi, not shure for this.
our php limits:
max execution time 600
memory_limit 12800M

In case I would set Batch Import Limit Global variable higher, then everything goes not via Scheduled import, but immidiatly and then everyging ends fine.

Please try to do this on the demo.corebos.com


RE: Batch Import issue - Svemar - 01-07-2021

(01-05-2021, 01:53 PM)Svemar Wrote: Hi, not shure for this.
our php limits:
max execution time 600
memory_limit  12800M

In case I would set Batch Import Limit Global variable higher, then everything goes not via Scheduled import, but immidiatly and then everyging ends fine.

Please try to do this on the demo.corebos.com
Hi,
Today tried to play on the copy of production system.

So I set Batch Import limt GV to 30000 and scheduled import GV to 30000.
Import went OK, but finally there was no UNDO button.