English 中文(简体)
我怎样才能等到 菜菜全部完成 才能继续我的代码?
原标题:How can i wait until all celery tasks are finished before continuing my code
I have a large csv file that I split into a list of chunks of 100000 rows each, pass each chunk to a function to do complex calculations, and append the result in a global_list. when the last chunk is finished, i take the global_list and do some statistic on. How can i ask celery to process all chunks in parallel but to wait until last task/last chunk is finished before executing the function complex_calc on the global_list? Thank you for your help for chunk in global_chunk_list: def func_calc.delay(chunk) #<<<<< use celery tasks complex_calc(global_list) #<<<<< should only start when processing last chunk is finished @celery.task(name= func_calc ) def func_calc(chunk): ... #save chunk in a global list global_list.append(result) def complex_calc(global_list): ...
问题回答
The appropriate method is to use the group primitive and the join method to wait on a set of parallel tasks to finish executing. task_group = group([func_calc.s(chunk) for chunk in global_chunk_list]) result_group = task_group.apply_async() results = result_group.join() # wait for all results See also the example from the docs. (One difference is using join rather than get, which waits for the tasks to finish) See also this answer. >>> from celery import group >>> from tasks import add >>> job = group([ ... add.s(2, 2), ... add.s(4, 4), ... add.s(8, 8), ... add.s(16, 16), ... add.s(32, 32), ... ]) >>> result = job.apply_async() >>> result.ready() # have all subtasks completed? True >>> result.successful() # were all subtasks successful? True >>> result.get() [4, 8, 16, 32, 64] To do this effectively, you ll need to have a result backend configured.




相关问题
Can Django models use MySQL functions?

Is there a way to force Django models to pass a field to a MySQL function every time the model data is read or loaded? To clarify what I mean in SQL, I want the Django model to produce something like ...

An enterprise scheduler for python (like quartz)

I am looking for an enterprise tasks scheduler for python, like quartz is for Java. Requirements: Persistent: if the process restarts or the machine restarts, then all the jobs must stay there and ...

How to remove unique, then duplicate dictionaries in a list?

Given the following list that contains some duplicate and some unique dictionaries, what is the best method to remove unique dictionaries first, then reduce the duplicate dictionaries to single ...

What is suggested seed value to use with random.seed()?

Simple enough question: I m using python random module to generate random integers. I want to know what is the suggested value to use with the random.seed() function? Currently I am letting this ...

How can I make the PyDev editor selectively ignore errors?

I m using PyDev under Eclipse to write some Jython code. I ve got numerous instances where I need to do something like this: import com.work.project.component.client.Interface.ISubInterface as ...

How do I profile `paster serve` s startup time?

Python s paster serve app.ini is taking longer than I would like to be ready for the first request. I know how to profile requests with middleware, but how do I profile the initialization time? I ...

Pragmatically adding give-aways/freebies to an online store

Our business currently has an online store and recently we ve been offering free specials to our customers. Right now, we simply display the special and give the buyer a notice stating we will add the ...

Converting Dictionary to List? [duplicate]

I m trying to convert a Python dictionary into a Python list, in order to perform some calculations. #My dictionary dict = {} dict[ Capital ]="London" dict[ Food ]="Fish&Chips" dict[ 2012 ]="...

热门标签