Question

背景:

I m利用邮政数据库开发一个Django项目。我们还在出现这种情况时使用 mo子,因为我的一些网络搜索提到了这个问题。在网络形式上,Django认为,工作需要大量时间(超过使用者希望等待的时间),因此,我们通过背景呼吁的系统放弃工作。目前正在进行的工作需要能够读写数据库。由于这项工作需要这么长时间,我们利用多处理方法同时管理部分。

问题:

The top level script has a database connection, and when it spawns off child processes, it seems that the parent s connection is available to the children. Then there s an exception about how SET TRANSACTION ISOLATION LEVEL must be called before a query. Research has indicated that this is due to trying to use the same database connection in multiple processes. One thread I found suggested calling connection.close() at the start of the child processes so that Django will automatically create a new connection when it needs one, and therefore each child process will have a unique connection - i.e. not shared. This didn t work for me, as calling connection.close() in the child process caused the parent process to complain that the connection was lost.

Other Findings:

一些 st 我读到的似乎表明,你确实能够这样做,多处理、 mo和詹戈没有很好地发挥作用。这似乎很难相信我的想法。

有些人建议使用芯片,这可能是一个长期的解决办法,但我目前无法在等待一些批准程序的情况下安装雕像,因此现在不能选择。

关于SO和其他地方的几篇关于持续数据库链接的文章,我认为这是一个不同的问题。

另外,还提到了心理学范围2. 集合体和药剂,以及一些关于 b的物品。诚然,我不理解我所阅读的多数内容,但我肯定不会像我所期待的那样跳出。

现行“工作-谈判”:

现在,我回过头来,只是连续地处理事情,但比我这样缓慢。

关于我如何利用多处理平行运作的任何建议? 比如,如果我有父母和两名孩子,他们都与数据库有独立联系,那么事情就会ok,但我似乎无法找到这种行动。

感谢并ry惜时间!

Answer 1

多份处理文件连接各过程,因为它预示了处理过程,因此复制了母过程的所有档案说明。尽管如此,与服务器的连接只是一个文件,你可以在文件/proc/fd/中看到这一链接。可在here查询。

My solution was just simply close db connection just before launching processes, each process recreate connection itself when it will need one (tested in django 1.4):

from django import db
db.connections.close_all()
def db_worker():      
    some_paralell_code()
Process(target = db_worker,args = ())

Pgbouncer/pgrix与多处理意义上的read没有关联。而是解决不按每项要求关闭联系的问题=加快连接高负荷状态。

Update:

彻底消除数据库链接方面的问题,只是把与数据库有关的所有逻辑移至db_worker——我想通过《空白》作为论点...... 更好的想法只是通过女方名单...... 见,QueryDict 和数值_list(id ,ng=True),并且不会忘记将其列入名单! 名单(QueryDict)通过后改为db_worker。由于这一原因,我们没有复制模型数据库链接。

def db_worker(models_ids):        
    obj = PartModelWorkerClass(model_ids) # here You do Model.objects.filter(id__in = model_ids)
    obj.run()


model_ids = Model.objects.all().values_list( id , flat=True)
model_ids = list(model_ids) # cast to list
process_count = 5
delta = (len(model_ids) / process_count) + 1

# do all the db stuff here ...

# here you can close db connection
from django import db
db.connections.close_all()

for it in range(0:process_count):
    Process(target = db_worker,args = (model_ids[it*delta:(it+1)*delta]))

Answer 2

在使用多个数据库时,你应关闭所有链接。

from django import db
for connection_name in db.connections.databases:
    db.connections[connection_name].close()

http://www.ohchr.org。

请采用“@lechup”的提法,以关闭所有联系(但自那以后又添加了这种方法):

from django import db
db.connections.close_all()

Answer 3

对Sharma 3和Django而言,这是对我来说行之有效的:

import multiprocessing
import django
django.setup() # Must call setup

def db_worker():
    for name, info in django.db.connections.databases.items(): # Close the DB connections
        django.db.connection.close()
    # Execute parallel code here

if __name__ ==  __main__ :
    multiprocessing.Process(target=db_worker)

注意:没有django.setup() 我无法做到这一点。我猜想,需要重新开始多处理。

Answer 4

在Django的运行中,我有“封闭的连接”问题。除测试外,还有另一个在试验执行期间有意修改数据库的程序。在每个测试案例中都开始了这一进程。

一种简单的固定办法是继承我从 >Transaction CRICase而不是/Case>。这使得数据库能够实际书写,而另一个过程对数据有最新的看法。

Answer 5

围绕你的问题采取的另一个方式是,利用下列手段,在被点击的进程中启动与数据库的新链接:

from django.db import connection    
connection.connect()

Answer 6

(不是伟大的解决办法,而是可能的工作方式)

如果你能够使用餐厅,你或许可以实施你自己的托儿系统,基本上把任务放在某个任务桌上,并有一个固定的跳板,让他们走下去和工作? (通过管理指挥)

Answer 7

我参加了这一问题,并能够通过以下方式解决这一问题(我们正在执行有限的任务制度)。

task.py

from django.db import connection

def as_task(fn):
    """  this is a decorator that handles task duties, like setting up loggers, reporting on status...etc """ 
    connection.close()  #  this is where i kill the database connection VERY IMPORTANT
    # This will force django to open a new unique connection, since on linux at least
    # Connections do not fare well when forked 
    #...etc

ScheduledJob.py

from django.db import connection

def run_task(request, job_id):
    """ Just a simple view that when hit with a specific job id kicks of said job """ 
    # your logic goes here
    # ...
    processor = multiprocessing.Queue()
    multiprocessing.Process(
        target=call_command,  # all of our tasks are setup as management commands in django
        args=[
            job_info.management_command,
        ],
        kwargs= {
             web_processor : processor,
        }.items() + vars(options).items()).start()

result = processor.get(timeout=10)  # wait to get a response on a successful init
# Result is a tuple of [TRUE|FALSE,<ErrorMessage>]
if not result[0]:
    raise Exception(result[1])
else:
   # THE VERY VERY IMPORTANT PART HERE, notice that up to this point we haven t touched the db again, but now we absolutely have to call connection.close()
   connection.close()
   # we do some database accessing here to get the most recently updated job id in the database

很糟糕的是,为了防止种族条件(同时有多个用户),最好在你召集这一进程之后尽快使用数据库。还有一个机会是,另一个用户在一线下完全向 d提出请求,然后有机会绕过数据库。

坦率地说,如果你不直接叫上指挥,那么就很可能是<>>>。

Answer 8


如果大家需要I/O平行,而不是处理平行问题,那么你可以通过把工作转而讨论来避免这一问题。 替换

from multiprocessing import Process


iii

from threading import Thread

www.un.org/chinese/sc/presidency.asp Procsess

Answer 9

如果你也利用联通,以下人员对我们工作,在被 for后强行关闭连接点。以前似乎没有帮助。

from django.db import connections
from django.db.utils import DEFAULT_DB_ALIAS

connections[DEFAULT_DB_ALIAS].dispose()

Answer 10

一种可能性是使用多处理spawn。儿童加工方法不会复制django s DB与儿童过程的联系细节。儿童进程需要从头rat锁上锁,但可以自由建立/连接自己的jan沟。

www.un.org/Depts/DGACM/index_spanish.htm 在呼吁守则中:

import multiprocessing
from myworker import work_one_item # <-- Your worker method

...

# Uses connection A
list_of_items = djago_db_call_one()

#  spawn  starts new python processes
with multiprocessing.get_context( spawn ).Pool() as pool:
    # work_one_item will create own DB connection
    parallel_results = pool.map(work_one_item, list_of_items)

# Continues to use connection A
another_db_call(parallel_results)

www.un.org/Depts/DGACM/index_spanish.htm In myworker.py:

import django. # <-
django.setup() # <-- needed if you ll make DB calls in worker

def work_one_item(item):
   try:
      # This will create a new DB connection
      return len(MyDjangoModel.objects.all())

   except Exception as ex:
      return ex

请注意,如果你在试验院内重新使用电话码,则不会向儿童进程宣传鞭).(需要重新应用)。

Answer 11

你们可以在Debian/Ubuntu为邮局提供更多资源:

nano /etc/postgresql/9.4/main/postgresql.conf

用你的话语取代9.4。

Here are some useful lines that should be updated with example values to do so, names speak for themselves :

max_connections=100
shared_buffers = 3000MB
temp_buffers = 800MB
effective_io_concurrency = 300
max_worker_processes = 80

Be careful not to boost too much these parameters as it might lead to errors with Postgre trying to take more ressources than available. Examples above are running fine on a Debian 8GB Ram machine equiped with 4 cores.

Answer 12

穿透镜头,并在深线结束时关闭所有行关。下面为我工作:

class MyThread(Thread):
    def run(self):
        super().run()

        connections.close_all()

def myasync(function):
    def decorator(*args, **kwargs):
        t = MyThread(target=function, args=args, kwargs=kwargs)
        t.daemon = True
        t.start()

    return decorator

如果你需要称职:

@myasync
def async_function():
    ...

task.py

ScheduledJob.py

友情链接