Question

我从我的数据库中输出了一些数据,数据形式是JSON, 基本上只是一个[清单],其中有一个星座(900K)(1月)。

现在,我正在生产服务器上进口,但我已搬进一些廉价的网络服务器。当我吃10分钟的所有资源时,他们不喜欢。

How can I split this file into smaller chunks so that I can import it piece by piece?

<><>Edit>: 实际上,它有一个邮政局数据库。我愿意接受其他建议,即我如何用草原出口所有数据。 I ve got phpPgAdmin安装在我的服务器上,据说可以接受CSV、Tabed和XML格式。

I had to fix phihag s script:

import json
with open( fixtures/PostalCodes.json , r ) as infile:
  o = json.load(infile)
  chunkSize = 50000
  for i in xrange(0, len(o), chunkSize):
    with open( fixtures/postalcodes_  + ( %02d  % (i//chunkSize)) +  .json , w ) as outfile:
      json.dump(o[i:i+chunkSize], outfile)

dump:

pg_dump -U username -t table database > filename

恢复:

psql -U username < filename

(我不了解黑板 p子做什么,但却给我留下错误)

The tutorials on this conveniently leave this information out, esp. the -U option which is probably necessary in most circumstances. Yes, the man pages explain this, but it s always a pain to sift through 50 options you don t care about.

I ended up going with Kenny s suggestion... although it was still a major pain. I had to dump the table to a file, compress it, upload it, extract it, then I tried to import it, but the data was slightly different on production and there were some missing foreign keys (postalcodes are attached to cities). Of course, I couldn t just import the new cities, because then it throws a duplicate key error instead of silently ignoring it, which would have been nice. So I had to empty that table, repeat the process for cities, only to realize something else was tied to cities, so I had to empty that table too. Got the cities back in, then finally I could import my postal codes. By now I ve obliterated half my database because everything is tied to everything and I ve had to recreate all the entries. Lovely. Good thing I haven t launched the site yet. Also "emptying" or truncating a table doesn t seem to reset the sequences/autoincrements, which I d like, because there are a couple magic entries I want to have ID 1. So..I d have to delete or reset those too (I don t know how), so I manually edited the PKs for those back to 1.

I would have ran into similar problems with phihag s solution, plus I would have had to import 17 files one at a time, unless I wrote another import script to match the export script. Although he did answer my question literally, so thanks.

Answer 1

假设你可以选择收回和出口数据......

pg_dump - 抽取SQL数据库,输入文字档案或其他档案。

pg_restore -recovery a PostgreSQL database from an file establishing by pg_dump.

如果没有使用,那么了解你在产出方面将做些什么,以便另一个建议能够打上标识,可能是有益的。

Answer 2

扎里

import json
with open( file.json ) as infile:
  o = json.load(infile)
  chunkSize = 1000
  for i in xrange(0, len(o), chunkSize):
    with open( file_  + str(i//chunkSize) +  .json ,  w ) as outfile:
      json.dump(o[i:i+chunkSize], outfile)

Answer 3

我将菲亚克的工作和标志变成了tiny script (gist) 。

还抄录如下:

#!/usr/bin/env python 
# based on  http://stackoverflow.com/questions/7052947/split-95mb-json-array-into-smaller-chunks
# usage: python json-split filename.json
# produces multiple filename_0.json of 1.49 MB size

import json
import sys

with open(sys.argv[1], r ) as infile:
    o = json.load(infile)
    chunkSize = 4550
    for i in xrange(0, len(o), chunkSize):
        with open(sys.argv[1] +  _  + str(i//chunkSize) +  .json ,  w ) as outfile:
            json.dump(o[i:i+chunkSize], outfile)

Answer 4

I know this is question is from a while back, but I think this new solution is hassle-free.

您可使用<条码>pandas 0.21.0<>>>>>,作为<条码>的一部分,支持一个星号参数。您可以随时间装上一只chu子,并拯救 j子:

import pandas as pd
chunks = pd.read_json( file.json , lines=True, chunksize = 20)
for i, c in enumerate(chunks):
    c.to_json( chunk_{}.json .format(i))

友情链接