Question

I ve seen a lot of similar questions to this, but nothing that really matched. Most other questions seemed to relate to speed. What I m experiencing is a single json dictionary that sits in a 1.1gig file on my local box taking up all of my 16 gigabytes of memory when I try to load it using anything along the lines of:

f = open(some_file, "rb")
new_dictionary = json.load(f)

不管图书馆一的使用是什么(我曾尝试过ujson、json、 yajl),不管我是否作为一流阅读东西。这对我来说是绝对的。 cra记忆的使用是什么,我是如何接手?

In case it helps, the dictionary is just a bunch of nested dictionaries all having ints point to other ints. A sample looks like:

{"0":{"3":82,"4":503,"15":456},"956":{"56":823,"678":50673,"35":1232}...}

<><>UPDATE: 当我用简单的json来做这件事时,实际上只有8克。没有人想到,为什么这个人比所有其他人少得多。

<><>>> 因此,我做了更多的调查。我用简单的json装上了字典,试图把所有钥匙变成ts(根据利里的建议,扼杀可能占用更多的空间)。空间保持在8克格。然后,我尝试了温斯顿·埃沃特建议,即管理一组。空间仍然保持在8克。最后,我听到了我新的数据结构,撤出了沙里,并重新上载。排位和持位数仍为8克。我猜测斯·沙尔只是想说,大的2名独裁者拥有很大的空间。我相信,如果我用简单的json装上它,那么我至少知道它不是一个“智者”问题。

Answer 1

Gabe really figured this out in a comment, but since it s been a few months and he hasn t posted it as an answer, I figured I should just answer my own question, so posterity sees that there is an answer.

不管怎么说,答案是,只有一位2名独裁者才占用了大片空间。每一名独裁者都用一些空间管理费用开车,由于其中有许多,其气球从1.1格格格到8格格不等,除试图使用不同的数据结构或获得更多的斜线外,别无他用。

Answer 2

You could try with a streaming API:

http://lloyd.github.com/yajl/

其中有一对on。

https://github.com/rtyler/py-yajl/

https://github.com/pykler/yajl-py”rel=“nofollow”>https://github.com/pykler/yajl-py

Answer 3

我的一点实验表明,在json物体被 par平后,打上了<代码>gc.( ),将记忆使用降至当初设计该物体时的位置。

这里是我获得的少量记忆使用结果:

Build. No GC
762912
Build. GC
763000
Standard Json. Unicode Keys. No GC
885216
Standard Json. Unicode Keys. GC
744552
Standard Json. Int Keys. No GC
885216
Standard Json. Int Keys. GC
744724
Simple Json. Unicode Keys. No GC
894352
Simple Json. Unicode Keys. GC
745520
Simple Json. Int Keys. No GC
894352
Simple Json. Int Keys. GC
744884

Basically, running gc.collect() appears to cleanup some sort of garbage producing during the JSON parsing process.

Answer 4

我不相信我会说这句话,但json实际上是一种非常简单的形式,建造你自己的教区将太困难。

尽管如此,只有:

You don t need the full dictionary at the end (i.e., you can consume the data as you read it)
You have a good idea what sort of structure the data is in (an arbitrarily deep dictionary would make this much more difficult)

友情链接