Question

Circumstances

I have a procedure which will construct a matrix using the given list of values! and the list starts growing bigger like 100 thousand or million values in a list, which in turn, will result in million x million size matrix.

在程序方面,根据行文、专栏或仅根据内容,在矩阵上进行一些增/次/半/多式业务。

Issues

由于汇总表如此之大,我不认为在记忆中进行整个操纵将会奏效。

<<>问题>

therefore, my question would be: how should i manipulate this huge matrix and the huge value list? like, where to store it, how to read it etc, so that i could carry out my operations on the matrix and the computer won`t stuck or anything.

Answer 1

First and foremost, such matrix would have 10G elements. Considering that for any useful operation you would then need 30G elements, each taking 4-8 bytes, you cannot assume to do this at all on a 32-bit computer using any sort of in-memory technique. To solve this, I would use a) genuine 64-bit machine, b) memory-mapped binary files for storage, and c) ditch python.

Update

我在下文中计算,如果你有2个投入矩阵和1个产出矩阵,即100 000×100 00032个轨道浮动/净值元素,即120 GB(而不是GiB,但)。 Assume,在家用电脑上,你可以实现100兆赫/秒I/O带宽的固定目标,任何业务,包括附加和减小,都必须使用矩阵的每个单一要素,业务绝对下限为120 GB /(100 MB/s) = 1200秒,或20分钟,用于单一矩阵操作。在C文中,尽可能高效地利用操作系统,豁免适用IO等。对百万人而言,每项行动需要100倍的时间,即1.5天。由于当时硬盘旋,计算机可能完全无法使用。

Answer 2

I suggest using NumPy. It s quite fast on arithmetic operations.

Answer 3

您是否考虑使用字典? 如果矩阵十分稀少,储存该矩阵可能可行

matrix = {
 (101, 10213) : "value1",
 (1099, 78933) : "value2"
}

Answer 4

你的数据结构不可能有阵列,太大。如果矩阵是双位矩阵,那么你就可以把较大的零区块与同一桶相隔,看一下储存的表述。

Update

友情链接