The V8 string size limit

Question

我有JSON文档,其大小数据为914MB。我将文件装上fs-extra并加上。但是,如果一行,就会产生错误。

不能造成超过0x1fffe8级的窒息性。

下面是法典

        const fs = require( fs-extra );
        const rawdata = fs.readFileSync( src/raw.json );
        const data = JSON.parse(rawdata);

我正在以<代码>npm开展这一项目,并运行在<编码> 包装.json以下。

"scripts": {
   
    "start:dev": "cross-env NODE_OPTIONS= --max-old-space-size=4096  ts-node -r tsconfig-paths/register ./src --env=development",
  
  }

Answer 1

`0x1fffffe8` is exactly 512MB.

很多评论家是正确的:你正在放弃制度限制。我同意“点”的说法,即它大多数可能有一个长限:fs-extra与限额无关。

无论如何,你将不得不处理猪肉。以下是这样做的不同方式。

`A:` Use a SAX-style JSON parser

你有许多教区选择。为了让大家开始,我发现在国家预防机制上有一对:

BFJ awalk function that does SAX-based parsing. 该基金已存档,但仍有数百万次每周下载。

`B:` Implement a Node Streams pipeline

Almost certainly your massive JSON data is a array at the root level. This approach uses a parser that can asynchronously process each element in that array individually, or in batches, whichever makes sense. It is based on the very powerful and flexible Node Streams API.

i) 如果你的数据为JSON阵列,而是一种被压缩的JSON物体的流体,那么它可能符合。见以下备选办法:<代码>D。

https://www.npmjs.com/ Package/JSONStream”rel=“noretinger”>JSONStream 允许你在其流层中按行道或规律进行过滤。它存档,但每周仍有数百万次下载。

BFJ——除了支持SAX-ecode外,它还有选择地进行目标流:

<代码>match 回归一个可读、物体-mode的流体,并按时间顺序排列,从投入的JSON流中分离出单个配对项目。
has a Pick pipeline operator that can pick desired items out of a stream, ignoring the rest. Many other options.

jsonparse

`C:` Manual chunking

? 如果你的数据支持,这很可能是最有效的。

这一选择与<条码>B一样,但不是使用一种流层子,而是使用king。如果初专干事数据阵列的要素非常固定,例如每个元件都完全按N行。你们可以很容易地在没有教条的情况下提取。

例如,如果你的数据表明:

{
  data: [
    { name: ...,
      address: ... },
    { name: ...,
      address: ... },
    { name: ...,
      address: ... },
    { name: ...,
      address: ... }
  ]
}

你们的工作就是这样:

Use a buffered reader to read the file. (DO NOT synchronously read it all into memory)
Discard the first two lines
Read the file in chunks, two lines at a time
If a chunk starts with {, remove any trailing comma and parse each individual {name:..., address:...} record.
If it doesn t, you have reached the end of the array. Discard the rest of the file or hand it off to some other process if you expect some other data there.

The details will depend on your data.

`D:` Use a JSON Streaming protocol parser

JSON 简化程序是一流中聚集的多目标。如果是你的话,你就应当使用支持该议定书的教官。

ndjson

Answer 2

The V8 string size limit

import NodeBuffer from "node:buffer";

NodeBuffer.constants.MAX_STRING_LENGTH

Represents the largest length that a string primitive can have, counted in UTF-16 code units.

这一数值可能取决于正在使用的联合材料发动机。

对<条码>载施加的限制并不涉及星数。它的0-x1FFFE8(229-24)UTF-16单位,载于我的Node.js 20.1.0。

Node.js使用V8发动机。 A Javascript string composed of 2-octet (uint16_t) UTF-16代码单位,以及在V8发动机中,这种装置的最大数量(string.length) 是

(2²⁸－16) (0x0FFFFFF0 UTF-16 code units / about 0.5 GiB) on a 32-bit system
and (2²⁹－24) (0x1FFFFFE8 UTF-16 code units / about 1 GiB) on a 64-bit system.

`node/lib/buffer.js`

const constants = ObjectDefineProperties({}, {
  MAX_LENGTH: {
    __proto__: null,
    value: kMaxLength,
    writable: false,
    enumerable: true,
  },
  MAX_STRING_LENGTH: {
    __proto__: null,
    value: kStringMaxLength,
    writable: false,
    enumerable: true,
  },
});

`node/src/node_buffer.cc`

  target
      ->Set(context,
            FIXED_ONE_BYTE_STRING(isolate, "kStringMaxLength"),
            Integer::New(isolate, String::kMaxLength))
      .Check();

`v8/include/v8-primitive.h`

/**
 * A JavaScript string value (ECMA-262, 4.3.17).
 */
class V8_EXPORT String : public Name {
 public:
  static constexpr int kMaxLength =
      internal::kApiSystemPointerSize == 4 ? (1 << 28) - 16 : (1 << 29) - 24;

`v8/include/v8-internal.h`

/**
 * Configuration of tagging scheme.
 */
const int kApiSystemPointerSize = sizeof(void*);

`0x1fffffe8` is exactly 512MB.

`A:` Use a SAX-style JSON parser

`B:` Implement a Node Streams pipeline

`C:` Manual chunking

`D:` Use a JSON Streaming protocol parser

The V8 string size limit

`NodeBuffer.constants.MAX_STRING_LENGTH`

`node/lib/buffer.js`

`node/src/node_buffer.cc`

`v8/include/v8-primitive.h`

`v8/include/v8-internal.h`

友情链接

0x1fffffe8 is exactly 512MB.

A: Use a SAX-style JSON parser

B: Implement a Node Streams pipeline

C: Manual chunking

D: Use a JSON Streaming protocol parser

The V8 string size limit

NodeBuffer.constants.MAX_STRING_LENGTH

node/lib/buffer.js

node/src/node_buffer.cc

v8/include/v8-primitive.h

v8/include/v8-internal.h

友情链接

`0x1fffffe8` is exactly 512MB.

`A:` Use a SAX-style JSON parser

`B:` Implement a Node Streams pipeline

`C:` Manual chunking

`D:` Use a JSON Streaming protocol parser

`NodeBuffer.constants.MAX_STRING_LENGTH`

`node/lib/buffer.js`

`node/src/node_buffer.cc`

`v8/include/v8-primitive.h`

`v8/include/v8-internal.h`