Streaming & Parsing Large JSON Files Without Crashing Memory: Node.js, Python & Web Workers

The Memory Wall: Why JSON.parse() Fails on Large Files

Standard JSON.parse() requires loading the entire JSON string into RAM and constructing a complete in-memory JavaScript object graph. For a 1GB JSON file, the resulting V8 memory footprint frequently expands to 3GB to 5GB due to object wrapper allocations, property descriptors, and hash table pointers. When memory consumption exceeds the default Node.js heap limit (typically 2GB to 4GB), the process terminates with an uncatchable JavaScript heap out of memory crash.

Streaming Architectures: AST vs Token-Based Parsers

1. Streaming Large JSON in Node.js (stream-json)

stream-json allows streaming multi-gigabyte JSON array payloads directly through Node.js pipeline streams with constant memory usage under 30MB:

import fs from 'fs';
import { parser } from 'stream-json';
import { streamArray } from 'stream-json/streamers/StreamArray';

const pipeline = fs.createReadStream('massive_dataset.json')
  .pipe(parser())
  .pipe(streamArray());

let processedCount = 0;

pipeline.on('data', ({ key, value }) => {
  // Process individual object item (e.g. database insertion)
  processedCount++;
  if (processedCount % 10000 === 0) {
    console.log(`Processed ${processedCount} records without memory bloat...`);
  }
});

pipeline.on('end', () => {
  console.log(`Successfully finished streaming ${processedCount} items!`);
});

2. Streaming Large JSON in Python (ijson)

import ijson

def process_large_json(file_path):
    with open(file_path, 'rb') as f:
        # Stream individual items from the root array
        items = ijson.items(f, 'item')
        for item in items:
            user_id = item.get('id')
            email = item.get('email')
            # Perform stream processing with constant O(1) RAM footprint

Preserving 64-Bit Integer & BigInt Precision

JavaScript numbers are IEEE 754 double-precision floats, limiting safe integers to $2^{53} - 1$ (Number.MAX_SAFE_INTEGER = 9,007,199,254,740,991). When parsing 64-bit integer IDs (such as Twitter/Discord Snowflake IDs or database BIGINT primaries), standard JSON.parse() silently truncates the least significant digits. Streaming parsers support custom tokenizers that deserialize large numeric strings directly into native BigInt primitives.