Skip to content

Split a large JSON file and merge many files

Split

Break a big file into several smaller ones:

jsonreshape split data.json --parts 10

This writes ten files with roughly equal numbers of records. Run jsonreshape split --help for splitting by size or by a fixed record count instead.

Merge

Combine many files into one, optionally removing duplicates:

jsonreshape merge "part_*.json" -o all.jsonl --dedupe-by id
  • The glob "part_*.json" matches every part — quote it so the shell does not expand it first.
  • --dedupe-by id keeps a single record per id.

Extract a sub-part

To pull one branch out of a large document:

jsonreshape extract big.json items.json --path data.items
jsonreshape extract data.json orders.jsonl --gather "orders[*]"