Detect Records in Date-Keyed JSON

This example shows how to read a JSON document whose keys are dates, one entry per day, as a stream of records. Because the keys change with every entry, there are no fixed field names for a detector to lock on to, and the standard detection would treat the three dates as three different fields. Calling enableDateTimeNodeNames() on the TreeDetectionStrategy adds a DateTimeNodeNameRule that classifies date and timestamp names as dynamic, so they collapse into one repeating node that becomes the record break. The date itself is kept as the events_id field.

Input JSON file

{
    "calendar": "product-team",
    "events": {
        "2024-01-15": { "title": "Kickoff", "attendees": 12 },
        "2024-01-16": { "title": "Design Review", "attendees": 8 },
        "2024-01-17": { "title": "Retrospective", "attendees": 15 }
    }
}

Java Code Listing

package com.northconcepts.datapipeline.foundations.examples.tree;

import java.io.File;

import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.foundations.pipeline.tree.Tree;
import com.northconcepts.datapipeline.foundations.pipeline.tree.detect.TreeDetectionStrategy;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.json.JsonReader;

public class DetectRecordsInDateKeyedJson {

    public static void main(String[] args) {
        File inputFile = new File("example/data/input/tree/date_keyed_events.json");

        TreeDetectionStrategy strategy = TreeDetectionStrategy.standard(true)
                .enableDateTimeNodeNames();

        Tree tree = Tree.loadJson(inputFile, strategy);

        JsonReader reader = new JsonReader(inputFile);
        tree.getAllFields().forEach(node -> reader.addField(node.getFieldName(), node.getXpathExpression(), node.isCascadeFieldValue()));
        tree.getAllRecordBreaks().forEach(node -> reader.addRecordBreak(node.getXpathExpression()));

        DataWriter writer = StreamWriter.newSystemOutWriter();

        Job.run(reader, writer);
    }

}

Code Walkthrough

1. inputFile points at date_keyed_events.json.
2. TreeDetectionStrategy.standard(true) returns the default detection strategy; the true argument turns on tree optimization, which merges dynamically named nodes into a single wildcard node while the tree is being built. enableDateTimeNodeNames() adds the DateTimeNodeNameRule so the 2024-01-15 style keys count as dynamic names.
3. Tree.loadJson(inputFile, strategy) loads the document into a Tree and runs the detection passes, which mark the collapsed event node as a record break and its leaves as fields.
4. A JsonReader is created on the same file and configured from the tree: tree.getAllFields() supplies the name and XPath expression of every detected field to addField(). The third argument, node.isCascadeFieldValue(), tells the reader to repeat a value that spans several records in each of them. Here that is calendar, which appears once at the top of the document and is copied onto every event.
5. tree.getAllRecordBreaks() supplies the XPath of the record node to addRecordBreak(), so one record is emitted per date.
6. Job.run() streams the records to a StreamWriter on the console.

Console Output

-----------------------------------------------
0 - Record (MODIFIED) {
    0:[calendar]:STRING=[product-team]:String
    1:[events_id]:STRING=[2024-01-15]:String
    2:[title]:STRING=[Kickoff]:String
    3:[attendees]:LONG=[12]:Long
}

-----------------------------------------------
1 - Record (MODIFIED) {
    0:[calendar]:STRING=[product-team]:String
    1:[events_id]:STRING=[2024-01-16]:String
    2:[title]:STRING=[Design Review]:String
    3:[attendees]:LONG=[8]:Long
}

-----------------------------------------------
2 - Record (MODIFIED) {
    0:[calendar]:STRING=[product-team]:String
    1:[events_id]:STRING=[2024-01-17]:String
    2:[title]:STRING=[Retrospective]:String
    3:[attendees]:LONG=[15]:Long
}

-----------------------------------------------
3 records