Use Structure-Based Sibling Collapse

This example shows how to read siblings with unrelated names but the same shape as one record type. The widget, gadget and doohickey objects share no naming pattern, yet each has the same sku, qty and price fields. Calling enableStructureSiblingCollapse() on the TreeDetectionStrategy adds a StructureSiblingSetRule to the sibling collapse pass so that structurally similar siblings collapse regardless of their names. The name is kept as the inventory_id field.

Input JSON file

{
    "inventory": {
        "widget": { "sku": "W-1", "qty": 10, "price": 2.50 },
        "gadget": { "sku": "G-7", "qty": 3, "price": 12.00 },
        "doohickey": { "sku": "D-4", "qty": 8, "price": 5.25 }
    }
}

Java Code Listing

package com.northconcepts.datapipeline.foundations.examples.tree;

import java.io.File;

import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.foundations.pipeline.tree.Tree;
import com.northconcepts.datapipeline.foundations.pipeline.tree.detect.TreeDetectionStrategy;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.json.JsonReader;

public class UseStructureBasedSiblingCollapse {

    public static void main(String[] args) {
        File inputFile = new File("example/data/input/tree/mixed_name_siblings.json");

        TreeDetectionStrategy strategy = TreeDetectionStrategy.standard(true)
                .enableStructureSiblingCollapse();

        Tree tree = Tree.loadJson(inputFile, strategy);

        JsonReader reader = new JsonReader(inputFile);
        tree.getAllFields().forEach(node -> reader.addField(node.getFieldName(), node.getXpathExpression()));
        tree.getAllRecordBreaks().forEach(node -> reader.addRecordBreak(node.getXpathExpression()));

        DataWriter writer = StreamWriter.newSystemOutWriter();

        Job.run(reader, writer);
    }

}

Code Walkthrough

1. inputFile points at mixed_name_siblings.json.
2. TreeDetectionStrategy.standard(true) returns the default detection strategy; the true argument turns on tree optimization, which merges dynamically named nodes into a single wildcard node while the tree is being built. enableStructureSiblingCollapse() adds the structure-based rule to the sibling collapse pass, inserting the pass if it is not present yet.
3. Tree.loadJson(inputFile, strategy) loads the document into a Tree with the three inventory objects merged into one node, which the detection passes mark as the record break.
4. A JsonReader is created on the same file and configured from the tree: tree.getAllFields() supplies the name and XPath expression of every detected field to addField(). inventory_id holds the original key.
5. tree.getAllRecordBreaks() supplies the XPath of the record node to addRecordBreak().
6. Job.run() streams the records to a StreamWriter on the console.

Console Output

-----------------------------------------------
0 - Record (MODIFIED) {
    0:[inventory_id]:STRING=[widget]:String
    1:[sku]:STRING=[W-1]:String
    2:[qty]:LONG=[10]:Long
    3:[price]:DOUBLE=[2.5]:Double
}

-----------------------------------------------
1 - Record (MODIFIED) {
    0:[inventory_id]:STRING=[gadget]:String
    1:[sku]:STRING=[G-7]:String
    2:[qty]:LONG=[3]:Long
    3:[price]:DOUBLE=[12.0]:Double
}

-----------------------------------------------
2 - Record (MODIFIED) {
    0:[inventory_id]:STRING=[doohickey]:String
    1:[sku]:STRING=[D-4]:String
    2:[qty]:LONG=[8]:Long
    3:[price]:DOUBLE=[5.25]:Double
}

-----------------------------------------------
3 records