Package com.northconcepts.datapipeline.parquet.tools

Class GenerateSchemaDefFromParquet

java.lang.Object
com.northconcepts.datapipeline.parquet.tools.GenerateSchemaDefFromParquet
public class GenerateSchemaDefFromParquet extends Object
Generates a SchemaDef from a Parquet schema: top-level fields go into a root EntityDef and each nested group becomes its own entity. Map types are not supported.

  • Constructor Details

    • GenerateSchemaDefFromParquet

      public GenerateSchemaDefFromParquet(String schemaName, String rootEntityName)

  • Method Details

    • generateSchemaDef

      public SchemaDef generateSchemaDef(File parquetFile)
      Reads the schema of the Parquet file and generates a schema definition from it.

    • generateSchemaDef

      public SchemaDef generateSchemaDef(InputFile inputParquetFile)
      Reads the schema of the Parquet input file and generates a schema definition from it.

    • generateSchemaDef

      public SchemaDef generateSchemaDef(MessageType schema)
      Generates a schema definition from the Parquet schema and keeps the schema for getParquetSchema().

    • getFieldTypeForPrimitiveType

      public FieldDef getFieldTypeForPrimitiveType(Type type)
      Creates the field definition for a primitive Parquet column from its primitive and logical type; INT96 columns use getInt96FieldType().

    • getInt96FieldType

      public FieldType getInt96FieldType()
      Returns FieldType for int96 data type of Parquet. Possible values are: FieldType.DATETIME, FieldType.BYTE invalid input: '&' FieldType.UNDEFINED

    • setInt96FieldType

      public GenerateSchemaDefFromParquet setInt96FieldType(FieldType int96FieldType)
      Set FieldType for int96 data type of Parquet. Possible values are: FieldType.DATETIME, FieldType.BYTE invalid input: '&' FieldType.UNDEFINED

    • getParquetSchema

      public MessageType getParquetSchema()
      Returns the Parquet schema used by the most recent generateSchemaDef call or null if there was none.