Package com.northconcepts.datapipeline.parquet

Class ParquetUtil

java.lang.Object
com.northconcepts.datapipeline.parquet.ParquetUtil
public class ParquetUtil extends Object
Utility methods to read Parquet schemas and to rewrite schema fields identified by column path.

  • Constructor Summary

    Constructors
    Constructor
    Description
    ParquetUtil()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static String
    makeOptionalFieldsRequired(List<ColumnPath> columnPaths, MessageType schema)
    Returns the schema as text with the given columns changed from optional to required.
    static String
    makeRequiredFieldsOptional(List<ColumnPath> columnPaths, MessageType schema)
    Returns the schema as text with the given columns changed from required to optional.
    static MessageType
    readSchema(File file)
    Parses the Parquet message type schema stored as text in the file, read with the platform default charset.
    static MessageType
    readSchema(Reader reader)
    Parses the reader's full text as a Parquet message type schema; the reader is closed afterwards.
    static String
    removeSchemaFields(List<ColumnPath> columnPaths, MessageType schema)
    Returns the schema as text with the declarations of the given columns removed.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • ParquetUtil

      public ParquetUtil()

  • Method Details

    • readSchema

      public static MessageType readSchema(Reader reader) throws IOException
      Parses the reader's full text as a Parquet message type schema; the reader is closed afterwards.
      Throws:
      IOException

    • readSchema

      public static MessageType readSchema(File file) throws IOException
      Parses the Parquet message type schema stored as text in the file, read with the platform default charset.
      Throws:
      IOException

    • makeRequiredFieldsOptional

      public static String makeRequiredFieldsOptional(List<ColumnPath> columnPaths, MessageType schema)
      Returns the schema as text with the given columns changed from required to optional.

    • makeOptionalFieldsRequired

      public static String makeOptionalFieldsRequired(List<ColumnPath> columnPaths, MessageType schema)
      Returns the schema as text with the given columns changed from optional to required.

    • removeSchemaFields

      public static String removeSchemaFields(List<ColumnPath> columnPaths, MessageType schema)
      Returns the schema as text with the declarations of the given columns removed.