Class ParquetPipelineInput

Method Details

• getName

  public String getName()

  Description copied from class: PipelineInput

  Returns a display name for this input, such as the source's name followed by its format.

  Overrides:

  getName in class FileSourcePipelineInput

• getFileSource

  public FileSource getFileSource()

  Specified by:

  getFileSource in class FileSourcePipelineInput

• setFileSource

  public ParquetPipelineInput setFileSource(FileSource fileSource)

  Specified by:

  setFileSource in class FileSourcePipelineInput

• createDataReader

  public DataReader createDataReader()

• generateJavaCode

  public void generateJavaCode(JavaCodeBuilder code)

  Description copied from interface: JavaCodeGenerator

  Appends Java source code representing this object to the given builder.

• isLineageSupported

  public boolean isLineageSupported()

  Description copied from class: PipelineInput

  Indicates if this input can save data lineage on the records it reads (false unless a subclass overrides it).

  Overrides:

  isLineageSupported in class PipelineInput

• isDebug

  public boolean isDebug()

  Indicates if debugging is enabled to print log statements.

• setDebug

  public ParquetPipelineInput setDebug(boolean debug)

  Indicates if debugging is enabled to print log statements.

• getConfiguration

  public Map<String,String> getConfiguration()

  Returns the Parquet configuration parameters.

• setConfiguration

  public ParquetPipelineInput setConfiguration(Map<String,String> configuration)

  Sets the Parquet configuration parameters.

• setConfiguration

  public ParquetPipelineInput setConfiguration(String key, String value)

  Sets a single Parquet configuration parameter, keeping the others.

• getFilter

  public MetadataFilter getFilter()

  Returns the metadata filter the reader applies when it opens the file (default is NO_FILTER).

• setFilter

  public ParquetPipelineInput setFilter(MetadataFilter filter)

  Sets the metadata filter the reader applies when it opens the file (default is NO_FILTER).

• getSchema

  public MessageType getSchema()

  Indicates the schema used to read the file.

• setSchema

  public ParquetPipelineInput setSchema(MessageType schema)

  Indicates the schema used to read the file.

• isMakeRequiredFieldsOptional

  public boolean isMakeRequiredFieldsOptional()

  Indicates if top-level required columns whose statistics report nulls are read as optional (default is false).

• setMakeRequiredFieldsOptional

  public ParquetPipelineInput setMakeRequiredFieldsOptional(boolean makeRequiredFieldsOptional)

  Indicates if top-level required columns whose statistics report nulls are read as optional (default is false).

• isMakeOptionalFieldsRequired

  public boolean isMakeOptionalFieldsRequired()

  Indicates if top-level optional columns with no nulls and equal min/max statistics may be read as required (default is false).

• setMakeOptionalFieldsRequired

  public ParquetPipelineInput setMakeOptionalFieldsRequired(boolean makeOptionalFieldsRequired)

  Indicates if top-level optional columns with no nulls and equal min/max statistics may be read as required (default is false).

• isRemoveFieldsWithoutColumnMetadata

  public boolean isRemoveFieldsWithoutColumnMetadata()

  Indicates if schema fields with no column metadata in any row group are dropped from the schema used for reading (default is false).

• setRemoveFieldsWithoutColumnMetadata

  public ParquetPipelineInput setRemoveFieldsWithoutColumnMetadata(boolean removeFieldsWithoutColumnMetadata)

  Indicates if schema fields with no column metadata in any row group are dropped from the schema used for reading (default is false).

• isRemoveFieldsWithoutValues

  public boolean isRemoveFieldsWithoutValues()

  Indicates if top-level columns that have no values in a row group are dropped from the schema used for reading (default is false).

• setRemoveFieldsWithoutValues

  public ParquetPipelineInput setRemoveFieldsWithoutValues(boolean removeFieldsWithoutValues)

  Indicates if top-level columns that have no values in a row group are dropped from the schema used for reading (default is false).

• toRecord

  public Record toRecord()

  Description copied from interface: RecordSerializable

  Converts this object's state to a record that RecordSerializable.fromRecord(Record) can load.

  Specified by:

  toRecord in interface RecordSerializable

  Overrides:

  toRecord in class PipelineInput

• fromRecord

  public ParquetPipelineInput fromRecord(Record source)

  Description copied from interface: RecordSerializable

  Loads this instance's state from a record and returns this (for fluid API call chaining). For fluid API call chaining, the overridden method should change the declared return type to its class.

  Specified by:

  fromRecord in interface RecordSerializable

  Overrides:

  fromRecord in class PipelineInput

  Parameters:

  source -

  Returns:

  this instance.

• toXmlElement

  public Element toXmlElement(Document document)

  Description copied from interface: XmlSerializable

  Returns an element, created with document, describing this object; the default implementation throws a DataException.

  Specified by:

  toXmlElement in interface XmlSerializable

  Overrides:

  toXmlElement in class PipelineInput

• fromXmlElement

  public ParquetPipelineInput fromXmlElement(Element pipelineInputElement)

  Specified by:

  fromXmlElement in interface XmlSerializable

  Overrides:

  fromXmlElement in class PipelineInput