Package com.northconcepts.datapipeline.csv

Class CSVReader

java.lang.Object
com.northconcepts.datapipeline.core.DataObject
com.northconcepts.datapipeline.core.Endpoint
com.northconcepts.datapipeline.core.DataEndpoint
com.northconcepts.datapipeline.core.DataReader
com.northconcepts.datapipeline.core.AbstractReader
com.northconcepts.datapipeline.core.TextReader
com.northconcepts.datapipeline.csv.CSVReader
public class CSVReader extends TextReader
Obtains records from a Comma Separated Value (CSV) or delimited stream.

  • Constructor Details

    • CSVReader

      public CSVReader(File file)

    • CSVReader

      public CSVReader(Reader reader)

  • Method Details

    • parseLine

      public static Record parseLine(String line, String fieldSeparator, String quoteChar, Boolean allowMultiLineText)
      Utility method to quickly parse a line of CSV text.
      Parameters:
      line - the delimited line of text to parse
      fieldSeparator -
      quoteChar -
      allowMultiLineText -
      Returns:

    • parseLine

      public static Record parseLine(String line, String fieldSeparator, String quoteChar, Boolean allowMultiLineText, Boolean trimFields)
      Utility method to quickly parse a line of CSV text.
      Parameters:
      line -
      fieldSeparator -
      quoteChar -
      allowMultiLineText -
      trimFields -
      Returns:

    • parse

      public static RecordList parse(String text, String fieldSeparator, String quoteChar, Boolean allowMultiLineText)
      Utility method to quickly parse lines of CSV text.
      Parameters:
      text - the delimited lines of text to parse
      fieldSeparator -
      quoteChar -
      allowMultiLineText -
      Returns:

    • parse

      public static RecordList parse(String text, String fieldSeparator, String quoteChar, Boolean allowMultiLineText, Boolean trimFields)
      Utility method to quickly parse lines of CSV text.
      Parameters:
      text -
      fieldSeparator -
      quoteChar -
      allowMultiLineText -
      trimFields -
      Returns:

    • getFieldSeparator

      public String getFieldSeparator()
      Returns the string used between each field value (defaults to comma).

    • setFieldSeparator

      public CSVReader setFieldSeparator(char columnSeparator)
      Assigns the string used between each field value (defaults to comma).

    • setFieldSeparator

      public CSVReader setFieldSeparator(String columnSeparator)
      Assigns the string used between each field value (defaults to comma).

    • getQuoteChar

      public String getQuoteChar()
      Returns the starting quote - same as calling getStartingQuote() (defaults to double quote - "). getStartingQuote()

    • setQuoteChar

      public CSVReader setQuoteChar(char quoteChar)
      Assigns startingQuote and endingQuote to the same value (defaults to double quote - ").

    • setQuoteChar

      public CSVReader setQuoteChar(String quoteChar)
      Assigns startingQuote and endingQuote to the same value (defaults to double quote - "). setStartingQuote(String) setEndingQuote(String)

    • getStartingQuote

      public String getStartingQuote()
      Returns the string used to mark the beginning of text values that may contain commas and other special characters (defaults to double quote - ").

    • setStartingQuote

      public CSVReader setStartingQuote(String startingQuote)
      Assigns the string used to mark the beginning of text values that may contain commas and other special characters (defaults to double quote - ").

    • getEndingQuote

      public String getEndingQuote()
      Returns the string used to mark the ending of text values that may contain commas and other special characters (defaults to double quote - ").

    • setEndingQuote

      public CSVReader setEndingQuote(String endingQuote)
      Assigns the string used to mark the ending of text values that may contain commas and other special characters (defaults to double quote - ").

    • getLineSeparators

      public String[] getLineSeparators()
      Returns the strings used to separate records or null if none were explicitly assigned.

    • setLineSeparators

      public CSVReader setLineSeparators(String... lineSeparators)
      Indicates the string or strings used to separate records.

    • setLineSeparator

      public CSVReader setLineSeparator(String lineSeparator)
      Indicates the string used to separate records.

    • isAllowMultiLineText

      public boolean isAllowMultiLineText()
      Indicates if line separators are allowed inside quoted fields.

    • setAllowMultiLineText

      public CSVReader setAllowMultiLineText(boolean allowMultiLineText)
      Indicates if line separators are allowed inside quoted fields.

    • isAllowQuoteInField

      public boolean isAllowQuoteInField()
      Indicates if unescaped quotes are allowed in fields (default to false).

    • setAllowQuoteInField

      public CSVReader setAllowQuoteInField(boolean allowQuoteInField)
      Indicates if unescaped quotes are allowed in fields (default to false).

    • isTrimFields

      public boolean isTrimFields()
      Indicates if space and tab characters at the beginning and ending of non-quoted-values should be removed. If line separators are explicitly set on this reader then any non-line-separator char matching Character.isWhitespace(int) will be removed (not only spaces and tabs). This is helpful when CSV files may contain spaces between the commas (field separators) and values.

      Whitespace characters used as field or line separators will not be removed.

      The default is true.

    • setTrimFields

      public CSVReader setTrimFields(boolean trimFields)
      Indicates if space and tab characters at the beginning and ending of non-quoted-values should be removed. If line separators are explicitly set on this reader then any non-line-separator char matching Character.isWhitespace(int) will be removed (not only spaces and tabs). This is helpful when CSV files may contain spaces between the commas (field separators) and values.

      Whitespace characters used as field or line separators will not be removed.

      The default is true.

    • getParser

      public LineParser getParser()

    • getColumnNumber

      public long getColumnNumber()
      Returns the 0-based character offset the parser has reached in the current line (not a field index).

    • isLineageSupported

      public boolean isLineageSupported()
      Description copied from class: DataReader
      Indicates if this reader can capture record and field lineage (false unless overridden by a reader that supports it).
      Overrides:
      isLineageSupported in class DataReader

    • addLineage

      protected Record addLineage(Record record)
      Description copied from class: DataReader
      Called by DataReader.read() for each record from DataReader.readImpl() while lineage is saved; the default copies recordLineage into every field along with its original index and name. Overrides set their source details on recordLineage first and end with super.addLineage(record).
      Overrides:
      addLineage in class TextReader

    • addExceptionProperties

      public DataException addExceptionProperties(DataException exception)
      Description copied from class: Endpoint
      Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.
      Overrides:
      addExceptionProperties in class TextReader

    • fillRecord

      protected boolean fillRecord(Record record) throws Throwable
      Description copied from class: AbstractReader
      Populates the given record with the next row's values; returns false at the end of the input.
      Specified by:
      fillRecord in class AbstractReader
      Throws:
      Throwable

    • setSkipEmptyRows

      public CSVReader setSkipEmptyRows(boolean skipEmptyRows)
      Description copied from class: AbstractReader
      Indicates that rows with only null values should not be returned by the reader (default is false).
      Overrides:
      setSkipEmptyRows in class AbstractReader

    • setFieldNames

      public CSVReader setFieldNames(String... fieldNames)
      Description copied from class: AbstractReader
      Sets the names given to each record's fields, taking precedence over names in the first row; must be called before AbstractReader.open().
      Overrides:
      setFieldNames in class AbstractReader

    • setFieldNames

      public CSVReader setFieldNames(Collection<String> fieldNames)
      Description copied from class: AbstractReader
      Sets the names given to each record's fields, taking precedence over names in the first row; must be called before AbstractReader.open().
      Overrides:
      setFieldNames in class AbstractReader

    • setFieldNamesInFirstRow

      public CSVReader setFieldNamesInFirstRow(boolean fieldNamesInFirstRow)
      Description copied from class: AbstractReader
      Indicates if the first row holds the field names instead of data (default is false); must be called before AbstractReader.open().
      Overrides:
      setFieldNamesInFirstRow in class AbstractReader

    • setStartingRow

      public CSVReader setStartingRow(int startingRow)
      Description copied from class: AbstractReader
      Sets the 0-based index of the first row to read; earlier rows are skipped when the reader is opened (default is 0).
      Overrides:
      setStartingRow in class AbstractReader

    • setLastRow

      public CSVReader setLastRow(int lastRow)
      Description copied from class: AbstractReader
      Sets the 0-based index of the last row to read or -1 to read to the end (default is -1).
      Overrides:
      setLastRow in class AbstractReader

    • setSaveLineage

      public CSVReader setSaveLineage(boolean saveLineage)
      Description copied from class: DataReader
      Indicates if record and field lineage is captured for each record read (default is false); enabling it throws if DataReader.isLineageSupported() is false or the product edition does not include lineage.
      Overrides:
      setSaveLineage in class DataReader