Package com.northconcepts.datapipeline.foundations.pipeline.dataset

Class Column

java.lang.Object
com.northconcepts.datapipeline.foundations.core.Bean
com.northconcepts.datapipeline.foundations.pipeline.dataset.Column
All Implemented Interfaces:
DataExceptionContributor, JsonSerializable, RecordSerializable, XmlSerializable, Serializable
Direct Known Subclasses:
TreeNode
public class Column extends Bean
Holds the metadata and statistics for each field in a dataset.
See Also:

  • Constructor Details

    • Column

      protected Column()

    • Column

      public Column(String name, int index)

    • Column

      public Column(String name, int index, Map<Object,LongAdder> uniqueValues)
      Creates a column that collects its unique values and their counts in the given map.

  • Method Details

    • getName

      public String getName()

    • setName

      public Column setName(String name)

    • getIndex

      public int getIndex()

    • setIndex

      public Column setIndex(int index)

    • getValueCount

      public long getValueCount()
      Returns the number of values seen, including nulls; array elements are counted individually.

    • setValueCount

      public Column setValueCount(long valueCount)
      Sets the number of values seen, including nulls; array elements are counted individually.

    • addValue

      public Column addValue(Object value)
      Adds the value to the value count and the null, blank, length, sample and unique-value stats without inferring its type.

    • getUniqueValues

      public Map<Object,LongAdder> getUniqueValues()
      Returns how often each distinct value occurred, with null keyed as "null". Only filled when the dataset collects unique values.

    • getUniqueValuesByCount

      public List<Map.Entry<Object,LongAdder>> getUniqueValuesByCount()
      Returns the entries of getUniqueValues() sorted by count, most frequent first.

    • getUniqueValueCount

      public int getUniqueValueCount()
      Returns the number of distinct values seen, which are tracked only when the dataset collects unique values.

    • getNullCount

      public long getNullCount()

    • setNullCount

      public Column setNullCount(long nullCount)

    • incNullCount

      public Column incNullCount()

    • getNonNullCount

      public long getNonNullCount()
      Returns the number of non-null values, blank ones included.

    • getNonNullNonBlankCount

      public long getNonNullNonBlankCount()

    • getBlankCount

      public long getBlankCount()
      Returns the number of empty-string values; whitespace-only values are not counted as blank.

    • setBlankCount

      public Column setBlankCount(long blankCount)
      Sets the number of empty-string values; whitespace-only values are not counted as blank.

    • incBlankCount

      public Column incBlankCount()

    • getInferredTextualValueCount

      public long getInferredTextualValueCount()
      Returns the number of string values not recognized as a boolean, number, date/time or UUID.

    • getInferredNumericValueCount

      public long getInferredNumericValueCount()
      Returns the number of string values recognized as numbers, excluding leading-zero whole numbers such as 007.

    • getInferredTemporalValueCount

      public long getInferredTemporalValueCount()
      Returns the number of string values that match at least one date/time pattern.

    • getInferredBooleanValueCount

      public long getInferredBooleanValueCount()
      Returns the number of string values equal to true, false, yes or no, ignoring case.

    • getInferredUuidValueCount

      public long getInferredUuidValueCount()
      Returns the number of string values recognized as UUIDs and not as a number or date/time.

    • getArrayValueCount

      public long getArrayValueCount()
      Returns the number of array values seen, null arrays included.

    • getMinimumArrayElements

      public int getMinimumArrayElements()
      Returns the fewest elements in a non-null array value or -1 if there were none.

    • getMaximumArrayElements

      public int getMaximumArrayElements()
      Returns the most elements in a non-null array value or -1 if there were none.

    • isTextual

      public boolean isTextual()
      Returns true if at least one string value was not recognized as another type.

    • isNumeric

      public boolean isNumeric()
      Returns true if at least one string value was recognized as a number.

    • isTemporal

      public boolean isTemporal()
      Returns true if at least one string value matched a date/time pattern.

    • isBoolean

      public boolean isBoolean()
      Returns true if at least one string value was true, false, yes or no.

    • isUuid

      public boolean isUuid()
      Returns true if at least one string value was recognized as a UUID.

    • isArray

      public boolean isArray()
      Returns true if at least one array value was seen.

    • getInferredTypeName

      public String getInferredTypeName()
      Returns the capitalized name of getInferredFieldType(), such as Int, or String when that is null.

    • getMinimumLength

      public int getMinimumLength()
      Returns the shortest string length of the non-null values or -1 if there were none.

    • setMinimumLength

      public Column setMinimumLength(int minimumLength)
      Sets the shortest string length of the non-null values (-1 if there were none).

    • getMaximumLength

      public int getMaximumLength()
      Returns the longest string length of the non-null values or -1 if there were none.

    • setMaximumLength

      public Column setMaximumLength(int maximumLength)
      Sets the longest string length of the non-null values (-1 if there were none).

    • getLengthSum

      public long getLengthSum()
      Returns the total string length of the non-null values.

    • setLengthSum

      public Column setLengthSum(long lengthSum)
      Sets the total string length of the non-null values.

    • getAverageLength

      public long getAverageLength()
      Returns getLengthSum() divided by the value count, nulls included, rounded down (0 if there are no values).

    • getFieldTypes

      public Map<FieldType,LongAdder> getFieldTypes()
      Returns the number of values seen for each field type, nulls included.

    • getInferredFieldType

      public FieldType getInferredFieldType()
      Returns the most frequently occurring type in a column.

      For example, if a column contains 10 integer values and 9 boolean values, the inferred field type for that column will be FieldType.INT.

    • getBestFitInferredFieldType

      public FieldType getBestFitInferredFieldType()
      Returns the best fit inferred field type for the column.

      If there is more than one inferred type, it will return the best fit field type based on the getBestFitFieldType() method.

    • setFieldTypes

      public Column setFieldTypes(Map<FieldType,LongAdder> fieldTypes)
      Sets the number of values seen for each field type, nulls included.

    • addField

      public Column addField(Field field, Dataset dataset)
      Adds the field's value (or each array element) to this column's stats, inferring string types as configured on the dataset.

    • getFieldTypeCount

      public int getFieldTypeCount()
      Returns the number of distinct field types seen.

    • getFieldType

      public FieldType getFieldType()
      Returns the most frequent field type, skipping an all-null UNDEFINED when others exist, or null if none were seen.

    • getNullValueFieldTypes

      public ConcurrentSkipListMap<FieldType,LongAdder> getNullValueFieldTypes()
      Returns the number of null values seen for each field type.

    • setNullValueFieldTypes

      public Column setNullValueFieldTypes(ConcurrentSkipListMap<FieldType,LongAdder> nullValueFieldTypes)

    • getBestFitFieldType

      public FieldType getBestFitFieldType()
      Returns a field type that can hold every value seen or null if none were seen. Mixed numeric types give a wider numeric type; any other mix gives STRING.

    • getTemporalPatterns

      public Map<DateTimePattern,LongAdder> getTemporalPatterns()
      Returns the number of string values that matched each date/time pattern.

    • setTemporalPatterns

      public Column setTemporalPatterns(Map<DateTimePattern,LongAdder> temporalPatterns)
      Sets the number of string values that matched each date/time pattern.

    • getTemporalPatternCount

      public int getTemporalPatternCount()
      Returns the number of distinct date/time patterns matched.

    • getTemporalFieldTypes

      public Map<FieldType,LongAdder> getTemporalFieldTypes()
      Returns the number of string values that matched a pattern of each temporal field type.

    • setTemporalFieldTypes

      public Column setTemporalFieldTypes(Map<FieldType,LongAdder> temporalFieldTypes)
      Sets the number of string values that matched a pattern of each temporal field type.

    • getTemporalFieldTypeCount

      public int getTemporalFieldTypeCount()
      Returns the number of distinct temporal field types matched.

    • getNumberDescriptor

      public NumberDescriptor getNumberDescriptor()
      Returns the sign and digit counts that fit every number recognized in string values or null if there were none.

    • setNumberDescriptor

      public Column setNumberDescriptor(NumberDescriptor numberDescriptor)
      Sets the sign and digit counts that fit every number recognized in string values.

    • getBigNumberDescriptor

      public NumberDescriptor getBigNumberDescriptor()
      Returns NumberDescriptor with necessary details for BigInteger invalid input: '&' BigDecimal numbers only.

    • setBigNumberDescriptor

      public Column setBigNumberDescriptor(NumberDescriptor bigNumberDescriptor)
      Set NumberDescriptor with necessary details for BigInteger invalid input: '&' BigDecimal numbers only.

    • getSampleValue

      public String getSampleValue()
      Returns the first non-null value added to this column, as a string.

    • setSampleValue

      public Column setSampleValue(String sampleValue)
      Sets the sample value, normally the first non-null value added to this column, as a string.

    • isNumber

      public static boolean isNumber(String string)
      Returns true if Commons Lang NumberUtils.isParsable accepts the string. Column type inference uses the pipeline's number detector instead.

    • isBoolean

      public static boolean isBoolean(String s)
      Returns true if the string is true, false, yes or no, ignoring case.

    • toRecord

      public Record toRecord()
      Description copied from interface: RecordSerializable
      Converts this object's state to a record that RecordSerializable.fromRecord(Record) can load.
      Specified by:
      toRecord in interface RecordSerializable
      Overrides:
      toRecord in class Bean

    • fromRecord

      public Column fromRecord(Record source)
      Description copied from interface: RecordSerializable
      Loads this instance's state from a record and returns this (for fluid API call chaining). For fluid API call chaining, the overridden method should change the declared return type to its class.
      Specified by:
      fromRecord in interface RecordSerializable
      Overrides:
      fromRecord in class Bean
      Parameters:
      source -
      Returns:
      this instance.

    • toXmlElement

      public Element toXmlElement(Document document)
      Description copied from interface: XmlSerializable
      Returns an element, created with document, describing this object; the default implementation throws a DataException.

    • fromXmlElement

      public Column fromXmlElement(Element element)

    • addExceptionProperties

      public DataException addExceptionProperties(DataException exception)
      Description copied from interface: DataExceptionContributor
      Adds this object's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.
      Specified by:
      addExceptionProperties in interface DataExceptionContributor
      Overrides:
      addExceptionProperties in class Bean