Package com.northconcepts.datapipeline.foundations.pipeline.dataset

Class JdbcDataset

java.lang.Object
com.northconcepts.datapipeline.foundations.core.Bean
com.northconcepts.datapipeline.foundations.core.FoundationObject
com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset
com.northconcepts.datapipeline.foundations.pipeline.dataset.JdbcDataset
All Implemented Interfaces:
DataExceptionContributor, JsonSerializable, RecordSerializable, XmlSerializable, Closeable, Serializable, AutoCloseable, Iterable<Record>
public class JdbcDataset extends Dataset
Caches the dataset's records on database.
See Also:

  • Method Details

    • createDataset

      public static JdbcDataset createDataset(JdbcDataset.DatabaseProvider databaseProvider, JdbcConnectionFactory factory, String tableNamePrefix, AbstractPipeline pipeline)
      Creates a persistent dataset stored in a database determined by the provided JdbcDataset.DatabaseProvider.

      The data and statistics tables will remain on the database even after the dataset is closed, unless setDropTablesOnClose(boolean) is set to true.

      Parameters:
      databaseProvider - the database provider defining database-specific SQL syntax
      factory - the JDBC connection factory for creating database connections
      tableNamePrefix - the prefix for the data and statistics table names
      pipeline - the pipeline to use for loading data into the dataset
      Returns:
      a new JdbcDataset instance

    • createDataset

      public static JdbcDataset createDataset(JdbcDataset.DatabaseProvider databaseProvider, JdbcConnectionFactory factory, String tableNamePrefix, DataReaderFactory dataReaderFactory)
      Creates a persistent dataset stored in a database determined by the provided JdbcDataset.DatabaseProvider.

      The data and statistics tables will remain on the database even after the dataset is closed, unless setDropTablesOnClose(boolean) is set to true.

      Parameters:
      databaseProvider - the database provider defining database-specific SQL syntax
      factory - the JDBC connection factory for creating database connections
      tableNamePrefix - the prefix for the data and statistics table names
      dataReaderFactory - the factory for creating the data reader to populate the dataset
      Returns:
      a new JdbcDataset instance

    • openDataset

      public static JdbcDataset openDataset(JdbcDataset.DatabaseProvider databaseProvider, JdbcConnectionFactory factory, String tableNamePrefix)
      Loads an existing persistent dataset stored in a database determined by the provided JdbcDataset.DatabaseProvider.

      This method opens a dataset that was previously created with createDataset(com.northconcepts.datapipeline.foundations.pipeline.dataset.JdbcDataset.DatabaseProvider, com.northconcepts.datapipeline.jdbc.JdbcConnectionFactory, java.lang.String, com.northconcepts.datapipeline.foundations.pipeline.AbstractPipeline) and reads the existing data and statistics from the database tables.

      Parameters:
      databaseProvider - the database provider defining database-specific SQL syntax
      factory - the JDBC connection factory for creating database connections
      tableNamePrefix - the prefix for the data and statistics table names
      Returns:
      a JdbcDataset instance containing the existing dataset

    • getDataTableName

      public String getDataTableName()
      Gets the name of the data table in the database.
      Returns:
      the data table name

    • getStatsTableName

      public String getStatsTableName()
      Gets the name of the statistics table in the database.
      Returns:
      the statistics table name

    • setDropTablesOnClose

      public JdbcDataset setDropTablesOnClose(boolean dropTablesOnClose)
      Sets whether to drop the database tables when this dataset is closed.

      If true, both the data and statistics tables will be dropped from the database when close() is called. If false (default), the tables will remain in the database.

      Parameters:
      dropTablesOnClose - true to drop tables on close, false otherwise
      Returns:
      this instance for method chaining

    • isDropTablesOnClose

      public boolean isDropTablesOnClose()
      Returns whether the database tables will be dropped when this dataset is closed.
      Returns:
      true if tables will be dropped on close, false otherwise

    • setAutoCloseConnection

      public JdbcDataset setAutoCloseConnection(boolean autoCloseConnection)
      Sets whether to automatically close the database connection when this dataset is closed.

      If true (default), the database connection will be closed when close() is called. If false, the connection will remain open and must be closed manually.

      Parameters:
      autoCloseConnection - true to auto-close the connection, false otherwise
      Returns:
      this instance for method chaining

    • isAutoCloseConnection

      public boolean isAutoCloseConnection()
      Returns whether the database connection will be automatically closed when this dataset is closed.
      Returns:
      true if the connection will be auto-closed, false otherwise

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Overrides:
      close in class Dataset

    • getRecordCount

      public long getRecordCount()
      Description copied from class: Dataset
      Returns the number of records loaded into this dataset so far.
      Specified by:
      getRecordCount in class Dataset

    • getRecord

      public Record getRecord(long index)
      Description copied from class: Dataset
      Returns the loaded record at the given 0-based index.
      Specified by:
      getRecord in class Dataset

    • getRecordList

      public RecordList getRecordList(long offset, int count)
      Description copied from class: Dataset
      Get a subset of the records cached in this dataset.
      Overrides:
      getRecordList in class Dataset

    • getColumnCount

      public long getColumnCount()
      Specified by:
      getColumnCount in class Dataset

    • getColumnNames

      public List<String> getColumnNames()
      Specified by:
      getColumnNames in class Dataset

    • getColumn

      public Column getColumn(int index)
      Specified by:
      getColumn in class Dataset

    • getColumn

      public Column getColumn(String name)
      Description copied from class: Dataset
      Returns the column with the given name or null if there is none.
      Specified by:
      getColumn in class Dataset

    • getOrCreateColumn

      protected Column getOrCreateColumn(String name, int index)
      Description copied from class: Dataset
      Returns the named column, creating it with the given 0-based field index if needed; called while collecting column stats.
      Specified by:
      getOrCreateColumn in class Dataset

    • getColumns

      public List<Column> getColumns()
      Specified by:
      getColumns in class Dataset

    • beforeLoad

      protected void beforeLoad()
      Description copied from class: Dataset
      Called at the start of the data loading process, but before any records or column stats have been loaded.
      Specified by:
      beforeLoad in class Dataset

    • afterLoad

      protected void afterLoad()
      Description copied from class: Dataset
      Called at the end of the data loading process after all the records and column stats have been loaded.
      Specified by:
      afterLoad in class Dataset

    • afterColumnStatsLoaded

      protected void afterColumnStatsLoaded()
      Description copied from class: Dataset
      Called during the data loading process after all the column stats have been loaded. The records would have already been loaded when this method is called since column stats require additional processing.
      Overrides:
      afterColumnStatsLoaded in class Dataset

    • createDataWriter

      protected DataWriter createDataWriter()
      Description copied from class: Dataset
      Writes records to this dataset's cache after clearing it.
      Specified by:
      createDataWriter in class Dataset

    • addExceptionProperties

      public DataException addExceptionProperties(DataException exception)
      Description copied from class: FoundationObject
      Adds this object's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.
      Specified by:
      addExceptionProperties in interface DataExceptionContributor
      Overrides:
      addExceptionProperties in class FoundationObject