Package com.northconcepts.datapipeline.jdbc

Class JdbcMultiUpsertWriter

java.lang.Object
com.northconcepts.datapipeline.core.DataObject
com.northconcepts.datapipeline.core.Endpoint
com.northconcepts.datapipeline.core.DataEndpoint
com.northconcepts.datapipeline.core.DataWriter
com.northconcepts.datapipeline.core.ProxyWriter
com.northconcepts.datapipeline.jdbc.JdbcMultiUpsertWriter
public class JdbcMultiUpsertWriter extends ProxyWriter
Writes records to a database table using the upsert idiom (attempt to insert, but update if duplicate key exists). It uses 1 or more connections; each connection writing asynchronously using a separate thread. The default strategy used is GenericUpsert, however several other are provided: MergeUpsert, VariableFieldsUpsert, MySqlUpsert.

  • Constructor Details

    • JdbcMultiUpsertWriter

      public JdbcMultiUpsertWriter(DataSource dataSource, int connections, int maxQueuedRecordsPerConnection, String tableName, IUpsert upsert, String... keyFieldNames)
      Creates a writer that obtains its connections from the data source when opened; at least one key field is required.

    • JdbcMultiUpsertWriter

      public JdbcMultiUpsertWriter(DataSource dataSource, int connections, int maxQueuedRecordsPerConnection, String tableName, String... keyFieldNames)
      Creates a writer using GenericUpsert that obtains its connections from the data source when opened.

    • JdbcMultiUpsertWriter

      public JdbcMultiUpsertWriter(JdbcConnectionFactory factory, int connections, int maxQueuedRecordsPerConnection, String tableName, IUpsert upsert, String... keyFieldNames) throws DataException
      Creates a writer that obtains its connections from the factory when opened; at least one key field is required.
      Throws:
      DataException

    • JdbcMultiUpsertWriter

      public JdbcMultiUpsertWriter(JdbcConnectionFactory factory, int connections, int maxQueuedRecordsPerConnection, String tableName, String... keyFieldNames) throws DataException
      Creates a writer using GenericUpsert that obtains its connections from the factory when opened.
      Throws:
      DataException

  • Method Details

    • getConnections

      public int getConnections()
      Returns the number of connections, each written to by its own thread.

    • getJdbcWriter

      public JdbcUpsertWriter getJdbcWriter(int index)
      Returns the writer for the connection at the 0-based index or null before open().

    • getAsyncWriter

      public AsyncWriter getAsyncWriter(int index)
      Returns the asynchronous writer feeding the connection at the 0-based index or null before open().

    • getBatchSize

      public int getBatchSize()
      Returns the number of records to chunk together in batch updates; 1 or less (the default is -1) disables batching.

    • setBatchSize

      public JdbcMultiUpsertWriter setBatchSize(int batchSize)
      Sets the number of records to chunk together in batch updates. Updates are not batched if this value is 1 or less. JdbcWriter defaults to non batching updates.

    • isCommitBatch

      public boolean isCommitBatch()
      Indicates if this writer should call commit on each connection after every batch is sent, otherwise, commit is called after the final batch update (default is false).

    • setCommitBatch

      public JdbcMultiUpsertWriter setCommitBatch(boolean commitBatch)
      Indicates if this writer should call commit on each connection after every batch is sent, otherwise, commit is called after the final batch update (default is false).

    • getNonUpdateFields

      public FieldList getNonUpdateFields()
      Returns the list of fields that must not be updated if the recored already exists (default null). This method may return null or a list of empty fields.

    • setNonUpdateFields

      public JdbcMultiUpsertWriter setNonUpdateFields(FieldList nonUpdateFields)
      Assigns the list of fields that must not be updated if the recored already exists (default null).

    • setNonUpdateFields

      public JdbcMultiUpsertWriter setNonUpdateFields(String... nonUpdateFields)
      Assigns the list of fields that must not be updated if the recored already exists (default null).

    • getUpsert

      public IUpsert getUpsert()
      Returns the upsert strategy; each connection's writer uses its own clone.

    • setJdbcType

      public JdbcMultiUpsertWriter setJdbcType(String fieldName, int jdbcType)

      The given Java object will be converted to the given targetSqlType before being sent to the database.

      Note that this method may be used to pass database-specific abstract data types.

      Parameters:
      fieldName -
      jdbcType - the object containing the JDBC Type parameter value

    • setJdbcType

      public JdbcMultiUpsertWriter setJdbcType(Class<?> type, int jdbcType)
      Forces field values of the specified Java class type to be sent to the database using PreparedStatement.setObject(parameterIndex, fieldValue, jdbcType) (instead of the appropriate PreparedStatement.setXXX() method for its field's type). This method may be used to pass database-specific abstract data types.
      Parameters:
      type - the Java class type to override.
      jdbcType - one of the Types constants or a database specific value.

    • getJdbcType

      public Integer getJdbcType(Class<?> type)
      The Types constant (or a database specific value) assigned to the specified Java class type using setJdbcType(Class, int) or null if none was set.
      Returns:
      one of the Types constants or a database specific value.

    • isDebug

      public boolean isDebug()
      Indicates if the generated SQL should be logged (default false).

    • setDebug

      public JdbcMultiUpsertWriter setDebug(boolean debug)
      Indicates if the generated SQL should be logged (default false).

    • open

      public void open() throws DataException
      Description copied from class: DataEndpoint
      Makes this endpoint ready for reading or writing.
      Overrides:
      open in class ProxyWriter
      Throws:
      DataException

    • createJdbcUpsertWriter

      protected JdbcUpsertWriter createJdbcUpsertWriter(Connection connection, String tableName, IUpsert upsertStrategy)

    • writeImpl

      protected void writeImpl(Record record) throws Throwable
      Description copied from class: DataWriter
      Overridden by subclasses to write the specified record to this DataWriter.

      Contract for subclasses (see also docs/authoring/DataWriter.md):

      Overrides:
      writeImpl in class ProxyWriter
      Throws:
      Throwable

    • getException

      public DataException getException()
      Returns an exception thrown by one of the internal AsyncWriters if any.

    • getExceptions

      public List<DataException> getExceptions()
      Returns all exceptions thrown by the internal AsyncWriters (the list will not be null, but may contain no elements).

    • rethrowAsyncException

      public void rethrowAsyncException() throws DataException
      Re-throws an exception thrown by one of the internal AsyncWriters or returns silently if no exceptions were thrown.
      Throws:
      DataException

    • addExceptionProperties

      public DataException addExceptionProperties(DataException exception)
      Description copied from class: Endpoint
      Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.
      Overrides:
      addExceptionProperties in class ProxyWriter