com.northconcepts.datapipeline.amazons3

Class AmazonS3FileSystem

java.lang.Object

com.northconcepts.datapipeline.core.DataObject

com.northconcepts.datapipeline.core.Endpoint

com.northconcepts.datapipeline.internal.lang.FileSystem

com.northconcepts.datapipeline.amazons3.AmazonS3FileSystem

public class AmazonS3FileSystem
extends FileSystem

Class for accessing an Amazon S3 file system.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AmazonS3FileSystem.CredentialsResolverBridge Adapts the backend-neutral CredentialsResolver seam to the AWS SDK's AwsCredentialsProvider so resolved credentials flow through the native client natively.

Field Summary

Fields

Modifier and Type	Field and Description
static long	CREDENTIALS_EXPIRY_SKEW_MILLISECONDS How long before a credentials snapshot's reported expiry the AmazonS3FileSystem.CredentialsResolverBridge re-resolves from the CredentialsResolver.
static int	DEFAULT_QUEUE_SIZE
static int	MINIMUM_PART_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

id, log, name, TIMESTAMP_FORMAT

Constructor Summary

Constructors

Constructor and Description
AmazonS3FileSystem() Constructs an Amazon S3 file system instance.

Method Summary

Modifier and Type	Method and Description
void	close() Closes the Amazon S3 connection.
void	deleteFile(String bucket, String key) Deletes an S3 object from the specified bucket.
boolean	exists(String bucket, String key) Checks if an S3 object exists in the specified bucket.
S3Client	getClient()
AwsCredentialsProvider	getCredentialsProvider()
String	getEndpointOverride()
String	getRegion()
boolean	isDebug()
List<S3Bucket>	listBuckets() Returns the buckets owned by the authenticated user.
S3ObjectListing	listFolder(String bucket, String folder) Returns the objects in the specified bucket's folder.
S3ObjectListing	listRootFolder(String bucket) Returns the root objects in the specified bucket.
S3ObjectListing	nextBatch(S3ObjectListing previousListing) Returns the next set of paged objects.
void	open() Opens a connection to the Amazon S3 file system.
InputStream	readFile(String bucket, String filePath) Reads the file in the S3 bucket.
AmazonS3FileSystem	setBasicAWSCredentials(String accessKey, String secretKey)
AmazonS3FileSystem	setClient(S3Client client)
AmazonS3FileSystem	setCredentialsProvider(AwsCredentialsProvider credentialsProvider)
AmazonS3FileSystem	setCredentialsResolver(CredentialsResolver credentialsResolver) Sets the backend-neutral resolver supplying this file system's credentials (see CredentialsResolver).
AmazonS3FileSystem	setDebug(boolean debug)
AmazonS3FileSystem	setEndpointOverride(String endpointOverride) Overrides the S3 service endpoint (e.g.
AmazonS3FileSystem	setRegion(String region) Sets the AWS region (e.g.
AmazonS3FileSystem	useProfileCredentialsProvider(String profileName) Authenticates using the named profile from the AWS profile files (instead of the default profile used by the default credentials provider chain).
OutputStream	writeMultipartFile(String bucket, String filePath)
OutputStream	writeMultipartFile(String bucket, String filePath, String contentType)

Methods inherited from class com.northconcepts.datapipeline.internal.lang.FileSystem

credentialValue, getCredentialsResolver, getCurrentCredentials, refreshCredentials, resolveCredentials

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

addElapsedtime, addExceptionProperties, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getNestedEndpoint, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getRootEndpoint, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription, toString

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

MINIMUM_PART_SIZE

public static final int MINIMUM_PART_SIZE

See Also:

Constant Field Values

DEFAULT_QUEUE_SIZE

public static final int DEFAULT_QUEUE_SIZE

See Also:

Constant Field Values

CREDENTIALS_EXPIRY_SKEW_MILLISECONDS

public static final long CREDENTIALS_EXPIRY_SKEW_MILLISECONDS

How long before a credentials snapshot's reported expiry the AmazonS3FileSystem.CredentialsResolverBridge re-resolves from the CredentialsResolver.

See Also:

Constant Field Values

Constructor Detail

AmazonS3FileSystem

public AmazonS3FileSystem()

Constructs an Amazon S3 file system instance. By default it authenticates using the standard AWS credentials provider chain (environment variables, system properties, profile files, container/instance credentials, etc.). Use one of the credential methods below to override.

Method Detail

setCredentialsResolver

public AmazonS3FileSystem setCredentialsResolver(CredentialsResolver credentialsResolver)

Sets the backend-neutral resolver supplying this file system's credentials (see CredentialsResolver). Resolved at open() — never at construction — so rotated secrets are picked up on the next connection; expiring credentials (for example STS session tokens) are additionally re-resolved mid-session when they report expiry within CREDENTIALS_EXPIRY_SKEW_MILLISECONDS.

When a resolver is set, it takes precedence over setCredentialsProvider(AwsCredentialsProvider) and the default provider chain.

Overrides:

setCredentialsResolver in class FileSystem

getCredentialsProvider

public AwsCredentialsProvider getCredentialsProvider()

setCredentialsProvider

public AmazonS3FileSystem setCredentialsProvider(AwsCredentialsProvider credentialsProvider)

useProfileCredentialsProvider

public AmazonS3FileSystem useProfileCredentialsProvider(String profileName)

Authenticates using the named profile from the AWS profile files (instead of the default profile used by the default credentials provider chain).

setBasicAWSCredentials

public AmazonS3FileSystem setBasicAWSCredentials(String accessKey,
                                                 String secretKey)

getRegion

public String getRegion()

setRegion

public AmazonS3FileSystem setRegion(String region)

Sets the AWS region (e.g. "us-east-1"). When unset, the region is resolved from the default region provider chain, falling back to us-east-1.

getEndpointOverride

public String getEndpointOverride()

setEndpointOverride

public AmazonS3FileSystem setEndpointOverride(String endpointOverride)

Overrides the S3 service endpoint (e.g. for S3-compatible stores or a specific region endpoint). Combine with setRegion(String) when signing requires it.

isDebug

public boolean isDebug()

setDebug

public AmazonS3FileSystem setDebug(boolean debug)

open

public void open()
          throws DataException

Opens a connection to the Amazon S3 file system.

Credentials are chosen in this order: the CredentialsResolver (if set), then the configured AwsCredentialsProvider (which defaults to the standard provider chain).

Overrides:

open in class Endpoint

Throws:

DataException

close

public void close()
           throws DataException

Closes the Amazon S3 connection. Clients built by open() are closed and released so the next open() rebuilds one — picking up rotated credentials; clients supplied through setClient(S3Client) are left untouched — their lifecycle belongs to the caller — and remain usable if this file system is reopened.

Overrides:

close in class FileSystem

Throws:

DataException

getClient

public S3Client getClient()

setClient

public AmazonS3FileSystem setClient(S3Client client)

readFile

public InputStream readFile(String bucket,
                            String filePath)

Reads the file in the S3 bucket. To specify a nested folder, use "/" as the path separator.

exists

public boolean exists(String bucket,
                      String key)
               throws DataException

Checks if an S3 object exists in the specified bucket.

Parameters:

bucket - the name of the S3 bucket

key - the key (path) of the S3 object

Returns:

true if the object exists, false otherwise

Throws:

DataException - if bucket or key is empty, or if an error occurs while checking existence

deleteFile

public void deleteFile(String bucket,
                       String key)
                throws DataException

Deletes an S3 object from the specified bucket.

Parameters:

bucket - the name of the S3 bucket

key - the key (path) of the S3 object to delete

Throws:

DataException - if bucket or key is empty, or if an error occurs while deleting the object

writeMultipartFile

public OutputStream writeMultipartFile(String bucket,
                                       String filePath)

writeMultipartFile

public OutputStream writeMultipartFile(String bucket,
                                       String filePath,
                                       String contentType)

listBuckets

public List<S3Bucket> listBuckets()
                           throws Throwable

Returns the buckets owned by the authenticated user.

Throws:

Throwable

listRootFolder

public S3ObjectListing listRootFolder(String bucket)
                               throws Throwable

Returns the root objects in the specified bucket.

Throws:

Throwable

listFolder

public S3ObjectListing listFolder(String bucket,
                                  String folder)
                           throws Throwable

Returns the objects in the specified bucket's folder.

Throws:

Throwable

nextBatch

public S3ObjectListing nextBatch(S3ObjectListing previousListing)
                          throws Throwable

Returns the next set of paged objects.

Throws:

Throwable