Class AmazonS3FileSystem

Method Details

• setCredentialsResolver

  public AmazonS3FileSystem setCredentialsResolver(CredentialsResolver credentialsResolver)

  Sets the backend-neutral resolver supplying this file system's credentials (see CredentialsResolver). Resolved at open() — never at construction — so rotated secrets are picked up on the next connection; expiring credentials (for example STS session tokens) are additionally re-resolved mid-session when they report expiry within CREDENTIALS_EXPIRY_SKEW_MILLISECONDS.

  When a resolver is set, it takes precedence over setCredentialsProvider(AwsCredentialsProvider) and the default provider chain.

  Overrides:

  setCredentialsResolver in class FileSystem

• getCredentialsProvider

  public AwsCredentialsProvider getCredentialsProvider()

  Returns the AWS SDK credentials provider used when no CredentialsResolver is set (defaults to the standard AWS credentials provider chain).

• setCredentialsProvider

  public AmazonS3FileSystem setCredentialsProvider(AwsCredentialsProvider credentialsProvider)

  Sets the AWS SDK credentials provider used when no CredentialsResolver is set (defaults to the standard AWS credentials provider chain).

• useProfileCredentialsProvider

  public AmazonS3FileSystem useProfileCredentialsProvider(String profileName)

  Authenticates using the named profile from the AWS profile files (instead of the default profile used by the default credentials provider chain), unless a CredentialsResolver is set, which takes precedence.

• setBasicAWSCredentials

  public AmazonS3FileSystem setBasicAWSCredentials(String accessKey, String secretKey)

  Authenticates with the given access key and secret key, replacing the configured credentials provider, unless a CredentialsResolver is set, which takes precedence.

• getRegion

  public String getRegion()

  Returns the AWS region set by setRegion(String), or null if it is resolved when connecting.

• setRegion

  public AmazonS3FileSystem setRegion(String region)

  Sets the AWS region (e.g. "us-east-1"). When unset, the region is resolved from the default region provider chain, falling back to us-east-1.

• getDelimiter

  public String getDelimiter()

  The delimiter sent with every listing. Unset by default, so listings include nested keys; FOLDER_DELIMITER lists one folder level at a time.

• setDelimiter

  public AmazonS3FileSystem setDelimiter(String delimiter)

  Sets the delimiter sent with every listing (unset by default, so listings include nested keys); FOLDER_DELIMITER lists one folder level at a time.

• getEndpointOverride

  public String getEndpointOverride()

  Returns the S3 service endpoint override or null if none is set.

• setEndpointOverride

  public AmazonS3FileSystem setEndpointOverride(String endpointOverride)

  Overrides the S3 service endpoint (e.g. for S3-compatible stores or a specific region endpoint). Combine with setRegion(String) when signing requires it.

• isDebug

  public boolean isDebug()

  Indicates if multipart uploads log each part and their completion or abort at debug level (default is false).

• setDebug

  public AmazonS3FileSystem setDebug(boolean debug)

  Indicates if multipart uploads log each part and their completion or abort at debug level (default is false).

• open

  public void open() throws DataException

  Opens a connection to the Amazon S3 file system. It stays open until close(): sources, sinks, and size lookups sharing this file system use the connection without closing it.

  Credentials are chosen in this order: the CredentialsResolver (if set), then the configured

  invalid reference

  AwsCredentialsProvider

  (which defaults to the standard provider chain).

  Overrides:

  open in class Endpoint

  Throws:

  DataException

• close

  public void close() throws DataException

  Closes the Amazon S3 connection — or, while a source or sink stream or a size lookup is still using it, as soon as the last of them finishes. Clients built by open() are closed and released so the next open() rebuilds one — picking up rotated credentials; clients supplied through setClient(S3Client) are left untouched — their lifecycle belongs to the caller — and remain usable if this file system is reopened.

  Overrides:

  close in class FileSystem

  Throws:

  DataException

• finalize

  protected void finalize() throws Throwable

  Disconnects even with leases outstanding: this file system being unreachable, the streams holding them were dropped without being closed.

  Overrides:

  finalize in class Endpoint

  Throws:

  Throwable

• getClient

  public S3Client getClient()

  Returns the AWS SDK client, or null if none was supplied and this file system is not connected.

• setClient

  public AmazonS3FileSystem setClient(S3Client client)

  Uses the given AWS SDK client instead of building one from this file system's credentials, region and endpoint settings; the caller remains responsible for closing it.

• readFile

  public InputStream readFile(String bucket, String filePath)

  Reads the file in the S3 bucket. To specify a nested folder, use "/" as the path separator.
  

• exists

  public boolean exists(String bucket, String key) throws DataException

  Checks if an S3 object exists in the specified bucket.

  Parameters:

  bucket - the name of the S3 bucket

  key - the key (path) of the S3 object

  Returns:

  true if the object exists, false otherwise

  Throws:

  DataException - if bucket or key is empty, or if an error occurs while checking existence

• getFileSize

  public long getFileSize(String bucket, String key) throws DataException

  Returns the size of an S3 object in bytes.

  Parameters:

  bucket - the name of the S3 bucket

  key - the key (path) of the S3 object

  Returns:

  the object's size in bytes, or -1 if the key is absent

  Throws:

  DataException - if bucket or key is empty, if the bucket does not exist, or if an error occurs while reading the object's metadata

• deleteFile

  public void deleteFile(String bucket, String key) throws DataException

  Deletes an S3 object from the specified bucket.

  Parameters:

  bucket - the name of the S3 bucket

  key - the key (path) of the S3 object to delete

  Throws:

  DataException - if bucket or key is empty, or if an error occurs while deleting the object

• writeMultipartFile

  public OutputStream writeMultipartFile(String bucket, String filePath)

  Returns a stream that uploads to the given bucket and key as a multipart upload, in parts of MINIMUM_PART_SIZE bytes sent from a background thread. Closing the stream completes the upload; a failed upload is aborted and its error rethrown by the next write or close.

• writeMultipartFile

  public OutputStream writeMultipartFile(String bucket, String filePath, String contentType)

  Same as writeMultipartFile(String, String), but also sets the object's content type unless contentType is null or blank.

• listBuckets

  public List<S3Bucket> listBuckets() throws Throwable

  Returns the buckets owned by the authenticated user.

  Throws:

  Throwable

• listRootFolder

  public S3ObjectListing listRootFolder(String bucket) throws Throwable

  Returns the objects in the specified bucket, including nested keys. With a delimiter set, keys containing it are rolled up into S3ObjectListing.getCommonPrefixes() instead.

  Throws:

  Throwable

• listFolder

  public S3ObjectListing listFolder(String bucket, String folder) throws Throwable

  Returns the objects under the specified folder, including nested keys. With a delimiter set, keys containing it below the folder are rolled up into S3ObjectListing.getCommonPrefixes() instead.

  Throws:

  Throwable

• nextBatch

  public S3ObjectListing nextBatch(S3ObjectListing previousListing) throws Throwable

  Returns the next set of paged objects.

  Throws:

  Throwable