Package com.northconcepts.datapipeline.filesystem.hdfs

Class HdfsFileSystem

java.lang.Object
com.northconcepts.datapipeline.core.DataObject
com.northconcepts.datapipeline.core.Endpoint
com.northconcepts.datapipeline.internal.lang.FileSystem
com.northconcepts.datapipeline.filesystem.hdfs.HdfsFileSystem
public class HdfsFileSystem extends FileSystem
Class for accessing the Hadoop Distributed File System.

  • Constructor Details

    • HdfsFileSystem

      public HdfsFileSystem(String hdfsUri, String userName)
      Enables access to Hadoop Distributed File System.
      URI Example: "hdfs://localhost:9000"
      Parameters:
      hdfsUri - the URI of the Hadoop Distributed File System

    • HdfsFileSystem

      public HdfsFileSystem(String hdfsUri)
      Enables access to Hadoop Distributed File System with the user name supplied through a CredentialsResolver (see setCredentialsResolver(CredentialsResolver)).
      URI Example: "hdfs://localhost:9000"
      Parameters:
      hdfsUri - the URI of the Hadoop Distributed File System

  • Method Details

    • setCredentialsResolver

      public HdfsFileSystem setCredentialsResolver(CredentialsResolver credentialsResolver)
      Sets the resolver supplying this file system's user name (Credentials.USERNAME) — resolved on each open(), never at construction, so rotated values are picked up on the next connection. When the resolved credentials don't contain a user name, the constructor-supplied value is used.
      Overrides:
      setCredentialsResolver in class FileSystem

    • open

      public void open() throws DataException
      Opens a connection to the Hadoop Distributed File System.
      Overrides:
      open in class Endpoint
      Throws:
      DataException

    • close

      public void close() throws DataException
      Closes the HDFS connection.
      Overrides:
      close in class FileSystem
      Throws:
      DataException

    • readFile

      public InputStream readFile(String folderName, String fileName) throws Throwable
      Reads the file in the specified HDFS folder. To specify a nested folder use "/" as separator.
      Example:
      • readFile("folderOne", "filename") - reads the file in folderOne
      • readFile("folderOne/folderTwo", "filename") - reads the file in folderTwo which is under folderOne
      Parameters:
      folderName - the name of the folder
      fileName - the file name to read.
      Returns:
      an InputStream
      Throws:
      Throwable

    • readFile

      public InputStream readFile(String fileName) throws Throwable
      Reads the specified file in the HDFS root folder.
      Parameters:
      fileName - the file name to read.
      Returns:
      an InputStream
      Throws:
      Throwable

    • listFolder

      public List<FileStatus> listFolder(String folderName) throws Throwable
      Returns the contents of the folder. To specify a nested folder use "/" as separator.
      Example:
      • listFolder("folderOne") - reads the contents of folderOne which is under the root folder
      • listFolder("folderOne/folderTwo") - reads the contents of folderTwo which is under folderOne
      Throws:
      Throwable

    • listRootFolder

      public List<FileStatus> listRootFolder() throws Throwable
      Returns a List of all HDFS files and folders under the root folder.
      Returns:
      the contents as a List
      Throws:
      Throwable

    • writeFile

      public OutputStream writeFile(String fileName) throws Throwable
      Writes the file to the HDFS File System. If the file exists, it is overwritten.
      Parameters:
      fileName - the name of the file
      Returns:
      an OutputStream for writing
      Throws:
      Throwable

    • writeFile

      public OutputStream writeFile(String folderName, String fileName) throws Throwable
      Writes the file to the HDFS File System. If the file exists, it is overwritten. To specify a nested folder use "/" as separator.
      Example:
      • writeFile("folderOne", "filename") - writes the file to folderOne
      • writeFile("folderOne/folderTwo", "filename") - writes the file to folderTwo which is under folderOne
      Parameters:
      folderName - the destination folder
      fileName - the name of the file
      Returns:
      an OutputStream for writing
      Throws:
      Throwable