com.northconcepts.datapipeline.filesystem.hdfs

Class HdfsFileSystem

java.lang.Object

com.northconcepts.datapipeline.core.DataObject

com.northconcepts.datapipeline.core.Endpoint

com.northconcepts.datapipeline.internal.lang.FileSystem

com.northconcepts.datapipeline.filesystem.hdfs.HdfsFileSystem

public class HdfsFileSystem
extends FileSystem

Class for accessing the Hadoop Distributed File System.

Field Summary

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

id, log, name, TIMESTAMP_FORMAT

Constructor Summary

Constructors

Constructor and Description
HdfsFileSystem(String hdfsUri) Enables access to Hadoop Distributed File System with the user name supplied through a CredentialsResolver (see setCredentialsResolver(CredentialsResolver)). URI Example: "hdfs://localhost:9000"
HdfsFileSystem(String hdfsUri, String userName) Enables access to Hadoop Distributed File System. URI Example: "hdfs://localhost:9000"

Method Summary

Modifier and Type	Method and Description
void	close() Closes the HDFS connection.
List<FileStatus>	listFolder(String folderName) Returns the contents of the folder.
List<FileStatus>	listRootFolder() Returns a List of all HDFS files and folders under the root folder.
void	open() Opens a connection to the Hadoop Distributed File System.
InputStream	readFile(String fileName) Reads the specified file in the HDFS root folder.
InputStream	readFile(String folderName, String fileName) Reads the file in the specified HDFS folder.
HdfsFileSystem	setCredentialsResolver(CredentialsResolver credentialsResolver) Sets the resolver supplying this file system's user name (Credentials.USERNAME) — resolved on each open(), never at construction, so rotated values are picked up on the next connection.
OutputStream	writeFile(String fileName) Writes the file to the HDFS File System.
OutputStream	writeFile(String folderName, String fileName) Writes the file to the HDFS File System.

Methods inherited from class com.northconcepts.datapipeline.internal.lang.FileSystem

credentialValue, getCredentialsResolver, getCurrentCredentials, refreshCredentials, resolveCredentials

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

addElapsedtime, addExceptionProperties, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getNestedEndpoint, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getRootEndpoint, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription, toString

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

HdfsFileSystem

public HdfsFileSystem(String hdfsUri,
                      String userName)

Enables access to Hadoop Distributed File System.
URI Example: "hdfs://localhost:9000"

Parameters:

hdfsUri - the URI of the Hadoop Distributed File System

HdfsFileSystem

public HdfsFileSystem(String hdfsUri)

Enables access to Hadoop Distributed File System with the user name supplied through a CredentialsResolver (see setCredentialsResolver(CredentialsResolver)).
URI Example: "hdfs://localhost:9000"

Parameters:

hdfsUri - the URI of the Hadoop Distributed File System

Method Detail

setCredentialsResolver

public HdfsFileSystem setCredentialsResolver(CredentialsResolver credentialsResolver)

Sets the resolver supplying this file system's user name (Credentials.USERNAME) — resolved on each open(), never at construction, so rotated values are picked up on the next connection. When the resolved credentials don't contain a user name, the constructor-supplied value is used.

Overrides:

setCredentialsResolver in class FileSystem

open

public void open()
          throws DataException

Opens a connection to the Hadoop Distributed File System.

Overrides:

open in class Endpoint

Throws:

DataException

close

public void close()
           throws DataException

Closes the HDFS connection.

Overrides:

close in class FileSystem

Throws:

DataException

readFile

public InputStream readFile(String folderName,
                            String fileName)
                     throws Throwable

Reads the file in the specified HDFS folder. To specify a nested folder use "/" as separator.
Example:

• readFile("folderOne", "filename") - reads the file in folderOne
• readFile("folderOne/folderTwo", "filename") - reads the file in folderTwo which is under folderOne

Parameters:

folderName - the name of the folder

fileName - the file name to read.

Returns:

an InputStream

Throws:

Throwable

readFile

public InputStream readFile(String fileName)
                     throws Throwable

Reads the specified file in the HDFS root folder.

Parameters:

fileName - the file name to read.

Returns:

an InputStream

Throws:

Throwable

listFolder

public List<FileStatus> listFolder(String folderName)
                            throws Throwable

Returns the contents of the folder. To specify a nested folder use "/" as separator.
Example:

• listFolder("folderOne") - reads the contents of folderOne which is under the root folder
• listFolder("folderOne/folderTwo") - reads the contents of folderTwo which is under folderOne

Throws:

Throwable

listRootFolder

public List<FileStatus> listRootFolder()
                                throws Throwable

Returns a List of all HDFS files and folders under the root folder.

Returns:

the contents as a List

Throws:

Throwable

writeFile

public OutputStream writeFile(String fileName)
                       throws Throwable

Writes the file to the HDFS File System. If the file exists, it is overwritten.

Parameters:

fileName - the name of the file

Returns:

an OutputStream for writing

Throws:

Throwable

writeFile

public OutputStream writeFile(String folderName,
                              String fileName)
                       throws Throwable

Writes the file to the HDFS File System. If the file exists, it is overwritten. To specify a nested folder use "/" as separator.
Example:

• writeFile("folderOne", "filename") - writes the file to folderOne
• writeFile("folderOne/folderTwo", "filename") - writes the file to folderTwo which is under folderOne

Parameters:

folderName - the destination folder

fileName - the name of the file

Returns:

an OutputStream for writing

Throws:

Throwable