COMMAND LINE

INTERFACE

BY DR. MONICA APTE

THE COMMAND-LINE INTERFACE

• The File System (FS) shell includes various shell-like commands that directly interact
with the Hadoop Distributed File System (HDFS) as well as other file systems that
Hadoop supports, such as Local FS, HFTP FS, S3 FS, and others.
• Below are the commands supported
• AppendToFile
• hadoop fs -appendToFile /home/testuser/test/test.txt /user/haas_queue/test/test.txt
• append the content of the /home/testuser/test/test.txt to the
/user/haas_queue/test/test.txt in the hdfs.
• cat
• Copies source paths to stdout.
• hadoop fs -cat hdfs://nameservice1/user/haas_queue/test/test.txt
• checksum
• Returns the checksum information of a file.
• hadoop fs -checksum hdfs://nameservice1/user/haas_queue/test/test.txt
• chgrp
• Usage: hadoop fs -chgrp [-R] GROUP URI [URI …]
• Change group association of files. The user must be the owner of files, or else a
super-user.
• The -R option will make the change recursively through the directory structure.
• chmod
• hadoop fs -chmod [-R] <MODE[,MODE]… |
• Change the permissions of files. With -R, make the change recursively through
the directory structure. The user must be the owner of the file, or else a super-
user.
• Options
• The -R option will make the change recursively through the directory structure
• chown
• Usage: hadoop fs -chown [-R] [OWNER][:[GROUP]] URI [URI ]
• Change the owner of files. The user must be a super-user.
• Options
• The -R option will make the change recursively through the directory structure.

• copyFromLocal
• hadoop fs -copyFromLocal /home/sa081876/test/* /user/haas_queue/test
• This command copies all the files inside test folder in the edge node to test folder in the hdfs
• Similar to put command, except that the source is restricted to a local file reference.

• Options:
• The -f option will overwrite the destination if it already exists.
• copyToLocal
• hadoop fs -copyToLocal [-ignorecrc] [-crc] URI <localdst>
• hadoop fs -copyFromLocal /user/haas_queue/test/* /home/sa081876/test
• This command copies all the files inside test folder in the hdfs to test folder in the edge
node.
• Similar to get command, except that the destination is restricted to a local file
reference.
• count
• hadoop fs -count [-q] [-h] [-v] <paths>
• Count the number of directories, files and bytes under the paths that match the specified file pattern. The output columns with
-count are: DIR_COUNT, FILE_COUNT, CONTENT_SIZE, PATHNAME
• The output columns with -count -q are: QUOTA, REMAINING_QUATA, SPACE_QUOTA, REMAINING_SPACE_QUOTA,
DIR_COUNT, FILE_COUNT, CONTENT_SIZE, PATHNAME
• The -h option shows sizes in human readable format.
• The -v option displays a header line.
• Example:
• hadoop fs -count hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2
hadoop fs -count -q hdfs://nn1.example.com/file1
hadoop fs -count -q -h hdfs://nn1.example.com/file1
hdfs dfs -count -q -h -v hdfs://nn1.example.com/file1
• cp
• Usage: hadoop fs -cp [-f] [-p | -p[topax]] URI [URI …] <dest>

• Copy files from source to destination. This command allows multiple sources as well in which case the destination
must be a directory.

• Options:

• The -f option will overwrite the destination if it already exists.

The -p option will preserve file attributes [topx] (timestamps, ownership, permission, ACL, XAttr). If -p is specified
with no arg, then preserves timestamps, ownership, permission. If -pa is specified, then preserves permission also
because ACL is a super-set of permission. Determination of whether raw namespace extended attributes are
preserved is independent of the -p flag.

• hadoop fs -cp /user/hadoop/file1 /user/hadoop/file2

hadoop fs -cp /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir
• createSnapshot
• HDFS Snapshots are read-only point-in-time copies of the file system. Snapshots can be taken on a
subtree of the file system or the entire file system. Some common use cases of snapshots are data
backup, protection against user errors and disaster recovery. For more information refer the link 
HdfsSnapshots.html
• hdfs dfs -createSnapshot <path> [<snapshotName>]
• path – The path of the snapshottable directory.
• snapshotName – The snapshot name, which is an optional argument. When it is omitted, a default
name is generated using a timestamp with the format “‘s’yyyyMMdd-HHmmss.SSS”, e.g.
“s20130412-151029.033”
• deleteSnapshot
• Delete a snapshot of from a snapshot table directory. This
operation requires owner privilege of the snapshot table directory.
• hdfs dfs -deleteSnapshot <path> <snapshotName>
• path – The path of the snapshottable directory.
snapshotName – The snapshot name.
• df
• Displays free space
• hadoop fs -df [-h] URI [URI …]
• Options:
• The -h option will format file sizes in a human-readable fashion.
Example:
• hadoop fs -df /user/hadoop/dir1
• du
• hadoop fs -du [-s] [-h] URI [URI …]
• Displays sizes of files and directories contained in the given directory or the length of a file in case its just a file.
• Options:
• The -s option will result in an aggregate summary of file lengths being displayed, rather than the individual files.
The -h option will format file sizes in a “human-readable” fashion (e.g 64.0m instead of 67108864)
Example:
• hadoop fs -du /user/hadoop/dir1 /user/hadoop/file1 hdfs://nn.example.com/user/hadoop/dir1
• expunge

• Empty the Trash.For more info refer the link HdfsDesign.html

• hadoop fs -expunge
• find
• hadoop fs -find <path> … <expression> …
• Finds all files that match the specified expression and applies selected actions to them. If no path is specified then defaults to the
current working directory. If no expression is specified then defaults to -print.
• hadoop fs -find / -name test -print
• get
• hadoop fs -get [-ignorecrc] [-crc] <src> <localdst>
• Copy files to the local file system. Files that fail the CRC check may be copied with the -ignorecrc option. Files and CRCs may be
copied using the -crc option.
• Example:
• hadoop fs -get /user/hadoop/file localfile
hadoop fs -get hdfs://nn.example.com/user/hadoop/file localfile
• getfacl
• hadoop fs -getfacl [-R] <path>
• Displays the Access Control Lists (ACLs) of files and directories. If a directory has a default ACL, then
getfacl also displays the default ACL.
• Options:
• -R: List the ACLs of all files and directories recursively.
path: File or directory to list.
Examples:
• hadoop fs -getfacl /file
hadoop fs -getfacl -R /dir
• getfattr
• hadoop fs -getfattr [-R] -n name | -d [-e en] <path>
• Displays the extended attribute names and values (if any) for a file or directory.
• Options:
• -R: Recursively list the attributes for all files and directories.
-n name: Dump the named extended attribute value.
-d: Dump all extended attribute values associated with pathname.
-e encoding: Encode values after retrieving them. Valid encodings are “text”, “hex”, and “base64”. Values encoded as text strings are
enclosed in double quotes (“), and values encoded as hexadecimal and base64 are prefixed with 0x and 0s, respectively.
path: The file or directory.
Examples:
• hadoop fs -getfattr -d /file
hadoop fs -getfattr -R -n user.myAttr /dir
• getmerge
• hadoop fs -getmerge [-nl] <src> <localdst>
• Takes a source directory and a destination file as input and concatenates files in src into the destination
local file. Optionally -nl can be set to enable adding a newline character (LF) at the end of each file.
• Examples:
• hadoop fs -getmerge -nl /src /opt/output.txt
• help
• hadoop fs -help
• Return usage output.
• mv
• hadoop fs -mv URI [URI …] <dest>
• Moves files from source to destination. This command allows multiple sources as well in which case the destination needs to be a directory. Moving
files across file systems is not permitted.
• put
• hadoop fs -put <localsrc> … <dst>
• Copy single src, or multiple srcs from local file system to the destination file system. Also reads input from stdin and writes to destination file system.
• renameSnapshot
• Rename a snapshot. This operation requires owner privilege of the snapshottable directory.
• hdfs dfs -renameSnapshot <path> <oldName> <newName>
• path The path of the snapshottable directory.
oldName The old snapshot name.
newName The new snapshot name.
• rm
• hadoop fs -rm [-f] [-r |-R] [-skipTrash] URI [URI …]
• Delete files specified as args.
• Options:
• The -f option will not display a diagnostic message or modify the exit status to reflect an error if the file does
not exist.
The -R option deletes the directory and any content under it recursively.
The -r option is equivalent to -R.
The -skipTrash option will bypass trash, if enabled, and delete the specified file(s) immediately. This can be
useful when it is necessary to delete files from an over-quota directory.
Example:
• hadoop fs -rm hdfs://nn.example.com/file /user/hadoop/emptydir
• rmdir
• hadoop fs -rmdir [–ignore-fail-on-non-empty] URI [URI …]
• Delete a directory.
• rmr
• hadoop fs -rmr [-skipTrash] URI [URI …]
• Recursive version of delete.
• setfacl
• hadoop fs -setfacl [-R] [-b |-k -m |-x <acl_spec> <path>] |[–set <acl_spec> <path>]
• Sets Access Control Lists (ACLs) of files and directories.
• setfattr
• hadoop fs -setfattr -n name [-v value] | -x name <path>
• Sets an extended attribute name and value for a file or directory.
• setrep
• hadoop fs -setrep [-R] [-w] <numReplicas> <path>
• Changes the replication factor of a file. If path is a directory then the command recursively changes the replication
factor of all files under the directory tree rooted at path
• stat
• hadoop fs -stat [format] <path> …
• Print statistics about the file/directory at <path> in the specified format.
• tail
• hadoop fs -tail [-f] URI
• Displays last kilobyte of the file to stdout.
• test
• hadoop fs -test -[defsz] URI
• text
• hadoop fs -text <src>
• Takes a source file and outputs the file in text format. The allowed formats are zip and TextRecordInputStream.
• touchz
• hadoop fs -touchz URI
• Create a file of zero length.
• truncate
• hadoop fs -truncate [-w] <length> <paths>
• Truncate all files that match the specified file pattern to the specified length.
• usage
• hadoop fs -usage command
• Return the help for an individual command.
 
• ls
• list files
• hadoop fs -ls [-d] [-h] [-R] <args>
• -d: Directories are listed as plain files.
-h: Format file sizes in a human-readable fashion (eg 64.0m instead of 67108864).
-R: Recursively list subdirectories encountered.
• lsr
• Recursive version of ls.
• hadoop fs -lsr <args>
• mkdir
• hadoop fs -mkdir [-p] <paths>
• Takes path as argument and creates directories.
• Options:
• The -p option behavior is much like Unix mkdir -p, creating parent directories along the path.
• moveFromLocal
• hadoop fs -moveFromLocal <localsrc> <dst>
• Similar to put command, except that the source localsrc is deleted after it’s
copied.
• moveToLocal
• hadoop fs -moveToLocal [-crc] <src> <dst>
• truncate
• hadoop fs -truncate [-w] <length> <paths>
• Truncate all files that match the specified file pattern to the specified length.