Scribd
Upload
3 views5 pages

HDFS and HAdoop command

The document provides fundamental commands for managing files and directories in Hadoop's HDFS, including uploading, listing, creating, copying, moving, and removing files. It also covers commands for checking Hadoop version, viewing cluster information, submitting and managing MapReduce jobs. Users are advised to replace placeholders with actual paths and filenames when executing commands.

Uploaded by

harshkheskwani
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
3 views5 pages

HDFS and HAdoop command

The document provides fundamental commands for managing files and directories in Hadoop's HDFS, including uploading, listing, creating, copying, moving, and removing files. It also covers commands for checking Hadoop version, viewing cluster information, submitting and managing MapReduce jobs. Users are advised to replace placeholders with actual paths and filenames when executing commands.

Uploaded by

harshkheskwani
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 5

HDFS is the primary storage system used in Hadoop for distributed storage and data

processing. Here are a few fundamental commands:

1. Upload a file to HDFS:


php

hdfs dfs -put <local-file> <hdfs-destination>

Example:

sql
hdfs dfs -put local-file.txt /user/hadoop/data/

List files and directories in HDFS:


bash
hdfs dfs -ls <hdfs-path>

Example:

bash
hdfs dfs -ls /user/hadoop/data/

Create a directory in HDFS:


arduino
hdfs dfs -mkdir <hdfs-directory>

Example:

bash
hdfs dfs -mkdir /user/hadoop/output/

Copy files between HDFS and local filesystem:


php
hdfs dfs -copyToLocal <hdfs-source> <local-destination>

Example:

sql
hdfs dfs -copyToLocal /user/hadoop/data/local-file.txt local-copy/

Move files within HDFS:


php
hdfs dfs -mv <hdfs-source> <hdfs-destination>

Example:

bash
hdfs dfs -mv /user/hadoop/data/local-file.txt /user/hadoop/archive/

Remove files or directories from HDFS:


bash
hdfs dfs -rm <hdfs-path>

Example:

bash
hdfs dfs -rm /user/hadoop/data/unwanted-file.txt

View the content of a file in HDFS:


bash
hdfs dfs -cat <hdfs-file>

Example:

bash
hdfs dfs -cat /user/hadoop/data/sample.txt

Get the summary of a file or directory in HDFS:


bash
hdfs dfs -du -s -h <hdfs-path>

Example:

bash
hdfs dfs -du -s -h /user/hadoop/data/

Change the replication factor of a file in HDFS:


php
hdfs dfs -setrep -w <replication-factor> <hdfs-file>

Example:
bash
hdfs dfs -setrep -w 3 /user/hadoop/data/high-replication-file.txt

Check the available disk space on HDFS:


hdfs dfsadmin -report

1.

Remember to replace placeholders like <local-file>, <hdfs-destination>, <hdfs-


path>, etc., with actual paths and filenames.

These basic commands should help you get started with managing files and directories in
HDFS. Hadoop HDFS commands often follow a similar structure to traditional Unix commands,
so if you're familiar with Unix/Linux commands, you'll find it relatively easy to work with HDFS.

Check Hadoop version:

hadoop version

View Hadoop cluster information:


hadoop dfsadmin -report

Submit a MapReduce job:

hadoop jar <jar-file> <main-class> <input-path> <output-path>

Example:

hadoop jar myjob.jar com.example.WordCount /user/hadoop/input


/user/hadoop/output

View the status of MapReduce jobs:


hadoop job -list

Kill a MapReduce job:


hadoop job -kill <job-id>

Example:

hadoop job -kill job_123456789_0001


View the content of a file in HDFS:

hadoop fs -cat <hdfs-file>

Example:

hadoop fs -cat /user/hadoop/data/sample.txt

List files and directories in HDFS:


hadoop fs -ls <hdfs-path>

Example:

hadoop fs -ls /user/hadoop/data/

Create a directory in HDFS:


hadoop fs -mkdir <hdfs-directory>

Example:

hadoop fs -mkdir /user/hadoop/output/

Copy files between HDFS and local filesystem:


hadoop fs -copyToLocal <hdfs-source> <local-destination>

Example:

hadoop fs -copyToLocal /user/hadoop/data/local-file.txt local-copy/

Remove files or directories from HDFS:


hadoop fs -rm <hdfs-path>

Example:

hadoop fs -rm /user/hadoop/data/unwanted-file.txt

View available disk space on HDFS:


hadoop fs -df -h
1.

Remember to replace placeholders like <jar-file>, <main-class>, <input-path>,


<output-path>, <hdfs-file>, <hdfs-path>, etc., with actual paths, filenames, and IDs.

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy