Thanks to visit codestin.com
Credit goes to www.scribd.com

0% found this document useful (0 votes)
6 views5 pages

Hadoop Commands

The document details a series of command line operations executed in a Windows environment for a Hadoop matrix multiplication project. It includes steps for starting Hadoop services, compiling Java classes, creating HDFS directories, and running a MapReduce job. The job successfully processes input data and outputs results to HDFS, demonstrating the use of Hadoop for matrix multiplication.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
6 views5 pages

Hadoop Commands

The document details a series of command line operations executed in a Windows environment for a Hadoop matrix multiplication project. It includes steps for starting Hadoop services, compiling Java classes, creating HDFS directories, and running a MapReduce job. The job successfully processes input data and outputs results to HDFS, demonstrating the use of Hadoop for matrix multiplication.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 5

Microsoft Windows [Version 10.0.26100.

4652]
(c) Microsoft Corporation. All rights reserved.

C:\Windows\System32>start-dfs.cmd

C:\Windows\System32>start-yarn.cmd
starting yarn daemons

C:\Windows\System32>jps
14756 DataNode
14536 NameNode
11292 ResourceManager
13980 Jps
14476 NodeManager

C:\Windows\System32>D:

D:\>cd MatrixMultiplicationProject

D:\MatrixMultiplicationProject>hdfs dfs -mkdir -p /user/hadoop/matrix_input

D:\MatrixMultiplicationProject>hdfs dfs -put matrix_data.txt /user/hadoop/matrix_input/


put: `/user/hadoop/matrix_input/matrix_data.txt': File exists

D:\MatrixMultiplicationProject>for /f "usebackq delims=" %i in (`hadoop classpath`) do set


HADOOP_CLASSPATH=%i

D:\MatrixMultiplicationProject>set
HADOOP_CLASSPATH=D:\Hadoop\hadoop-3.3.6\etc\hadoop;D:\Hadoop\hadoop-3.3.6\share\h
adoop\common;D:\Hadoop\hadoop-3.3.6\share\hadoop\common\lib\*;D:\Hadoop\hadoop-3.3.6\
share\hadoop\common\*;D:\Hadoop\hadoop-3.3.6\share\hadoop\hdfs;D:\Hadoop\hadoop-3.3.6\
share\hadoop\hdfs\lib\*;D:\Hadoop\hadoop-3.3.6\share\hadoop\hdfs\*;D:\Hadoop\hadoop-3.3.6\
share\hadoop\yarn;D:\Hadoop\hadoop-3.3.6\share\hadoop\yarn\lib\*;D:\Hadoop\hadoop-3.3.6\s
hare\hadoop\yarn\*;D:\Hadoop\hadoop-3.3.6\share\hadoop\mapreduce\*

D:\MatrixMultiplicationProject>echo %HADOOP_CLASSPATH%
D:\Hadoop\hadoop-3.3.6\etc\hadoop;D:\Hadoop\hadoop-3.3.6\share\hadoop\common;D:\Hadoo
p\hadoop-3.3.6\share\hadoop\common\lib\*;D:\Hadoop\hadoop-3.3.6\share\hadoop\common\*;
D:\Hadoop\hadoop-3.3.6\share\hadoop\hdfs;D:\Hadoop\hadoop-3.3.6\share\hadoop\hdfs\lib\*;D:
\Hadoop\hadoop-3.3.6\share\hadoop\hdfs\*;D:\Hadoop\hadoop-3.3.6\share\hadoop\yarn;D:\Had
oop\hadoop-3.3.6\share\hadoop\yarn\lib\*;D:\Hadoop\hadoop-3.3.6\share\hadoop\yarn\*;D:\Had
oop\hadoop-3.3.6\share\hadoop\mapreduce\*
D:\MatrixMultiplicationProject>javac -classpath %HADOOP_CLASSPATH% -d classes
MatrixMultiplyMapper.java MatrixMultiplyReducer.java MatrixMultiplyDriver.java

D:\MatrixMultiplicationProject>jar -cvf matrixmultiply.jar -C classes/ .


added manifest
adding: MatrixMultiplyDriver.class(in = 1916) (out= 1009)(deflated 47%)
adding: MatrixMultiplyMapper.class(in = 2775) (out= 1204)(deflated 56%)
adding: MatrixMultiplyReducer.class(in = 2562) (out= 1162)(deflated 54%)

D:\MatrixMultiplicationProject>hdfs dfs -mkdir -p /user/hadoop/matrix_input

D:\MatrixMultiplicationProject>hdfs dfs -put matrix_data.txt /user/hadoop/matrix_input/


put: `/user/hadoop/matrix_input/matrix_data.txt': File exists

D:\MatrixMultiplicationProject>hadoop jar matrixmultiply.jar MatrixMultiplyDriver


/user/hadoop/matrix_input /user/hadoop/matrix_output 2 2 2
Exception in thread "main" java.lang.UnsupportedClassVersionError: MatrixMultiplyDriver has
been compiled by a more recent version of the Java Runtime (class file version 55.0), this
version of the Java Runtime only recognizes class file versions up to 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.util.RunJar.run(RunJar.java:321)
at org.apache.hadoop.util.RunJar.main(RunJar.java:241)

D:\MatrixMultiplicationProject>javac -classpath %HADOOP_CLASSPATH% -source 1.8 -target


1.8 -d classes MatrixMultiplyMapper.java MatrixMultiplyReducer.java MatrixMultiplyDriver.java
warning: [options] bootstrap class path not set in conjunction with -source 8
1 warning

D:\MatrixMultiplicationProject>jar -cvf matrixmultiply.jar -C classes/ .


added manifest
adding: MatrixMultiplyDriver.class(in = 1916) (out= 1010)(deflated 47%)
adding: MatrixMultiplyMapper.class(in = 2511) (out= 1087)(deflated 56%)
adding: MatrixMultiplyReducer.class(in = 2562) (out= 1162)(deflated 54%)

D:\MatrixMultiplicationProject>hdfs dfs -mkdir -p /user/hadoop/matrix_input

D:\MatrixMultiplicationProject>hdfs dfs -put matrix_data.txt /user/hadoop/matrix_input/


put: `/user/hadoop/matrix_input/matrix_data.txt': File exists

D:\MatrixMultiplicationProject>hdfs dfs -rm /user/hadoop/matrix_input/matrix_data.txt


Deleted /user/hadoop/matrix_input/matrix_data.txt

D:\MatrixMultiplicationProject>hdfs dfs -put matrix_data.txt /user/hadoop/matrix_input/

D:\MatrixMultiplicationProject>hdfs dfs -rm -r /user/hadoop/matrix_output


rm: `/user/hadoop/matrix_output': No such file or directory

D:\MatrixMultiplicationProject>hadoop jar matrixmultiply.jar MatrixMultiplyDriver


/user/hadoop/matrix_input /user/hadoop/matrix_output 2 2 2
2025-08-07 12:49:23,924 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to
ResourceManager at /0.0.0.0:8032
2025-08-07 12:49:24,278 WARN mapreduce.JobResourceUploader: Hadoop command-line
option parsing not performed. Implement the Tool interface and execute your application with
ToolRunner to remedy this.
2025-08-07 12:49:24,380 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding
for path: /tmp/hadoop-yarn/staging/cse/.staging/job_1754550781839_0001
2025-08-07 12:49:24,716 INFO input.FileInputFormat: Total input files to process : 1
2025-08-07 12:49:24,993 INFO mapreduce.JobSubmitter: number of splits:1
2025-08-07 12:49:25,168 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1754550781839_0001
2025-08-07 12:49:25,169 INFO mapreduce.JobSubmitter: Executing with tokens: []
2025-08-07 12:49:25,277 INFO conf.Configuration: resource-types.xml not found
2025-08-07 12:49:25,277 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2025-08-07 12:49:25,625 INFO impl.YarnClientImpl: Submitted application
application_1754550781839_0001
2025-08-07 12:49:25,656 INFO mapreduce.Job: The url to track the job:
http://DESKTOP-N1CRCIS:8088/proxy/application_1754550781839_0001/
2025-08-07 12:49:25,657 INFO mapreduce.Job: Running job: job_1754550781839_0001
2025-08-07 12:49:34,813 INFO mapreduce.Job: Job job_1754550781839_0001 running in uber
mode : false
2025-08-07 12:49:34,816 INFO mapreduce.Job: map 0% reduce 0%
2025-08-07 12:49:38,895 INFO mapreduce.Job: map 100% reduce 0%
2025-08-07 12:49:43,969 INFO mapreduce.Job: map 100% reduce 100%
2025-08-07 12:49:45,005 INFO mapreduce.Job: Job job_1754550781839_0001 completed
successfully
2025-08-07 12:49:45,133 INFO mapreduce.Job: Counters: 54
File System Counters
FILE: Number of bytes read=198
FILE: Number of bytes written=556117
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=199
HDFS: Number of bytes written=28
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=2189
Total time spent by all reduces in occupied slots (ms)=2551
Total time spent by all map tasks (ms)=2189
Total time spent by all reduce tasks (ms)=2551
Total vcore-milliseconds taken by all map tasks=2189
Total vcore-milliseconds taken by all reduce tasks=2551
Total megabyte-milliseconds taken by all map tasks=2241536
Total megabyte-milliseconds taken by all reduce tasks=2612224
Map-Reduce Framework
Map input records=8
Map output records=16
Map output bytes=160
Map output materialized bytes=198
Input split bytes=127
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=198
Reduce input records=16
Reduce output records=4
Spilled Records=32
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=56
CPU time spent (ms)=841
Physical memory (bytes) snapshot=647811072
Virtual memory (bytes) snapshot=1043279872
Total committed heap usage (bytes)=588251136
Peak Map Physical memory (bytes)=347095040
Peak Map Virtual memory (bytes)=528916480
Peak Reduce Physical memory (bytes)=300716032
Peak Reduce Virtual memory (bytes)=514457600
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=72
File Output Format Counters
Bytes Written=28

D:\MatrixMultiplicationProject>hdfs dfs -ls /user/hadoop/matrix_output


Found 2 items
-rw-r--r-- 1 cse supergroup 0 2025-08-07 12:49 /user/hadoop/matrix_output/_SUCCESS
-rw-r--r-- 1 cse supergroup 28 2025-08-07 12:49 /user/hadoop/matrix_output/part-r-00000

D:\MatrixMultiplicationProject>hdfs dfs -cat /user/hadoop/matrix_output/part-r-00000


0,0 19
0,1 22
1,0 43
1,1 50

D:\MatrixMultiplicationProject>

You might also like