4장. 맵리듀스 프로그램

6월 10, 2017

4장. 맵리듀스 프로그램

맵리듀스 프로그래밍 예제

클래스파일 3개 작성

① WordCountMapper.java

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

package pj01.hadoop.chapter04;

import java.io.IOException;

import java.util.StringTokenizer;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

public class WordCountMapper extends

        Mapper<LongWritable, Text, Text, IntWritable> {

    private final static IntWritable one = new IntWritable(1);

    private Text word = new Text();

    public void map(LongWritable key, Text value, Context context)

            throws IOException, InterruptedException {

        StringTokenizer itr = new StringTokenizer(value.toString());

        while (itr.hasMoreTokens()) {

            word.set(itr.nextToken());

            context.write(word, one);

        }

    }

} 

Colored by Color Scripter
cs

② WordCountReducer.java

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

package pj01.hadoop.chapter04;

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Reducer;

public class WordCountReducer extends

        Reducer<Text, IntWritable, Text, IntWritable> {

    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values, Context context)

            throws IOException, InterruptedException {

        int sum = 0;

        for (IntWritable val : values) {

            sum += val.get();

        }

        result.set(sum);

        context.write(key, result);

    }

} 

Colored by Color Scripter
cs

③ WordCount.java

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

package pj01.hadoop.chapter04;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class WordCount {

    public static void main(String[] args) throws Exception {

        Configuration conf = new Configuration();

        if (args.length != 2) {

            System.err.println("Usage: WordCount <input> <output>");

            System.exit(2);

        }

        Job job = new Job(conf, "WordCount");

        job.setJarByClass(WordCount.class);

        job.setMapperClass(WordCountMapper.class);

        job.setReducerClass(WordCountReducer.class);

        job.setInputFormatClass(TextInputFormat.class);

        job.setOutputFormatClass(TextOutputFormat.class);

        job.setOutputKeyClass(Text.class);

        job.setOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(job, new Path(args[0]));

        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        job.waitForCompletion(true);

    }

} 

Colored by Color Scripter
cs

Jar파일 생성

Jar파일 생성확인

1

2

[hduser@hdstudy01 hadoop]$ ll alzio*

-rw-rw-r--. 1 hduser hduser 3545 2017-06-11 00:21 alzio-hadoop-examples.jar
cs

입력데이터 생성 및 HDFS업로드

-작성
$ vi input2.txt

1

2

Hello World

Bye World
cs

-업로드

1

[hduser@hdstudy01 hadoop]$ hadoop fs -put input2.txt input2.txt
cs

맵리듀스 프로그램 실행

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

[hduser@hdstudy01 hadoop]$ hadoop jar alzio-hadoop-examples.jar input2.txt wordcount_output5

17/06/11 00:21:22 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

17/06/11 00:21:22 INFO input.FileInputFormat: Total input paths to process : 1

17/06/11 00:21:22 INFO util.NativeCodeLoader: Loaded the native-hadoop library

17/06/11 00:21:22 WARN snappy.LoadSnappy: Snappy native library not loaded

17/06/11 00:21:23 INFO mapred.JobClient: Running job: job_201706091514_0006

17/06/11 00:21:24 INFO mapred.JobClient:  map 0% reduce 0%

17/06/11 00:21:30 INFO mapred.JobClient:  map 100% reduce 0%

17/06/11 00:21:37 INFO mapred.JobClient:  map 100% reduce 33%

17/06/11 00:21:39 INFO mapred.JobClient:  map 100% reduce 100%

17/06/11 00:21:40 INFO mapred.JobClient: Job complete: job_201706091514_0006

17/06/11 00:21:40 INFO mapred.JobClient: Counters: 29

17/06/11 00:21:40 INFO mapred.JobClient:   Map-Reduce Framework

17/06/11 00:21:40 INFO mapred.JobClient:     Spilled Records=8

17/06/11 00:21:40 INFO mapred.JobClient:     Map output materialized bytes=52

17/06/11 00:21:40 INFO mapred.JobClient:     Reduce input records=4

17/06/11 00:21:40 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=3872464896

17/06/11 00:21:40 INFO mapred.JobClient:     Map input records=3

17/06/11 00:21:40 INFO mapred.JobClient:     SPLIT_RAW_BYTES=109

17/06/11 00:21:40 INFO mapred.JobClient:     Map output bytes=38

17/06/11 00:21:40 INFO mapred.JobClient:     Reduce shuffle bytes=52

17/06/11 00:21:40 INFO mapred.JobClient:     Physical memory (bytes) snapshot=252497920

17/06/11 00:21:40 INFO mapred.JobClient:     Reduce input groups=3

17/06/11 00:21:40 INFO mapred.JobClient:     Combine output records=0

17/06/11 00:21:40 INFO mapred.JobClient:     Reduce output records=3

17/06/11 00:21:40 INFO mapred.JobClient:     Map output records=4

17/06/11 00:21:40 INFO mapred.JobClient:     Combine input records=0

17/06/11 00:21:40 INFO mapred.JobClient:     CPU time spent (ms)=1300

17/06/11 00:21:40 INFO mapred.JobClient:     Total committed heap usage (bytes)=160501760

17/06/11 00:21:40 INFO mapred.JobClient:   File Input Format Counters

17/06/11 00:21:40 INFO mapred.JobClient:     Bytes Read=23

17/06/11 00:21:40 INFO mapred.JobClient:   FileSystemCounters

17/06/11 00:21:40 INFO mapred.JobClient:     HDFS_BYTES_READ=132

17/06/11 00:21:40 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=115065

17/06/11 00:21:40 INFO mapred.JobClient:     FILE_BYTES_READ=52

17/06/11 00:21:40 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=22

17/06/11 00:21:40 INFO mapred.JobClient:   Job Counters

17/06/11 00:21:40 INFO mapred.JobClient:     Launched map tasks=1

17/06/11 00:21:40 INFO mapred.JobClient:     Launched reduce tasks=1

17/06/11 00:21:40 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=9157

17/06/11 00:21:40 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0

17/06/11 00:21:40 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=6512

17/06/11 00:21:40 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0

17/06/11 00:21:40 INFO mapred.JobClient:     Data-local map tasks=1

17/06/11 00:21:40 INFO mapred.JobClient:   File Output Format Counters

17/06/11 00:21:40 INFO mapred.JobClient:     Bytes Written=22

Colored by Color Scripter
cs

맵리듀스 출력디렉토리 확인

1

2

3

4

5

6

[hduser@hdstudy01 hadoop]$ hadoop fs -ls wordcount_output5

Found 3 items

-rw-r--r--   1 hduser supergroup          0 2017-06-11 00:21 /user/hduser/wordcount_output5/_SUCCESS

drwxr-xr-x   - hduser supergroup          0 2017-06-11 00:21 /user/hduser/wordcount_output5/_logs

-rw-r--r--   1 hduser supergroup         22 2017-06-11 00:21 /user/hduser/wordcount_output5/part-r-00000

Colored by Color Scripter
cs

맵리듀스 출력결과물 확인

1

2

3

4

5

[hduser@hdstudy01 hadoop]$ hadoop fs -cat wordcount_output5/part-r-00000

Bye     1

Hello   1

World   2

Colored by Color Scripter
cs

각 단계별 로그파일과 설정정보가 함께 저장되기 때문에 별도의 파일에 출력한다.

- 웹로그 확인
http://hdstudy:50030/jobtracker.jsp

-완료-

이 블로그 검색

단다의 데이터과학 이야기

4장. 맵리듀스 프로그램

댓글

댓글 쓰기

가장 많이 본 글

7장. 맵리듀스 정렬구현

Keras - RNN