共计 5544 个字符，预计需要花费 14 分钟才能阅读完成。
这篇文章主要介绍“hadoop 重写方法有哪些”，在日常操作中，相信很多人在 hadoop 重写方法有哪些问题上存在疑惑，丸趣 TV 小编查阅了各式资料，整理出简单好用的操作方法，希望对大家解答”hadoop 重写方法有哪些”的疑惑有所帮助！接下来，请跟着丸趣 TV 小编一起来学习吧！
1. 下载（略）
2. 编译（略）
3. 配置（伪分布、集群略）
4. Hdfs
1. Web interface:http://namenode-name:50070/(显示 datanode 列表和集群统计信息)
2. shell command dfsadmin comman
3. checkpoint node backup node
1. fsimage 和 edits 文件 merge 原理
2. （猜测是早期版本的特性）手动恢复宕掉的集群：import checkpoint；
3. backupnode: Backup Node 在内存中维护了一份从 Namenode 同步过来的 fsimage，同时它还从 namenode 接收 edits 文件的日志流，并把它们持久化硬盘，Backup Node 把收到的这些 edits 文件和内存中的 fsimage 文件进行合并，创建一份元数据备份。Backup Node 高效的秘密就在这儿，它不需要从 Namenode 下载 fsimage 和 edit，把内存中的元数据持久化到磁盘然后进行合并即可。
4. banlancer: 平衡各 rock 和 datanodes 数据不均衡
5. Rock awareness：机架感知
6. Safemode：当数据文件不完整或者手动进入 safemode 时，hdfs 只读，当集群检查达到阈值或手动离开安全模式时，集群恢复读写。
7. Fsck：块文件检查命令
8. Fetchdt: 获取 token（安全）
9. Recovery mode: 恢复模式
10. Upgrade and Rollback：升级、回滚
11. File Permissions and Security
12. Scalability
13.
5. Mapreduce
1.
public class MyMapper extends Mapper Object, Text, Text, IntWritable {
private Text word = new Text();
private IntWritable one = new IntWritable(1);
// 重写 map 方法
@Override
public void map(Object key, Text value, Context context)
throws IOException, InterruptedException {
StringTokenizer stringTokenizer = new StringTokenizer(value.toString());
while(stringTokenizer.hasMoreTokens()){
word.set(stringTokenizer.nextToken());
//（word,1）进行传递
context.write(word, one);
}
}
}
public class MyReducer extends Reducer Text, IntWritable, Text, IntWritable {
private IntWritable result = new IntWritable(0);
// 重写 reduce 方法
@Override
protected void reduce(Text key, Iterable IntWritable iterator,
Context context) throws IOException, InterruptedException {
int sum = 0;
for(IntWritable i : iterator){
sum += i.get();
}
result.set(sum);
// reduce 输出的值
context.write(key, result);
}
}
public class WordCountDemo {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, word count
job.setJarByClass(WordCountDemo.class);
// 设置 map、reduce class
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReducer.class);
job.setCombinerClass(MyReducer.class);
// 设置最终输出的格式
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
// 设置 FileInputFormat outputFormat
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
2. Job.setGroupingComparatorClass(Class).
3. Job.setCombinerClass(Class),
4. CompressionCodec
5. Map 数：Configuration.set(MRJobConfig.NUM_MAPS, int) = dataSize/blockSize
6. Reducer 数：Job.setNumReduceTasks(int).
With 0.95 all of the reduces can launch immediately and start transferring map outputs as the maps finish. With 1.75 the faster nodes will finish their first round of reduces and launch a second wave of reduces doing a much better job of load balancing.
7. Reduce- shuffle: Input to the Reducer is the sorted output of the mappers. In this phase the framework fetches the relevant partition of the output of all the mappers, via HTTP. – reduce 是 mapper 排序后的输出的结果。在这一阶段，框架通过 http 抓取所有 mapper 输出的有关分区。
8. Reduce – sort：The framework groups Reducer inputs by keys (since different mappers may have output the same key) in this stage. The shuffle and sort phases occur simultaneously; while map-outputs are being fetched they are merged.- 在这一阶段，框架按照输入的 key（不同的 mapper 可能输出相同的 key）分组 reducer。Shuffle 和 sort 会同时发生，当 map 输出被捕捉时，他们又会进行合并。
9. Reduce – reduce：
10. Secondary sort
11. Partitioner
12. Counter ：Mapper and Reducer implementations can use the Counter to report statistics.
13. Job conf：配置 – speculative manner ( setMapSpeculativeExecution(boolean))/setReduceSpeculativeExecution(boolean)), maximum number of attempts per task (setMaxMapAttempts(int)/ setMaxReduceAttempts(int)) etc.
Or
Configuration.set(String, String)/ Configuration.get(String)
14. Task executor environment – The user can specify additional options to the child-jvm via the mapreduce.{map|reduce}.java.opts and configuration parameter in the Job such as non-standard paths for the run-time linker to search shared libraries via -Djava.library.path= etc. If the mapreduce.{map|reduce}.java.opts parameters contains the symbol @taskid@ it is interpolated with value of taskid of the MapReduce task.
15. Memory management – Users/admins can also specify the maximum virtual memory of the launched child-task, and any sub-process it launches recursively, using mapreduce.{map|reduce}.memory.mb. Note that the value set here is a per process limit. The value for mapreduce.{map|reduce}.memory.mb should be specified in mega bytes (MB). And also the value must be greater than or equal to the -Xmx passed to JavaVM, else the VM might not start.
16. Map Parameters …… (http://hadoop.apache.org/docs/r2.6.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#MapReduce_Tutorial)
17. Parameters ()
18. Job submission and monitoring:
1.Job provides facilities to submit jobs, track their progress, access component-tasks reports and logs, get the MapReduce cluster s status information and so on.
2. The job submission process involves:
1. Checking the input and output specifications of the job.
2. Computing the InputSplit values for the job.
3. Setting up the requisite accounting information for the DistributedCache of the job, if necessary.
4. Copying the job s jar and configuration to the MapReduce system directory on the FileSystem.
5. Submitting the job to the ResourceManager and optionally monitoring it s status.
3. Job history
19. Job controller
1. Job.submit() || Job.waitForCompletion(boolean)
2. 多 Mapreduce job
1. 迭代式 mapreduce（上一个 mr 作为下一个 mr 的输入，缺点：创建 job 对象的开销、本地磁盘读写 io 和网络开销大）
2. MapReduce-JobControl：job 封装各个 job 的依赖关系，jobcontrol 线程管理各个作业的状态。
3. MapReduce-ChainMapper/ChainReduce：（chainMapper.addMap(). 可以在一个 job 中链接多个 mapper 任务，不可用于多 reduce 的 job）。
20. Job input output
1. InputFormat TextInputFormat FileInputFormat
2. InputSplit FileSplit
3. RecordReader
4. OutputFormat OutputCommitter
到此，关于“hadoop 重写方法有哪些”的学习就结束了，希望能够解决大家的疑惑。理论与实践的搭配能更好的帮助大家学习，快去试试吧！若想继续学习更多相关知识，请继续关注丸趣 TV 网站，丸趣 TV 小编会继续努力为大家带来更多实用的文章！
