# MapReduce 教程 [TOC] ## Purpose 目標 This document comprehensively describes all user-facing facets of the Hadoop MapReduce framework and serves as a tutorial. 這個文檔綜合地描述了所有面向用戶的Hadoop MapReduce 框架和服務。 ## Prerequisites必要準備 Ensure that Hadoop is installed, configured and is running. More details: 請確保安裝、配置并運行起來 Hadoop，更多的請參考環境安裝環節（下面的是原文鏈接，英文不錯的可以試試） * [Single Node Setup](http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html)for first-time users. * [Cluster Setup](http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html)for large, distributed clusters. ## Overview Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner. A MapReduce*job*usually splits the input data-set into independent chunks which are processed by the*map tasks*in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the*reduce tasks*. Typically both the input and the output of the job are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks. Typically the compute nodes and the storage nodes are the same, that is, the MapReduce framework and the Hadoop Distributed File System (see[HDFS Architecture Guide](http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html)) are running on the same set of nodes. This configuration allows the framework to effectively schedule tasks on the nodes where data is already present, resulting in very high aggregate bandwidth across the cluster. The MapReduce framework consists of a single masterResourceManager, one slaveNodeManagerper cluster-node, andMRAppMasterper application (see[YARN Architecture Guide](http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YARN.html)). Minimally, applications specify the input/output locations and supply*map*and*reduce*functions via implementations of appropriate interfaces and/or abstract-classes. These, and other job parameters, comprise the*job configuration*. The Hadoop*job client*then submits the job (jar/executable etc.) and configuration to theResourceManagerwhich then assumes the responsibility of distributing the software/configuration to the slaves, scheduling tasks and monitoring them, providing status and diagnostic information to the job-client. Although the Hadoop framework is implemented in Java?, MapReduce applications need not be written in Java. * [Hadoop Streaming](http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/streaming/package-summary.html)is a utility which allows users to create and run jobs with any executables (e.g. shell utilities) as the mapper and/or the reducer. * [Hadoop Pipes](http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/pipes/package-summary.html)is a[SWIG](http://www.swig.org/)\-compatible C++ API to implement MapReduce applications (non JNI? based).