当我们使用Hadoop的MapReduce框架来运行一个示例程序的时候,例如:

 hadoop jar hadoop-mapreduce-examples-3.4.0.jar grep input output 'dfs[a-z.]+'

报出以下错误:

一开始还以为是电脑的配置不太行,然后找了很多东西,发现只需要在yarn-site.xml文件中加入以下配置:

首先你通过hadoop classpath获取到这个位置

然后将这个位置复制,写入yarn-site.xml中

<property>
        <name>yarn.application.classpath</name>
        <value>/opt/hadoop-3.4.0/etc/hadoop:/opt/hadoop-3.4.0/share/hadoop/common/lib/*:/opt/hadoop-3.4.0/share/hadoop/common/*:/opt/hadoop-3.4.0/share/hadoop/hdfs:/opt/hadoop-3.4.0/share/hadoop/hdfs/lib/*:/opt/hadoop-3.4.0/share/hadoop/hdfs/*:/opt/hadoop-3.4.0/share/hadoop/mapreduce/*:/opt/hadoop-3.4.0/share/hadoop/yarn:/opt/hadoop-3.4.0/share/hadoop/yarn/lib/*:/opt/hadoop-3.4.0/share/hadoop/yarn/*
</value>
    </property>

同时yarn-site.xml文件中要对外暴露端口,我完整的yarn-site.xml文件如下:

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
    </property>

    <property>
        <name>yarn.resourcemanager.address</name>
        <value>node1:8032</value>
    </property>

    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>node1:8030</value>
    </property>

    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>node1:8031</value>
    </property>
    <property>
        <name>yarn.nodemanager.memory-mb</name>
        <value>4GB</value>
    </property>
    <property>
<name>yarn.nodemanager.cpu-vcores</name>
        <value>4</value>
    </property>
   <property>
        <name>yarn.application.classpath</name>
        <value>/opt/hadoop-3.4.0/etc/hadoop:/opt/hadoop-3.4.0/share/hadoop/common/lib/*:/opt/hadoop-3.4.0/share/hadoop/common/*:/opt/hadoop-3.4.0/share/hadoop/hdfs:/opt/hadoop-3.4.0/share/hadoop/hdfs/lib/*:/opt/hadoop-3.4.0/share/hadoop/hdfs/*:/opt/hadoop-3.4.0/share/hadoop/mapreduce/*:/opt/hadoop-3.4.0/share/hadoop/yarn:/opt/hadoop-3.4.0/share/hadoop/yarn/lib/*:/opt/hadoop-3.4.0/share/hadoop/yarn/*
</value>
    </property>
</configuration>

提供参考

重新start-yarn.sh

结果如下,发现成功了:

然后访问自己的   ip地址+8088端口

发现没有问题

然后查看Browse Directory查看是否生成output文件

同时通过命令:hdfs dfs -cat /user/root/output/part-r-00000查看运行后的内容

至此,问题解决

Logo

技术共进,成长同行——讯飞AI开发者社区

更多推荐