往hive中的表插入数据的时候,不能直接运行成功,报一下错误

错误原因:
第一种,namenode内存空间不够
原因:JVM剩余内存空间不够新job运行所致

错误提示

0: jdbc:hive2://hadoop101:10000> insert into table student values(1002,"zss");
INFO  : Compiling command(queryId=root_20210909172055_08738d9c-4ded-4067-a2ac-6a64572ad49b): insert into table student values(1002,"zss")
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null), FieldSchema(name:col2, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=root_20210909172055_08738d9c-4ded-4067-a2ac-6a64572ad49b); Time taken: 0.425 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=root_20210909172055_08738d9c-4ded-4067-a2ac-6a64572ad49b): insert into table student values(1002,"zss")
WARN  : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
INFO  : Query ID = root_20210909172055_08738d9c-4ded-4067-a2ac-6a64572ad49b
INFO  : Total jobs = 3
INFO  : Launching Job 1 out of 3
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Number of reduce tasks determined at compile time: 1
INFO  : In order to change the average load for a reducer (in bytes):
INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
INFO  : In order to limit the maximum number of reducers:
INFO  :   set hive.exec.reducers.max=<number>
INFO  : In order to set a constant number of reducers:
INFO  :   set mapreduce.job.reduces=<number>
INFO  : number of splits:1
INFO  : Submitting tokens for job: job_1631178391888_0002
INFO  : Executing with tokens: []
INFO  : The url to track the job: http://hadoop101:8088/proxy/application_1631178391888_0002/
INFO  : Starting Job = job_1631178391888_0002, Tracking URL = http://hadoop101:8088/proxy/application_1631178391888_0002/
INFO  : Kill Command = /opt/bagdata/hadoop-3.1.3/bin/mapred job  -kill job_1631178391888_0002
INFO  : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
INFO  : 2021-09-09 17:21:13,094 Stage-1 map = 0%,  reduce = 0%
INFO  : 2021-09-09 17:21:53,093 Stage-1 map = 100%,  reduce = 100%
ERROR : Ended Job = job_1631178391888_0002 with errors
ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
INFO  : MapReduce Jobs Launched: 
INFO  : Stage-Stage-1: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
INFO  : Total MapReduce CPU Time Spent: 0 msec
INFO  : Completed executing command(queryId=root_20210909172055_08738d9c-4ded-4067-a2ac-6a64572ad49b); Time taken: 59.838 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2

解决办法1:

set hive.exec.mode.local.auto=true;

错误原因
第二种,yarn资源不足
原因:
该错误是YARN的虚拟内存计算方式导致,上例中用户程序申请的内存为1Gb,YARN根据此值乘以一个比例(默认为2.1)得出申请的虚拟内存的值,当YARN计算的用户程序所需虚拟内存值大于计算出来的值时,就会报出以上错误。调节比例值可以解决该问题。具体参数为:yarn-site.xml中的yarn.nodemanager.vmem-pmem-ratio

解决方法:
调整hadoop配置文件yarn-site.xml中值:

<property>
   <name>yarn.scheduler.minimum-allocation-mb</name>
   <value>2048</value>
   <description>default value is 1024</description>
</property>

增加yarn.scheduler.minimum-allocation-mb 数量,从缺省1024改为2048;上述运行问题即刻得到解决

Logo

技术共进,成长同行——讯飞AI开发者社区

更多推荐