Hive问题

GC overhead limit exceeded

  • 问题描述：

DataGrip连接hiveserver2执行带MR的sql不停。

  • 报错信息：

两个文件的日志信息

  1. Job Submission failed with exception 'java.io.InterruptedIOException(Retry interrupted)'
  2. FAILED: command has been interrupted: during query execution: 
  3. null
  4. OK
  5. OK
  6. OK
  7. OK
  8. OK
  9. Exception in thread "HiveServer2-Handler-Pool: Thread-431" java.lang.OutOfMemoryError: GC overhead limit exceeded
  10.     at java.nio.ByteBuffer.wrap(ByteBuffer.java:373)
  11.     at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
  12.     at org.apache.hadoop.hive.serde2.thrift.ColumnBuffer.toTColumn(ColumnBuffer.java:317)
  13.     at org.apache.hive.service.cli.ColumnBasedSet.toTRowSet(ColumnBasedSet.java:165)
  14.     at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:791)
  15.     at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
  16.     at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
  17.     at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
  18.     at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
  19.     at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
  20.     at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
  21.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  22.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  23.     at java.lang.Thread.run(Thread.java:748)
  26. java.io.FileNotFoundException: File does not exist: /tmp/atguigu/operation_logs/756591cf-0199-454f-8464-30ab702b8a5e
  27.     at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2275) ~[commons-io-2.4.jar:2.4]
  28.     at org.apache.hive.service.cli.session.HiveSessionImpl.cleanupSessionLogDir(HiveSessionImpl.java:793) ~[hive-service-3.1.2.jar:3.1.2]
  29.     at org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:754) ~[hive-service-3.1.2.jar:3.1.2]
  30.     at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.close(HiveSessionImplwithUGI.java:93) ~[hive-service-3.1.2.jar:3.1.2]
  31.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212]
  32.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212]
  33.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
  34.     at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
  35.     at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.2.jar:3.1.2]
  36.     at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.2.jar:3.1.2]
  37.     at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.2.jar:3.1.2]
  38.     at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_212]
  39.     at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_212]
  40.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.3.jar:?]
  41.     at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.2.jar:3.1.2]
  42.     at com.sun.proxy.$Proxy37.close(Unknown Source) ~[?:?]
  43.     at org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:552) ~[hive-service-3.1.2.jar:3.1.2]
  44.     at org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:241) ~[hive-service-3.1.2.jar:3.1.2]
  45.     at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService$1.deleteContext(ThriftBinaryCLIService.java:141) ~[hive-service-3.1.2.jar:3.1.2]
  46.     at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300) ~[hive-exec-3.1.2.jar:3.1.2]
  47.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212]
  48.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212]
  49.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
  50. 2022-02-27T21:55:59,699  INFO [756591cf-0199-454f-8464-30ab702b8a5e HiveServer2-Handler-Pool: Thread-444] session.SessionState: Resetting thread name to  HiveServer2-Handler-Pool: Thread-444
  • 解决方案：

错误原因：
由java.lang.OutOfMemoryError: Java heap space 可知 hiveserver2运行出现 JVM内存溢出
This Error happened during connect hiveserver2 via beeline, both happened from hiveserver node and remote node.
解决方法：
heapsize太小了，可以适当的调大些
Root cause, the heapsize of hadoop opts is too small, need to increase the size in hive-env.sh as bold.

  1. 修改hive-env.sh配置文件，详情修改内容如下
  2. cd hive/conf
  3. mv hive-env.sh.template  hive-env.sh
  5. # Hive Client memory usage can be an issue if a large number of clients
  6. # are running at the same time. The flags below have been useful in 
  7. # reducing memory usage:
  8. #
  9. if [ "$SERVICE" = "cli" ]; then
  10.    if [ -z "$DEBUG" ]; then
  11.      export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -Xmx4096m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
  12.    else
  13.      export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -Xmx4096m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"
  14.    fi
  15. fi
  17. # The heap size of the jvm stared by hive shell script can be controlled via:
  18. #
  19. export HADOOP_HEAPSIZE=4096
  20. #
  21. # Larger heap size may be required when running queries over large number of files or partitions. 
  22. # By default hive shell scripts use a heap size of 256 (MB).  Larger heap size would also be 
  23. # appropriate for hive server.