Hi BODS Gurus,
My scenario is load data to HIVE via DataServices from ODP. Before load to HIVE, DataServices will generate large temp file very slow, and at the same time, there will be java threads consume huge memory up to 10G.
When I run 4 jobs at the same time the job server will crash and the error info as following:
--------------------------------------------------------------------------------------------------------------------------------
/usr/local/hive/bin/hive-config.sh: fork: retry: Resource temporarily unavailable
/usr/local/hadoop/bin/../libexec/hadoop-config.sh: fork: retry: Resource temporarily unavailable
/usr/local/hadoop/bin/../libexec/hadoop-config.sh: fork: retry: Resource temporarily unavailable
Logging initialized using configuration in file:/usr/local/hive/conf/hive-log4j.properties
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
---------------------------------------------------------------------------------------------------------------------------------
1. How cloud I separate the large temp file into several smaller temp files and load them to HIVE?
2. What caused the java threads consume huge memory? how can I reduce the memory consumption?
Thanks in advance.
Best Regards,
Andy