异常
2022-10-10 19:41:55,541 INFO [IPC Server handler 17 on 33142] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1663912973547_0434_m_000000_0: Error: java.io.IOException: File copy failed: hdfs://10.255.128.198:8020/user/hive/warehouse/vaas_dws.db/dws_ac_estimate_soh_yuce_partion/model_estimate_date=2022-09-27/000000_0 --> hdfs://10.246.97.4:8020/user/hive/warehouse/vaas_dws.db/dws_ac_estimate_soh_yuce_partion/dws_ac_estimate_soh_yuce_partion/model_estimate_date=2022-09-27/000000_0
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:259)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:217)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://10.255.128.198:8020/user/hive/warehouse/vaas_dws.db/dws_ac_estimate_soh_yuce_partion/model_estimate_date=2022-09-27/000000_0 to hdfs://10.246.97.4:8020/user/hive/warehouse/vaas_dws.db/dws_ac_estimate_soh_yuce_partion/dws_ac_estimate_soh_yuce_partion/model_estimate_date=2022-09-27/000000_0
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:256)
... 10 more
Caused by: java.io.IOException: java.nio.channels.UnresolvedAddressException
at org.apache.hadoop.hdfs.ExceptionLastSeen.set(ExceptionLastSeen.java:45)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:829)
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:101)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
解决方案：(异常的原因是因为跨集群的时候无法解析跨集群的namenode的主机导致的)
需要把两个集群的hosts合并重新distcp后数据正常