错误日志:
Epoch: [229] Total time: 0:17:21
Test: [ 0/49] eta: 0:05:00 loss: 1.7994 (1.7994) acc1: 78.0822 (78.0822) acc5: 95.2055 (95.2055) time: 6.1368 data: 5.9411 max mem: 10624
WARNING::Received 1 death signal, shutting down workers
WARNING::Sending process 44348 closing signal SIGHUP
WARNING::Sending process 44349 closing signal SIGHUP
WARNING::Sending process 44350 closing signal SIGHUP
WARNING::Sending process 44351 closing signal SIGHUP
WARNING::Sending process 44352 closing signal SIGHUP
WARNING::Sending process 44353 closing signal SIGHUP
WARNING::Sending process 44354 closing signal SIGHUP
Traceback (most recent call last):
File "/home/biometrics/miniconda3/envs/torch/bin/torchrun", line 33, in <module>
(load_entry_point('torch==1.12.0.dev20220502', 'console_scripts', 'torchrun')())
File "/home/biometrics/miniconda3/envs/torch/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
return f(*args, **kwargs)
File "/home/biometrics/miniconda3/envs/torch/lib/python3.7/site-packages/torch/distributed/", line 761, in main
run(args)
File "/home/biometrics/miniconda3/envs/torch/lib/python3.7/site-packages/torch/distributed/", line 755, in run
)(*cmd_args)
File "/home/biometrics/miniconda3/envs/torch/lib/python3.7/site-packages/torch/distributed/launcher/", line 131, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/biometrics/miniconda3/envs/torch/lib/python3.7/site-packages/torch/distributed/launcher/", line 236, in launch_agent
result = ()
File "/home/biometrics/miniconda3/envs/torch/lib/python3.7/site-packages/torch/distributed/elastic/metrics/", line 125, in wrapper
result = f(*args, **kwargs)
File "/home/biometrics/miniconda3/envs/torch/lib/python3.7/site-packages/torch/distributed/elastic/agent/server/", line 709, in run
result = self._invoke_run(role)
File "/home/biometrics/miniconda3/envs/torch/lib/python3.7/site-packages/torch/distributed/elastic/agent/server/", line 850, in _invoke_run
(monitor_interval)
File "/home/biometrics/miniconda3/envs/torch/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/", line 60, in _terminate_process_handler
raise SignalException(f"Process {()} got signal: {sigval}", sigval=sigval)
: Process 44343 got signal: 1
网上的解决办法是: