Unable to run interactive_hosted locally
Created by: suchenzang
To reproduce:
- ssh to node 131 on cluster
- confirm that mp 175B shards exist in
/mnt/scratch/175B/reshard_no_os/
- permissions on dir has been opened to all (chmod 777)
- metaseq is on main
Resulting stacktrace:
(metaseq-test-apr7) susanz@<node 131 on cluster>:~/staging/metaseq$ python -m metaseq_cli.interactive_hosted
2022-05-08 20:09:42 | INFO | metaseq_cli.interactive | Local checkpoint copy already exists, skipping copy
2022-05-08 20:09:42 | INFO | metaseq.tasks.language_modeling | dictionary: 50272 types
2022-05-08 20:09:42 | INFO | metaseq.hub_utils | loading model(s) from /mnt/scratch/175B/reshard_no_os/reshard.pt
Traceback (most recent call last):
File "/shared/home/susanz/miniconda3/envs/metaseq-test-apr7/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/shared/home/susanz/miniconda3/envs/metaseq-test-apr7/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/shared/home/susanz/staging/metaseq/metaseq_cli/interactive_hosted.py", line 304, in <module>
cli_main()
File "/shared/home/susanz/staging/metaseq/metaseq_cli/interactive_hosted.py", line 300, in cli_main
dist_utils.call_main(cfg, worker_main, namespace_args=args)
File "/shared/home/susanz/staging/metaseq/metaseq/distributed/utils.py", line 263, in call_main
return main(cfg, **kwargs)
File "/shared/home/susanz/staging/metaseq/metaseq_cli/interactive_hosted.py", line 156, in worker_main
models = generator.load_model() # noqa: F841
File "/shared/home/susanz/staging/metaseq/metaseq/hub_utils.py", line 485, in load_model
models, _model_args, _task = checkpoint_utils.load_model_ensemble_and_task(
File "/shared/home/susanz/staging/metaseq/metaseq/checkpoint_utils.py", line 473, in load_model_ensemble_and_task
state = load_checkpoint_to_cpu(filename, arg_overrides)
File "/shared/home/susanz/staging/metaseq/metaseq/checkpoint_utils.py", line 408, in load_checkpoint_to_cpu
paths_to_load = get_paths_to_load(local_path, suffix="shard")
File "/shared/home/susanz/staging/metaseq/metaseq/checkpoint_utils.py", line 348, in get_paths_to_load
if not _is_checkpoint_sharded(checkpoint_files):
File "/shared/home/susanz/staging/metaseq/metaseq/checkpoint_utils.py", line 339, in _is_checkpoint_sharded
size_ratio = max(sizes) / min(sizes)
ValueError: max() arg is an empty sequence
@stephenroller last time we saw this, it seemed to be a read permissions issue?