Fix inclusion of valid subset in opt-baseline (!313) · Merge requests · Administrator / metaseq

Merged Administrator requested to merge susan/fix_benchmark_data into main Aug 20, 2022

Created by: suchenzang

Brittle logic here, but valid subset currently exists if args.data is not None (as in, args.data is only None when args.benchmark is set ). Without this fix, throws:

  File "/shared/home/susanz/repos/metaseq/slurm_snapshot_code_oss/2022-08-20T22_54_06.361518/metaseq/tasks/streaming_language_modeling.py", line 275, in get_shard_str
    int(shard_id) not in shards
ValueError: invalid literal for int() with base 10: 'BookCorpusFair'