Created by: ruanslv
Patch Description Added a few config options so we can kick off an arbitrary evals command during training. This allows us to keep logic in OSS land without adding dependencies to our metaseq-internal evals code.
Instead of having the two command requirements, we could also add logic for them in metaseq directly, but they are cloud env specific, so it would mean more of our azure internal logic getting added to OSS land.
Instead, for now internally we can keep the logic in the eval scripts.
Testing steps
Using our baseline sweeps setting the new params. Example:
--eval_command = f""" python metaseq_internal/scripts/eval/schedule_jobs_few_shot_opt_evaluation.py -o {output_path} -t opt_eval_tasks --model-name {args.prefix} --model-template bf16_sharded_config --slurm-partition zetta """