[Cleanup] Continue cleaning decoder args: 10.6/N

Created by: suchenzang

[ Note this will be rebased onto main after https://github.com/facebookresearch/metaseq/pull/372 and upstream diffs are merged independently ]