Skip to content
GitLab

Initialization changes

Merged Administrator requested to merge susan/init_changes into main

Created by: suchenzang

  • Flag for scaling positional embedding init
  • Truncate init for embeddings too
  • Enable zeros init if sigma is small enough