> there are no regularizers in any modern LLMs. Using a large & diverse training... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		visarga on Sept 5, 2023 \| parent \| context \| favorite \| on: Chinchilla’s death > there are no regularizers in any modern LLMs. Using a large & diverse training set is the best regulariser, but I think there is also weight decay and dropout in transformers

euclaise on Sept 6, 2023 [–]

RWKV also uses some sort of L2-esque regularization, which was supposedly an idea taken from PaLM (although I can't find a source on this point, other than some message in the RWKV discord)

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact