Good benchmarks are costly to build even for mid-large corporations. And once th... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

charlieyu1 14 days ago | parent | context | favorite | on: Goodhart's Law Comes for Every Benchmark You Trust

Good benchmarks are costly to build even for mid-large corporations. And once the benchmark is used on models you really can’t tell if the problems would be scrapped for training

fluoridation 14 days ago [–]

I was speaking in general, not just about AI.

pixl97 13 days ago | [–]

Any benchmark that has a selection condition can be discovered. The more money involved, the more effort that will go into discovering said benchmark.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact