Conversation

Notices

1. Baldur Bjarnason (baldur@toot.cafe)'s status on Tuesday, 21-Mar-2023 10:10:01 JST Baldur Bjarnason

   in reply to

   • Cory Doctorow

   @pluralistic

   That's a really useful post but I think they're wrong on one point.

   Training data leakage actually seems to be _the norm_. Most of the field ignores the cardinal rule of not testing on your training data, and it's caused a reproducibility crisis in ML-based science https://reproducible.cs.princeton.edu/

   That OpenAI pulled this stunt isn't a mistake. It's par for the course. This is how the AI industry oversells the capabilities of its products.

   • Cory Doctorow (pluralistic@mamot.fr)'s status on Tuesday, 21-Mar-2023 10:10:03 JST Cory Doctorow

     GPT-4 and professional benchmarks: the wrong answer to the wrong question

     https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks

     Adrian Cochrane repeated this.