Tallies with my experience. It's absolutely terrible with anything requiring spatial awareness, confusing near and far, left and right. All wordcel, zero shape rotator. Apparently requires a physique immersed in space.
I saw a nice rendering of it floating around, maybe on Facebook somewhere?
It's sorta unsurprising to me that these types of construction results are the things that the AI is specifically good at, since you can get a lot further just by trying a lot of things with tireless effort.
The lack of geometric intuition of these models still continues to surprise me. I have a test problem that keeps stumping even the good models (write a program that, given a binary matrix, produces the area of the largest orthogonally connected region made entirely of ones without having 3 consecutive ones in any row or column). They write horrible solutions without making the trivial structural observation.
if we actually look at Jacob Tsimerman's comment in the paper he states: "I actually briefly worked on this problem and tried to make a counterexample, but failed to make progress. On Boris Alexeev’s suggestion, I thought about this problem with the idea of making a counterexample stemming from a varying family of bounded degree number fields.” i.e. the exact approach that yielded the counterexample and guess who Boris Alexeev works for? - OpenAI. My guess is that this is one of the reasons why OpenAI's proof does not include a list of authors/contributors. I know OpenAI claims that the problem was "one-shotted" by a "general purpose internal model" but I think at this stage trusting company announcements/demos is naive at best.
I wonder if a model that is in part trained to operate some kind of body in the world may be more likely to develop some kind of geometric intuition? There’s a level at which geometry is an idealization of the physical — it underpins basic architecture and engineering. Arches, cantilevers, etc. Not to mention ballistics. (A cat may not be able to factor a conic section, but they have a better intuitive understanding of parabolic arcs than most humans who can.)
More training on geometric things would undoubtedly help. It rapidly gets trickier as the number of dimensions goes up because there's just so much space.
Tallies with my experience. It's absolutely terrible with anything requiring spatial awareness, confusing near and far, left and right. All wordcel, zero shape rotator. Apparently requires a physique immersed in space.
I saw a nice rendering of it floating around, maybe on Facebook somewhere?
It's sorta unsurprising to me that these types of construction results are the things that the AI is specifically good at, since you can get a lot further just by trying a lot of things with tireless effort.
The lack of geometric intuition of these models still continues to surprise me. I have a test problem that keeps stumping even the good models (write a program that, given a binary matrix, produces the area of the largest orthogonally connected region made entirely of ones without having 3 consecutive ones in any row or column). They write horrible solutions without making the trivial structural observation.
Haha I'm not going to make the obvious observation because I don't want to contaminate future training but yeah that's a trivial linear pass.
if we actually look at Jacob Tsimerman's comment in the paper he states: "I actually briefly worked on this problem and tried to make a counterexample, but failed to make progress. On Boris Alexeev’s suggestion, I thought about this problem with the idea of making a counterexample stemming from a varying family of bounded degree number fields.” i.e. the exact approach that yielded the counterexample and guess who Boris Alexeev works for? - OpenAI. My guess is that this is one of the reasons why OpenAI's proof does not include a list of authors/contributors. I know OpenAI claims that the problem was "one-shotted" by a "general purpose internal model" but I think at this stage trusting company announcements/demos is naive at best.
I wonder if a model that is in part trained to operate some kind of body in the world may be more likely to develop some kind of geometric intuition? There’s a level at which geometry is an idealization of the physical — it underpins basic architecture and engineering. Arches, cantilevers, etc. Not to mention ballistics. (A cat may not be able to factor a conic section, but they have a better intuitive understanding of parabolic arcs than most humans who can.)
More training on geometric things would undoubtedly help. It rapidly gets trickier as the number of dimensions goes up because there's just so much space.