Embodied Cognition
The currently in-vogue view of how humans think and process the world around them is called "embodied cognition," and you can read its wikipedia page if you want. However, I've never really been properly taught what the term means, but still like to use it, so I'll be defining what it means to me. To me, it is a view that the ways we think and the way we talk are determined by our experiences in the world. The words we use don't have clear and defined meanings, but are instead all referential to experiences common to people. In practical terms, this means I don't think any AI will be able to hold a conversation or do near-human-quality translation until it is at human intelligence, and can be given human experiences (e.g. walking around, picking up objects, looking at things with two high-resolution optical sensors that perceive the visible wavelengths of light). This isn't because translation and conversation require intelligence per se, but because otherwise it won't get the references that make up language. If you've ever heard two people talking about some shared special interest (a movie, a game, an academic field, etc.), you'll know that you can't understand them, let alone translate their words or respond intelligently. Yes, this is partially due to jargon, but the omnipresence of jargon in any interest or field shows that we need jargon, we need to invent new words, to cover new experiences. Embodied cognition isn't just about language, however. Math, especially fundamental math (logic), is also based on embodied experience. Why do we take the law of the excluded middle as an axiom? Why do we define an addition, subtraction, multiplication, etc? Why did Euclid pick the axioms he picked? There are many many possible "maths" which could exist, and the reason we study the ones we do is because they align with, or arise from, our embodied experience. We pick out individual objects and say that some things are the same as others, so it makes sense to count how many objects of one type you have, and so we get numbers and addition and subtraction. If we did not view objects as belonging to categories, maybe we would not have this kind of arithmetic. Contrast "you have 6 apples, you take away 3, how many do you have left" with (and language limits me here, since this is trying to express a thought very disconnected from embodied expereince) "you have a green and lumpy, a red and shiny, and a red and matte. The red and matte is taken by a friend. What do you have left?"
We also understand abstract concepts in terms of things familiar to our experience. Notice the tendency to personify evolution: we almost always say "giraffes evolved to have long necks" or "evolution favored finches with beaks for opening nuts," when it would be more accurate to say something like "giraffes with longer necks survived, leading to a population with longer necks overall." This isn't to say the first version is wrong, as it makes it easier to teach, quicker to say, and people still understand what is being described, if they take a moment to think it through. It's just that we see individual organisms change all the time (people growing up, bones getting broken, trees flowering) but have significantly less experience seeing populations change. Metaphor is key to learning for this reason - we need a connection to embodied experience to have understanding. Proteins and cell permeability are very hard to imagine, since we can't perceive it in any way, but bring in a comparison to locks and keys (objects used every day) and suddenly it clicks. Hours of study, understanding every single charge carrier and bond involved, would grant a greater ability to do further computation or prediction than the key metaphor, but it would still not grant more understanding, in my experience.
A major takeaway from this, for me, is the breaking of the illusion of logic, the idea that humans have access to this sacred domain of Truth that transcends our biases and messy emotions, a Truth that is universal and in some way embedded into the nature of reality itself. The idea that there is nothing subjective about "1+1=2" or "the sun is hot" or "F = mv", something more than human about them. The idea that any machine intelligence in the future, any alien we meet, or even anyone from an alternate dimension pulled into our own, would agree on these statements. Embodied cognition says no, that is not true, and in fact they would not even understand these statements enough to form an opinion on them, not unless they had significantly similar experiences to humans. These are not truths of reality, but truths of human umwelt or human culture. They may be special in that anyone who can understand them would agree on them, or in that they require very little shared experience, but they are not properties of the universe, just properties of us. This isn't to say that any communication with, say, aliens, is impossible. The Voyager Golden Record is a wonderful example of trying to minimize the shared embodiement required to understand a message. Obviously the aliens who pick it up need to be able to hear and see to appreciate the contents of the record (which is already quite a steep requirement) but the diagram showing the location of our sun only requires that they can understand a pictographic representaion of something, that they know what hydrogen is and what electron spin is, and that they can observe pulsars and measure their frequency. These are reasonable experiences for any intelligent life anywhere in our universe to have, and so, it's not absurd to suppose the voyager record could communicate with them. However, it would be entirely misunderstood by any entity from a different universe where hydrogen has a different fundamental transition. It would also be incomprehensible to anything that doesn't perceive time as we do, or doesn't perceive space like us.