Euclid's Definition 1
We normally think of a geometrical point as being smaller than anything, because of where Euclid placed it in the hierarchy of geometry. Building up from simple to more complex, his first definition is the simplest: "A point is that which has no part." Clearly, Euclid intended for anyone reading to be thinking of something simple, like a pebble in the hands of a sculptor, with "parts" being removed, getting smaller and smaller until it has no parts.
Mathematics is considered to happen outside the bounds of physics, but it is clear that his definition presupposes the existence of something like matter which he then eliminates, leaving a point so small that it no longer has no material existence. Since the matter (the "part") is eliminated from existence even as it is mentioned into existence, there is no need to discuss it further. In other words, although matter is being used to define the point, the point itself has no matter within it.
At the time, this approach was a clever way of managing two problems with Greek mathematical understanding.
Problem 1: Horror vacui
First was a strong opposition to the idea of "the void." Wikipedia says:
In philosophy and early physics, horror vacui (Latin: horror of the vacuum) — commonly stated as "nature abhors a vacuum", for example by Spinoza — is a hypothesis attributed to Aristotle that nature contains no vacuums because the denser surrounding material continuum would immediately fill the rarity of an incipient void. Aristotle also argued against the void in a more abstract sense: since a void is merely nothingness, following his teacher Plato, nothingness cannot rightly be said to exist. Furthermore, insofar as a void would be featureless, it could neither be encountered by the senses nor could its supposition lend additional explanatory power.
Following Aristotle, Euclid would have understood that geometry could never be founded upon the void, even though it would be the one thing conceptually more simple than "that which has no part." Remember this was in the age where a man was drowned at sea for telling the hidden truth about irrational numbers. Because the horror vacui was strong enough (including the fact that there were negative religious connotations involved) it was likely conceptually impossible for Euclid to understand the point that "that which has no part" and "the void" were identical. In our age, we can see they are essentially talking about the same thing, but Euclid likely imagined that he was creating a technical description of a partless mathematical entity which was unrelated to the void.
Problem 2: Measuring infinitesimals
The second problem with Greek mathematical understanding would have helped hide the first problem even from a penetrating mind like Euclid. This problem required twenty more centuries to pass before it began to be resolved with the invention of calculus by Newton and Leibniz. It's the problem of the infinitesimal. This is the problem that Euclid was solving by saying "A point is that which has no part" instead of "A point is that which has the least-measurable-sized part," which would be an awkward but precise way of capturing the basic intuition that happens when we normally think of a point. Until we learn geometry, we do not think of point-like things as "having no part," but Euclid does, and he does it to avoid any reference to a measurable part.
Why? Infinitesimals are problematic. Euclid would have known about Zeno's paradoxes which involve an infinite number of infinitesimals to measure any distance -- the existence of which can be used to prove that motion is impossible. Infinitesimals in his time were understood as the mathematical inversion of infinity -- the opposite of infinity -- and they were already known to be logically nontrivial. Euclid knew he could not rely on an infinitesimal for the Definition 1 of geometry.
So this is why "that which has no part" is used instead of "that which has the tiniest possible size" or anything like it. And in using this phrasing, Euclid sidestepped the problem with infinitesimals. Curiously, the similarity of this description to infinitesimals helped cover up Problem #1 by drawing attention away from any obvious void analogies.
The Solution: Neither here nor there is better than either here or there
So those are the two problems that Euclid faced. He invented a clever solution which evaded both horns of the dilemma simultaneously.
At heart is a semantic trick which, when unraveled by logic, is not itself a better solution than either the void or the infinitesimal, being a hybrid of both which is also neither. For the purpose of comparing with the two problematic solutions, let's call this semantic trick the neither void nor infinitesimal because it's not really a "third" alternative so much as it's a clever combination of the first two in a way that pits them against each other and makes it possible to think a genuine third alternative exists.
All three solutions are roughly equal to the task of defining the simplest thing upon which to build the rest of geometry. That was Euclid's aim while creating Definition 1. But Euclid's choice, "that which has no part," has the paradoxical advantage that it brings an infinitesimal "that" into being and then immediately eliminates "that" by saying it has no being. We know it has no being because it has "no part." We're supposed to politely ignore the fact that something which has no part necessarily has no whole. This all happens within a simple phrase.
Any arguments talking about the simple phrase's similarity to the void can be easily dismissed, as well as any arguments talking about its similarity to the infinitesimal.
This hybrid solution which is half-infinitesimal and half-void while also being neither infinitesimal nor void while also being essentially simple must have seemed so clever to Euclid. It worked well enough to be the first definition in geometry for many centuries. However, by the time we get to our era, we've developed delicate logical scalpels which are sharper than the more concrete-minded ancient Greek mathematical tools, and it frankly doesn't work any more, although few actually take the time to question it.
Even though armed with greater precision, we now have a problem that Euclid's geometry didn't have when it was created: It has been around for a couple dozen centuries and its logical assumptions have pervaded the way we think in fundamental ways. It's hard to think outside of its constraints. The advent of non-Euclidean geometry in the 1800s was a good start in shaking these solid foundations. But we can do better.
Now it's time to unravel something even more fundamental than the 5th postulate: the 1st definition.