Doesn't sound a very difficult question, does it? A mammal is a warm-blooded, air-breathing vertebrate that, crucially, feeds its young with milk from its mammary glands. Right? Well, kind of...
For the last twenty years or so, animals have been placed into groups on the basis of something called cladistics. Essentially, the idea is that a proper, meaningful, group of animals will be one that contains a single common ancestor and all of its descendants. Or, to put it another way, that everything in the group has to be more closely related to other animals in that group than it is to anything else. Which seems pretty straightforward and common-sense, and, indeed, is a very useful way of doing things. But applying it strictly does sometimes lead to some fairly surprising results.
Vertebrates first freed themselves entirely from the water when they evolved a way of making their own little pools of water, surrounded by a protective shell, and leaving their tadpoles inside that pool with a supply of yolk to feed off until they became developed enough to hatch. The animals that evolved this feature are called amniotes. Later on, some mammals evolved a way of doing away with the shell, and keeping the pond inside a membranous sac in the mother's body, but the principle is the same, and they still count as amniotes.
Not long after the amniotes first appeared, they split into two great evolutionary lines. One led to the reptiles and birds, and the other, called the Synapsida, led to the mammals.
Wait a minute, I can hear you saying, but didn't mammals evolve from reptiles? Well, it depends what you mean by "reptile". Certainly, if we could look at the early synapsids today, most people would probably call them reptiles. They were cold-blooded, hairless, laid eggs, wouldn't have produced milk, and anyway, they just kind of looked reptilian. Take a look at this one, for example. By most people's standards, that's a reptile.
But here's how the early amniotes evolved into the animals we have today:
Crocodiles Birds Other Turtles Mammals
^ ^ ^ ^ ^
| | | | |
| | | | |
------------- | | |
| | | |
| | | |
------------------ | |
| | |
DIAPSIDA ANAPSIDA SYNAPSIDA
| | |
Bearing in mind out definition for what constitutes a "group" of animals (a monophyletic clade if we want to get technical - but let's not) there are two problems with this chart. The first is that reptiles aren't actually a proper group at all - they can't be, because crocodiles are more closely related to birds than they are to, say, turtles. Which isn't to say that crocodiles are particularly close to birds, admittedly, just that they're even further from turtles. (This is because birds evolved from dinosaurs, which were fairly close to crocodiles).
When we say that a proper biological group has to consist of a common ancestor, and all of its descendants, the key word is "all". The common ancestor of all living reptiles is at the point I've marked (A), and if we want to include all of its descendants, then we have to count the birds. Either birds are reptiles, or reptiles don't really exist as a meaningful group. Bummer!
Be that as it may, the other problem is that even if the reptiles are a group, their last common ancestor, and therefore the first reptile, is the creature at (A), and mammals didn't evolve from that. So mammals evolved from creatures that certainly looked very reptilian, but which weren't, in a strict scientific sense, reptiles as we understand them today. They are entirely their own line.
But, at any rate, it's fairly clear that while the early synapsids may not have been reptiles, they weren't mammals, either. At some point, they evolved into mammals, and all of the earlier forms of reptilian-looking synapsids died out. So at what point did that happen? When did not-mammals become mammals?
The obvious answer is "when they developed mammary glands and began producing milk". Which is all very well, but a bit of a bugger when all you've got to go on is fossil bones. How do you tell from the bones whether the animal produced milk or not? You can't, pretty much. So palaeontologists have to use a different definition.
|Skull of a turtle - note that the lower jaw consists of at least four different bones. (d = dentary, ar = articular)|
Eventually, the dentary formed its own joint with the skull, and the lower jaw actually had two joints on each side for a while. That's not much of a problem, so long as the joints are lined up properly, but there's really no need for it, so eventually, the only remaining other bone in the lower jaw, the articular, began to shrink as well. In the end, the dentary was the only bone left in the lower jaw at all; and in mammals, we call it the mandible.
|Lower jaw of a mammal - note the absence of separate bones|
And that, at least when you're looking at fossils, is the defining characteristic of a mammal: that it has one bone on each side of the lower jaw, and three bones in each middle ear.
[Pictures from Wikimedia Commons]