|The skull of Diademodon, a 6 foot long cynodont|
Instead, as I've discussed elsewhere, we have to pick a feature that survives in skeletons. Any decision we come to has to be arbitrary, and it's not going to line up with the very first point that mammals started giving milk to their young, since we don't know when that was. But one has to draw the line somewhere, and the one we've picked is the structure of the middle ear.
The mammalian middle ear consists of a cavity in the skull crossed by three tiny bones, connecting the outer ear (the visible bit, and the tube that runs in from it) to the inner ear, where sound is converted into neural signals that can be sent to the brain for interpretation. The first of these three bones is the malleus, or "hammer", which attaches to the inside of the eardrum at one end and to the second bone, the incus or "anvil", at the other. Finally, the incus connects to the innermost bone, the stapes, or "stirrup", which is attached, at its other end, to a particular part of the inner ear called the fenestra ovalis or "oval window".
The reason for all this is that the inner ear is filled with fluid, a high density medium, while sound is, for most mammals, likely to be reaching the ear through the far lower density medium of the air. The three connected bones of the middle ear serve to transmit sound from the eardrum to the inner ear with a minimum loss of energy, aided in part by the fact that the eardrum is so much larger than the oval window, and the three bones taper as they approach the latter, like increasingly small cogs in a gearbox. (The actual details are a bit more complicated than this, but that's the gist of it).
But what's interesting is that this is only true in mammals. Birds and reptiles have only one bone crossing the middle ear, which performs the same function, but is likely less efficient. Indeed, birds are only about half as good as humans at discriminating individual frequencies, and they can't detect very high-pitched sounds at all - although this is presumably less of a problem than one might think, given the existence of birdsong. (As I've also noted recently, their inner ear is less complex too; the two features likely go hand in hand, since there may not be much point improving one without the other).
In turns out that it's the malleus and incus that are the newcomers. In mammalian embryos these first form together with bones that would otherwise be associated with the jaw, and we can see the same thing happen in the fossils of mammalian ancestors. In reptiles, the jaw joint is comprised of the articular bone at the back of the lower jaw, and the quadrate bone in the skull proper. In the line that led to mammals, a new jaw joint formed as the structure of the lower jaw changed, and the articular and quadrate migrated upwards and away into the ear, becoming the malleus and incus, respectively.
With evidence from both embryology and fossils, we can be confident that this is how mammals evolved their unique three-bone middle ear structure. But what of the stapes in all this? This is the "original" bone of the middle ear, found in all reptiles and birds, and most amphibians - crucially including all of the really old fossil tetrapods from which reptiles and so on later evolved. Although it can get a bit weird in living amphibians, which have modified the original structure more than somewhat over the course of their evolution, in general, the stapes sits in the middle ear cavity and directly connects the eardrum to the oval window of the inner ear.
In the course of the change from the single-bone middle ear of mammalian ancestors to the three-bone structure of mammals themselves, the stapes has to do two things. Firstly, it has to get smaller and more slender, forming the last, and most delicate part of the chain that transmits and modifies the sound waves. This has quite clearly been observed in the fossils of the closest mammalian relatives. But the other thing it has to do is detach itself from the eardrum, allowing the malleus to take over that role. And, since the eardrum itself doesn't fossilise, quite what happened to it as the structures underlying it shifted is rather less clear.
To try and figure that out we need to look, not at mammals themselves, but at the fossils of their relatives, the non-mammalian synapsids. This is a very large group, and to really unpick the details we'd probably need to look at a fair few of them, but perhaps one of the best places to start is with the cynodonts. Not to be confused with the similarly-named dicynodonts, these are the very closest relatives of mammals, the creatures that were almost - but not quite - over the line into mammalian status.
We can't, as noted, actually look at their eardrums, but we can look at the structure of the stapes, to see how it might have worked, and what it might have been connected to. Of course, being such a delicate bone, it doesn't often preserve well in fossils of this age - many cynodonts were, in fact, older than the very oldest dinosaurs. Yet, perhaps surprisingly, there are a few fossils in which it is preserved, and a survey looking at the detailed structure of this one tiny bone in this one group of long-gone creatures was published just a couple of weeks ago.
There have been may theories about how the middle ear would have worked in these animals, while it was in the process of shifting from one shape to another. Perhaps the earliest popular theory, dating from the 1940s, is that the stapes connected to the incus at one end, and the inner ear at the other, while having some sort of side branch connecting to the eardrum. While it's possible that that side-branch might have been made of cartilage or something else that hasn't preserved, the survey finds no evidence for its existence.
But if that's not the case, what are we left with? If the eardrum had already lost its connection to the stapes by this point in evolution, then what was it attached to? After all, it has to be connected to something, or the ear won't work very well, which rather defeats the point. There are at least three possible answers to this.
The first, advanced in the 1970s, is that, in mammalian ancestors, the eardrum was attached to the jaw, rather than where we'd reasonably expect it to be. This has the advantage that sound would originally have been transmitted to the inner ear the same way it is now, from malleus to incus to stapes. The main difference being that the "malleus" was still, at this point, the articular bone, and part of the jaw.
An alternative is that the eardrum changed in shape as the bones re-arranged themselves, and, by the time of the cynodonts stretched all the way from the jaw to its original location near the stapes. Or, finally, cynodonts could have had two eardrums on each side, one broadly where it is in reptiles, and the other newly formed down by the jaw joint. If so (and, personally, I'm not convinced), it's that one, not the original, that survives today in mammals.
It's hard to decide between these three possibilities. Certainly, there are appropriate spaces in the skull where you could put an eardrum down by the jaw. Furthermore, the lack of any apparent connection between the stapes and an eardrum by this stage of evolution would tend to suggest that there had to be something eardrum-like down by the jaw, whether it was a new structure or an extension of the pre-existing one. If it was a completely new thing, and the old one still existed, the latter probably didn't do much, a vestigial structure soon to vanish forever, yet still left over from earlier ancestors.
It may seem odd that cynodonts, the closest relatives of true mammals, heard through bones that were still part of their jaw - albeit, only just. But we do know that these are the bones that now reside in our own middle ear, so the possibility that they took up that role early on makes a lot of sense. By having another function, these two bones could survive, where all the other bones that we see today in the reptilian lower jaw vanished, bar the one that holds the teeth, and which today is the only one we have left.
[Photo by FunkMonk, from Wikimedia Commons]