But there once used to be a third subfamily of cats, and they represent, with the possible exception of mammoths, what are probably the most widely known of all extinct mammals. In fact, I'd guess that they are the only extinct mammals that come at all close to dinosaurs in terms of popularity. I refer, of course, to the sabretooth cats.
Chances are, especially if you're American, the sabre-tooth cat you've heard most about is Smilodon. This is actually three different species of cat, albeit very closely related. The largest was about the size of a lion, but with a short tail, like a bobcat. They used to live in both North and South America, and died out only around 10,000 years ago, perhaps not coincidentally shortly after humans became numerous on those continents. (Not, one assumes, that humans would have hunted Smilodon themselves - but they may well have eaten much of the cats' preferred prey).
But Smilodon is far from the only species of sabre-tooth. We have Megantereon, also from America, Homotherium from Europe, and Machairodus, also from the Europe, to name just three genera. The sabretooths were a broad and successful group, surviving for millions of years from the early Miocene, around 15 million years ago, to a mere blink of the geological eye into the past, at the end of the Ice Ages. Given the timing involved, its hard to avoid the conclusion that it's only because of our own species that they aren't still around today.
Megantereon Homotherium &
| Paramachairodus ^
| ^ |
| | | Metailurus
| | | ^
------------- | | Felines
| | | ^
| | | | Pantherines
---------------------- | | ^
| | | |
| | | |
--------------------- | |
As mentioned, there are other sabre-tooth cats besides the ones shown here. Note that the felines and pantherines (that is, all living cats, plus some extinct non-sabre-tooth forms, such as European jaguars and the like) are more closely related to each other than they are to animals like Smilodon.
It's worth noting that the sabre-tooths were not the earliest cats, nor any modern cats descended from them. They were a diverse, but entirely separate, evolutionary line within the cat family, that, for a long time, would have lived alongside animals that more closely resemble today's cats. Indeed, the very oldest cat fossils cannot easily be placed into any of the three felid subfamilies - presumably because they're the ancestors of all of them.
The most noticeable thing about the sabretooths is, of course, their teeth. No animal today possesses canines quite like them. Yet it's evidently a good feature for at least certain sort of carnivore to have, because it evolved at least four times among cat-like animals. The sabretooths themselves represent one of those times, but there were some very cat-like animals that lived earlier, and also had the same sort of teeth. Yet these other animals, of which the best known is probably Dinictis, were not really cats at all, any more than a hyena is a dog - they just looked rather similar.
But even if we restrict ourselves to sabretooths that were indisputably cats, there were plenty of them. They were at least as different from one another - and probably more so - as jaguars and lions are today. They weren't a single type of animal, and not just because they lived at different times and places, because many of them didn't. Which raises the question of how they interacted with each other.
A review by Manuel Salesa and co-workers of the fossil remains from the Las Casiones fossil beds near the city of Teruel in eastern Spain gives us one snapshot of these interactions. The fossil beds date from the late Miocene, a little over 6 million years ago, so roughly half way through the evolutionary history of sabretooths. They include the fossils of many animals other than cats, with especially large numbers of three-toed horses (well, two toes and a hoof - this is late enough in evolutionary terms that only the hoof actually reached the ground). Even if the only thing you're interested in is the cats, those other animals are still relevant, because it gives you a good clue as to what they would have been eating.
An unusual feature of the site, commented on in the paper, is that many of the fossil bones are lined up in more or less the same direction, over a wide area. That may seem rather odd, but remember that, when an animal dies, even if its remains are fortunate enough to be fossilised, they won't necessarily vanish straight into the mud (or whatever) in perfect life-like formation. Even ignoring post mortem changes in the body itself, bones can get separated, carried about, and so forth, by other animals or by conditions in the local environment.
Rivers are a common example. If an animal falls into a river as it dies, the bones may well be scattered for some distance before they come to rest in a mudbank or other environment that happens to preserve them. When that happens, there's a good chance that the bones will come to line up along the direction of the river, because of the way they drift downstream. That's probably not what happened here, but the process is similar, suggesting a relatively light flow of water, gently lapping against the bones after the carcass decomposed, or was eaten by scavengers. In other words, this is probably the shoreline of an ancient lake, rather than the bed of a river.
As for the cats themselves, they discovered remains of three different species of sabretooth, all living in the same area, at, so far as we can tell, the same time. Paramachairodus and Metailurus were both leopard-sized animals, with the latter being slightly the larger of the two, while Machairodus giganteus was much larger, about the same weight as a lion, although with shorter legs and lower shoulders. How did three different sabretooths share the same environment?
Perhaps the short answer is 'not very comfortably'. Lions and leopards share the same environment today, and there are also places in Asia where tigers and leopards meet. When this happens, it is the lion or tiger that usually wins out. In Africa, lions can easily chase leopards away from their prey, and get first choice of most things they come across - there really isn't much that will fight off a lion. Leopards can partly avoid this by hiding freshly killed prey up trees, where lions can't reach it, but on the ground, they're really no match, and both species know it.
It is likely that a similar relationship existed between the large Machairodus and the smaller cats that they shared the lakefront with. That probably implies some degree of open woodland, allowing the smaller cats to avoid detection by their larger, and presumably more aggressive kin. Sure, all of these animals were sabretooths, and would doubtless look pretty fearsome to us today, but there can have been little doubt as to which one was the more dangerous.
As for the two leopard-sized cats, the picture is less clear. One was slightly larger, which may have given it an edge, but it's perhaps more likely that they hunted in different ways. Perhaps one was more active in the day, and the other at night, or perhaps they pursued slightly different prey. That would allow both to live side by side without one completely replacing the other.
It may also be relevant that Paramachairodus has an unusual groove in one of it's ankle bones. That may not sound like much, but, from its position, there would have been a muscle attached there which helps to flex and bend the toes. The reason it's significant is not just that Metailurus didn't have such a visible groove in it's skeleton, but that neither do almost any other cats. Nor do hyenas, or dogs, but we do find just such an adaptation in carnivores such as pine martens, civets, bears, and red pandas, among others. It keeps their feet more flexible, and for most (though not all) of those species, that's very handy, because they happen to climb trees.
Now, an animal the size of a leopard isn't going to do its hunting in the branches, but it is entirely plausible that these particular sabretooths were better at climbing than big cats are today, and, more importantly, better at it than their own neighbours. That may well have given them a slightly different habitat to exploit, again, avoiding too much fighting over resources.
It's also worth noting that, when I say that bigger cats drive smaller ones away from their food and extend that to fossil species, I'm assuming that the animals lived on their own. Most cats do, after all... but lions are an exception. There's no particular reason to assume this is so, but if one of the smaller sabretooths happened to live in prides, and the larger didn't, the balance of power may have been rather different.
There was plenty for the cats to eat, and, again, they may have hunted different animals, as leopards and lions do, to some extent, today. There were three different kinds of those three-toes horses found in the deposits, with the largest the size of a zebra, and the smallest the size of a small gazelle. Machairodus would have surely preferred the zebra-sized animals, while the others could have preyed on, if not the really small ones, at least the medium-sized species. There were also three different kinds of antelope, and two deer, that left fossils in the same locality, all of which would have been fair prey for sabretooths. There was even a hippo, apparently small enough for Machairodus to take on, but which surely can have had no other predators.
The story doesn't quite end there, because the sabretooths were not alone. In the same fossil beds, palaeontologists also uncovered the remains of Felis attica. This isn't a sabretooth, but a true feline, about the size of a bobcat. Significantly smaller than even the smallest sabretooths in the area, and, of course, with rather different teeth, it probably spent a lot of time hiding in the undergrowth. In fact, it was sufficiently small that it may nor have bothered the sabretooths much at all, except perhaps when they were scavenging, and would likely have eaten rabbits, birds, and just possibly the smallest of those gazelle-sized horses.
A few remains - too scanty to be assigned to any particular species - belong to a second feline cat. This would have been, so far as we can judge, somewhere between an ocelot and a domestic cat in size. Which means that we can, at this one site, see a range of cats from the potentially hippo-eating Machairodus, down to something that would have been perfectly happy with mice.
[Image by "Ghedo" from Wikimedia Commons. Cladogram adapted from Meloro 2012.]