|Dromomeryx, a palaeomerycid|
Probably the most distinctive thing about deer is that they have antlers (and not horns). However, especially when we're looking at fossil species, it's important to remember that not all deer have antlers. Granted, the enormous majority do, and those that don't have lost them during their evolutionary history, rather than being holdovers from some ancient form that never had antlers in the first place. (This isn't true of musk deer, but that's partly why they aren't considered to be true members of the deer family).
And even that's assuming your fossil belongs to a male. (Or a female reindeer, oddly enough).
Antlers first appear on deer fossils in the early Miocene, on the general order of 20 million years ago. Still, in the grand scheme of the Age of Mammals, which has so far lasted 66 million years, that isn't all that long. It's a lot closer to us than it is to the dinosaurs, at any rate. So we ought to have a reasonable idea of what some of these first antlered forms look like.