Yes, the mountain sides have a number of caves in various sizes, some are old bear caves while larger caves were once used by Native Americans. Perhaps the most famous of these caves is called the Great Cave, a cave located along an abandoned loop off of the Precipice trail. It has an opening that is 100 feet high and the cave goes back into the mountain side about 100 feet. Even though it has been abandoned by the Park Service, a number of sites on line tell how to reach The Great Cave.
The second most famous cave in the park is yet another abandoned cave, Anemone Cave, an ancient sea cave that can only be entered at low tide. People have become trapped and drowned in this sea cave so pay close attention to the tide.
The Bear's Den is an abandoned bear cave within sight of the Park Loop Road and hundreds pass it every day without ever knowing its even there.
There are many other caves you can find on line, including the caves of Day Mountain.