Oooh!
I know
one nice place.
Mammoth Cave. Located in central Kentucky, is the longest cave system in the world. I've been twice. Not only is the cave itself amazing, the surrounding areas are gorgeous. Particulary in autumn, the trees are gorgeous.
There are various different tours you can take inside the cave. The first time I went was a school field trip, so of course we went on the Historic tour. The natural entrance to the cave is just a gaping hole in the side of a hill. You could stand on top of the hill and have no idea there was a cave beneath you.

Natural Entrance
The Historic tour takes you through all the historically significant places in the cave. Places where mining took place, and places where people suffering from tuberculosis camped out hoping the cold air would cure them.
There's also the Frozen Niagara tour. Which requires a lot more walking, but is breathtaking. This is the tour where you get to see the most beautiful natural formations.

Frozen Niagara
It literally looks like a waterfall made of stone. That photo doesn't do it justice at all, it's gorgeous in person.

More Formations
The cave itself is mostly small passageways, but there are several large areas, as large as a theatre. There are also bottomless pits. And underground rivers. And bats. And scary fish with no eyes.
Also, the tour guides like to turn off the lights at at least one point in every tour. You don't know true darkness until you're 300 feet underground with no lights. Scary, but oh so fun.
Is that the one where they set off a firecraker at one point, or am I thinking of somewhere else? Because I've don't remember going to Kentucky, but it looks like a cave system that I went to...