I respond by telling them that nobody is going to ask someone how old they are when they submit a manuscript for publication. I've even read that it is wrong to mention one's age in a query letter, so I've never done that. There is no need for it. If the writing is good, it will stand on its own, whether the author is 15, 30, or 99 years old. Because age is just a number, right?
But for me it's more than that. My grandfather passed away from a heart attack when he was 35 years old, and his son--my uncle--passed away after a short battle with brain cancer when he was only 27. I am 23. I know that tomorrow is not promised. The clock is ticking and will not continue forever. We can't keep putting off things because if we do, they may never happen.
I write now because I know that if I put it off, I may never have the chance to do it later on. People have told me that they know people who want to be writers "someday," to which I respond, "There is no 'someday.' You're either a writer or you're not."
Just DO it. Don't wait until you're thirty, don't wait until the mortgage is paid off, don't wait until the kids are grown and you're retired. You want to pen the next great novel? Get cracking. Git-r-dun. Start right. And start right now.
Because you won't be young and goofy-looking forever, either.