Is college worth it? This may seem like a loaded question - but when a former U.S. Secretary of Education is asking it, chances are people will pay attention.

William Bennett and David Wilezol ask this $1 trillion dollar question in their new book, “Is College Worth It? A Former United States Secretary of Education and Liberal Arts Graduate Expose The Broken Promise of Higher Education. “

The cost of higher ed is skyrocketing faster than inflation, and student loan debt is growing by the second. And, with a job market full of unemployed and underemployed grads—perhaps there’s never been a better time to ask this question.

Experts say that, yes, college is worth it, but it all depends on where you go and the degree to which you apply yourself when there.

So what is the return on this massive investment for students and families, asks Andrew Kelly, resident scholar of Education Policy Studies at the American Enterprise Institute.

“There are two things going in opposite directions—tuition is going up and wages seem to be going in the other direction,” Kelly says. “There is a cyclical nature to this conversation. It seems to come about during recessions or recoveries, for obvious reasons.”

Wilezol says it is worth it, but just not for everyone.

“As a society, we have bought into this narrative that we need a four-year BA to be personally and financially successful in life,” he says. “I don’t know if that is entirely true, especially as we look at the Class of 2011 graduates, with 50% unemployed or underemployed, and with $25,000, on average, student loan debt. There are a lot of people waiting tables, and working at Starbucks.”

It’s Not Whether College is Worth it—But Which College is Worth it

While school isn’t for everyone, better colleges produce better grads with higher earning potential, hands-down, Wilezol says.

“They attract better talent professionally,” he says. “These institutions produce graduates that are able to graduate on time, and get jobs. It does matter where you go, but it’s not the end all to be all of what makes a successful college grad.”

That would be how much and how hard you work, and the major you choose, he says.

“You have to consider a major that is paying better than others, and legitimately do your academic work,” Wilezol says. “People in college sometimes don’t know why they are there, or only go because their friends are going, or take on something that’s not the best fit for them.”

Isabel Sawhill, senior fellow at the Brookings Institute, says it matters hugely where you choose to go, and agrees that students coming out of more selective institutions will fare better than those with degrees from less selective schools.

“The job market recognizes that,” Sawhill says. “But student debt is over $1 trillion, and that is a lot of money to be borrowing to do something that is not guaranteed.”

But while name does matter, Sawhill says the message about higher education in America needs to be re-written.

“It’s not ‘everyone should go to college,’” she says. “It should be ‘everyone needs marketable skills in today’s economy and they need to be better prepared by their primary and secondary education.’”

MBAs, PhDs and Beyond

If experts are questioning the value of four-year degrees what does that mean for even more distinguished scholastic achievements? Wilezol says they can produce increased income over time. But it’s necessary to be selective and wise when choosing which programs to participate in.

“Be very careful about what you are doing,” he says. “If you are going to spend $60,000 or $80,000 on a masters degree at Columbia, you better have a really certain expectation of earnings potential.”

Kelly says these degrees, and law school degrees as well, are the “canaries in the coal mine” in terms of the greater debate about the ROI on college education.

“We are giving out more masters degrees than we were 15 or 20 years ago,” Kelly says. “And I think the laws of supply and demand suggest that there are more people with Masters’ degrees in cities like New York, D.C. and San Francisco, so the less employers will have to pay for these people, because they are plentiful.”

But Sawhill says these degrees are still what set applicants apart in a crowded market.

“The best and brightest used to have just a BA, but now they will have an MA or a PhD,” she says.

The price of tuition has escalated 1100% since 1978, Wilezol says. So the bottom line is, spend wisely.

“Higher education is becoming the most expensive product in America,” he says. “What are we seeing for it? Students individually are not totally equipped as well as they should be for the workforce. There is a lot that needs to change, and we have a lot of thinking to do.”

Follow Kate Rogers on Twitter at @KateRogersNews