Everyone knows they should be saving for retirement. But that is easier said than done.

According to the annual Capital One ShareBuilder’s Financial Freedom Survey, which polls around 1,000 Americans about their retirement plans, 93% of working Americans said they know they should be contributing to their retirement, but only 72% are doing anything about it.

What’s more, many Americans identify saving for their kid’s education as a bigger priority than their retirement—and that can cause financial hardships later in life.  

“Everybody would like to put their kids through college, but it’s not a financial option for everyone,” says Dan Greenshields, president of Capital One ShareBuilder, who points out retirement can last 30-plus years now. “Your child can get a loan for school, but you can’t get a loan for retirement.”

In the years following the Great Recession that were filled with high unemployment and turmoil in the housing market, 60% of people were still saving for their life after work, according to the ShareBuilder survey. But now that the labor market is stabilizing and the housing market has recovered, retirement savings are still sluggish.

“People spend more time picking out what vacation they are going to go on or what restaurant they might eat at than they do  on their overall retirement plan,” says Greenshields. “The things they do today can have a huge impact.” 

To better plan, he recommends workers determine the type of lifestyle they wish to have in retirement. City living is mostly likely going to come at an added cost than rural town middle America, but it can come with more benefits and amenities.

Once a lifestyle is determined, it’s time to calculate its cost and create a savings plan, which needs to be re-evaluated periodically. “The biggest thing is don’t treat it as a task,” Greenshields says. “You’ve got to set aside as much as you can and really think about retirement.”

According to the survey, even those that are saving, they aren’t putting away enough: with only 6% of their income going into their nest egg instead of the recommended 12%.

What’s more, 58% of Americans plan to retire by age 65, yet close to the same percentage fear they won’t have enough money saved by that birthday. In 2012, only 23% of people polled claimed to be concerned about not saving enough.

The top financial concerns for Americans as they move through 2014 include supporting their children and saving for their college (34%), while 13% named retirement at their concern and 10% pointed to credit card debt.

Back in the beginning of 2013, 20% were most concerned with paying for college, while 10% were worried about a job loss and 14% were fretting over keeping up with the daily household bills.

“Unfortunately, saving for retirement is often put on the back-burner for what seem like more pressing financial priorities, such as paying for college,” said Greenshields. “Now more than ever, Americans are responsible for ensuring their own financial security during retirement, and the earlier you begin to plan and save, the better.”