When it comes to deciding whether or not to continue to work with your current financial advisor, time is money.

Many people drag their feet over the decision to find a new money manager. After all, finding an advisor, submitting documents, getting set up and creating and discussing a new financial plan requires a lot of time.

Switching to a new professional not only means reams of paperwork, but there’s also the constant worry the new person will be worse than the old one.  These fears cause people to procrastinate making the decision—and then they hit their boiling point.

“You might be tempted to call up your advisor and say I’m really unhappy cash me out and send me my money back but if the account is transferred over to another advisor it won’t be a taxable event,” says Jeremy Kisner, president of Surevest Wealth Management.

But knee-jerk reactions could result in a tax penalty or fees.

Whether it’s poor communication or underperforming returns, there are many signs it’s time to find a new financial advisor.

 “The No.1 reason people fire an advisor is a lack of communication or responsiveness,” says Kisner. He says clients shouldn’t be forced to wait for hours in a waiting room to meet with their advisor or have more than a day go by to have a phone call returned.

To avoid having to pay taxes or fees by ending a relationship abruptly, experts suggest looking for early signs that the relationship isn’t going to last.

A professional’s performance is also a main reason people stop working with an advisor, but experts advise having too short of an evaluation period.

“I believe in looking at a rolling five year period,” says Kisner. “If an advisor says, ‘I think we can do 7% a year in a normal market environment’ and in the first year it’s 2% and in the second it’s 15% that’s great,” he says.  On the flip side, if the advisor consistently underperforms, or worse yet loses money in a normal environment, that’s a big red flag and time to take action.

According to Matt Matrisian, senior vice president and director of practice management for AssetMark, a good financial advisor will set realistic expectations around the services provided and how the funds will be invested.

An advisor should take the time to sit down with the client to make sure he or she understands the client’s goals and personal financial situation and preferences to come up with an investment plan.

“You want someone who is looking at all aspects of a client’s personal life,” says Matrisian.

Money managers should be upfront with how and when they will communicate and take into consideration the client’s wishes as well, he says.

Before working with a professional, clients should know what types of services are provided along with performance expectations. The more information customers know from the start, the less likely there will be misunderstanding and disputes later in the process.  Clients working with pros without these qualities should re-think the relationship.

Also be weary of any advisor who doesn’t take the time to explain investments in a clear manner, says Eleanor Blayney, CFP Board's consumer advocate.

“Advisors are obligated to explain things to you in a way you’ll understand. They should check in with you frequently and say things like, ‘Does this make sense?’ or ‘Do you understand?’  If you are continually confused about what your advisor is saying, mistrust will build up and that’s a surefire way for the relationship to sour.