While big banks like J.P. Morgan Chase (JPM) and Citigroup (C) couldn’t return their TARP funds fast enough, dozens of publicly-traded lenders and thrifts are still sitting on nearly $5 billion in bailout cash some four years after the Great Recession ended.

These TARP holdouts include medium-sized institutions like $2.3 billion Synovus Financial (SNV) and Puerto Rican lender Popular (BPOP), as well as much smaller lenders such as Atlantic Bancshares, which is on the hook for just $2 million.

Banks that have been unable or unwilling to escape TARP now run the risk of being hit by a looming spike in dividend rates at the five-year anniversary of entering the government program as well as being stigmatized by customers and counterparties alike.

“If after four years they still haven’t repaid their money, that is a sign of an inherent weakness,” said Anthony Michael Sabino, a professor at St. John’s University. “Maybe it’s time to urge those banks to seek out a merger partner.”

According to SNL Financial, there are 75 publicly-traded banks and thrifts that remain in the TARP program, which was hastily cobbled together by the U.S. in the fall of 2008 following the implosion of investment bank Lehman Brothers. A recent report by the Treasury Department lists $4.68 billion in outstanding TARP payments as of June 30.

Virtually all of the biggest financial institutions, from Bank of America (BAC) and Goldman Sachs (GS) to Wells Fargo (WFC), have repaid their bailouts, with interest and dividends.

But some regional lenders, many of which struggled during the downturn more than their big-bank cousins, remain on the TARP list, raising questions about their overall health.

“A lot of the banks that have the ability to repay have probably done so,” said Andrew Wolcott, an analyst at SNL Financial.

Still on the Hook

Synovus, which is headquartered in Columbus, Ga., has not yet repaid the $967.9 million in TARP funds it received. A spokesman reiterated that Synovus expects to repay the money during the third quarter.

San Juan, Puerto Rico-based First BanCorp (FBP) owes $222.7 million of the original $400 million the U.S. provided. 

John Pelling III, an investor relations officer at First BanCorp, referred questions about a TARP exit to the Treasury Department because the U.S. converted its preferred shares in the lender into common stock during a recapitalization. "We cannot control when they decide to sell their common shares -- only they can," he said. 

A spokesman from the Treasury Department declined to comment, but pointed to comments made last week by Timothy Massad, Treasury's assistant secretary for financial stability.

"Our economy is in a stronger position because of our efforts, and we will continue to wind down the remaining CPP investments in a way that helps support community banks and protects taxpayer interests," Massad said. 

Other public TARP holders include Bluffton, S.C.-based Atlantic Bancshares, El Monte, Calif.-based Cathay General Bancorp (CATY), which owes $129 million, and Anchor BanCorp Wisconsin (ABCW), which is also undergoing a recapitalization. 

None of those lenders responded to a request for comment.

A spokesman from Popular, which is still on the hook for $935 million, declined to talk about TARP repayment because the lender is in the midst of a "quiet period" ahead of its earnings report on Thursday. However, the representative said the bailout repayment is likely to be a topic of conversation during the earnings call. 

Some have criticized the Treasury Department for not concentrating enough on TARP now that the big banks have exited the program. 

“Most of Treasury’s attention under TARP has focused on the bailout of and repayment of TARP funds by the largest banks and AIG, while notably less attention has been paid to the plight of smaller TARP banks and to TARP housing programs," said a spokesperson at the Office of the Special Inspector General for the TARP program. 

Looming Rate Hike

In recent months, a number of lenders have either repaid their TARP funds or enacted plans to do so.

That’s at least partially because the dividend payments on preferred stock that banks are required to pay the U.S. are scheduled to soar from 5% to 9% at the fifth anniversary of their participation in TARP.

“It hops up and that creates an incentive for them to get out. You really don’t want to be paying that higher dividend,” said Ernie Patrikis, a partner at White & Case who previously served as general counsel of the New York Fed.

For some early TARP recipients, that deadline is quickly approaching.

For example, both 1st Financial Services, the bank holding company for Mountain 1st Bank & Trust Company, and Los Angeles-based Broadway Financial Corp. (BYFC) would see their dividend rates jump in November.

“A lot of people are trying to rush before that happens because that’s a pretty big jump in cost of capital,” said Wolcott.

Neither bank responded to a request for comment, but Wayne Bradshaw, CEO of Broadway Financial, recently told SNL his thrift is putting the finishing touches on a recapitalization.

“We're trying to put the bank in a better position to move forward as an institution, and in that process the TARP debt and everything else in the capital structure would be addressed," he said.

Moral Hazard Concerns Linger

While some banks scramble to repay TARP before their five-year anniversary, others simply may not be healthy enough to do so.

“I think the Federal Reserve and the Office of the Comptroller of the Currency have to exert a little pressure,” said Sabino. “A bank that hasn’t yet repaid its TARP money has its own internal problems that might only be solved by agreeing to merge with a stronger rival."

Despite the bailout cash still owed by some lenders, the U.S. has managed to turn a profit on the banking portion of TARP.

“The Treasury has made a pile of money. This is just the tail wagging the dog. These things will work out one way or another over time,” said Patrikis.

According to the Treasury Department, $271 billion has been recovered from TARP’s banking programs through repayments, dividends, interest and other income, compared with $245 billion originally invested in these companies.

TARP “worked,” Treasury said in its monthly report to Congress. “It helped stop widespread financial panic, it helped prevent what could have been a devastating collapse of our financial system, and it did so at a cost that is far less than what most people expected at the time the law was passed.”

Others are more concerned about the less-quantifiable costs of TARP, such as moral hazard.

“Banks screwed up royally and the government bailed them out,” said Sabino. “That’s the wrong message to send and that message is still resonating throughout the banking community. It sets the stage for repetition.”

Follow Matt Egan on Twitter @MattMEgan5