With President Obama hinting this week that Ben Bernanke is ready to retire, the race to replace the chairman of the Federal Reserve is set to heat up this fall.

While Janet Yellen, Larry Summers and Tim Geithner get all of the attention, there’s actually a long list of dark-horse candidates who could be handed the keys to the Fed’s $3 trillion balance sheet.

The lesser-known names include former Fed vice chairman Roger Ferguson, who would be the first African-American Fed chief; former CEA chairs Christina Romer and Austan Goolsbee; and even Stanley Fischer, the current governor of the Bank of Israel.

“He’s got to pick a candidate that doesn’t scare the markets,” said Ernie Patrikis, former general counsel of the New York Fed. “This is going to be a very tough job. What Bernanke did was easy compared with what his successor will have to do. How do we get out of this huge portfolio?” 

Whoever Obama nominates to succeed Bernanke when his term expires in January will be charged with plotting the delicate exit from incredibly easy-money policies at a pace that won’t derail the economic recovery or spark an inflationary spike.

“You need a person Wall Street likes, someone with monetary policy experience and somebody close to the president. I think that trims any long list pretty fast,” said David Jones, president of DMJ Advisors and a former Fed economist.

Yellen, 66, the current vice chair of the Fed, is considered to be the front-runner for the top job, positioning her to become the first female chairman. Forty-six percent of economists surveyed by The Wall Street Journal in early April predicted Obama will promote Yellen, who is considered a reliable dove that wouldn’t rush to unwind quantitative easing.

In a recent analysis of Fed chief candidates, UBS (UBS) economists assign a 26% probability that Yellen will get the job.

What If It’s Not Yellen?

But Obama could also nominate a female economist he knows even better than Yellen: Romer.

The former chair of the Council of Economic Advisors is best known for being a chief architect of    Obama’s economic stimulus plan, which could hurt her case given criticism on the left and the right of the effectiveness and size of that $787 billion plan. Still, UBS sees a 20% chance of Romer, 54, becoming the next Fed chair.

Instead of expressly throwing his support behind a notable dove like Yellen, Romer would allow Obama to pick a Fed outsider.

“From a political perspective, I think Romer provides him with insulation. On top of it, she’s qualified,” said Drew Matus, an economist at UBS.

“You need a person Wall Street likes, someone with monetary policy experience and somebody close to the president."

- David Jones of DMJ Advisors

As FOX Business’s Charlie Gasparino previously reported, the White House is also considering Ferguson, currently the CEO of the Teachers Insurance and Annuity Association - College Retirement Equities Fund.

It’s easy to see why Obama would zero in on Ferguson’s name as the 61-year-old has both a law degree and a Ph.D. from Harvard, a relationship as an adviser to the White House and real-life Fed experience as a former vice chairman.

“I think he’s a very strong candidate. If you’re checking boxes, he checks them all,” said Matus, who pegged the probability of Ferguson being nominated at 14%.

Ferguson is also credited with guiding the Fed’s initial response on 9/11 when Alan Greenspan, then the Fed chief, was out of Washington.

In the Running?

An intriguing candidate for the top job at the Fed is Goolsbee, 43, the affable former chair of the CEA. Goolsbee is seen as strong on macro policy, but not all that experienced on monetary policy.

His young age would allow Obama to appoint someone with the potential to have a long-lasting impact on the Fed.

"He has enough credentials to be respected," said Jones. "That is a strong candidate in a weak bench."

Matus said Goolsbee just "hits the minimum criteria," but he "isn't exactly what most people would consider the first choice."

Another potential Fed chief is already a central banker: Bank of Israel chief Stanley Fischer. There is precedence for such a trans-national move as former Bank of Canada governor Mark Carney is set to head the Bank of England.

Princeton professor Alan Blinder’s name has also been floated as a possible Fed chief. Blinder, 67, is another ex-vice chair of the Fed (1994-1996) and member of the CEA. Yet he doesn’t have nearly as much recent policy experience as Yellen or Ferguson.

Star Power Questioned

Some have voiced concern about the depth of the list of potential candidates to replace Bernanke.

"There really isn't a good bench in terms of picking a successor," said Jones.

But Matus notes that neither Bernanke nor Greenspan were “household names” before they took the reins at the Fed.

Two of the most recognizable candidates to run the Fed are seen as long-shots: Summers and Geithner.

While few doubt the intellectual heft of Summers, the former Harvard president doesn't seem to have the temperament for a consensus-building role like Fed chief.

"He's the guy who comes into the room and says he's the smartest guy in the room. In fact, he is the smartest guy in the room -- except you don't want someone to tell you that," said Jones.

Third Term for Big Ben?

Geithner is another relic of the first Obama term who clearly has the credentials to be Fed chief.

However, Gasparino reported this week that Geithner, who has spent virtually his entire career receiving a government paycheck, doesn't want the job, seeking instead to cash in on his impressive resume in the private sector.

"If I were Tim, I'd be tired," said Patrikis, now a partner at White & Case. "I know what it feels like when you've been at the Fed for a long time and you have kids wanting to go to school."

UBS listed a number of other candidates who meet the minimum credentials for Fed chair, including outgoing CEA Chair Alan Krueger, BlackRock (BLK) exec Peter Fischer, NY Fed President William Dudley, Fed governor Jeremy Stein, ex-vice chair Donald Kohn and current Fed presidents Eric Rosengren and James Bullard.

Of course, it’s still possible Obama and Bernanke decide the best person to exit QE is the guy who started it in the first place.  

“My feeling has been that he’s going to twist Bernanke’s arm as much as possible to get him to stay for at least part of a third term,” said Jones.

Follow Matt Egan on Twitter @MattMEgan5