Larry Summers has a message for Janet Yellen: Get out of the stock-picking business.

At least that was the takeaway from comments made last week by the former Clinton Treasury Secretary, Obama economic adviser and wannabe Fed chairman about the current Fed chairwoman during an off-the-record meeting attended by about 100 people at the Greenwich Ct., mansion of hedge fund impresario Steve Cohen.

The event, sponsored by the Stamford, Ct., Chamber of Commerce, featured a debate between Summers and Columbia University business school dean Glenn Hubbard on various issues including financial reform, Obamacare and taxes.

But Summers raised some eyebrows when he took issue with Yellen’s recent biannual economic report to Congress that focused on the valuation of stocks in the social media and biotechnology sectors.

The report, which was released as Yellen was scheduled to give testimony to lawmakers, stated that, “Valuation metrics in some sectors do appear substantially stretched–particularly those for smaller firms in the social media and biotechnology industries, despite a notable downturn in equity prices for such firms early in the year.”

Shares of stocks in both sectors tanked and  Yellen was criticized for turning a report on the health of the U.S. economy into a stock picking manual.

Summers apparently agreed. Without referring to Yellen by name, he said during the debate that the Federal Reserve “shouldn’t be trying to value” various parts of the stock market, according to two people who were present.

“Summers basically said it was a dumb idea to comment on subsectors of the stock market, and Hubbard agreed,” said one person who attended.

Both Summers and Hubbard declined to comment.

Yellen was nominated Fed chair by President Obama late last year after Summers dropped out from consideration amid significant opposition; Democrats in Congress associated him with the wave of 1990s financial deregulation that took place when he served in the Clinton administration.

Many Democrats believe deregulation was the root cause of the 2008 financial crisis. 

Yellen, of course, isn’t the only Fed chair to weigh in on the direction of the markets. During the technology stock bubble of the 1990s, former Fed chairman Alan Greenspan famously remarked that he feared investors’ “irrational exuberance has unduly escalated asset values, which then become subject to unexpected and prolonged contractions.”

Still, Summers comments about Yellen took many in the crowd of investors and business types by surprise. The debate was largely a cordial affair with Hubbard, a Republican economic adviser to Mitt Romney, and Summers, a Democrat, agreeing on many issues including tax reform and the need to bring confidence back to the markets.

At one point, Summers quipped that while he supported Obamacare, he wished it was enacted after the economy had more fully recovered.

Since leaving the Obama Administration in 2010, and failing to garner enough support to gain senate confirmation to become Fed chairman, Summers has been on the speaking circuit, often before well-heeled groups of bankers and investors.

During such events he’s known for speaking his mind, and at times ruffling feathers. Earlier in the year, he debated economist Nassim Taleb at the SkyBridge Alternatives Conference in Las Vegas, where he defended the deregulation he helped enact during the Clinton Administration, stating that it had little to do with the 2008 financial collapse.

When SkyBridge managing partner and moderator Anthony Scaramucci quipped that when he attended Harvard law school, it was a “communist” institution, Summers jokingly remarked “It still is.”

Summers was forced to resign as President of Harvard in 2006 in part because he remarked that the dearth of women in scientific fields is possibly the result of "different availability of aptitude at the high end.”

Charles Gasparino joined FOX Business Network (FBN) in February 2010 as Senior Correspondent.