One of the first big hedge funds to try to profit from a rebound in the U.S. housing market by investing in foreclosed homes is looking to cash out, even as other institutional investors are still getting in.

Och-Ziff Capital Management Group LLC, the $31 billion hedge fund led by Daniel Och, recently told its investment partner, 643 Capital Management, that it wants to exit from the foreclosed homes business, said several people familiar with the matter.

The hedge fund is looking to make a profit on a portfolio of about 300 foreclosed homes in northern California that were acquired at distressed prices, said the sources, who did not want to be identified because they were not authorized to discuss the matter.

Though the total value of the portfolio is not known, its business model involved buying homes at an average price of about $100,000 apiece. In addition, Och-Ziff and its partner needed to spend tens of thousands of dollars on each home for potential renovations before renting them out.

The decision by Och-Ziff to exit the market after just a year is notable since the hedge fund was one of the first institutional investors to see the large supply of foreclosed homes in the United States as a new asset class that could generate consistent income if operated as rental properties.

Earlier this year, proponents of investing in foreclosed homes were projecting a return of at least 8 percent a year from renting them out.

But the New York-based hedge fund is looking to sell now because the returns it is generating from rental income are less than expected and it is looking to take advantage of a recent rebound in home prices in northern California, the sources said. It's not clear what kind of return Och-Ziff had expected to earn from renting out homes.

The average cost of renting a home nationally grew at 0.9 percent in the third quarter from the previous three-month period. That growth rate is down from 1.3 percent in the second quarter, according to Reis Inc., a commercial real estate research firm.

Meanwhile, the median price paid for all new and resale houses and condos in the San Francisco Bay Area in August was $410,000 -- up 10.8 percent from $370,000 in August 2011, according to DataQuick.

Och-Ziff's move could indicate that institutional investors may have to dial back their expectations, especially with regards to rental income.

"It's not surprising that some investors may have overestimated rental returns," said Rick Sharga, executive vice president with Carrington Mortgage Holdings, a division of Carrington Capital, which has been buying and renting foreclosed homes since 2007. "If you are an investor getting into this cold you were probably making assumptions based on models rather than experience."

Gregor Watson, founder of 643 Capital Management, which is Och-Ziff's partner, said he is not marketing the portfolio of homes but declined to comment further. An Och-Ziff representative declined to comment.

Over the past year, other high-profile Wall Street names, such as Blackstone Group, Oaktree Capital Management, Colony Capital and TCW, have all committed money to investing in foreclosed homes. Oaktree has invested in a fund managed by Carrington.

Wall Street analysts estimate that this year alone, private equity firms, hedge funds and other investment firms have raised between $6 billion and $8 billion to acquire single-family homes at either foreclosure auctions or from banks. So far, private equity giant Blackstone has emerged as one of the biggest buyers, spending more than $1 billion to gobble up foreclosed homes.

Och-Ziff, whose flagship fund is up a little over 8 percent this year, originally entered the foreclosed home market as part of a joint venture with McKinley Partners, a California real estate investment firm. Watson, founder of 643 Capital Management, was a principal with McKinley, but left this year to open his own shop, taking Och-Ziff with him.

Sharga said it was not surprising that an early market entrant like Och-Ziff would look to get out and take advantage of the recent appreciation in home values. He said other early financial investors also may change direction and decide to cash in sooner rather than later.

"This is a normal winnowing-out process," Sharga said. "We will see other early entrants depart."

(Reporting By Matthew Goldstein and Jennifer Ablan; Editing By Martin Howell and David Gregorio)