Nasdaq OMX Group (NDAQ) recently held discussions with private equity firm Carlyle Group about taking the stock exchange private, just months after its main rival, the New York Stock Exchange, agreed to be purchased by the Atlanta-based IntercontinentalExchange (ICE), the FOX Business Network has learned.

Nasdaq held these discussions about three weeks ago, according to two people with direct knowledge of the deal. One person with knowledge of the deal says the Nasdaq isn't "currently in discussions with any private equity firm" though this person said that could change. These people say the discussions between Carlyle and Nasdaq were preliminary and price appeared to be the main sticking point.

The move comes as the Nasdaq faces major competitive challenges. In December, the ICE announced that it would purchase NYSE Euronext (NYX) for $8.2 billion, combining the second-largest futures exchange with NYSE's core business of listing corporations and market making, or matching buyers and sellers of stocks.

People at the NYSE say a sharp decline in stock trading volumes and investors increasingly sending trading orders to alternative markets -- such as so-called dark pools -- and away from both the major exchanges all but forced it to find a willing buyer to remain competitive. Nasdaq faces similar challenges and in recent years its chief executive, Robert Greifeld, has attempted to diversify the exchange's operations away from the equities business. In December, for example, the Nasdaq bought a group of corporate consulting and information businesses from Thomson Reuters (TRI) for $390 million.

But analysts worry whether such moves are enough to keep the exchange competitive, particularly against much larger rivals, and given the competitive landscape. In 2012, the Nasdaq won the listing for the initial public offering for Facebook (FB), meaning that the big IPO opened for trading on the exchange, and earning Nasdaq a substantial fee. But given the new competitive demands of the market, having a stock listed on any exchange doesn't mean it will trade there after the IPO.

Moreover, Nasdaq, which once dominated the market for listing of technology companies, faces an increased threat in that department from the NYSE.

People close to the deal say Carlyle initiated the talks with Nasdaq, but these same people say the cloudy business environment for stock exchanges caused Greifeld to mount serious preliminary discussions with the private equity firm for a deal to take Nasdaq private. One of the key players in the negotiations was Carlyle chief financial officer Adena Friedman, who left Nasdaq in 2011 as its CFO and head of corporate strategy.

Friedman joined the Nasdaq in 1993, and led the exchange's major deals -- its purchase of the Boston Stock Exchange, the Philadelphia Stock Exchange, and the OMX, a group of Nordic exchanges.

A Nasdaq spokesman said: "We don't comment on market rumors or speculation." A Carlyle spokesman also declined to comment.

Nasdaq has a market value of around $5 billion and a deal to take Nasdaq private could reach around $8 billion, according to one analyst with knowledge of the possible deal. Shares of Nasdaq have risen about 17% over the past year, touching a new 52-week high intraday Monday,  and the company just announced record earnings.

But questions remain about the Nasdaq's future and whether it has the ability to compete with larger players, while some analysts are not convinced that Greifeld's business model will produce significant earnings growth. In 2011, Greifeld teamed up with the ICE to mount a hostile bid for the NYSE, which was engaged in merger talks with Germany's Deutsche Borse.

US regulators blocked the potential Nasdaq-NYSE deal over antitrust issues since most of the stock listing business would be housed in one place. Later, European regulators blocked the NYSE-Deutsche Borse combination, forcing NYSE chief Duncan Niederauer to sell to the Intercontinental Exchange.

One rationale for taking Nasdaq private would be that it would allow Greifeld to continue efforts to remake the exchange and not face the short-term earnings pressure of a public company in the process. With interest rates low, such deals can be cobbled together cheaply using debt. Computer maker Dell (DELL), teaming up with private equity firm Silver Lake Partners, recently announced a $24.5 billion management buyout for similar reasons, also with a heavy reliance on debt.

One key difference from the Dell deal is the fact that founder, Michael Dell, will remain in control of the private company. Michael Dell is the 17th-richest person in the United States, according to Forbes, with a net worth $14.6 billion. To remain in control of the company, he put $750 million into the deal. It's unclear if Greifeld has the means to complete a similar deal that guarantees he remains in control of a private Nasdaq.

Greifeld is considered one of Wall Street's better managers and most proficient cost cutters, and his desire to remain CEO of whatever formation the Nasdaq ultimately finds itself in is well known. His independent streak has scuttled other possible deals, analysts note, including the possibility of merging with the London Stock Exchange. It's unclear how Greifeld's desire to remain in control came into play with Carlyle.

Charles Gasparino joined FOX Business Network (FBN) in February 2010 as Senior Correspondent.