Nasdaq OMX on Wednesday agreed to pay $10 million, the largest penalty ever levied against a stock exchange, to settle civil charges stemming from mistakes it made during Facebook's initial public offering last year, the U.S. Securities and Exchange Commission said.

The SEC said that Nasdaq's series of "ill-fated decisions" on the day of the IPO led to a series of regulatory violations.

As a result, more than 30,000 Facebook orders remained stuck in Nasdaq's system for more than two hours when they should have been either executed or canceled, leaving investors in the lurch and causing market makers to lose an estimated $500 million.

The exchange operator agreed to settle the charges without admitting or denying the allegations.

(Reporting by Sarah N. Lynch; Editing by Kenneth Barry)