The Federal Reserve Bank of New York has joined a group of influential bond holders demanding that Bank of America (NYSE: BAC) buy back bad mortgages packaged into $47 billion in bonds and sold to investors.

The group, which reportedly also includes Pimco and Black Rock, says it holds more than 25% -- or $16.5 billion -- of the $47 billion in outstanding mortgage-backed securities contained in the bonds.

The bonds were put together by Countrywide Financial, a unit of Bank of America.

The possible ramifications – that other banks might also be forced to buy back bad mortgages packaged into bonds – contributed to a broad selloff in stocks. The Dow Jones Industrial Average closed down 165.07 to 10978.62.

A law firm representing the group issued a statement Tuesday alleging that Countrywide failed to properly service the loans.

The statement did not name the bond holders making the complaint, but Fox Business Network has confirmed that the New York Fed is part of the group. Pimco and BlackRock were named by Bloomberg News. Pimco did not immediately respond to a request for comment, and a BlackRock spokeman declined to comment.

Bank of America also didn’t immediately respond to a request.

The letter, to Bank of New York Mellon Corp. (NYSE: BK), the trustee for the bonds, and Bank of America, cited the latter’s "failure to observe and perform, in material respects" its duties as the servicer for the bond deals.

The failure to properly handle the loans "has materially affected the rights" of bondholders, the letter said.

The letter urges Bank of New York Mellon as trustee “to enforce Countrywide’s obligations to service loans prudently by maintaining accurate loan records, demanding the repurchase of loans that were originated in violation of underwriting guidelines, and compelling the sellers of ineligible or predatory mortgages to bear the costs of modifying them for homeowners or repurchasing them” if necessary.

As mortgage servicer, Bank of America is responsible for collecting loan payments and working with troubled borrowers.

The letter seems to lay the groundwork for what could be one of the first lawsuits by mortgage-bond investors seeking to enforce their contract rights, including loan buybacks, in response to the current foreclosure crisis.

Charlotte, North Carolina-based Bank of America will “defend our shareholders” by fighting unjustified demands it buy back defective mortgages, Chief Executive Officer Brian T. Moynihan said today on Bloomberg Television.