A less-than-smooth unwinding of the Federal Reserve's $2.9 trillion balance sheet could do real harm to the U.S. economy or to employment, a top Fed official warned on Saturday.

"It may go very smoothly, that would be very nice," Philadelphia Fed President Charles Plosser said at the American Economic Association's annual meeting.

Plosser has been an outspoken critic of the Fed's large-scale asset purchases and has publicly worried about the difficulty of scaling back the Fed's balance sheet when the time comes.

"If it doesn't go smoothly, we may do more damage to the economy in terms of instability, or a rise in unemployment," he said. "I think those concerns are real and we should take them into account." (Editing by Andrea Ricci)