The final vote was 52-48. Moderate Democratic Sens. Joe Manchin of West Virginia and Jon Tester of Montana joined Republicans to vote in favor of the repeal.
Even with Senate approval, the GOP-backed resolution is unlikely to overturn the mandate. The Democrat-controlled House is not expected to take up the measure and President Biden would likely veto the bill if it cleared Congress.
Republicans brought the repeal to the Senate floor under the "Congressional Review Act," which allows Congress to review presidential executive orders. Sen. Mike Braun, R-Ind., who sponsored the resolution, said the mandate was an example of the "heavy hand of government" hurting businesses.
"It’s got Main Street America scared," Braun said prior to the vote. "They’re worried about, well, what does this mean on other issues? Anybody who thinks this is a good idea, imagine the next time it happens when you’re on the wrong side on whatever the merits of the case would be."
The Biden administration’s mandate requires private companies with 100 or more employees to ensure their employees are vaccinated against COVID-19 or undergo regular testing. Firms that do not comply face steep fines.
GOP lawmakers argue the mandate is too broad and constitutes federal overreach. Business groups who oppose the mandate say it is too burdensome given strained economic conditions.
"It's daunting to families as they're facing higher bills for their gas and their heating," Sen. Shelley Moore Capito, R-W.Va., said ahead of the vote. "They are very concerned about what this would do to their long-term ability to get a job, keep a job. I think they realize that this is an invasion into their own abilities to make decisions about themselves in their health care."
The mandate prompted immediate legal challenges from several states as well as business and religious groups. Last month, a federal appeals court enacted a temporary hold on enforcement of the mandate pending the outcome of litigation.
Senate Majority Leader Chuck Schumer, D-NY., slammed the Republican effort to repeal the mandate.
"Some of the anti-vaxxers here in this chamber remind me of what happened 400 years ago when people were clinging to the fact that the sun revolved around the Earth. They just didn’t believe science. Or 500 years ago when they were sure the Earth was flat," Schumer said earlier in the day.
A fight over the mandate nearly derailed efforts to fund the government prior to a key deadline last week.
The Senate approved a continuing resolution to fund the government with just hours to spare but rejected a Republican-backed amendment to strip funding from Occupational Safety and Health Administration, the entity responsible for implementing the mandate.
The Associated Press contributed to this report.