Nothing riles Americans quite like taxes: who pays what, whether the government needs to raise them or might slash them, and how much they’re eating away at our paychecks.

And when it comes to tax avoidance — the legal means of minimizing one’s tax burden, not to be confused with its illegal cousin, tax evasion — a growing chorus of critics say high tax rates are to blame, and an overly complex tax code isn’t helping. The emerging point of view: The higher tax rates get, the more people will try to figure out ways to stop paying them.

As evidence of that, analysts point to the relatively steady chunk of gross domestic product that can be attributed to tax revenues, even as nominal rates rise and fall. From 1950 until 2010, all federal taxes hovered between 15% and 20% of GDP, though top income tax rates varied wildly (from 28% to 91%) during that period. And that suggests a greater reliance on loopholes when rates are higher, experts say.

“When tax rates are that high, nobody is going to pay,” says Alan Dlugash, an accountant at Marks Paneth & Shron LLP in New York. “You’re going to find a way to get out of it,” he says. In fact, he says his high-end clients in New York are holding onto stocks and avoiding selling their business for fear of being bulldozed by a giant capital gains tax bill.

Big business appears to be following suit. The New York Times reported in May that loopholes in the tax code have meant U.S. multinationals are paying far less than the corporate tax rate (one of the highest in the world), leading one expert to conclude that U.S. businesses were “world leaders in tax avoidance.”

The Internal Revenue Service doesn’t have specific figures on the revenue lost each year to tax avoidance, but nearly a decade ago it reported a tax-gap of $345 billion, the vast majority of which was lost to under-reporting. That figure hasn’t been updated since 2001, though the IRS says plans are underway to release more current stats.

Tax critics, meanwhile, are busily trying to prove the harmful effects of high tax rates. For example, raising $1 trillion in tax revenue costs the economy and taxpayers an additional $110 to $150 billion, according to a 2009 report by Robert Carroll, a fellow at the conservative-leaning Tax Foundation. And a 2008 IMF paper argues that lowering the tax rate would effectively increase tax compliance and raise revenues.

Nina Olson, national taxpayer advocate at the IRS, says she hasn’t seen any definitive evidence that high rates contribute to tax avoidance; the problem, she says, is ambiguity: the tax code has undergone a staggering 4,428 revisions in the last decade. The code itself is some 3.8 million words, creating endless opportunities to exploit the system and confusing even those who want to comply into unintentional non-compliance.

“People find it confusing. And if you don’t understand it, then you’re going to feel ripped off,” Olson says, adding that a person whose marginal tax rate is 28% could actually pay only 5% once all the various tax benefits are factored into the equation.

Olson’s office is examining the root causes for non-compliance, looking specifically at attitudes towards tax, education, the influence of tax professionals, enforcement and even civic duty.
Not everyone is convinced that lowering tax rates would stem the wave of tax avoidance. Ted Gayer, a senior fellow at the Brookings Institute, acknowledges that high taxes impact the economy through consequences such as lower labor supply and disincentive to invest. But slashing taxes won’t bring a tide of new revenue, he says.

“When you change taxes, you’re going to get a behavioral response,” Gayer says. “But we shouldn’t fool ourselves into thinking we can get a free lunch by lowering tax rates and collecting more revenue. I don’t buy it.”