The Short Answer Can the IRS take your passport over tax debt? Indirectly, yes — and summer is exactly when people find out the hard way. If you owe the IRS more than $66,000 (counting penalties