With around 36,000 votes counted, official figures showed the 'no' camp in the lead with 56.8 pct of the vote compared with 43.2 pct in favor of the accord, state television said.
Iceland has 230,000 voters but a turnout figure was not immediately available.
"These are disappointing numbers. It looks like the 'no' side will have it," Prime Minister Johanna Sigurdardottir told Iceland's state television.
The debt was incurred when Britain and the Netherlands compensated their own nationals who lost savings in online "Icesave" accounts owned by Landsbanki, one of three Icelandic banks that collapsed in late 2008.
Icelandic lawmakers in February backed the repayment plan agreed with the creditors in December but the president refused to sign the bill, triggering the referendum.
In March 2010, Iceland rejected an earlier Icesave repayment blueprint in a referendum.
Policymakers hope solving the Icesave issue will help Iceland, whose economy went into deep recession after its bank crisis, get back into financial markets to fund itself.
That is a condition for it to remove controls on capital flows it imposed in 2008 to stabilize a tumbling currency.
(Reporting by Anna Ringstrom and Omar Valdimarsson; editing by Philippa Fletcher)