The Prince, 73, is said to be particularly frustrated because he is due to represent the Queen at the Commonwealth heads of government meeting in the Rwandan capital Kigali this month.
The spokesperson said: “We would not comment on supposed anonymous private conversations with the Prince of Wales, except to restate that he remains politically neutral.
This comes after the judge at the High Court refused to grant an injunction sought by campaigners to block the one-way flight to Rwanda next Tuesday.
“Matters of policy are decisions for Government.”
According to The Times, a source heard the Prince of Wales expressing his opposition to the plans in a private conversation.
However, Clarence House spokesperson insisted that he had not tried to influence the Government.
The campaign groups said they would appeal against the decision on Monday.
More than 30 people who arrived in the UK illegally are due to be on the plane to Rwanda the following day.