The EU Court of Justice ruled that jailing migrants contradicts an EU directive whose main objective is "to set up an effective policy to drive out and repatriate third country nationals whose stay (in the EU) is irregular, while respecting their fundamental rights," the court said in a statement.
The court is tasked with ensuring that EU directives are interpreted and applied in the same way in all EU countries
Under Italy's 2009 law, migrants who enter Italy illegally and refuse to leave face a prison sentence of from one to four years and fines of up to 10,000 euros, followed by immediate expulsion.
A Italian court in the northern city of Trento had referred to the EU court the case of an Algerian, Hassen El Dridi, who in 2010 was ordered to leave Italy within five days because he did not have a residence permit.
El Dridi ignored the court order and was given a one-year jail term which El Dridi appealed.
The EU court said that following its ruling, judges in Trento should "disapply" the jail terms contained in Italy's immigration legislation.
Italy's interior minister Roberto Maroni said he was "dissatisfied" with the court ruling and was considering taking action against it.
"In the coming days, I will evaluate the consequences of this sentence and see what can be done to remedy it," he said.
"The European Court of Justice's decision leaves me dissatisfied because there are other European countries that have made illegal immigration a crime and have not been censured for this," Maroni said.
"Second, if illegal immigration is legitimised, its decriminalisation together with an EU directive on repatriation will make it impossible to deport migrants," he added.
The ruling was welcomed by the head of the Vatican's migration body, Antonio Maria Veglio.
"The sentence shows attention and sensitivity towards human dignity, even when the human being concerned is in an irregular position," said Veglio.