Chief Justice David Maraga said the 8 August election had not been "conducted in accordance with the constitution" and declared it "invalid, null and void".
He said the verdict was backed by four of the six Supreme Court judges.
The announcement drew cheers from opposition supporters both inside and outside the courtroom.
The court ruling did not attribute any blame to President Kenyatta's party or campaign.
What did the judges say was wrong?
Justice Maraga said the election commission had failed "to conduct the presidential election in a manner consistent with the dictates of the constitution".
He said the commission had committed irregularities "in the transmission of results", adding that the court would provide details in a full judgment within 21 days.
Dissenting judges said that the Nasa opposition alliance - which had petitioned the Supreme Court - failed to prove claims that the polls had been rigged.
International election monitors from the EU, the African Union and the US said there had been no major fraud on election day and urged Mr Odinga to concede.
The election sparked days of sporadic protests, in which at least 28 people were killed. The vote had raised fears of major political violence - as was the case after a disputed poll in 2007.
How have the two political sides reacted?
Mr Odinga said the ruling marked "a historic day for the people of Kenya and by extension for the people of the continent of Africa".
He added that he had "no faith at all in the electoral commission as currently constituted" and called for the prosecution of its members.
President Kenyatta, in a televised address, said that it was "important to respect the rule of law even if you disagree with the Supreme Court ruling".
He called for calm, saying: "Your neighbour will still be your neighbour, regardless of what has happened... My primary message today to every single Kenyan is peace. Let us be people of peace."
He added: "We are ready to go back again to the people with the same agenda that we delivered to the people."
But Senator Irungu Kang'ata, who is from Mr Kenyatta's Jubilee Party, told Kenyan TV: "The judgement is fake. There is an admission on the part of the chief justice that he has only read a part of the evidence. "
One of the two dissenting judges, Jackton Ojwang, said: "There is not an iota of merit in invalidating the clear expression of the Kenyan people's democratic will."
And the electoral commission?
Chairman Wafula Chebukati noted the ruling and said there would be "changes to personnel" ahead of the new election.
He invited the director of public prosecutions "to prosecute any of our staff that may have been involved in violations".
But he ruled out resigning, saying he had not been accused of wrongdoing.
A huge victory for Odinga - by Dickens Olewe, BBC News
Raila Odinga will feel vindicated against accusations that he was just being a bad loser in challenging President Kenyatta's win.
However, this historic decision is a massive indictment of the electoral commission. It is therefore no surprise that the opposition Nasa coalition is now calling for a new team to manage the next elections.
This is also a setback for the international, and some local, election observers, who profusely praised the election as free, fair and credible.
People will be watching for the reaction of former US Secretary of State John Kerry, who was the head of the mission for US NGO, the Carter Centre, whose positive assessment of the election was used in court.
Regardless of the winners and losers following the ruling, this is a proud moment for Kenya. The litigation and debate on the merits of the election was done at the Supreme Court and not on the streets.
Chief Justice Maraga said it best in his opening statement: "The greatness of a nation lies in its fidelity to the constitution and the strict adherence to the rule of law."