Congress requires the US president to certify every 90 days that Iran is upholding its part of the agreement. Mr Trump had already recertified twice, but refused to sign a third time ahead of a Sunday deadline.
Congress now has 60 days to decide whether to pull out of the nuclear deal by re-imposing sanctions.
Some advocates of the deal, signed between Iran and six international powers - the UK, US, Russia, France, Germany, and China - had feared that Mr Trump would withdraw the US entirely.
Instead he essentially passed the ball to Congress, which will now decide whether to rewrite the framework in accordance with Trump's wishes. The president made it clear that if it did not, he would cancel the deal.
"In the event we are not able to reach a solution working with Congress and our allies, then the agreement will be terminated," he said. "It is under continuous review and our participation can be cancelled by me, as president, at any time."
What changes does he want?
Mr Trump is seeking is the end to the nuclear deal's so-called "sunset clauses", one of which allows for the lifting of restrictions on Iran's nuclear enrichment programme after 2025.
He also called for new sanctions on Iran's elite Revolutionary Guards, which he called the "corrupt personal terror force of Iran's leader", and restrictions on Iran's ballistic missile programme, which is not covered by the deal.
The president said that congressional leaders were already drafting amendments that would curb the ballistic missile development and eliminate expiry dates on restrictions to Iran's nuclear development.
How did key players respond?
Iranian President Hassan Rouhani said the US was "more than ever isolated" and could not change the nuclear deal.
"As long as our rights are guaranteed, as long as our interests are served, as long as we benefit from the nuclear deal, we will respect and comply with the deal," Mr Rouhani said.
Yukiya Amano, head of the International Atomic Energy Agency (IAEA), said Iran was implementing the deal and was subject to "the world's most robust nuclear verification regime".
European diplomats warned that any such unilateral changes to the agreement were likely to trigger the deal's collapse and a return to a nuclear standoff in the Middle East.
EU foreign policy chief Federica Mogherini called the deal "robust" and said there had been "no violations" by Iran. She said it was not in the power of "any president in the world" to terminate the agreement.
In a joint statement, the UK, Germany and France said they were "concerned" by Mr Trump's move but remained committed to the deal. They said they "shared concerns about Iran's ballistic missile programme and regional activities".
Russia said it remained committed to the deal and was opposed to the use of "aggressive and threatening rhetoric in international relations".
Israeli Prime Minister Benjamin Netanyahu congratulated Mr Trump, who he said had "boldly confronted Iran's terrorist regime". Saudi Arabia also backed the US president's "firm strategy".
What has changed?
By Lyse Doucet, chief international correspondent
President Trump has recast the list of Middle East threats, with Iran replacing Islamic State as Enemy Number One.
That world view is shared by his strongest supporters in the region, including Israel and Gulf Arab leaders who have long seen Iran as their primary threat, and a rival with vast sway across the Middle East.
They resented Washington's focus on the Iran deal during President's Obama administration. Like President Trump, they want to undo his legacy. The new approach imposes new sanctions but stops short of designating Iran's elite Revolutionary Guards as a terrorist group - a step Iran says would be tantamount to a declaration of war.
The urgent question now is whether the new strategy will embolden Iran's hardliners including the Revolutionary Guards. Like US forces, they're involved in battles to defeat IS in Iraq and Syria, and may also see a new enemy.
Trump hands Congress a hot potato
By Anthony Zurcher, BBC News Washington
Donald Trump, trying to reconcile a bluntly delivered campaign promise with the tricky realities of governing, is taking a half-step - and handing the mess to Congress.
Legislators do not make for great caretakers, however, and without firm White House guidance Iran may prove a daunting challenge. The president has decertified Iran's compliance, but Congress will have to decide how to fix the deal to his liking.
The administration recommends establishing "triggers" that would automatically impose penalties on the Iran. That will take a lot of legislative manoeuvring, not Washington's strong suit lately.
There are signs of progress in Congress, but with tax reform and budget negotiations continuing, the schedule is packed. At some point Mr Trump could again be on the spot. He says if there's no further action, he will officially nix the deal.
The original Iran deal legislation was a way to allow congressional Republicans to object to the agreement without killing it. Now, it seems, Mr Trump wants new provisions that will allow him to kill it - or keep it - without getting his hands dirty.
What is the nuclear deal?
Formally known as the Joint Comprehensive Plan of Action, it is designed to prevent Iran developing a nuclear weapon.
It lifted some sanctions that stopped Iran from trading on international markets and selling oil.
The lifting of sanctions is dependent on Iran restricting its nuclear programme. It must curb its uranium stockpile, build no more heavy-water reactors for 15 years and allow inspectors into the country.