The two men had a rocky relationship. In 2017, Mr Rosenstein appointed Special Counsel Robert Mueller to probe claims of Russian interference in the 2016 presidential election.
In a resignation letter, he wrote: "We enforce the law without fear or favour because credible evidence is not partisan."
He will leave the post on 11 May.
Correspondents say Mr Rosenstein had been expected to resign in March, following the appointment of William Barr as attorney general.
However, he stayed in the job longer to help Mr Barr manage the public release of the special counsel's findings from the investigation, which has dominated much of Mr Trump's presidency so far.
Mr Trump has repeatedly called the investigation a "witch hunt".
The relationship between Mr Rosenstein and the president became particularly fractured last September, when the New York Times published a story claiming the deputy attorney general had discussed ousting Mr Trump.
Citing anonymous sources, the paper said Mr Rosenstein had suggested secretly recording the president in order to prove he was dysfunctional, and had argued that it was permitted in line with the constitution.
Mr Rosenstein denied the claims, and a source told the BBC at the time that the comment was sarcastic.