A government spokesman later declared nationwide public holidays for Thursday and Friday.
With only a few seats left to count, official results showed Mr Mahathir's Pakatan Harapan alliance, along with an ally in Sabah state, Borneo, had won 115 seats with BN on 79 seats.
Opposition supporters poured on to the streets in celebration as the results became clear.
The campaign pitted Mr Mahathir's opposition group against the BN, led by incumbent Prime Minister Najib Razak.
The BN and its major party, the United Malays National Organisation (UMNO), have dominated Malaysian politics since the country won independence from Britain in 1957, but the once-powerful coalition has seen its popularity decline in recent years.
In the previous election, in 2013, the opposition made unprecedented gains, winning the popular vote, but it failed to win enough seats to form a government.
In a dramatic turn of events, then-opposition leader Anwar Ibrahim was sentenced to five years jail on sodomy charges, which he said were part of a political smear campaign.
As he left, he said he was "embarrassed" to be associated with a party "that is seen as supporting corruption".
Mr Najib has been embroiled in a corruption scandal, which saw him accused of pocketing some $700m from the 1Malaysian Development Berhad, a state investment fund. He has vehemently denied all allegations and been cleared by Malaysian authorities.
The fund is still being investigated by several countries and Mr Najib has been accused of stifling Malaysian investigations by removing key officials.
The government recently passed a law redrawing election boundaries, leading to accusations that it had gerrymandered constituencies to ensure they were filled by Malay Muslims, who are traditionally BN supporters.
In the days before the poll, election reform group Bersih 2.0 accused the Election Commission (EC) of multiple "electoral crimes", including irregularities in postal voting and failing to remove dead people from the electoral roll.
A controversial fake news law was also recently introduced, which critics say could be used by the authorities to muffle dissent.
Mr Mahathir is himself being investigated under that law after alleging that his plane had been sabotaged.
The government had insisted the election would be free and fair, with Mr Najib saying that the EC acted "for the good of all".
Voters were electing 222 members of parliament as well as state assembly members in 12 of the 13 states.
Malaysia uses a first-past-the-post electoral system, where the party that gets the most seats in parliament wins even if it does not win the popular vote.