About 500 armed police officers were involved in the operation.
Dozens of addicts reacted in anger, vandalising cars and looting shops.
Sao Paulo's centre-right Mayor Joao Doria said the operation marked the end of impunity in the area, widely known as Cracolandia or Crackland.
Critics say the move will merely push the problem to other parts of the city.
Mr Doria promised to knock down many buildings and redevelop the streets near the Luz train station that have become an open-air drugs market over the past decade.
"Crackland doesn't exist any more and it won't come back. The government won't allow it," said Mr Doria during the operation.
He also announced that CCTV cameras would be installed in that part of the city.
But he later admitted in a press conference that "it will be difficult to put an end to a historical problem".
"Police will be deployed here permanently and the problem will be reduced," Mr Doria said.
The governor of Sao Paulo, Geraldo Alckmin, said state rehab centres had 3,000 places available for addiction treatments.
The mayor and the governor recently announced a plan to combat drug trafficking in the region.
The previous left-wing mayor of Sao Paulo, Fernando Haddad, had a programme that tried to solve addiction through therapy and without the use of force by police.
But many were critical and felt that something had to be done about Crackland, which became a no-go area for most residents of Brazil's largest city.