Before it emerged that Mr Lochte had left Brazil, a judge ordered that the four have their passports confiscated pending further police questioning, amid reports of inconsistencies in the men's accounts of the alleged robbery.
Gunnar Bentz and Jack Conger were taken off a US-bound plane at Rio de Janeiro airport on Wednesday night. Team-mate James Feigen had remained in Brazil.
Our correspondent says the swimmers - who have repeatedly changed their accounts of what happened - could be charged with falsely communicating a crime.
The three men remaining in Brazil - who have been ordered by a judge to surrender their passports - are expected to speak to investigators on Thursday.
What is said to have happened?
The Civil Police of Rio de Janeiro told the BBC's Wyre Davies that about 06:00 (09:00 GMT) on Sunday, the four men arrived by taxi at a petrol station in Barra da Tijuca, 16km from the Olympic Park.
One of the the athletes broke the door to the bathroom, the police said, and petrol station attendants asked the Americans to pay for the damage.
A verbal dispute is said to have ensued with the attendants, and security guards were called to contain the incident. The police were also called.
While police were on their way, another customer at the gas station served as interpreter for the athletes and helped agree payment for the damage.
When police arrived, the athletes had already gone after paying for the broken door, and they returned to the Olympic Village.
Video from CCTV appears to show the athletes being detained and ordered to sit on the ground.
What have the swimmers said?
Mr Lochte admitted on Wednesday to some inaccuracies in his original account of being robbed at gunpoint in the early hours of Sunday, but vehemently denied making the story up.
"I wouldn't make up a story like this nor would the others - as a matter of fact we all feel it makes us look bad," he told US TV network NBC.
Accounts of what happened to the swimmers have been confusing from the beginning.
News of the incident emerged after Mr Lochte's mother told US media about it.
Mr Lochte himself gave an initial account of the events to NBC on Sunday, saying he and the other swimmers had been in a taxi returning from a club in the early hours when they were pulled over by men wearing police badges.
He said they had pulled a gun and told the swimmers to get on the ground. "I refused... and then the guy pulled out his gun, he cocked it, put it to my forehead..."
Mr Lochte has since slightly altered his account, telling NBC on Wednesday that the taxi had not been asked to pull over - they had been robbed while making a stop at a petrol station - and he said the gun had not been pointed directly at his forehead.
He called the inconsistencies a "traumatic mischaracterisation" caused by the stress of the incident.
Police and the judge investigating the case found inconsistencies in the men's accounts.
CCTV footage of their return to the athletes' village appears to show the swimmers laughing and joking, and handing over their wallets, phones and accreditation, as they go through the security screens. The judge said they had not show signs of being affected by a robbery.
What has been the reaction?
US Olympic Committee (USOC) spokesman Patrick Sandusky said in a statementthat Mr Bentz, Mr Conger and Mr Feigen were co-operating with authorities and looking to speak to them on Thursday.
Mr Lochte's lawyer told the BBC he had returned to the US two days ago before the controversy broke.
"He was never asked to remain for further investigation or for any other purpose after he met with Brazilian authorities after he gave a statement," said Jeffrey Ostrow.
Who are the swimmers?
Lochte is one of the most successful swimmers in history, with 12 Olympic medals, and he once had his own reality television show in the US. In Rio, he swam in two events, winning gold in the 4x200m freestyle relay.
Feigen won gold in the 4x100m freestyle relay.
Bentz competed in the 4x200m preliminaries, but not the final. He still received a gold medal after the US team's win.