Police said they had arrested a man near the jail and later confirmed to Danish media it was Madsen.
Pictures from the scene showed armed police surrounding him near the prison west of Copenhagen.
Madsen was jailed for life for Kim Wall's murder. She boarded his homemade submarine for an interview in 2017.
Her mutilated body was discovered on a beach by a passer-by 11 days later.
A video, taken by a photographer from the Ekstrabladet website, showed the convicted killer sitting against a fence with a belt around his waist.
Danish media said he had told police at Herstedvester jail, to the west of Copenhagen, that he was carrying a bomb.
Who was Kim Wall?
Madsen's claim that the journalist's death on board his submarine in 2017 was an accident was rejected at his trial. He later admitted to a journalist that he was to blame.
Kim Wall was a freelance journalist who reported around the world, from North Korea to Uganda, for a variety of quality publications.
When she climbed on Madsen's 40-tonne submarine, it was supposed to be a final Danish story before moving to the Chinese capital Beijing with her Danish partner.
What happened at the prison?
West Copenhagen police said little about what had happened at the jail in Albertslund, only that a man had been arrested while trying to escape.
But it was clear from the image of a man sitting propped up against a fence in a residential street that it was Madsen.
Unconfirmed reports said bomb disposal experts were at the scene.
Madsen was given a life sentence for murder, which normally would mean up to 17 years in jail, according to Danish public TV.