Danica Roem, a 32-year-old journalist, ousted pro-Trump incumbent Republican Bob Marshall.
Mr Marshall had described himself as Virginia's "chief homophobe".
The district that elected Ms Roem, which includes outer suburbs of Washington DC, strongly supported Hillary Clinton in the 2016 presidential election.
Ms Roem championed gay, transgender and immigrant rights during her campaign, but the race mostly focused on the state's transport infrastructure.
Headlines of the night
- In Virginia, Democrat Ralph Northam defeated Republican Ed Gillespie in a bitter governorship race
- Within minutes of Northam winning, President Trump criticised Gillespie
- A Democrat will succeed Republican Chris Christie as New Jersey governor
Diverse politicians chosen across the US
Reasons for Republicans to worry
Analysis by Anthony Zurcher, BBC News, Washington
Democrats finally have a victory of the non-moral kind. After coming up short in a handful of special elections across the US, they went to the polls in a battleground state and posted a huge win.
How the Democrats, from governor candidate Ralph Northam on down, swept through election night in Virginia should be particularly concerning to Republicans across the US.
Turnout from Democratic supporters surged. They ran up huge margins with college-educated voters and residents in the wealthy Northern Virginia suburbs. The legions of rural voters who turned out for Donald Trump in 2016 were a non-factor.
Democrats won legislative races that were considered to be in play only in the rosiest of Democratic wave scenarios. Exit polls show a plurality of Virginians went to the polls to send a message to Mr Trump. Their top issue was healthcare. At least in Virginia, the president's dismal approval ratings translated into ballot-box poison.
The stage is now set for the midterm elections in 2018. Republicans will have a year to brace for what could be an anti-Trump tsunami forming on the horizon. What they - and Mr Trump - do next could decide their fate.
Mr Marshall had a strong record of opposing equality legislation.
The veteran Republican co-sponsored a 2006 Virginia constitutional amendment banning same-sex marriage.
He also introduced a bill that would have banned transgender school students from using the bathrooms of their choice.
Ms Roem is not the first openly transgender state lawmaker in the United States.
Stacie Laughton won a seat in New Hampshire's legislature in 2012, but she resigned before taking office.
A closeted transgender woman, Althea Garrison, won a seat in the Massachusetts legislature in 1992. Ms Garrison was later outed by a conservative newspaper.
Also on Tuesday former television anchor and Democrat Chris Hurst, whose girlfriend was fatally shot on-air in 2015, unseated a Republican incumbent in Virginia's House of Delegates.
Mr Hurst's girlfriend, Alison Parker, and Adam Ward were killed by a former employee of their station WDBJ in Roanoke while broadcasting.
A year since Trump upset the odds