|2019 French Open
|Venue: Roland Garros, Paris Dates: 26 May-9 June
|Coverage: Live text and radio commentary on selected matches on the BBC Sport website and app.
Williams, 37, was seeded 10th but lost 6-2 7-5 to her 20-year-old compatriot, who was at times booed by the crowd.
The defeat means Williams' quest to equal Margaret Court's record of 24 Grand Slam singles titles continues.
"I'm so happy with this win - obviously you can tell with these emotions," a crying Kenin told the crowd on Philippe Chatrier.
"Playing against Serena, you have to fight for every point."
Williams' defeat followed the shock exit by world number one Naomi Osaka a few hours earlier and leaves only three top-10 players in the women's draw.
Earlier in the day, defending champion Simona Halep enjoyed a straightforward 55-minute victory, beating Ukraine's Lesia Tsurenko 6-2 6-1 to reach the last 16.
World number 35 Kenin will play Australian eighth seed Ashleigh Barty in the fourth round, while Halep faces unseeded 18-year-old French Open debutant Iga Swiatek.
Kenin booed en route to victory
Kenin, who received a walkover in the second round after Bianca Andreescu withdrew, was fired up from the outset and has now won 11 of her last 13 matches against compatriots.
But the crowd became irritated with the young American, booing Kenin as she repeatedly checked ball marks to see whether the umpire had made the right call and even rubbed out one mark with her foot.
After Kenin threw her racquet on the clay in celebration, she exchanged a firm handshake with Williams at the net.
Yet some boos were still audible as Kenin lifted both arms in celebration as she went to retrieve her racquet, although it did not stop her offering thanks to the Chatrier crowd for their "great support".
While it was a mixed response for Kenin, Williams left the court to deafening applause.
Williams struggles from the start
Kenin is in the French Open main draw for just the second time, while Williams was making her 17th appearance. But the older American looked frustrated from the very beginning of the match.
Williams saved four break points in her second service game, but stumbled as Kenin played a drop shot and the 23-time Grand Slam champion was broken on the fifth attempt.
The world number 10 had break point in the next game, but failed to convert and was then broken again by her compatriot, who held serve to win the set.
Kenin started strongly in the second set, immediately breaking Williams' serve.
Though Philippe-Chatrier showed their support for the 2002, 2013 and 2015 champion, Williams failed to win four break points, but was successful with the fifth to level things at 3-3.
Momentum then swung Williams' way momentarily, as she served three aces to hold in the next game.
But her opponent got another break at 5-5 and, though Williams had break point in the final game, Kenin kept her composure to seal victory.
Women's draw blown wide open
Williams's exit means only three of the top 10 seeds are left in the draw, highlighting the unpredictable nature of the women's singles game.
That is contrasted with the fact all of the men's top 10 seeds have reached the last 16, only the third time this happened after the 1969 French Open and the 1970 Australian Open.
The top half of the women's draw has been blown wide open by the defeats for Williams and top seed Osaka on Saturday.
Third seed Halep, Australian eighth seed Ashleigh Barty and American 14th seed Madison Keys are the only seeded players left in that half.
Williams loses another chance as she chases Court
Three-time French Open champion Williams has won fewer of her 23 Grand Slam titles at Roland Garros than she has at the Australian Open (seven), Wimbledon (seven) and US Open (six).
So, coupled with the fact she appeared to be hampered by her movement against Kenin, it is perhaps unsurprising she fell early as she continues to chase the 24th Grand Slam singles title that would draw her level with Australian Court's all-time record.
After giving birth to daughter Olympia in September 2017, Williams made her Slam return at last year's Roland Garros and pulled out before her scheduled last-16 match against old foe Maria Sharapova with an injury.
Yet she returned to the grass a few weeks later with a run to the Wimbledon final, German 11th seed Angelique Kerber spoiling her comeback at the All England Club with a 6-2 6-2 win.
Another chance beckoned at the US Open in September, yet she suffered another defeat in the final - this time in more controversial circumstances against Japan's Naomi Osaka.
An angry Williams accused the umpire of being a "thief" in some of the most dramatic scenes at a Grand Slam final, going on to lose 6-2 6-4 in New York after being docked a game in the second set.
Four months later she returned to the Australian Open, where Court has won 11 of her 24 titles and has a stadium named after her.
Another dramatic exit followed for Williams as she missed four match points as Czech Karolina Pliskova won the final six games to reach the semi-finals in three sets.
BBC tennis correspondent Russell Fuller
Sofia Kenin is having a fine year, and at the age of just 20 and with her illustrious compatriot at the other end of the court, showed great composure to serve out the match.
Williams, it has to be said, was a long way from her best.
Her movement was way below par, the legacy of a knee problem and a reduced schedule which saw her complete just eight matches in the eight months leading into Roland Garros.
Wimbledon starts exactly a month today. Williams says she hopes that will prove enough time to get her back into the form required, but knows she need matches, and hinted she might even consider playing a warm-up event in the lead up to the Championships.