Mr Pence briefly encountered Mr Kim but they tried to avoid directly facing each other, Yonhap news agency reports.
Meanwhile South Korean President Moon Jae-in has shaken hands with North Korean leader Kim Jong-un's sister at the Winter Olympics opening ceremony.
The Games are taking place amid tension over North Korea's nuclear programme.
Mr Pence has brought to South Korea as a guest Fred Warmbier, the father of a young American who died after being released from prison in North Korea.
Meanwhile on Thursday North Korea held a military parade that had been scheduled for April but was brought forward.
What happened at the dinner?
Mr Pence and Kim Yong-nam were being hosted by South Korean President Moon Jae-in before the opening ceremony in Pyeongchang.
But the US vice-president left the reception venue after five minutes, South Korea's Yonhap said.
While Mr Moon and Japanese Prime Minister Shinzo Abe both shook hands with Mr Kim, Mr Pence did not, South Korean officials said.
At the event Mr Moon said he hoped the Winter Olympics would be remembered as the "day peace began". He is due to meet the North Korean delegation for talks on Saturday according to Yonhap.
At the opening ceremony, Mr Pence, North Korean leader Kim Jong-un's sister Kim Yo-jong and Kim Yong-nam were seated in close proximity to each other.
Who is Kim Yo-jong?
The highest profile member of the North Korean delegation to the Games, she is the first immediate member of the North's ruling family to visit the South since the 1950-1953 Korean war.
Ms Kim, who is said to be very close to her brother, was promoted to the North's powerful politburo last year.
She is on a US sanctions list over alleged links to human rights abuses in North Korea.
Ms Kim is thought to be about 30 years old, around four years younger than her brother.
Her visit is being seen as a sign that Kim Jong-un is serious about improving ties with the South, the BBC's South Korea correspondent Laura Bicker reports.
She adds that some are also speculating that Ms Kim might be bringing a message from her brother.
Meet North Korea's first family
How will the Koreas compete at the Games?
Athletes from both North and South Korea marched under one flag at the opening ceremony, bringing spectators to their feet.
They are also fielding a joint women's ice hockey team.
Alongside 22 athletes, Pyongyang has sent more than 400 delegates to the Games, including a team of cheerleaders and an orchestra.
However the opening ceremony was not shown on North Korean state TV, which was broadcasting patriotic songs and slogans celebrating industry and the armed forces.
Have relations warmed?
The sports diplomacy comes at a time of improved ties between the two Koreas, although experts have cautioned that it does not put an end to underlying regional tensions.
The Korean peninsula has been divided since the 1950-53 war and the two sides have never signed a peace treaty.