The referendum was held on Sunday in Saint-Apollinaire, a town of about 5,000 located just outside Quebec City.
Provincial rules meant only 49 people were eligible to vote; the nays won 19-16 and one ballot was rejected.
The cemetery was proposed by the Quebec Islamic Cultural Centre, which was the site of a shooting that killed six people and injured 19 in January.
"We never thought people could oppose the installation of a cemetery," the centre's president Mohamed Labidi told Radio-Canada. "What are they afraid of?"
The Islamic cultural centre had purchased a plot of land in a wooded area next to an existing cemetery after the shooting. The only Muslim cemetery in Quebec is in Laval, hours from Quebec City.
The town's decision to oppose the cemetery has led to an outcry amongst Muslims and civil-rights advocates across the country and may lead to a human rights complaint, Mr Labidi said.
The mayor of the town supported the cemetery and has said he fears his town's reputation has been hurt.
"They do not know these people so they base their decisions on hearsay," Mayor Bernard Ouellet told the Canadian Broadcasting Corporation.
Opponents went door to door to gather signatures to call for the referendum, since building the cemetery would require a minor zoning change. A provincial law allows referendums to be held on zoning matters, with only people who live in the affected area eligible to vote.
That meant only 49 people in a town of 5,000 were eligible to vote, and only 36 people cast ballots.
"We need cemeteries that welcome everybody, no matter their religion, where they are from, their skin colour, their culture. You have to think about that because in 20 years it is going to be a problem," opponent Sunny Létourneau told the CBC.
She says she only supports non-denominational cemeteries.