If results hold, the Sweden Democrats, a nationalist party with Neo-Nazi roots, will become the third-largest party in parliament, handing the governing Social Democrats a stinging setback.
With most ballots counted, the ruling center-left Social Democrats had 28 percent of the vote, trailed by the Moderates with 19 percent and the nationalist Sweden Democrats with nearly 18 percent.
With 84.4 percent of votes counted, the Sweden Democrats had 17.6 percent of the vote compared to 28.4 for the Social Democrats, according to the Sweden public broadcaster SVT.
This significantly reduces the probability the centre-right Alliance coalition will secure a Parliamentary majority via an official or unofficial deal with the Sweden Democrats.
The new government, which could now take weeks to form, will need either cross-bloc alliances between centre-right and centre-left parties, or an accommodation with the Sweden Democrats, long shunned by all the other parties due to their extremist roots, to pass legislation - potentially giving the populists a say in policy.
Prime Minister Stefan Lofven's ruling center-left bloc maintained a slight lead with 40.6% of the vote, with the center-right alliance winning 40.3%. Sunday's general election was the first since the country of 10 million took in a record 163,000 refugees in 2015 as mass migration to Europe rose dramatically. The number is far lower than the asylum-seekers Germany accepted that year, but highest per capita of any European nation.
Immigration was the hot topic of the campaign, helping the steady rise in popularity of the Sweden Democrats.
The potential for an immigration backlash to result in a big boost for the far-right Sweden Democrats inspired fear among many Swedes before the election.
Sweden Democrats leader Jimmie Akesson declared victory at a supporters' rally saying, "We will gain huge influence over what happens in Sweden during the coming weeks, months and years".
"This government we have had now. they have prioritised, during these four years, asylum-seekers", Akesson said, giving an exhaustive list of things he says the government has failed to do for Swedish society because of migrants.
On a good night for the smaller parties, the ex-communist Left almost doubled its score to 9.8% and the centre-right Centre and Christian Democrat parties both advanced. The leader of the Moderates party that came in second, Ulf Kristersson, already had called on Lofven to resign and claimed the right to form Sweden's next government.
Arrivals dropped the following year to levels seen in previous years.
Mr Akesson had labelled the vote a choice between immigration and welfare in a campaign that was unusually antagonistic.
People watch and react at the Social Democratic Party's election party in Stockholm.
"We have a moral responsibility".
Around 18.5 percent of Sweden's population of 10 million was born overseas, according to Statistics Sweden.
He said the election result marked "the death of bloc politics" in Sweden.
Speaking to supporters late Sunday Kristersson said he planned to build a government that would "unite our country and take responsibility". "Integration is one of the biggest questions for the future in Sweden".