Novak failed to reach the season-ending event semis for the first time since 2011.
Roger Federer defeated second seed Novak Djokovic 6-4, 6-3 to reach the last four at the Nitto ATP Finals.
Both players were 1-1 in round-robin play entering the decisive third match on Thursday.
Federer broke Novak in the third game and never faced a break point to take the opening set 6-4.
The third-seeded Swiss earned two break points early in the second set, but Djokovic saved them to hold his service game. The five-time former champion from Serbia then failed to convert his break point chance in the fourth game. A couple of minutes after saving the break point, Federer broke for a 6-4, 3-2 lead. He closed the set and match following a break to love in the ninth game, that brought him his 23rd win in 49 career meetings with Novak.
Match statistics: aces (2-12), double faults (3-2), winners (14-23), unforced errors (13-5), total points won (44-67).
While Dominic Thiem and Roger Federer advanced from Group Bjorn Borg into the last four, things are still up for grabs in the Andre Agassi Group. In Friday action at The O2, Rafael Nadal will face Stefanos Tsitispas (qualified), and defending titlist Alexander Zverev takes on Daniil Medvedev .
Novak will next lead 2010 champion Serbia in the new season-ending Davis Cup by Rakuten Finals, to be held on indoor hard courts at the Caja Mágica in Madrid, Spain, from 18 to 24 November.
18 teams will compete in six round-robin groups of three teams. Serbia is in Group A with France and Japan.
Wednesday, November 20: SERBIA vs JAPAN (11:00 CET)
Thursday, November 21: SERBIA vs FRANCE (11:00 CET)
* Each tie will consist of two singles matches and one doubles match
Photo: ATP Tour
