Novak and Viktor saved one match point en-route to a 5-7, 6-1, 11-9 victory over Pierre-Hugues Herbert and Nicolas Mahut. They face Klaase/Ram in quarter-finals.
The defending champion goes on to face no.31 seed Juan Martin del Potro for a place in the ATP Masters 1000 event last 16.
Novak advanced past the Argentine 4-6, 6-4, 6-4 in two hours and 38 minutes to move to 12-4 in their ATP Head2Head series.
Wildcard Denis Istomin prevailed in five grinding sets after four hours and 48 minutes of play at Rod Laver Arena on Thursday.
Novak eased past Argentine Horacio Zeballos 6-3, 6-4 in 72 minutes on Wednesday to advance into the last eight in Doha.
The reigning Paris Masters champion beat world no.34 Gilles Muller 6-3, 6-4 and next faces no.14 seed Grigor Dimitrov (Thursday).
Nole began his title quest at the ATP Masters 1000 event with a comfortable 6-3, 6-3 victory over world no. 50 Fabio Fognini.
The top seed and defending champion will face Mikhail Youzhny (RUS) in the third round on Friday.
Nole defeated Gilles Muller 7-5, 7-6 (3) for a place in the third round of the Canada Masters on Wednesday.
Nole captured his 30th straight Grand Slam win beating Frenchman Adrian Mannarino 6-4, 6-3, 7-6 (5).