The correct choice to complete the text and conform to the conventions of Standard English is:
(Choice A) new. Sequencing
So the completed sentence would read: "In the past, researchers were limited to studying the marks found on the fossilized teeth of skeletons, but in 2017 a team led by Laura Weyrich of the Australian Centre for Ancient DNA tried something new. Sequencing the DNA found in Neanderthals’ fossilized dental plaque."