The correct choice to complete the text so that it conforms to the conventions of Standard English is:
(Choice A) new. Sequencing
So the complete sentence is: "In the past, researchers were limited to studying the marks found on the fossilized teeth of skeletons, but in 2017 a team led by Laura Weyrich of the Australian Centre for Ancient DNA tried something new. Sequencing the DNA found in Neanderthals’ fossilized dental plaque."
This choice correctly uses a period to separate the two complete sentences.