Question

1. I can't do the science project myself. It's too difficult?

What does it refer to? #2 or #3?

2. doing the science project myself
3. the science project
---------------------------------
4. I'm going to Tom's house to work on the science project.

5. I'm going to Tom's house to do the science project.

(Are both the same? What is the difference between #4 and #5?)

Answer 1

In #1, "it" would have to refer to the science project. The word "project" is the noun to which the pronoun "it" refers.
http://www.chompchomp.com/terms/antecedent.htm

There's not much difference between 4 and 5. Sentence 4 implies that they may or may not finish the project at that time. Sentence 5 implies that they'll work on it until it's finished.