Markus does market research for Interbrew, a large international beer conglomerate. His job is to run hypothesis testing to improve marketing strategies, such as packaging or advertising. He runs, on average 20,000 tests a year, half for packaging, which are run at level 5% and half for advertising which are run at level 1%. Which of the following assertions are true on average? (Choose all that apply.)

Question

At most 600 of the tests are rejected erroneously

At least of 600 of Markus's tests are incorrect

At most 600 of Markus's test are incorrect

Among all the tests that Markus should have rejected, at most 5% were not rejected

Among all the tests that Markus should not have rejected, at most 5% were rejected

Answer 1

To analyze the validity of the assertions based on the information provided, we first need to clarify some key terms:

1. Level of Significance (α): This indicates the probability of making a Type I error (rejecting a true null hypothesis). A 5% significance level means that, on average, 5% of the tests that should not have been rejected (true null hypotheses) will be erroneously rejected.

2. Total Tests: Markus runs an average of 20,000 tests a year, with half for packaging (10,000 tests at 5% significance level) and half for advertising (10,000 tests at 1% significance level).

Now, let's evaluate each assertion:

1. At most 600 of the tests are rejected erroneously:
   In the packaging tests (10,000 at 5% level), the expected number of Type I errors is 5% of 10,000 = 500.
   In the advertising tests (10,000 at 1% level), the expected number of Type I errors is 1% of 10,000 = 100.
   Total expected erroneous rejections = 500 + 100 = 600.
   This statement is true.

2. At least 600 of Markus's tests are incorrect:
   This statement suggests that at least 600 tests are false positives (Type I errors). Since the expected number of erroneous rejections is exactly 600, there may very well be cases with fewer than 600 false positives if he performs better than the expected rate. Thus, this assertion is false.

3. At most 600 of Markus's test are incorrect:
   As established, the expected number of erroneous rejections is 600. Therefore, it is reasonable to assert that the number of incorrect rejections will not exceed this figure on average. This statement is true.

4. Among all the tests that Markus should have rejected, at most 5% were not rejected:
   This statement pertains to the power of the test but seems to misapply the context. In general, "should have rejected" refers to the power of the test (the probability of correctly rejecting a false null hypothesis), which is not addressed directly in the given information. Hence, we cannot determine the validity of this statement without more specific data. This assertion is ambiguous.

5. Among all the tests that Markus should not have rejected, at most 5% were rejected:
   This is directly tied to the significance level of the tests. For the packaging tests (5% significance), it is correct that at most 5% of the true nulls could be rejected. Thus, this statement holds. For the advertising tests (1% level), it similarly suggests that at most 1% would be rejected, but the phrasing is based on mixing them together (which can be misleading). Thus, considering each test's individual significance level, the statement is true.

Final Assertions:

• True:

  • At most 600 of the tests are rejected erroneously.
  • At most 600 of Markus's tests are incorrect.
  • Among all the tests that Markus should not have rejected, at most 5% were rejected.

• False:

  • At least 600 of Markus's tests are incorrect.

• Ambiguous:

  • Among all the tests that Markus should have rejected, at most 5% were not rejected.