The story is more complicated than that: Does Dairy Cause or Prevent Cancer? An Objective Look
Casein is a protein, and it gets digested, so it's hard for it to have much effect one way or the other. The page I linked to discusses stuff that doesn't get digested, like calcium.