A childhood belief about sport

When I was a child, and my father had first tried to explain (American) Football to me, he said how a team has "four downs to gain ten yards". In my first attempt to understand that, I got the idea that it meant a team, in advancing 10 yards, had to knock down four players of the opposing team or they would lose the ball. So at the first several football games I watched, I was counting how many players on the defensive team fell down, and trying to see if four fell down by the time the offense advanced ten yards, to see if the offensive team would retain possession of the ball. Naturally it was confusing, as often a team retained possession when I clearly thought they shouldn't have, and vice-versa.