A childhood belief about sport

When my father and little brother used to watch football on TV, the commentator would often make references to players being in the "penalty box". This was a specific area of the pitch from which penalty shots were taken.

I thought the "penalty box" was a big metal or wooden box that stood at the side of the pitch (near the manager's stand), and that players who'd committed a foul were locked in there until the end of the game. There was a little window so they could watch how their team were doing, but they had to sit in the box and reflect on their actions.