I thought the phrase "throw the towel in" had to do with throwing one's swimming towel into the pool/ocean/lake/river.
A childhood belief about speaking

others believed things like this
- I used to believe that the waves in the ocean were made from whales swimming by...— Tina
- I used to think that a "pool shark" was an actual shark that lived in a swimming poolAnon
- I believed that the ladder on the back of a trailer/rv led up to a swimming pool.— marilese
- I used to believe that if you farted while swimming in the pool, it could propel you foward really fast.— Rick
- Up until I was in 3rd grade I believed my bathing cap is what kept me afloat in the swimming pool! Anon
- When I was a kid I used to think the "carpool lane" was an actual pool cars went swimming in— skeez