Brown played basketball and baseball for four years at South Dayton and was a starting guard on the 1941-42 sectional championship team that had an undefeated record. After a three-year stint in the Army, Brown attended Cortland State where he played baseball and soccer for three years and was the right halfback on the 1948 All-America Soccer Team. During that time, Brown also played baseball with the Steel Partition Bombers who won the New York State Championship and went on to compete in the national toumament at Wichita, Kansas.
In 1949, he signed a contract with the Brooklyn Dodgers and played second base with their farm team in Nashua, New Hampshire. The manager was former Jamestown Falcons manager Greg Mulleavy and some of the Brown's teammates were Billy Hunter, Billy Loes and Gino Cimoli. In one game Brown set a record for handling 17 chances, nine putouts and eight assists, without making an error.