It was during his freshman year that he played both fall and spring baseball as a catcher. The summer of 1931 saw Brown playing in various leagues in Chautauqua and Cattaraugus Counties.
In the fall of 1931, Brown returned to Michigan and he played varsity for manager Ray Fisher, who was an outstanding major leaguer for the New York Yankees and the Cincinnati Reds. However, a knee injury concluded Brown's collegiate career.
Brown returned to South Dayton and began officiating both baseball and basketball games for several years. Ruth Morley became his wife in 1934 and their son Richard Brown, the former Southwestern boys and girls basketball coach, was born in 1935. Times were tough in those days with Brown still able to play baseball on weekends after working 12-18 hour days.