Known as "Big Augie", Carlson was born April 7, 1902, in Sweden, then moved to the Jamestown area in 1903 and became a lifelong Chautauqua County resident.
Shortly after coming to the United States, Carlson suffered a tragedy when his father passed away. Thus he spent his early years in the Randolph Orphanage. His mother couldn't speak or write English when she came over, but later was able to get the family with seven children back together.
Carlson played baseball and football at Celoron Park with the Northsiders in his late teens, but his real love was boxing.