Swanson was born on June 10, 1965. He quickly developed an affinity for baseball and, by the age of 5, began participating in the Farm League in 1970, followed by Little League play in 1976-77. By the age of 11, he was already making headlines as an all-around player, but showed exceptional promise as a talented left-handed pitcher. ln 1978, Swanson helped lead his Vac-Air team to the division title in East Lake Babe Ruth action and, later on, was the starting pitcher for the 1983 16-18 Upstate New York tournament team that finished in second place.
During his high school sports career at Frewsburg, Swanson's name became synonymous with success and leadership while playing football and basketball for the Bears, in addition to building his reputation as one of the area's top baseball players, both in pitching and as a first baseman. He was an all-league selection and an all-star performer every year until his graduation with high honors in 1983.