Olsen attended Utah State University where he became a member of the Sigma Chi fraternity and was a three-year letterman in football as a defensive tackle. He graduated from the College of Business and Social Sciences at USU with a bachelor’s degree in Finance in 1962 and a master’s degree in Economics in 1971. He later received an honorary doctorate degree in business from the Huntsman School.

In football, as a senior, he was a consensus All-American selection (making the vast majority of All-America teams) and was the winner of the Outland Trophy. After Olsen’s junior year of 1960, he was also named All-American by the Football Writers Association of America and Newspaper Enterprise Association. He was also All-Conference in both 1960 and 1961. Olsen and Utah State were in the 1960 Sun Bowl, losing to New Mexico State, 20–13. Led by Olsen, the Aggie defense held the New Mexico State Aggies to just 44 rushing yards on 32 carries.

The Aggie defense Olsen anchored as a senior gave up an average of 50.8 rushing yards (which led the nation), 88.6 passing yards, and 139.4 total yards which all still stand as school records for defense. The 1961 Aggie defense gave up an average of 7.8 points a game, which is second in team history behind Olsen’s 1960 team, which allowed 6.5 points per game. Additionally, the Aggie defense held four opponents to less than 100 total yards. One, the University of Idaho, was held to a school-record of 23 total yards, with the Aggies winning 69–0.

The Aggies, not known as a national power football program, finished 10th in both the AP and UPI post-season polls, the only time that has occurred in school history. The Aggies had a combined 18–3–1 record during Olsen’s junior and senior seasons under coach John Ralston and were conference champions those two seasons as well.