As a sophomore at Syracuse University (1954), Brown was the second-leading rusher on the team. As a junior, he rushed for 666 yards (5.2 per carry). In his senior year in 1956, Brown was a consensus first-team All-American. He finished fifth in the Heisman Trophy voting and set school records for highest season rush average (6.2) and most rushing touchdowns in a single game (6). He ran for 986 yards—third-most in the country despite Syracuse playing only eight games—and scored 14 touchdowns. In the regular-season finale, a 61–7 rout of Colgate, he rushed for 197 yards, scored six touchdowns, and kicked seven extra points for a school-record 43 points. Then in the Cotton Bowl, he rushed for 132 yards, scored three touchdowns, and kicked three extra points, but a blocked extra point after Syracuse’s third touchdown was the difference as TCU won 28–27.

Perhaps more impressive was his success as a multisport athlete. In addition to his football accomplishments, he excelled in basketball, track, and especially lacrosse. As a sophomore, he was the second-leading scorer for the basketball team (15 ppg) and earned a letter on the track team. In 1955, he finished in fifth place in the Nation Championship decathlon. In his junior year, he averaged 11.3 points in basketball and was named a second-team All-American in lacrosse. His senior year, he was named a first-team All-American in lacrosse (43 goals in 10 games to rank second in scoring nationally). He is in the Lacrosse Hall of Fame. The Carrier Dome has an 800 square-foot tapestry depicting Brown in football and lacrosse uniforms with the words “Greatest Player Ever.”