Despite fielding a fairly solid team, playing in Nickerson doomed the team in Boston. The stadium had been built in 1955 (though parts of it dated to 1915), and had not aged well. It was so small that the Breakers lost money even when they sold out as visiting teams got a portion of the gate proceeds. The Breakers and Washington Federals were the only teams to draw less than 14,000 per game in 1983. The other 10 teams drew over 18,000 per game. (What fans did come to the games were generally passionate; the documentary Small Potatoes: Who Killed the USFL? made note of a particular Breakers victory in which fans stormed the field afterward.)

Concluding that Nickerson Field was not suitable even for temporary use, Matthews again approached Harvard, but the school refused again. He then hashed out a deal to move to Foxborough, but ultimately decided against being a tenant of an NFL team. After floating offers to move to Seattle, Honolulu and Portland, Matthews ultimately decided to move to New Orleans. He sold a 31 percent interest to real estate developer Joe Canizaro, and the move to New Orleans was approved by the USFL on October 18, 1983. Matthews later sold his remaining stake to Canizaro, but Vataha remained as team president.