"It's a difficult subject to talk about because everyone feels horrible about how it went and how it ended, but there's also great glimmers of fond remembrance," the actor tells EW.