"I grew up watching re-runs of 'Perry Mason' and a lot of old black and white stuff and I liked it, but I didn't love it and I didn't love it because I wasn't in it," the actor explained