Let's use Josh Wilson as an example because he's the first one that comes to my mind. When he played he was the position player on his teams that would be called on to pitch after they run out of hurlers. There's scores of players like this if you look around on baseball reference close enough.
He was a shortstop and pitched 3 innings in the MLB over 3 different appearances. He gave up 3 runs (and a home run) in one of those appearances for an overall ERA of 9.00. So the other two times he made it through an inning without giving up a run thus giving his team another chance to win it. He didn't just go out and serve up batting practice.