Well, I know of one reliever who bitched at me for 10 "years" or so because he wanted to start and I kept him in the bullpen because he had very marginal endurance; he could scarcely last five innings in a game. Finally had to let him go, and he went to the Mets for his final year. He started 32 games for them, went 7-13 with 5.20 ERA, presumably lasted at least the required 5 innings for 7 wins, and retired happy that at least he finally got a chance to start.
All of which has nothing to do with your question, of course.
If you change a pitcher to reliever when his expectation is starting rotation (see the guy above, for example) then yes, it will annoy him eventually. Not right away, but soon.
The question that this begs is, what triggers his annoyance? Is it the position change or being taken out of the starting rotation on the pitching chart? I'll post later on this.