"Out of character" is a phrase used in entertainment to differeniate between a person playing a character and the character; when out of character, the person is being him-or-herself. The opposite, when the person is acting as the character, is "in character".
Occasionally, a film, TV show, or book will break the fourth wall by having one of the characters step out of character and comment on the story or the other characters.
Often, the phrase is used in fanfiction to describe a character in the story who is acting differently from how he appeared in the story's source material.