We investigated the role of the prefrontal areas in recognizing hand actions performed in social contexts. We used video clips depicting the interaction between two individuals: one person (requester) requested the other person to pass one of two objects. The other person (responder) then passed the object that was congruent to the request in the congruent condition or incongruent in the incongruent condition. Both requester and responder appeared on screen in the two-person condition and only the responder appeared in the one-person condition. Participants were required to observe the clips and to judge whether the responder’s actions were congruent or incongruent. Functional MRI showed the incongruency effect in the prefrontal cortex. In particular, the brain areas including the bilateral dorsolateral prefrontal cortex showed the interaction effect; the magnitude of activation in the incongruent condition was significantly higher than that in the congruent condition, but this difference was found only in the two-person condition. These results suggest that the dorsolateral prefrontal cortex play an important role in the detection of incongruency between the requester’s request and the responder’s action depending on social contexts.