What does hamlet mean when he says, “let me comply with you in this garb, lest my extent to the players (which i tell you must show fairly outwards) should more appear like entertainment than yours”? why is this statement ironic?

Hamlet means that he must act like he is happy to see Rosencrantz and Guildenstern, otherwise people might think he is not happy to see them.

Of course, this line is ironic -- Hamlet is NOT happy to see them. he knows Rosencrantz and Guildenstern are there because Claudius and Gertrude want his friends to spy on him. 

The line is also ironic because the idea of "acting" is a big motif in the play. Hamlet does not want to appear as though he's happier to see the actors, which is why he must shake hands with his friends. He must appear (or act) as happy to see them as he is to see the real actors. 

Ironically, Hamlet, Rosencrantz, and Guildenstern are all acting here -- Hamlet is acting that he's happy to see his friends (or that he isn't losing his mind, or that he isn't plotting revenge) and R&G are acting as though they aren't supposed to spy on their friend.

Abetter tresfgyts