The story begins with Holmes and Watson out for a walk on an early spring day. When they return, they find that a client has come and gone in their absence, leaving behind only his pipe. Holmes makes several rapid deductions about the man based upon the pipe, all of which prove to be correct. The man, one Grant Munro, returns in a high state of agitation, requesting Holmes’ professional aid and advice. Munro breaks into a rather lengthy explanation of his problem, which concern the mysterious actions of his wife, Effie. Munro explains that his wife was a widow, whose husband and child had died in America from the yellow fever. After being married happily for three years, Munro’s wife had come asking him for money. He gave it to her, but never learned for what it was used.
A short time later, an empty house near the Munro’s was tenanted, and passing by the place one day, Grant had seen a hideous yellow face in the window. Not long after that, Munro had been awakened by Effie leaving the house in the middle of the night, only to return in a high state of excitement and worry. The next day, passing by the mysterious house again, Grant had witnessed his wife hurrying away from the place. The man had attempted to go in, but was restrained by Effie. She had refused to explain what was going on, but had asked Grant to trust her, which the man had relented to do. Some days afterwards however, having learned that his wife had frequented the house again, Munro had broken into the house and searched it, finding nothing of consequence except a large photo of Effie.
After Grant’s narrative, Holmes questions him, particularly about the face in the window. Munro describes it as “hideous” and “unnatural.” Holmes promises to look into the matter, and Grant leaves. Holmes explains to Watson that he believes Effie’s first husband is still alive, and is somehow blackmailing her. The two journey that night to Norbury, the town where the Munro’s live.
Accompanied by Grant, they break into the mysterious house, despite the objections of Effie and a strange woman who lives in the house. Charging upstairs, the three find a child with a hideous yellow face. Reaching forward, Holmes removes what proves to be a mask from the child’s face, revealing a little Negro girl. Effie explains that her husband in America was in fact a Negro, and that she hid her child away, making her wear the yellow mask to keep the neighbors from gossiping. Effie claims that she acted as she did for fear of how her husband would respond. After a long pause, the man scoops up the little girl, Lucy, stating: “I am not a very good man, Effie, but I think I am a better one than you have given me credit for being.”
Holmes and Watson quietly depart, and Holmes, realizing he was wrong in his deductions, advises Watson that if he (Sherlock) should ever become too overconfident in his own ability to simply whisper “Norbury” in his ear.
