More than 50 House members, representing more than 10 percent of Congress, have said they'll skip this week's festivities