A: The only way any person would "let it go" is if he were forced to hold on to it for a while. In defense of your husband, I wouldn't have been able to grasp how chaotic life with a baby can be before I had a baby and it was my job to keep everything together. Now, to indict him for one moment, he should realize that you wouldn't keep a messy house to spite him. Ergo, there must be a reason he doesn't come home to a mint on his pillow.
Permit me to use a nonbaby example to explain. During construction of my friend's new home, her design consultants extolled the virtues of a well-lit home. She agreed and installed recessed ceiling lights, undermounted fluorescents, lamps, and chandeliers with the understanding that her husband would be the designated lightbulb changer. But he never stayed on top of his job, which so frustrated her that she took it over. Soon after, my friend had a new complaint: "Why do we have so many %*& lights!" So one way to illustrate your predicament to your husband would be to take a mini-vacation without him or your kids. It doesn't have to be for the weekend, but it does have to be for most of one or two days. Be clear that things need to get done to his standards. He cannot simply watch the kids; he needs to do what you do, including laundry, making meals, running errands, playtime with the kids, and, of course, cleaning.
Even if a role reversal isn't possible, you need to talk to your husband and tell him that his expectations are stressing you out. A resolution may take repeated discussions or mini-vacations, but someday you may actually be allowed to rest your case when your husband utters the words that all moms cherish: "Honey, I don't know how you do it."