Actually, no. As long as your baby is healthy to begin with, going outside on a mild day can be a boon for you both (talk to your doctor first if your baby is a preemie, or has a weak immune system or other health problems). The nice weather and sunshine are naturally invigorating, and lots of moms swear that babies sleep better after getting some fresh air (many squeeze in naptime outside, too). Being outdoors is far better for your baby than being in a room with poor ventilation -- especially if someone in that room is sick; some European hospitals place newborns outside for short stints daily! So don't worry too much about taking your infant out for a stroll. And don't beat yourself up if he does get ill: You can't filter every microbe that floats by, and most healthy babies will have six to nine colds in their first year alone.
Tips for taking your baby out
Keep well-meaning admirers at bay: Pull down the stroller cover when they approach!
Don't overdress your baby -- put him in one more layer than you yourself need.
Avoid strong sunlight since soft baby skin burns easily. Or dress him in a brimmed hat and long cotton sleeves. Dab a bit of infant-safe sunblock on his face and hands (Coppertone's WaterBabies continuous spray sunscreen is a good one to try).