“I am thinking about this product but am concerned on sizing. I need this for a 6', 180 lb man, he's a size 32 pants, usually a large shirt, but the medium size robe is 38-40, which seems big. If he's average size will a medium be too big?”

It is based on chest size, not pants size. A small is typically for a chest size of 34- to 36-inches. A medium is for chest size 38 to 40. In a robe, a medium should be fine for him.