Size Medium will have room for layering. That's the size I got and am glad I did. I've worn them over jeans, or jeans and longies, or just longies, or sweats, or sweats and longies. They aren't a skinny tight fit, as you can tell, but still look nice with that little bit of extra room....not boxy. I'm a women's size 8 / 10 and in men's have a 32-34" waist and 32-33" inseam, and am 5'10" tall. The waist might be a tiny bit roomy, but there are wide belt loops or you could take it in. I find the cut of the pants stay up without a belt and allow movement at the same time. Sweater could be tucked in if necessary and the extra bit at the waist makes layering comfy. The waist is flat fronted but has 2 1/2" wide elastic from each side and across the back. The elastic in the casing is stitched in rows so it won't roll. The outer overall length might be a tiny bit long but you could turn under the hem. There is an inner "gater" with a grip elastic so that can be put anywhere under / over a boot.
Hope this helps you!