I think it depends on body shape. I am taller and weigh more than the numbers above, and the small fits me well. I actually got the xs at first when that was the only size in stock and even that was ok, except that the elastic in the waist was a bit too snug. When the other sizes became available I tried small and medium and ended up sticking with the small, as I found the medium a bit too baggy. Hope this helps!
I am 5'4" and 130 lbs. I ordered the medium and the lining felt tight even though the dress didn't look too tight. I sent it back and ordered the large and that is much more comfortable for me.