Swell builds up over time given a constant wind direction, this is why the sea state often lags the wind speed.
If the wind suddenly swings 180 degrees, it will have the effect of starting to kill the swell, if the wind swings across it at an angle, you will end up with a confused swell.
A backing wind usually indicates the approach of a Low Pressure which is what brings the bad weather (ass opposed to a High which gives us the good.) If you stand with your back to the wind, it means the Low pressure is on your Left hand side.
http://www.geocities.com/freighterman.geo/weather_at_sea.html