You will need a wind-blocking layer. The wind pullovers that are made for playing golf are inexpensive and effective. Wearing one of those inside your outer layer works well.