In a flemish twist string, some twisting is necessary to keep the string from unraveling. Even with a continuous loop string (served ends), it's usually necessary to do some twisting to adjust brace height. If you have one to three twists per inch of string length, you're OK. If you have lots more than that, that's too much twisting, and you'll need to get a shorter string. If you take the string off your bow and it starts to curl up like an overly twisted rubber band, it's twisted too much.
If the bow is marked AMO 54 inches, then yes, you order a string three inches shorter. Not all bows are marked according to AMO standards, however. The rule of thumb for non-AMO marked bows is 4 inches shorter than the bow length. That works most of the time, but not always.
Measure your brace height with the string the way you have it. Then take the string off and measure its length. With those two numbers, a good string maker should be able to make a string the correct length for the brace height you want to achieve. Good luck.