How can I minimise FX losses if I were to transfer USD out of my broker and back to SGD?

I use Transferwise for such things. You can easily create a USD account in Transferwise.

Withdraw the money from your broker into this USD account. Once you have the money in your USD account then just send it to your SGD account via Transferwise.

Obviously, there are some fees but it’s definitely much cheaper than withdrawing the money directly out.

Some brokers maybe using Transferwise in their back end to do such transfers also. If that’s the case with your broker, then fees will be very similar.