How to determine if a resale or HDB BTO is a better choice in terms of ROI?

Location is one of the major determinants of value. Not location in and of itself, but location with its attendant perks - accessibility, facilities, amenities, etc.

Right time, right place is also very important. A location may not be considered "good" today, but in 10 years' time, thanks to whatever developments, that can change.

Worth noting also that a new flat will generally appreciate after projection completion while the trend for resale is more of a decaying value tied to a decaying lease. Generally speaking.