For a family of 5, which would be more cost-effective- a family insurance plan or individual insurance?

Depends on the coverage, and whether that's enough for your needs. Focus on that first before looking at price.

'Group' policies are given by employers. You don't own the policy and are subjected to the terms and conditions of the group policy.

For personal insurance needs always make sure you have individual private policies in your own name and then any group policy can be used as extra cover.

You have more control over individual benefits and coverage.