A color is randomly assigned when you add a subgroup to a group chat, making it a Super Group. To change a Super Group's color, open the Super Group from the Groups tab by swiping left or by tapping the # located in the top right of any subgroup chat. Tap the Settings icon at the top right to open Super Group Info. Tap Color to select a new color.
Custom color is unique to Super Groups. To turn your group chat into a Super Group, add a subgroup from the # menu at the top right of your group chat.