Sex is determined by the 23rd set of chromosomes. If the set contains two X chromosomes, then the infant is female. If it contains an X and a Y, then the infant is male. The mother's egg cell is always an X. Therefore, the father's sperm determines the sex of the infant since the sperm can carry either an X or a Y chromosome.
Color blindness is an X-linked recessive gene. This means that it is always passed on from a mother on the X chromosome. Since boys have one X and one Y chromosome, if they inherit one recessive gene on the X chromosome, they have no dominant chromosome to overpower it on the Y chromosome. This makes them much more apt to be color-blind. Girls have two X chromosomes. This means that they will usually have a dominant gene on the other X chromosome. Thus, they may carry the trait but will not experience it themselves.