In a population in which homozygotes (AA and aa) mate preferentially with individuals of the same genotype, which frequency will tend to become smaller with each generation of selective mating?

It will remain constant...

Explanation:

In accordance with the Mendel's law of segregation, independent assortment as well as Hardy-Weinberg Theorem., no matter what type of cross within the homozygotic characteristics takes place, the frequency of dominant and recessive will remain constant over the whole population even if selective breeding strategy is applied.

For example, the ratio for a single trait of allele will remain 3:1 for a whole population of large numbers

the denominator has a very specific role.