First of all; the only silly questions are the ones not asked!
As for what determines fry size; there are at least five:
1. As you correctly surmised, without the proper genes your goldfish will never be large no matter how well you care and feed them.
2. How much good goldfish care your goldfish receive in the first 120 days of life is a very important factor that will determine their growth rate and size.
3. Proper nutrition the rest of their lives is a must!! The goldfish food and goldfish care you give your goldfish is vital to their growth rate.
4. Water Quality: Excellent water quality which is a part of good goldfish care is a must for raising nice big goldfish. That means 0 ammonia, 0 nitrites, a good solid pH and very low nitrates.
5. Over crowding: Fish give off certain hormones that inhibit growth (self preservation). When forced to live in over crowded conditions they will not grow (at the very best, slowly). Also, in over-crowded conditions, fighting off numerous diseases is a constant problem.
There are many others I am sure, but these are the most important.