They start soon after flowers and blossoms reach their peak, and after juveniles have grown to full body size. This varies slightly from year to year depending on the weather. As Bill says, the males don't wait for the juveniles to grow so they leave first. The females stick with the juveniles, and the juveniles need as much time as they can get to grow fully because migration uses up too much energy to allocate any calories to growth. By the end of September, there's hardly anything left in bloom. This sets up a balancing act between gaining as much growth as possible before migrating, while facing risks of starvation from a dwindling food supply, or sacrificing growth (and chances of surviving migration) to migrate sooner to gain a more predictable food supply.