When the silk just starts to turn brown. I was told this by a field supervisor one year when I was about 14. There was crew of us out there to harvest their crop by hand. She told us this was more a rule of thumb than a law set in stone. But all these years later, I'm sure it still applies.