How does Eye P.A. calculate Retry Rate?

All packets that are NOT Control, Beacons or Management-Request (Association/Reassociation/Probe) are counted as possible packets that count toward the retry calculation denominator.

If a packet has a retry bit set, it is counted toward the calculation numerator.

The rate is simply numerator/denominator.

Of course the group of packets that go into the calculation is determined by the current selection. It is calculated as the treepie for that selection is created.