All packets that are NOT Control, Beacons or Management-Request (Association/Reassociation/Probe) are counted as possible packets that count toward the retry calculation denominator.
If a packet has a retry bit set, it is counted toward the calculation numerator.
The rate is simply numerator/denominator.
Of course the group of packets that go into the calculation is determined by the current selection. It is calculated as the treepie for that selection is created.