Competition formats resistant to bad prior rating information

[oiuyt]

Originally Posted by Arbe

To get a handle on this, I looked at how A-rated fencers did in this year's senior sectional (North Atlantic):

While it's off Peter's topic, it might be interesting to look at another example of correlation between classification and results. The PdF Memorial Day Weekend round-robins, by their nature, don't protect high seeds the way most tournaments do. With a standard pools->DE format it could be reasonably argued that the top (seeded) fencers have an easier time posting top results because they end up with relatively easier pools. This is most noticible with it being easier for an overseeded fencer to approach living up to his/her faulty seed than for an equal-skill competitor without the inflated rating.

Okay, let's look at some results tables.

In MF the highest non-A/B fencer was 11th (of 28). All of the (12) A/B fencers finished in the top 15. The A's finished 1st, 2nd, 3rd, 5th, 8th, and 9th. The D and under fencers finished 14th, 21st, 23rd, 25th, 26th, 27th, and 28th, and were in correctly sorted order by entering classification. Except for the outlier in 14th they clustered at the bottom with only a couple of C's mixed with the top D's.

In MS the highest non-A/B is 8th (of 28). All of the (12) A/B fencers finished in the top 14. The A's finished 1st, 2nd, and 9th (9th place was injured and not fencing close to his typical level). Except for a few outliers (C's placing 25th, 26th, a U placing 16th) the results are close to following sorted order.

Epee, of course, is where we would expect greatest variation from seeding. In ME, the highest non-A/B fencer was 9th (of 37). The A's placed 1st, 2nd, 5th, 7th, 8th, and 26th (which is obviously an EXTREME outlier from expectations). The rest of the top-8 were all B's (3rd, 3rd, 6th), but then the B's spread out a bit more (12th, 15th, 16th, 20th), intermixed with C's (and 1 D, 14th). All of the E/U fencers finished 28th or lower and only 1 D (32nd) finished below that point. So while there's a fair degree of mixing in the middle, primarily with B/C fencers (plus the one D with a strong result and the one A with a very weak result), otherwise we have another fairly well-sorted results list.

This is the format which is most resistant to poor initial seeding (there is no reliance on that seeding whatsoever). These results are one of the best examples that I've seen to date on how good our current system actually is. Does New England division (and all of the various visitors from out of area, including myself) have over/under-classified individuals? Of course. But it sure seems like it's at most a very minor issue.

I haven't bothered to go through the women's results as well. I would not be at all surprised if they were not as decisively clear as the men's results, if for no other reason than just how clean the men's results were. Even if the women's results showed similar properties I'd expect them to be somewhat noisier.

Okay, back to the topic at hand. Feel free to ignore my brief interlude if you don't like drift or sidebar discussions. :)

-B

[DJ Apostrophe]

Originally Posted by oiuyt

...clip...

In MS the highest non-A/B is 8th (of 28). All of the (12) A/B fencers finished in the top 14. The A's finished 1st, 2nd, and 9th (9th place was injured and not fencing close to his typical level). Except for a few outliers (C's placing 25th, 26th, a U placing 16th) the results are close to following sorted order.

Epee, of course, is where we would expect greatest variation from seeding. In ME, the highest non-A/B fencer was 9th (of 37). The A's placed 1st, 2nd, 5th, 7th, 8th, and 26th (which is obviously an EXTREME outlier from expectations). The rest of the top-8 were all B's (3rd, 3rd, 6th), but then the B's spread out a bit more (12th, 15th, 16th, 20th), intermixed with C's (and 1 D, 14th). All of the E/U fencers finished 28th or lower and only 1 D (32nd) finished below that point. So while there's a fair degree of mixing in the middle, primarily with B/C fencers (plus the one D with a strong result and the one A with a very weak result), otherwise we have another fairly well-sorted results list.

-B

Just as a point of reference... the 20th place B in ME (myself) is also injured... damned plantar fasciitis...

I was okay in the beginning of the day (i.e. it didn't hurt too badly) until about round 5 or so, I fought through it for rounds five and six, iced it during my bye round (round 7), it felt better for a round or two, then was back with a vengance for the rest of the day. It sucks to not be able to push of your back foot... That combined with mono, has kept me out of a regular training schedule since Feb. sometime...


Also, Diuene (16th place U in MS) isn't your typical U fencer... he had been fencing for a few years in Brazil before moving to the US... If this had been more than his first tournament in the US, he probably would have had a rating going into the event... so that explains that outlier.

-w

[tallntender]

This year's NYAC Hangover Open had an idiosyncratic round robin format (five touch bouts diminishing to three touch and finally to one touch bouts as time ran out). It was a mixed competition.

The results are available on AskFRED, and they show a pretty consistent pattern too. The results correlated fairly well with classification.

Right again, oiuyt!

[keropie]

Originally Posted by tallntender

Right again, oiuyt!

You know, if we're not careful his mask won't fit anymore. . .

Edit: Added smiley

[oiuyt]

Originally Posted by DJ Apostrophe

Just as a point of reference... the 20th place B in ME (myself) is also injured... damned plantar fasciitis...

Mmmm, yes, forgot about that.

Originally Posted by DJ A

Also, Diuene (16th place U in MS) isn't your typical U fencer... he had been fencing for a few years in Brazil before moving to the US... If this had been more than his first tournament in the US, he probably would have had a rating going into the event... so that explains that outlier.

Heh, and I thought it was rediculously good even without having to go through and find reasons for each of the outliers.

-B

[mrbiggs]

Originally Posted by PeterGustafsson

Once you reach the DE stage, you can only be sure of your upcoming opponent - the next one is decided by a bout outside your control, and often not yet fenced.

Yeah, but a fencer knows that it's one of two fencers, and a good deal of the time the fencer can be pretty sure if which one it will be. It helps fencers keep track of what's happening in the tournament, promoting efficiency.

[tbryan]

Originally Posted by Arbe

If I were given the task of generating an accurate results list, then the bout I would be least interested in would be some A-rated fencer facing some U-rated fencer. I would be most interested in some A-rated fencer facing some other A-rated fencer. Yet our current system actively avoids such pool bouts between equally rated opponents.

That's because the final results are determined by the DE bracket. The pools are just there to seed the DEs. You don't seem to understand the current tournament format or its features/problems very well.

Originally Posted by Arbe

There are several possibly valid reasons for doing this. Maybe tournaments have other goals beyond producing accurate results lists. Maybe the ratings aren't accurate enough to be given much strength in determining DE seeding.


One alternative pool assignment system would just put the top 7 rated fencers in pool 1, the next 7 in pool 2, and so on.

Okay...I just have to respond to this part.

Imagine that you have 15 fencers, and we know their relative strengths. In the curent system, we create 3 pools of 5 fencers each. In each pool, we have one fencer who wins all of his bouts, one fencer who loses just one bout, ... down to one fencer who lost all four bouts.

So, the fencers who won all of their bouts takes spots 1, 2, and 3 in the DE table. The fencer who lost just one bout take spots 4, 5, and 6 in the DE table, etc.

Now, look at a DE Table. Fencers with seeds 1 and 2 don't fence each other until the finals (assuming that they win all of their other bouts). Fencers 1 and 2 don't fence fencers 3 and 4 until the round of 4 (assuming these four win all of their bouts).

To produce an accurate final result, you'd like your best two fencers (assuming that there are no upsets, etc.) to fence each other in the finals. If you put your five best fencers in one pool in the current system, they'll be seeded across the entire DE bracket. Your best two guys have a good chance of hitting each other in the round of 4. One of them may fence your third best guy in the round of 8. Of course, there will be a lot of upsets since one of our pools has the 5 lowest seeded fencers, and one of them took seed 2 or 3. The guy who beats the second seed takes his spot in the future bouts, but that might be the 5th strongest fencer at your event (the one who lost the pool in the pool of the best fencers) instead of the 2nd strongest fencer at your event.

So, the way it is currently done does encourage a bout between your two best guys. It just happens in the final. We spread the top-seeded fencers across the pools so that (in this example), the 3 highest seeds are in separate pools. They're each trying to beat the rest of the fencers in their pools by as many touches at possible to get a 4-0 record with +20 indicators and take the top seed. If they win all of their bouts in pools, these three guys won't fence each other until the final 4. That's how the current format is meant to work.

The problem that was being discussed is that if we have a bunch of fencers with equivalent ratings, they are seeded randomly. Really, there's probably a broad range of skill among six A06 fencers. If you seed them randomly, you might put your best two fencers in the same pool and your weakest two As in another pool. That means that your 5th or 6th weakest fencer has a good chance of snagging the 2nd seed.

Now, there are a few of the responses we get on this forum:

• The rating system is broken. Let's come up with a different rating system that doesn't end up making a bunch of fencers equivalent. (OROD's thread is the most recent one of these.)
• The tournament format is broken. Let's come up with a workable format that is less vulnerable to the problems with the way the initial seed is used to create the pools as the current format is. (You're reading this thread.)
• The rating system and the format are fine and workable systems. Just go win all of your DEs, and you win the event. (People who think that we've already reached a pretty good compromise format and rating system. They generally don't start threads, but they respond in the other two types of threads.)

Anyway, I'm probably more in the third camp, but I'm a bit less flippant. I think that the initial seeding and the seeding into DEs does matter quite a lot when the field of fencers has a broad range of skill. (In an event where the skill level of all participants is roughly even, I believe that it matters much less.) I just think that we've reached a fairly good compromise between what's ideal and what's practical in both format and seeding. I'm happy to see people discuss alternatives, but I'm encouraged by oiuyt's posts that suggest that the current USFA ratings actually are a pretty good seeding mechanism.

[Arbe]

Originally posted by oiuyt
While it's off Peter's topic, it might be interesting to look at another example of correlation between classification and results. The PdF Memorial Day Weekend round-robins, by their nature, don't protect high seeds the way most tournaments do.

I agree that classification is a good predictor of tournament results (at least for the tournaments I looked at). I looked at 4 tournaments, and there were only 5 extremely unusual results out of almost 200 fencers (for instance a U-rated fencer finishing 3rd). In the smaller round robin tournaments (about 30 fencers), the results differed from those predicted by classification by an average of 3. In the larger tournaments (about 60 fencers), the results differed from those predicted by classification by an average of 7 for MF and 11 for ME.



I looked at the how the round robin results compared to classification. I calculated the difference between the round robin result and the rating rank. For example, an A06 fencer finished 5th, and their rank based on rating alone was 1st, giving a delta of 4.

For MF, there was an average delta of 3.3. Dividing by the number of fencers (28) gives 12% delta.
For ME, there was an average delta of 3.0. Dividing by the number of fencers (35) gives 8% delta.

Note that for ME, 2 outliers (outside 3 sd) were dropped. There were no outliers in MF.

Not including the outliers, the largest deltas were:
MF: 21st rated C04 finished 12th, 23rd rated D05 finished 14th, 17th rated C06 finished 24th, 18th rated C05 finished 11th.
ME: 22nd rated C05 finished 11th, 23rd rated D06 finished 14th, 24th rated D06 finished 18th.

The correlation between the results list and the ratings list is 0.87 for MF and 0.93 for ME. (not including outliers)


I ran the same calculations on the senior NA sectionals.

For MF, there was an average delta of 6.5. Dividing by the number of fencers (61) gives 11% delta.
For ME, there was an average delta of 10.7. Dividing by the number of fencers (62) gives 17% delta.

Note that for MF, 1 outlier (outside 3 sd) were dropped. There were 2 outliers in ME.

Not including the outliers, the largest deltas were:
MF: 11th rated B06 finished 34th, 32nd rated C06 finished 55th, 38th rated D06 finished 20th.
ME: 5th rated A04 finished 42nd, 14th rated B06 finished 47th, 47th rated E06 finished 16th.

The correlation between the results list and the ratings list is 0.89 for MF and 0.73 for ME. (not including outliers)

[KD5MDK]

tbryan - I think you are too hard on Arbe's idea. Specifically, he's very aware you want the top seeds to end up seeded at the top after the first round, assuming things are accurate. For a 3 pool, 15 person tournament, his plan would have all the fencers from pool 1 be ranked 1-5 after pools, regardless of their pool performance, all the fencers from pool 2 would be ranked 6-10, and pool 3 would contain the 11-15 seeds. The pool just determines which fencer in their pool takes 1st and which takes 5th.
Now, what it doesn't provide for is for the 6th rated person to be a stronger fencer than the 5th, and so forth. However, I think it's an interesting idea for an experiment.

[tbryan]

Originally Posted by KD5MDK

tbryan - I think you are too hard on Arbe's idea. Specifically, he's very aware you want the top seeds to end up seeded at the top after the first round, assuming things are accurate. For a 3 pool, 15 person tournament, his plan would have all the fencers from pool 1 be ranked 1-5 after pools, regardless of their pool performance, all the fencers from pool 2 would be ranked 6-10, and pool 3 would contain the 11-15 seeds. The pool just determines which fencer in their pool takes 1st and which takes 5th.
Now, what it doesn't provide for is for the 6th rated person to be a stronger fencer than the 5th, and so forth. However, I think it's an interesting idea for an experiment.

Not at all--What was that? Oh, look, it's the entire point of Arbe's post flying over my head.

I was scanning this thread between doing other things. I clearly didn't read Arbe's post well.

I apparently read this part.

Originally Posted by Arbe

One alternative pool assignment system would just put the top 7 rated fencers in pool 1, the next 7 in pool 2, and so on.

And missed this part.

Originally Posted by Arbe

The top 7 rated fencers would get the top 7 DE seeds, with the order determined by the pool results.

So, I got it stuck in my head that Arbe was going to seed just like the current format but start with the best five fencers in one pool. Thus, the looong explanation in my post about how the current format works. I was thinking that Arbe just didn't "get it."

At least I was right in one respect. There's now one post in this thread where the author didn't "get it."

Sorry, Arbe. I'll read more carefully next time.

[Arbe]

Originally posted by Poulet
What's the incentive to try hard in pools under that system? Being seeded 1st or 6th doesn't really ever matter, except in tiny competitions.

Originally posted by tbryan
The problem that was being discussed is that if we have a bunch of fencers with equivalent ratings, they are seeded randomly. Really, there's probably a broad range of skill among six A06 fencers. If you seed them randomly, you might put your best two fencers in the same pool and your weakest two As in another pool. That means that your 5th or 6th weakest fencer has a good chance of snagging the 2nd seed.

Does being seeded 1st or 6th matter?

Let's see (if there are no upsets except for the 6th seed winning all bouts):
the 1st seed is going to face the 64th seed, 32nd seed, 16th seed, 8th seed, 4th seed, and 6th seed.
the 6th seed is going to face the 59th seed, 27th seed, 11th seed, 3rd seed, 2nd seed, and 1st seed.

Not much difference until the round of 16. A large difference in the round of 8. If the seeding is accurate, then the 1st seed will probably coast into the final 4. The 6th seed will probably face a moderate challege in the round of 16, and a severe challenge in the round of 8.

I'd try hard for the 1st seed.

[mrbiggs]

Originally Posted by KD5MDK

tbryan - I think you are too hard on Arbe's idea. Specifically, he's very aware you want the top seeds to end up seeded at the top after the first round, assuming things are accurate. For a 3 pool, 15 person tournament, his plan would have all the fencers from pool 1 be ranked 1-5 after pools, regardless of their pool performance, all the fencers from pool 2 would be ranked 6-10, and pool 3 would contain the 11-15 seeds. The pool just determines which fencer in their pool takes 1st and which takes 5th.
Now, what it doesn't provide for is for the 6th rated person to be a stronger fencer than the 5th, and so forth. However, I think it's an interesting idea for an experiment.

I think it makes earning new ratings too difficult. We all know that ratings aren't all that accurate, especially in adjacent letters. Under the current system, for example, a largish tournament will usually have a B with an A. If the B beats the A, he gets a high seeding and an easy DE path. If he doesn't, he'll get a medium seeding and path. And, of course, if he loses to a C or D, he'll be in trouble. This proposal eliminates that. The B can only be ranked in a certain range whether he blows his pool bouts or wins them. The proposal pretty much makes pools kind of moot, as a matter of fact.

[KD5MDK]

I'm not saying it's a preferred competitive format. I'm just saying it would make for an interesting experiment. One thing the current system does is prevent there from being many bouts among the very top fencers. For example, in a perfect tournament the top eight fencers will only fence other top eight fencers at most 3 times, and that's only the best two. 4 will only fence 1 top 8 opponent, 2 will fence 2, and 2 will fence 3. It might be useful to see what those top 8 fencers would look like if most of them met each other in pools.

[tbryan]

Originally Posted by KD5MDK

I'm not saying it's a preferred competitive format. I'm just saying it would make for an interesting experiment. One thing the current system does is prevent there from being many bouts among the very top fencers.

I remember Rudy Volkman (I think) chatting with some fencers at a Div I NAC and saying how the NACs really showed that our better fencers needed to get more practice fencing each other in 5 touch bouts.

At many events with a wide range of skill (and ratings) among the compeitors, the better fencers only see each other in late rounds of DEs. Fencing a bunch of (relatively) hard fencers in 5 touch bouts during pools is something that not everyone gets to practice that often. I would love to try this format (now that I understand it).

[PeterGustafsson]

Hi!

Originally Posted by fencerX

Before complaining about classifications affecting seeding which could possibly affect results, you really need to go back and figure out what do you want the results to do for you in the first place.

There are many features that one might want from a competition. Many of the not easily, or at all, compatible. A partial list, in no particular order:
1. Go fast - low number of total bouts
2. Give some minimum # of bouts to every fencer
3. Build momentum - the bouts get progressively more important, and the final ones decide the top places
4. Never reward a loss with a better final result (surprisingly, a competition format once used in Greco-Roman wrestling World Championships failed this criterion!)
5. Never punish a win with a worse final result
6. Produce a final results list where all ties are resolved
7. Be simple for the BC
8. Have a high % of close bouts
9. Keep the number of times a fencer has to move form one piste to another low
10. Make it easy for a fencer to predict which, and when, his next bout will be
11. Reduce the amount of necessary data flow piste <-> BC
12. Produce a balanced loads of bouts on each piste, either with or without other concurrent events
13. Make it easy for fencers to fence multiple events
14. Make the whole event go fast, measured in elapsed time.
15. Be good at dealing with fencers whose relative albilities are unknown.
16. Fill in your wished feature here!

Originally Posted by fencerX

Is the purpose to identify (solely) the best fencer that day's event (to hand them the prize, etc)? The best X top fencers? Is that the best X fencers in that day's event or over time and multiple events? The best Y% fencers to qualify them for something or other? Do you require or want all of these top X or Y fencers in order? Do you really wish or expect or is it necessary to be able to place every competitor in the "true" position of their "relative performance" in the event?

Different event (and qualification and seeding) approaches can help you do different things, but first you need to figure out and agree on the point of the entire exercise.

If one is considering a qualification event, as the one prompting the "epee seeding close to breakdown" thread, one could say that for a qualifier the really necessary features are #4,5, and 7. #6, while nice to have, is not a necessity for a qualifier, since rating ties can be resolved with results from other events or by lots. A qualifier must separate between the qualified and the non-qualified, everything else is just extras.

If one then goes to design a competition format which fulfills #4,5, and 7, and optimizes according to #1, one could get this:

1. List fencers according to previous rating.
2. Set up an ordinary DE table.
3. Run the DE until a winner is produced. He gets slot#1 of the qualification list.
4. Let the one who lost to the winner in Round#1 fence against the one who lost in round #2.
5. The winner in step#4 against the loser to the ultimate winner in the next higher round.
6. Repeat step#5 until the one who lost against the winner in the final has fenced. The winner of that bout gets slot#2 of the qualification list.
7. Repeat steps #4-6, with the losers of the latest qualifier, until all qualification slots have been filled

This is quite similar to the format used in Judo, BTW.

This format also fulfills criteria #9-11. DE´s are easy to manage and understand, and all bouts after the first winner has been found can be, actually must be, fenced on a single piste. This clears up the other pistes for other events. OTOH, many fencers will only fence one single bout. For those who have not qualified, it will not produce a well-sorted results list. Furthermore, a single out-of-whack result can affect the total qualification list quite a lot.

So there you are. A format intended to be specifically a good qualification format, producing a list of qualified in quite few bouts.


Have a nice time!

Peter Gustafsson

[Arbe]

The Judo format is interesting. The total amount of fencing time needed is about the same as the current format, but the results should be better. (The Judo format would have to be modified somewhat to allow multiple pistes to be used, otherwise the actual tournament time will be too long.)

I ran some numbers with a 64 fencer tournament, 16 qualifiers. Assuming 5 minutes for a 5 touch bout, and 15 minutes for a 15 touch bout. Total fencing time for the tournament then is:

Current system: Pools 870 minutes, DE 945 minutes, Total 1815 minutes.
Judo system: DE 945 minutes, next qualifier 75 minutes, Total 2060 minutes.

Of course, the actual tournament time is strongly affected by the number of pistes being used, and the data flow from the BC. I would say your feature #11 (Reduce the amount of necessary data flow piste<->BC) and #12 (balanced piste load) are very important in keeping the total tournament time at a reasonable level.

[D+F+P=Hadouken!]

based on my observations of the Sr mens Epee points list, the higher up on the points list you are, the higher you tend to place. Of course, its not absolute, but its alot more reliable then simple letter ratings.

Perhaps an automated local points system would be in order. A points system that uses some sort of formula to calculate points.

Then you come to the issue of regional differences in strength. For this, you could have a strength factor for each region to multiply the points total by.

So for example:

NYC-2.0
Socal-1.8
Pacific Northwest-1.5
Southwest-1.4
Southeast-1.2
Great lakes- 0.8

This would make points transferable from region to region, and hopefully, provide a more accurate seeding.