still something wrong with the way we advance in rating

[xavier]

Let me guess, epee?

yeah...
Is it specific to epee?

[peet]

But wow, if only someone had access to say, a good starting collection of data about fencers and their competitive results, then they could use that to provide the system you say would be nice.

I'm just sayin'...

Funny you might say that...

-p

[noodle]

Funny you might say that...

-p

dare you to update the world on how thats going

[dramamine]

I have always felt like the biggest problem with US ratings is how easily they are earned, probably because they only require one result to obtain. It's also worth noting that it is significantly harder to earn a letter at say, a NAC, than a local tournament. My first ever tournament in the states was a local event in Portland that I went to when I was around 15, at which I earned a B. There were some good fencers there for sure, maybe 12 strong A's or so, but then a whole bunch of people with ratings that blew my mind. For some reason I also get the feeling that one of the people I fenced in DEs was given an A because he was from France, not because he earned it in a tournament, but I could be totally wrong. Furthermore, the reason that the competition became whatever it was (I think an A4, the highest possible rating) was because someone's mom agreed to fence (for the first time ever) in order to make the event have 64 people. I probably would have ended up with a C or something instead if she hadn't fenced.

I mean A is supposed to be pretty good, right? If the person isn't on the points list, then that's the best they can be (I don't really know how the US rating system works so correct me if I'm wrong). I feel like you should have to perform with a certain level of consistency in order to earn an A, so that fluke inflations don't happen. Or in my case, so you don't get an inflated rating because someone's mom was talked into fencing for the first time in her life.

[peet]

dare you to update the world on how thats going

That would fall squarely in the intersection of these two areas:

1. Not ready for prime time.
2. Not my responsibility to present to the public.

(sorry...)

-p

[K O'N]

That would fall squarely in the intersection of these two areas:

1. Not ready for prime time.
2. Not my responsibility to present to the public.

(sorry...)

-p

Can you tell those of us not in on the original conversation what you're planning to do, when you get the time? Not trying to crowd you, just curious what the project is.

K O'N

[peet]

Can you tell those of us not in on the original conversation what you're planning to do, when you get the time? Not trying to crowd you, just curious what the project is.

K O'N

Since it's been mentioned in several Board of Directors agendas, I guess it's not letting too many cats out of bags to say that the USFA has a committee looking at alternative fencer rating systems. I am assisting with technical aspects, using FRED's sizable database of results and my own development time to simulate different rating systems so that they may be compared against each other and evaluated against certain criteria. I am not designing the rating systems myself, nor am I in charge of anything at all. I'm just helping out.

-P

[wahrman]

No, you seem to understand it just fine.

If the US a system more like Canada or the UK then Galen would have earned a well deserved, large dose of points but not triggered the high water mark A.

I'm not quite sure what the opposition is to such a system, maybe it's just inertia.

I have always felt like the biggest problem with US ratings is how easily they are earned, probably because they only require one result to obtain. It's also worth noting that it is significantly harder to earn a letter at say, a NAC, than a local tournament. My first ever tournament in the states was a local event in Portland that I went to when I was around 15, at which I earned a B. There were some good fencers there for sure, maybe 12 strong A's or so, but then a whole bunch of people with ratings that blew my mind. For some reason I also get the feeling that one of the people I fenced in DEs was given an A because he was from France, not because he earned it in a tournament, but I could be totally wrong. Furthermore, the reason that the competition became whatever it was (I think an A4, the highest possible rating) was because someone's mom agreed to fence (for the first time ever) in order to make the event have 64 people. I probably would have ended up with a C or something instead if she hadn't fenced.

I mean A is supposed to be pretty good, right? If the person isn't on the points list, then that's the best they can be (I don't really know how the US rating system works so correct me if I'm wrong). I feel like you should have to perform with a certain level of consistency in order to earn an A, so that fluke inflations don't happen. Or in my case, so you don't get an inflated rating because someone's mom was talked into fencing for the first time in her life.

[K O'N]

Since it's been mentioned in several Board of Directors agendas, I guess it's not letting too many cats out of bags to say that the USFA has a committee looking at alternative fencer rating systems. I am assisting with technical aspects, using FRED's sizable database of results and my own development time to simulate different rating systems so that they may be compared against each other and evaluated against certain criteria. I am not designing the rating systems myself, nor am I in charge of anything at all. I'm just helping out.

-P

Huh. Ok. Thanks for the information.

Do you have available how predictive ratings are right now? I mean, clearly we don't think they could ever be very (high r value) predictive, since people improve between competitions and the measurements are too coarse, but how good or bad are they, I wonder?

K O'N

[DangerMouse]

One of the other issues that has not really been discussed here is how to design a system where ratings make sense in other events. For example, while there are many very strong women epeeists in my region, they consistently finish below the strong male epeeists, although they are all A rated fencers. Similar problems exist for transfering ratings between veterans events and opens. A points system makes this issue even harder as the question as to which points count and how much they count for arises. Note this also skews the r value measument K O'N is asking about.

[peet]

Do you have available how predictive ratings are right now? I mean, clearly we don't think they could ever be very (high r value) predictive, since people improve between competitions and the measurements are too coarse, but how good or bad are they, I wonder?

Sorry, I haven't gotten to that part yet, but it'll come. Clearly a rating system's ability to predict bout outcome is a very important measure of its effectiveness.

One of the other issues that has not really been discussed here is how to design a system where ratings make sense in other events. For example, while there are many very strong women epeeists in my region, they consistently finish below the strong male epeeists, although they are all A rated fencers. Similar problems exist for transfering ratings between veterans events and opens. A points system makes this issue even harder as the question as to which points count and how much they count for arises. Note this also skews the r value measument K O'N is asking about.

A numeric system only makes this harder if you have more than one set of ratings. It's important to note that while a points ranking system like the current NRPS must have one points list for each gender/age, a numeric rating system need not have more than one "list" since it's not a ranking.

This is the difference between a "ranking" and a "rating". A ranking is an ordered list of competitors (I'm 1st in the country, you're 2nd, he's 3rd, etc...) and so cannot have mixed genders/ages. A rating is not an ordered list, it's just a designation of competitive strength. So anyone of X strength can be compared against anyone else of X strength in a given weapon regardless of age or gender.

As you noted, how well a given system's designations (be they numeric or otherwise) equate across ages, genders, geographic areas, etc is one criterion of its effectiveness.

-p

[piste off]

One of the other issues that has not really been discussed here is how to design a system where ratings make sense in other events. For example, while there are many very strong women epeeists in my region, they consistently finish below the strong male epeeists, although they are all A rated fencers. Similar problems exist for transfering ratings between veterans events and opens. A points system makes this issue even harder as the question as to which points count and how much they count for arises. Note this also skews the r value measument K O'N is asking about.

You would also have to differentiate for regional biases as well.

Earning a classification (in particular the higher ones) is a lot more difficult in an area like Metro.

R-

[K O'N]

No, you seem to understand it just fine.

If the US a system more like Canada or the UK then Galen would have earned a well deserved, large dose of points but not triggered the high water mark A.

I'm not quite sure what the opposition is to such a system, maybe it's just inertia.

Inertia, sure, and some of the ideas people throw around are so dreadful one worries that we might end up with something far worse. For example, if you could damage your rating by fencing you might be tempted to 'preserve' it by staying home. See, that's idiotic.

The current system is supposed to be for seeding events. It does a reasonably good job at that. For example, I picked the most recent >= A1 event in my section off askfred, which turned out to be this:

http://askfred.net/Results/results.p...ament_id=10877

Assign A = 1, B = 2, ect, to U = 6. Looking at mixed epee, the Pearson r value between ratings and finishing place is .75. That's pretty good when we consider that fencers are expected to get better between events, so the ratings are expected to be wrong in predictions of placement when those people improve.

A common complaint I hear about ratings is that they're "too easy to get". It's a reasonable thing to say, if you're a good fencer and you look around and see people with ratings whom you think suck. Or if you get a rating and you think it should have been harder to do so.

The thing to think about is, if you're a good fencer, any rating system expected to seed people will have to give some ratings to people you think are crappy. I'm sorry about that, but "everyone in this room is crappy, here, have a pin from the World Cup" is not a very good seeding system. In the above event, there were two As, four Bs, three Cs. That's not top heavy, that indicates to me that As are pretty hard to get. Perhaps in other areas As are easier to get? But even there we have national points to look at. Is this really broken?

An A shouldn't mean to you that you're a Black Belt in fencing, it should mean you're in the top 15% or 20% or so in your weapon, and should be seeded there, if we try to split rated fencers into fifths. If "pretty good" has emotional overtones you don't like, then a B shouldn't mean you're pretty good, it should mean you're in the second 20% and should be seeded there. I think in actuality less than 1/5 of fencers are As, so by that measure As are too hard to get, not too easy.

On the other hand, we might be more concerned with rating the higher level fencers for everyone's sake, since there are probably two or three people in every event whom you would be kind of unhappy to see clumped together in your pool, so it's probably good to segregate them away from the rest; thus, maybe an A should mean you're in the top 5%. Whatever. In any case, if you got a B in epee and think you don't deserve it, ask yourself, if I go to another epee event, will I end up in the middle of the Bs? Yes? Then you deserve a B.

A numerical system like chess would give finer gradings of fencers. But I think every chess system, certainly Elo ratings, punish you if you do badly in a competition. Even in chess where a small difference in skill is a large predictor of victory there are problems in ratings hoarding and avoiding competition if you get a high rating. In fencing I predict this would be magnified, since small differences in skill in fencing are not good predictors of victory ("in epee, anyone can beat anyone", for example); let's suppose I have a good day. I get a high rating, higher, in my mind, than I really rate. It is four months to SN. I decide to train and save my rating rather than compete, to help my seeding at SN. I know that if I compete I will have some bad days, and on those days I will lose my artificially high rating. See, this is a disaster. The current "high water system" encourages fencing, a chess style system would discourage it.

Elo ratings are very good for chess, since a higher rated player can be counted on to beat a lower rated player the vast majority of the time. There's no incentive to stay home if you think you can win all the bouts you should win. And an Elo system is very good for other games like rating football teams, since those teams have to play. But in fencing we have a large variability in performance and we don't have to play on a schedule. Be careful of adopting a system that makes me want to stay home, I'm lazy enough as it is.

K O'N

[parrythis]

a 'C' by the name of Galen Jackson (whom I don't know) placed fifth and got no change in rating in spite of defeating a 'B', a 'C', and an 'A' in the pools, and a 'B' and an 'A' in the DE's, thereby placing ahead of nine 'A's and eight 'B's in the final results.

The current rating system is not an indication of how good a fencer is. It is an indication of how well you did at a particular tournament. Therefore, the rating carries with it some elements of the fencer's skills on a particular day AND some elements of what his competition was on that same day.

If Mr. Jackson went to this tournament with the intention of improving his rating, he went to the wrong tournament. It was probably top-heavy. If a fencer wants to improve their rating, most of the effort lies in improving their skill, but some portion of the effort lies in selecting the right tournament to do so.

[Allen Evans]

I am assisting with technical aspects, using FRED's sizable database of results and my own development time to simulate different rating systems so that they may be compared against each other and evaluated against certain criteria. I am not designing the rating systems myself, nor am I in charge of anything at all. I'm just helping out.

For some reason, I find this totally hilarious.

The only sad part about this is that you're probably helping for free.

*slap*

[peet]

The only sad part about this is that you're probably helping for free.

Not entirely.

-p

[Morale Officer]

Not entirely.

-p

Beer doesn't count. lol!

[VorpalCat]

Beer doesn't count. lol!

But it's better than nothing!

[fdad]

yeah...
Is it specific to epee?

In some weapons (e.g. Women's Sabre) it is hard to get a higher rating, relative to other weapons (e.g. Men's Epee). Some would claim the problem at the event you are complaining about was not that you didn't get a rating for beating those higher rated fencers, but that they should not have such high ratings.

[NGV]

Some would claim the problem at the event you are complaining about was not that you didn't get a rating for beating those higher rated fencers, but that they should not have such high ratings.

That might be true for some tournaments - but at the specific tournament in question, a bunch of very strong fencers finished outside of the top 12 in men's epee.

http://askfred.net/Results/results.p...nament_id=9072

Again, the issue in this particular case is that the A2 category incorporates tournaments of vastly different strength levels, because it's pretty easy to come up with 25 fencers for a tournament, and quite hard to come up with 64 (and there's nothing in between).