PointControl: a new fencing rating system

Discussion in 'Fencing Discussion' started by David Ma, Sep 15, 2015.

  1. David Ma

    David Ma Rookie

    Joined:
    Sep 5, 2015
    Messages:
    7
    Likes Received:
    5
    PointFirst, Gav and erooMynohtnA like this.
  2. jkormann

    jkormann Podium

    Joined:
    Nov 8, 2002
    Messages:
    2,947
    Likes Received:
    191
    Nifty. Thanks for doing the work!
     
  3. David Ma

    David Ma Rookie

    Joined:
    Sep 5, 2015
    Messages:
    7
    Likes Received:
    5
    Thanks for using it! We've served 550 queries already :)
     
  4. jdude97

    jdude97 Podium

    Joined:
    Apr 10, 2013
    Messages:
    1,835
    Likes Received:
    206
    Awesome! I loved reading the technical article too, and understood most of it thanks to my AP Stats class. One note on using historical AskFred data to assign initial ratings is that AskFred includes lots of tournaments that aren't official USFA tournaments, including non-approved formats, high school tournaments, club private tournaments, and just tournaments that weren't sanctioned for whatever reason, plus it's missing tournaments for which the organizers never submitted final results plus the national tournaments themselves.

    I'd say to fix that problem if you want to make your system a legitimate proposal to the USFA would be to assign initial ratings based on their current USFA letter classification, since anything you get on AskFred is unofficial and missing certain data. For the first year or two then, the system would effectively be the same, since you'd just be a 38 instead of an A or a 22 instead of a D (just guesstimating numbers there - I'm sure you can calculate pretty easily what would be good and fair starting values). But as people fence more and more bouts, the system forgets about those initial anchors of letter classifications and becomes it's own thing as intended.

    One thing I didn't see you touch on was ease of use in both introducing the system and using it on a day to day basis. People have often been concerned that the USFA and tournament organizers would have a hard time working with a national rankings list because they already struggle with just managing 100 person div1 NRPS lists. What would be your response to those who rightfully have such a concern?

    Also, one element of the current classifications system is it's based on how well you do in tournaments, not on whom you beat. Since fencing is results oriented after all, it makes sense that someone who consistently reaches the semifinals should be higher than someone who always crashes out in the quarters. Perhaps this system accounts for that, or at least that information is successfully relayed in the data, but I might propose that certain bouts get some sort of extra strength factor. But what I wouldn't want to see is someone losing a semifinal going down in ranking unduly to make the winner go up extra. There'd have to be a way to balance that out. Basically such a system would both reward doing well in each individual bout in the abstract as well as a tournament as a whole, because winning your first round match isn't the same as winning the final, due to a number of factors including the pressure situation and fatigue from having fenced a whole tournament. All those factors must be included.

    With all that said, really great job on this. Thanks for putting in your own time and effort and money (did anyone pay you for this?) to help better the fencing community as a whole.
     
  5. David Ma

    David Ma Rookie

    Joined:
    Sep 5, 2015
    Messages:
    7
    Likes Received:
    5
    Hey jdude97, thanks for your thoughtful response!

    > One note on using historical AskFred data to assign initial ratings

    The system assigns the same rating (25) to everybody that is "new".

    > AskFred includes lots of tournaments that aren't official USFA tournaments, including non-approved formats, high school tournaments, club private tournaments, and just tournaments that weren't sanctioned for whatever reason, plus it's missing tournaments for which the organizers never submitted final results plus the national tournaments themselves.

    Unfortunately that's all the data I can find and have access to. In terms of non-official tournaments, I don't currently see a way of programatically distinguishing between official and non-official. However, if people are as serious in those tournaments as in official ones, I would say let's count them too for more accuracy!

    > a legitimate proposal to the USFA would be to assign initial ratings based on their current USFA letter classification

    Yes that's a great idea to bridge over from the old to the new system! Though, I think it's a long shot that the USFA would implement this. :(

    > One thing I didn't see you touch on was ease of use in both introducing the system and using it on a day to day basis. People have often been concerned that the USFA and tournament organizers would have a hard time working with a national rankings list because they already struggle with just managing 100 person div1 NRPS lists. What would be your response to those who rightfully have such a concern?

    I don't think PointControl can replace the national rankings list because as I understand it, it's actually a "countrywide competition". The difference between competitions and rankings is that competitions are fun one-time events where a winner gets crowned, whereas ranking system try their best to determine who is the most skilled at that point in time.
    About PointControl being easy to use and maintain, I think the UX/UI right now is pretty intuitive, and it was built and being maintained by my brother (mostly tech) and I (mostly stats).

    > [On tournament rankings]

    The PointControl ranking is actually computed using every single bout and is agnostic to where you finish in a tournament.

    > Thanks for putting in your own time and effort and money

    You're welcome :) I'm just happy to see others use sometime we've built. Nobody funded us, it's still cheap enough to maintain by myself currently.
     
  6. K O'N

    K O'N Podium

    Joined:
    Aug 14, 2006
    Messages:
    3,752
    Likes Received:
    559
    I hate to be the grump here, especially since this is all so nice and math-y, and I like math a lot. My degrees are in math. I'm a math guy, math is excellent. But I have a couple of questions about this particular application:

    Have you considered the effect on fencer behavior? In particular, in our current system there's no reason to ever throw a bout. There's no reason to ever lose on purpose. In this system, how will you calculate eligibility for Div II events? If it's below some cutoff, there will be an overwhelming temptation to go lose in some local event and get under 22, or whatever the cutoff is. That's a bad effect.

    If you calculate Div II by some kind of high water mark for the last two years or something, as some chess tournaments do with high-Elo ratings to let people in to tournaments with Elo cutoff scores (similar to Div II events), you then in effect have a two-tiered system. My rating is 23 with a max of 25.5 18 months ago, so I seed as a 23 but I can't enter Div II events for another six months; what about a Div II qualifier? If I'll be Div II eligible in six months at SN, can I qualify for Div II? It really does open a decent can of worms, which is why I think it's not universal in chess.

    Another thing to think about in any you-can-lose-points-by-fencing system is the effect it has on high level fencers and local events. So, for example, I live near a good club with a lot of elite kids. I fenced in a local event recently. Two of the kids who fenced in the event from this club were 1st and 2nd in a Div 1 NAC last year, these are genuinely elite fencers. They came out to fence at this local thing... why? I don't know, just to fence. Why not? They had fun, it's not a problem, right? Some of my students got to fence them, I had one of them in my pool and even got a hand touch on him. Yay! He's a nice kid, it was all great. And some non-elite kids got to fence some top juniors, excellent.

    But in a system where an elite kid will have a rating of way, way more than anyone else in the event, would they come out and fence and risk their rating at a local thing? What do they have to gain? I mean, anyone can lose, right? In fact one of them went 3-2 in the pools and lost early in the DE tree. That can happen. So in our current high water mark system, who cares? But in a system where he can lose rating points and then be seeded lower in his next NAC, he'd be crazy to come out to a local thing and risk losing in the pools to a vet fencer with a far lower rating than his. That would be insane, and elite fencers would learn quickly not to do it, and hey! Thanks. We have marginally better seeding in this local thing, which honestly who gives a crap, and we just lost all the elite kids who are staying home at their club and fencing each other rather than coming out a few times a year and fencing the great unwashed masses. That's excellent, that's really good. That just made fencing a lot worse, with no measurable benefit except that now we're using more math.

    For any policy change, you should examine the logical behavior changes under this new set of rules. What would a logical fencer do under the set of rules you're proposing? Is that behavior desirable and good for fencing?
     
  7. DangerMouse

    DangerMouse Podium

    Joined:
    Mar 13, 2003
    Messages:
    3,202
    Likes Received:
    198
    Nice work on this! This is the best presentation of an alternative to the current ranking system that I've seen.

    To preface my comments, as most on this board know, I don't consider the current system to be broken. Yes, it could be more granular. And yes, it could use a little tweaking. But, I see very few cases where it is really out of whack with reality.

    That said, I have a couple observations on your PointControl system:
    1. My PointControl score rose significantly after I switched to focusing on coaching instead of competing. During that time I only went to a few tournaments and they were just for fun. When I was competition focused, I would seek out the most difficult tournaments. What this indicates to me is that the PointControl score rewards fencing easier events and fewer events. Or, to turn that around, it disincentivizes the best training methods of seeking challenges instead of chasing results. The current system has no risk for attending tournaments purely for training, but PointControl and other similar (Elo, etc.) based systems have penalties for fencers that are not seeking to win any given tournament. This is bad for fencer development in general.

    2. You measure the predictive nature of PointControl against final results from tournaments, but you also say that PointControl is agnostic to where you finish in a tournament. My stats knowledge is a bit weak, but this seems like a disconnect to me. Shouldn't it either be based on final results or use head to head predictions as a quality measure?

    Edit: Per usual, K O'N and I are in agreement and he both beat me to the punch with his post and said it more eloquently.
     
  8. K O'N

    K O'N Podium

    Joined:
    Aug 14, 2006
    Messages:
    3,752
    Likes Received:
    559
    Mine had the same effect. I confess I hadn't looked at the actual numbers before I wrote my response above; if you look at Ari Simmons' track, he just gained points (!!) for finishing 20th out of 43 in a little local A2! This is a kid who won a Div I NAC not too long ago. How on earth can that be right?

    My objections above were aimed at a working system with Elo-like behavior. Under such a system Ari would have lost points this weekend. I think there's something wrong with the implementation here, if someone on his level actually gained points for finishing so low in a little local event.
     
  9. DangerMouse

    DangerMouse Podium

    Joined:
    Mar 13, 2003
    Messages:
    3,202
    Likes Received:
    198
    Since the NAC data is not included in PointControl, I actually don't think that is necessarily an implementation issue, but probably more of a data issue. My general objections still stand though.
     
  10. bbower

    bbower DE Bracket

    Joined:
    Feb 20, 2003
    Messages:
    508
    Likes Received:
    128
    ELO type scoring systems have been discussed for fencing for over 20 years. I'll be more blunt than the other posters in this thread. Any point system that encourages fencers to avoid participating in tournaments for fear of decreasing their rating will NEVER be adopted by the USFA.
     
  11. Privateer

    Privateer Podium

    Joined:
    Feb 8, 2009
    Messages:
    1,080
    Likes Received:
    135
    I ask because I don't know, but usually my same question, how do other countries handle fencer ranking/rating/pool seeding? What drawbacks have they encountered?
     
  12. fdad

    fdad Podium

    Joined:
    Jul 9, 2008
    Messages:
    4,129
    Likes Received:
    214
    I agree. However ... even though nobody should be choosing events based on the potential for a ratings increase, many people do. Just like the problem with ELO type systems with people avoiding competitions because they might end up with a lower rating, the current system results in people avoiding competitions that wont potentially increase their rating. Perhaps something like the current system, with more granularity, an unbounded top end, and a faster decay would be worth considering.
     
  13. K O'N

    K O'N Podium

    Joined:
    Aug 14, 2006
    Messages:
    3,752
    Likes Received:
    559
    No other country has the number and diversity of fencers the US does. Systems that work for smaller, more homogeneous populations may not scale well into the very large, very heterogeneous US fencing world.
     
  14. K O'N

    K O'N Podium

    Joined:
    Aug 14, 2006
    Messages:
    3,752
    Likes Received:
    559
    I guess. Either his rating reflects his ability or it does not; if it does not, the problem is clear. If it does, then he should have lost points this weekend. You can't have it both ways.
     
  15. Privateer

    Privateer Podium

    Joined:
    Feb 8, 2009
    Messages:
    1,080
    Likes Received:
    135
    “You are not special. You're not a beautiful and unique snowflake. You're the same decaying rating as everything else. We're all part of the same ranking heap."
     
  16. Mitchell

    Mitchell is a Verified Fencing ExpertMitchell hi Staff Member

    Joined:
    Feb 24, 2003
    Messages:
    3,279
    Likes Received:
    485
    That isn't the point he was making, I think. Many people do choose events based no the potential for a ratings increase. But if you introduce an ELO system, many people will purposefully avoid events in order to *avoid losing rating*. There's a difference between strategically picking which events to attend with the intent of boosting your rating, and strategically avoiding events you would've likely attended otherwise because there's potential to lose rating.
     
  17. jkormann

    jkormann Podium

    Joined:
    Nov 8, 2002
    Messages:
    2,947
    Likes Received:
    191
    Or attending an event that starts as a higher rating, then plummets when key people don't make it into the finals. The pools were just as hard, but didn't finish hard due to luck or fortune. With a 'get a point / give a point' system the fencer accumulates more points for winning, and that's the 'point'.
     
  18. David Ma

    David Ma Rookie

    Joined:
    Sep 5, 2015
    Messages:
    7
    Likes Received:
    5
    Hey everybody, thanks for all the feedback. I wasn't able to reply to any posts during the day. I'll reply to general ideas rather than any person in particular.

    > Replacing the letter based system would be bad because it disincentivizes highly rated fencers from actual fencing.

    Replacing the letter based ranking is not my goal. I might have struck some chords by starting my article with "The current letter-based fencing rating system sucks.", but it sure did get people talking :)
    I recognize the downsides of replacing the old system. There are a ton of operational stuff that rely on the current system (e.g. nationals qualifications, seeding software, mindsets) and changing that would require a huge amount of work for marginal benefits.
    I also agree that a rating that can go down might disincentivize highly rated fencers to fence tournaments, but only if qualification to nationals is based on ratings.

    In fact, one of my motivations to create PointControl was actually to track my own improvement while I lurk in the U area. In fact, over 60% of active epeeists (has fenced in a tournament in the last year) are Unrated. I hope to bring some feedback mechanism for those fencers.

    > You measure the predictive nature of PointControl against final results from tournaments, but you also say that PointControl is agnostic to where you finish in a tournament. My stats knowledge is a bit weak, but this seems like a disconnect to me. Shouldn't it either be based on final results or use head to head predictions as a quality measure?

    The data that a model uses to predict doesn't have to correspond to the actual thing being predicted. For example, if model A used the height of fencers to predict tournament results, we can still compare to model B that uses letter ratings (model B would probably do better).

    Nonetheless, I initially wanted to make head to head predictions and compare it to the letter rating model. It was much more computationally intensive, and the letter model would have given a lot of ties, so I settled for tournament predictions.

    > Ari Simmons

    The data in PointControl hasn't updated to include his/her last result. The last tournament I have on file for this person is 2015-08-08. I'll work on the updates...

    > PointControl is giving funky ratings!

    That's very possible. With every rating system, there will be cases where ratings don't seem to match reality. You can definitely think of cases with the letter system.
    However, if I can think of a way to systematically find those cases and have a systematic fix, I would be happy to work on it. Currently, I'm not smart enough to figure it out.
     
  19. Eledex

    Eledex Made the Cut

    Joined:
    Dec 2, 2009
    Messages:
    134
    Likes Received:
    13
    I think it is fun. Most useful (imo) for comparing A ranks, they have a wider range of skill than other rankings.
     
  20. jdude97

    jdude97 Podium

    Joined:
    Apr 10, 2013
    Messages:
    1,835
    Likes Received:
    206
    There are two main issues with the current system. One is that a fencer has a single good day and then is overly rated for the next many years. Two is that there's not enough granularity within ratings.

    I see the latter as the bigger problem and also the one best fixed by a numerical system. The biggest issues with lack of granularity are at the top and bottom of the spectrum.

    60% of tournament fencers according to David are U's. These fencers range from people in their first tournament to vets fencing for thirty years to a plethora of youth fencers themselves ranging from a nationally top ranked Y10er to a Y12 national medalist to a Y14er fencing for 4 years to of course absolute beginners of all ages.

    Then on the other end of the spectrum, A's range from olympic champions to div1 NAC medalists to people who insistently earn points at div1 NACs to people who just barely make the cut at div1 NACs to people who are good local fencers to people who got their rating on a flukey day and now go 3-3 in the pools of an open.

    Just as 60% of fencers are U's, at some tournaments 80% of fencers are A's. At certain four seasons cup events at alliance in houston, your average men's epee pool will be as hard if not harder than a pool at a div1 NAC because over half the entrants are consistently in contention to medal at a junior or div1 NAC.

    On that same basis, while we use national points to seed div1 NACs, thus this saturation of A's not being detrimental, at big local events with lots of A's or even if not especially at smaller events with almost all A's, not being able to seed within classification and year leads to vastly uneven pools. Such is what occurs when an Olympian is considered equal to a person who just makes the cut at div1 NACs to person who beat a single A and two B's to get their A at a mixed 15-person A1 in the eyes of the initial seeding.
     

Share This Page