A new referee ratings process

[catwood1]

I'm sure many people have seen a referee who has a surprisingly good rating, and wondered why they have it, or who gave it to them. (Maybe people think that about me )

Me and a friend of mine had an idea of a way to improve the system, and I wanted to get other people's opinions on it. This would result in many fewer "soft" 4s, 5s, and 6s.

We leave the top and bottom of the scale the same. There aren't many "soft" 1s, 2s, or 3s, because a vote is necessary to give them. Also, 10s, 9s, 8s, and 7s are sometimes given out unpredictably, but when they are, there really aren't many bad consequences. Those are all just stepping stones to stronger ratings.

However, the difference between a 7 and a 4 is huge, and there are some FOCs who give out very hard ratings, and some that give out soft ratings. And the consequences of this are pretty big. Things like who refs the top bouts at big local events, and who gets hired for NACs are all directly based on these ratings. I have seen a 7 from one FOC who is a better ref than a 4 by another FOC.

Our idea is to make a hybrid of the two ratings systems necessary for the mid range ratings. Make it so that to get a 4, 5, or 6, you need to have TWO FOCs, or instructors give you that rating, instead of just one. This will mean that no longer can one soft FOC give out strait 4s across the board to a new ref. However, any solid ref won't have a problem getting these better ratings. Now, to get these ratings, it would take either repeated exposure over multiple events to different FOCs, or solid performances at NACs or sectionals. This would probably increase the focus on sectionals. NACs would still be a great place to get ratings, because there are always at least 2 head FOCs. This means if they agreed, they could give out ratings.

This would also put a "lip" at the 7 rating. It would show than any ref who was higher rated than a 7 was a solid ref, who had been seen by multiple FOCs. Currently, seeing any rating on this side of a 4 still can have some dispute over whether the ref is actually competent.

Let me know what you guys think!

[KD5MDK]

What do you do with everyone who already has a rating? Move us all down to 7s?

It's hard enough to find high level bouts to get an increase in many areas. Making it so you had to perform at that level in two separate times while being observed (because when will two FOCs have the opportunity to observe you at the same time) is far less likely, which just decreases the number of people who get ratings they deserve.

[peet]

The proposal makes it harder to award a "soft" rating, but also makes it harder to earn a rating at all.

How about a way to make the ratings more accurate, but no more scarce?

Mind you I have no idea what that might be...

-p

[keith]

It's hard enough to find high level bouts to get an increase in many areas. Making it so you had to perform at that level in two separate times while being observed (because when will two FOCs have the opportunity to observe you at the same time) is far less likely, which just decreases the number of people who get ratings they deserve.

Surely the whole point is to shift the system so that it takes longer for a ref to get a higher rating?

After all any ref who deserves a higher rating will end up with one - it may take longer but a referee who deserves a higher rating will get one.

[peet]

Oh, and:

This assumes that ref rating inflation is a problem serious enough to warrant changes in the system, and I'm not sure it is. It may very well be, but I don't know it to be.

For instance, I think that hiring and assigning are often done as much on the head ref's personal knowledge/feelings about the ref in question than their rating. More so in high level events/bouts.

Just a thought.
-p

[formerfencer]

I think it's a good start and something definitely needs to change.

Now, to get these ratings, it would take either repeated exposure over multiple events to different FOCs, or solid performances at NACs or sectionals. This would probably increase the focus on sectionals. NACs would still be a great place to get ratings, because there are always at least 2 head FOCs. This means if they agreed, they could give out ratings.

I think that one of the problems we currently have is that all you need is one good weekend to get your rating. Someone could have a great sectionals, be seen by 2 members of the FOC, have everyone agree they are a 4 and proptly melt down at SN and the subsequent 3 NACs. The person will still be a 4.

I'd like to see ratings of 5 or higher only given by consensus of 2 members over the course of 3 tournaments. Probably harsher than many would like, but I think it would more accurately reflect skill level.

[keith]

This assumes that ref rating inflation is a problem serious enough to warrant changes in the system, and I'm not sure it is. It may very well be, but I don't know it to be.

How about we just pretend it's a serious problem until the election is over?

[bigdawg2121]

I think it's a good start and something definitely needs to change.



I think that one of the problems we currently have is that all you need is one good weekend to get your rating. Someone could have a great sectionals, be seen by 2 members of the FOC, have everyone agree they are a 4 and proptly melt down at SN and the subsequent 3 NACs. The person will still be a 4.

I'd like to see ratings of 5 or higher only given by consensus of 2 members over the course of 3 tournaments. Probably harsher than many would like, but I think it would more accurately reflect skill level.

Gawsh, in consideration of how I came about all of my ratings your system would have sucked for me personally. However, not everyone gets to be as lucky as I've managed to be in the past . In all seriousness though, three things:

1) You don't have to be an FOC to give a 7,6,5, or even a 4. I'm not sure it would help things to make it a stringent requirement that the observer be an FOC.

2) Doing it your way would probably make it excessively hard for people, especially outside of some specific regions, to ever get ratings. How many legit 5 or better tournaments are there in a given season? About 5 NACs, possibly sectionals, Summer Nationals (sorta). If you require two actual FOCs to concur on those ratings you can probably get rid of sectionals and most large regional events save 2 or 3 (e.g. Pomme de Terre). That's not really a lot of tournaments per year. How many 7/6 rated refs are going to pay their own ways to get to the number of NACs necessary to get upgrades?

3) TBA

[mrbiggs]

this would make it even more difficult for aspiring referees who live far from examiners to get ratings.

[keith]

this would make it even more difficult for aspiring referees who live far from examiners to get ratings.

Only a problem if living far from examiners does not correlate with living far from high quality bouts.

[bigdawg2121]

Only a problem if living far from examiners does not correlate with living far from high quality bouts.

There's not always a correlation. One could easily referee at a saber tournament where I am and have an event with 5-7 As another 4Bs, and enough Cs, Ds, etc to make a very strong event with nationally competitive athletes and people that have made finals in pts events etc. We have one examiner that's even kind of in the area and that assumes he doesn't have to referee and chooses not to fence.

[keith]

There's not always a correlation.

..but when a theory and reality fail to coincide the only possible explanation is that your perception of reality is flawed. You just need better drugs.

[Peach]

..but when a theory and reality fail to coincide the only possible explanation is that your perception of reality is flawed. You just need better drugs.

This explains a number of your posts

hee hee hee hee

[formerfencer]

Gawsh, in consideration of how I came about all of my ratings your system would have sucked for me personally.

Me too, as you know

1) You don't have to be an FOC to give a 7,6,5, or even a 4. I'm not sure it would help things to make it a stringent requirement that the observer be an FOC.

I was responding to the OP. I agree with Catwood on this method. He and I just disagree on the 6s.

2) Doing it your way would probably make it excessively hard for people, especially outside of some specific regions, to ever get ratings. How many legit 5 or better tournaments are there in a given season? About 5 NACs, possibly sectionals, Summer Nationals (sorta). If you require two actual FOCs to concur on those ratings you can probably get rid of sectionals and most large regional events save 2 or 3 (e.g. Pomme de Terre). That's not really a lot of tournaments per year. How many 7/6 rated refs are going to pay their own ways to get to the number of NACs necessary to get upgrades?

Okay,
A#1: It should be hard. By definition, a 5 is capable of handling a DivI pool. Do you truly believe that all the refs rated 5 and higher in the USFA are capable at that level? When you go to a NAC as a fencer, how many of those refs do you want to work your pool? You and I have that perspective. There aren't 3s that make you cringe as they walk toward you pool with a clipboard? How about that little dance of joy you do when they pass your strip? And the look of apology you give your teammate who is in that pool?

B#2: Along with changing the ways to get ratings, you change the perception of the rating in the cadre. A 5 will become much more valuable to a ref, although not as attainable. Maybe since a 5 means you can handle a divI pool, you should see one before you get a 5. Not that you can't, just that you need to prove you can.

3) TBA

Even I don't talk enough to refute a non-existent argument (...ok, at least not on fnet)

[catwood1]

Correct, you don't have to be an FOC to give out a 4, 5, or 6. You can also be an instructor, and I'm not sure what ratings examiners can give out, I think it is through a 7. I just said FOC because it was easier to type, when I mean everyone who is qualified to give those ratings.

For everyone that already has a rating, it would be unfair move them down. We would have to leave them where they are. However, to renew the year on their rating, they would need 2 instructor/FOC's recommendation.

Ratings are also given out for bouts that are not of the "proper" level. I have seen a 5 given out in an event that was a girls qualifier that only had one rated fencer in the event. Also, I received one of my ratings for my work over the course of a weekend, not one specific bout. Ratings are not always given out directly based on 1 high level bout.

I absolutely believe its a problem worth fixing. Refereeing is one of the primary ways I interact with the sport, so I find it wrong when a better ref has a lower ratings than a worse ref, simply because he has had the "wrong" observer. Its also unfair when the wrong refs is given the gold medal bout in an event based on bad ratings. Ultimately, it is doing a disservice to the fencers, by having better qualified refs standing by, while worse refs continue to do the top bouts. While at the top levels, bouts are assigned on personal knowledge, at the lower levels ratings can be very important. Also, ratings are a way for hirers to do not know the ref in question to gauge
their ability.

Yes this would make it harder to get ratings, however, it would also add legitimacy to the 4/5/6 ratings. Currently, if you tell someone you are a 6, they don't know what to think. Because depending on the observer, a 6 can mean anything between "fantastic ref" and "not totally blind." With my system, it will add a lot of legitimacy to the 4/5/6 ranks.

By increasing the legitimacy of these ratings, it would likely mean that 6s could almost always get hired for NACs. Really, it would greatly increase the importance of sectionals. Now refs may have to drive a distance to get better ratings. I think that is a reasonable price to pay for a much better and more accurate system.

Overall, I think the good refs will get their good ratings regardless. Whether it takes them 1 year, or 2 years, they will get it. However, this will keep the bad refs from getting ratings like 5s, or 6s.

[formerfencer]

How about a way to make the ratings more accurate, but no more scarce?

They won't be more scarce, they'll just be 7s, 8s, 9s and 10s. What's wrong with that?

[catwood1]

I was responding to the OP. I agree with Catwood on this method. He and I just disagree on the 6s.

The thing about 6s, currently, is that they sometimes are given out when a referee simply "isn't blind." There are MANY refs who I have seen that are decent refs. If I have them reffing me in my little local div 3 event I won't be complaining. Thats about what I would expect. But if I got them in a really strong event I wouldn't be happy.

Let me give this scenario:

40 person event. 1A, 3Bs, 10Cs, and alot of Ds, Es, and Us. That makes it a B2 event I believe. By definition, a 6 should be able to referee the finals. While some 6s would be perfectly fine in this bout, I know MANY who absolutely by no means should be reffing this bout.

[tchwojko]

Okay,
A#1: It should be hard. By definition, a 5 is capable of handling a DivI pool. Do you truly believe that all the refs rated 5 and higher in the USFA are capable at that level? When you go to a NAC as a fencer, how many of those refs do you want to work your pool? You and I have that perspective. There aren't 3s that make you cringe as they walk toward you pool with a clipboard? How about that little dance of joy you do when they pass your strip? And the look of apology you give your teammate who is in that pool?

If magically the ratings for all the refs were exactly what you thought they should be, would a different set of refs be at the NACs? Would you get a shortage of refs 5 or better available?

(I'm asking because I don't know. I have no idea what the distribution of ref quality is, or supply and demand of said refs.)

[formerfencer]

Let me give this scenario:

40 person event. 1A, 3Bs, 10Cs, and alot of Ds, Es, and Us. That makes it a B2 event I believe. By definition, a 6 should be able to referee the finals. How many 6s do you know that should be refereeing the finals of that event? I can't think of any that I know...

After looking at current 6s in foil, 7 in the country. And I understand your point. I just think if you're going to make a national-level delineation for FOC/examiners to have a consensus, make it where the definition becomes DivI.

Or, maybe the definitions should be changed...

[IHateMrPotatohead]

We have one examiner that's even kind of in the area and that assumes he doesn't have to referee and chooses not to fence.

Actually, you have another examiner within driving distance who doesn't coach and rarely fences.