Looking for Judges! Payload Contest

[Ravidge]

Hello. The contest build period is nearly completed, and the testing phase will begin shortly. Every major contest we have a panel of judges, made up of regular members from the community who agreed to help out.

This time is no different. Judges are needed!

What is expected of a judge: (in short)

To play every contest entry to a degree where you feel you can make a good review of it. The amount you decide to play on each map is irrelevant.
Review all the maps, this means typing out some text where you motivate your scoring and opinions of the map. You're free to write as much as you want, but all that is required is a paragraph or two where you motivate your scores.
Score all the maps in the 5 different categories. Each judge is asked to take extra care to judge all maps against a certain standard of their choosing (what standard is irrelevant, just as long as all maps are judged against the same one), and use all of the 1-10 range of scores to differentiate good from bad.


Why do we need a judge panel?
To make sure we have a steady base of complete votes with motivations/reasons behind them.

I'm not good enough to be on some judge panel...
We are not looking for professional reviewing, we just need dedicated people who want to help the community.
I would like to receive as many judge applications as possible, and then leave it to me/staff to decide whether you're fit for the job!

How do I apply?
Send me (Ravidge) a forum PM!
You only really need to say: "Hello, I would like to apply for a position on the contest judge panel" or similar, it's that easy.

[Sel]

Can we please do 1 to 100 scoring this time?

[henke37]

How about putting the maps up on the server for a few days in a loop and getting some real player data to assist the judges?

[yyler]

We do that too. Historically the judge's ratings carry more weight, though.

[Ravidge]

If you've sent in a application, good, I will be going through them in the next couple days and starting to assembly a proper judge panel.

If you haven't sent a application in, WHERE IS YOUR EXCUSE!??!? (it's not too late)


Anyway, just a heads up, so you people don't feel forgotten!

[ics]

I'd like to reserve a right to be a judge if my entry will be dropped off from contest due to pleminary vote (which is a sucky way to do it btw). I have already ran most of these contest maps (last batch goes next friday) on my community servers so i have fair knowledge on how these maps work in-game. Just that, don't drop my entry, pl_curve_b8 off for this. Being in contest is still my 1st wish.

[LeSwordfish]

(which is a sucky way to do it btw).

What else is your suggestion? Complete analysis of nearly thirty maps? Try and make everyone play thirty maps before rating?

[ics]

It's a sucky way because none of the contestants who are new here knew about pre-ranking off some of the maps in advance. There was no mention about such thing in the rules and if there are future contests, note about this should be added to contest rules so similiar mess cannot occur again.

Those people who submitted map put work, effort and much time to their entries and some of them, perhaps even mine won't get to the main contest and the mappers won't get the feedback they deserve because pleminary votes will rank them off. They never hear detailed explanations on what is wrong and how they can learn from their mistakes. That's the most saddest part in this.

[LeSwordfish]

Well, they're very welcome to do the whole gameday shizzle everybody else does. But if you post a map three days bfore the contest, soley in the submission thread, and not anywhere else, you can't expect either to have a good enough map to stand up to ones that have been online and under testing for months, or to get as much feedback as them.

[Fr0Z3n]

It's a sucky way because none of the contestants who are new here knew about pre-ranking off some of the maps in advance. There was no mention about such thing in the rules and if there are future contests, note about this should be added to contest rules so similiar mess cannot occur again.

Those people who submitted map put work, effort and much time to their entries and some of them, perhaps even mine won't get to the main contest and the mappers won't get the feedback they deserve because pleminary votes will rank them off. They never hear detailed explanations on what is wrong and how they can learn from their mistakes. That's the most saddest part in this.

In Ravidge's defense, in the Prelim voting therad, he does say that this is NOT the norm for contests, and that this is due to the over-estimated amount of maps that entered. No one (besides ravidge) knew that a Prelim vote was going to occur until post-submission.


And Swordfish is right, if you want some detailed feedback after the contest, enter it into a gameday.

[Tarry H Sruman]

If elected, I will personally give detailed feedback to all of those whose maps were not accepted, because that is what Ronald Reagan would do.

SRUMAN 2012

[yyler]

Well, they're very welcome to do the whole gameday shizzle everybody else does. But if you post a map three days bfore the contest, soley in the submission thread, and not anywhere else, you can't expect either to have a good enough map to stand up to ones that have been online and under testing for months, or to get as much feedback as them.

There is nothing more correct than this. Sorry, ics.

[Tarry H Sruman]

As was said to the people who missed the submission thread, it is not our responsibility to make sure you comply with the procedures. Ravidge put out a comprehensive description of this contest over 4 months ago, and you had all that time to test it. You cannot complain about your map being eliminated because you neglected to test it. Some of the submissions we received were from very inexperienced mappers and of sub-par quality, you can't honestly expect us to give every one of them a full judging when half of them are fundamentally broken, can you?

[henke37]

I sure hope there were no broken entries. If a map can't be won against a dummy bot then it is broken. If the map lacks lighting, cubemaps or packed in custom content then it is broken. Anything else is just very bad.

[grazr]

...you can't honestly expect us to give every one of them a full judging when half of them are fundamentally broken, can you?

Still, it'll be nice.

I know some people will judge all the maps, including myself, although the maps that don't get past the preliminary vote probably wont receive particularly comprehensive judging.

It's a sucky way because none of the contestants who are new here knew about pre-ranking off some of the maps in advance. There was no mention about such thing in the rules and if there are future contests, note about this should be added to contest rules so similiar mess cannot occur again.

Those people who submitted map put work, effort and much time to their entries and some of them, perhaps even mine won't get to the main contest and the mappers won't get the feedback they deserve because pleminary votes will rank them off. They never hear detailed explanations on what is wrong and how they can learn from their mistakes. That's the most saddest part in this.

If you're worried about publicity, your map will still be featured so you're not being ignored; the only real difference is how comprehensively the map will be judged. If you don't pass preliminary then you weren't gonna win in the first place so it's not like you're being disqualified arbitrarily. The prelims are only to increase the time in which the judges can deliver their comprehensive reviews. Not passing prelims in itself is feedback and has nothing to do with being new or not.

If the people submitted maps with the intention of receiving feedback... we've been running gamedays that have been providing that service for 4 months and will continue to provide that service until the final judgements.

[yyler]

Not to mention we run gamedays all the time anyway, so if testing and feedback is what you want you could have gotten it regardless of the contest.

[ics]

If you're worried about publicity, your map will still be featured so you're not being ignored; the only real difference is how comprehensively the map will be judged. If you don't pass preliminary then you weren't gonna win in the first place so it's not like you're being disqualified arbitrarily. The prelims are only to increase the time in which the judges can deliver their comprehensive reviews. Not passing prelims in itself is feedback and has nothing to do with being new or not..

I'm not worried about publicity. I'm not trying to be famous. I do maps for fun and always have. It's my way on contributing games and to the community i run and also to other servers that might find something i do usefull.

However, this is no way directed to the mappers who did the maps and i don't want to point fingers but since there are maps in the contest which have either seethrough textures or devtextures and my map, which is basically done without any huge flaws, doesn't get through to the contest and the maps that have these things i mentioned, might get in, i'm wondering how that could be possible.

But we aren't there yet.

[LeSwordfish]

I'm not worried about publicity. I'm not trying to be famous. I do maps for fun and always have. It's my way on contributing games and to the community i run and also to other servers that might find something i do usefull.

However, this is no way directed to the mappers who did the maps and i don't want to point fingers but since there are maps in the contest which have either seethrough textures or devtextures and my map, which is basically done without any huge flaws, doesn't get through to the contest and the maps that have these things i mentioned, might get in, i'm wondering how that could be possible.

But we aren't there yet.

If you have a problem with an individual map, then post explaining why no-one should vote for it. Or don't vote for it yourself. Ultimately, however, this is a democratic process. It's up to the voters to choose which maps they want to go through and which they don't. Saying "But this is unfair, I'm better than those ones." isn't going to change anything. If Obama were to go on stage and say "Oh, come on now. Look at Romney. The man's an idiot." he'd not do too well out of it.

[ics]

If you have a problem with an individual map, then post explaining why no-one should vote for it. Or don't vote for it yourself. Ultimately, however, this is a democratic process. It's up to the voters to choose which maps they want to go through and which they don't. Saying "But this is unfair, I'm better than those ones." isn't going to change anything. If Obama were to go on stage and say "Oh, come on now. Look at Romney. The man's an idiot." he'd not do too well out of it.

Presidential race has nothing to do with TF2 mapping. Thats a very bad example. I just think its unfair if unfinished map is voted up on the contest over very much near finished one. Thats all i'm saying.

[Wilson]

Some of those "unfinished" maps are superior to some of those "finished" maps.
I don't care if mapper didn't reach rc before contest ended, i voted based on how good the map is, not on amount of detail etc.

[LeSwordfish]

Finishing surely isn't everything, and I say that as someone who didn't finish, yes. But it's a vote. People choose the ones they think are best. Their criteria might be different to yours.

Dealwithitmole.jpg


Or possibly dealwithitmole.gif. I think the gif was better.

http://dl.dropbox.com/u/1293763/dealwithitmole.gif -You're welcome. *Fr0z3n

[Fr0Z3n]

... there are maps in the contest which have either seethrough textures or devtextures and my map, which is basically done without any huge flaws, doesn't get through to the contest and the maps that have these things i mentioned, might get in

Note this part here.
The people who are voting are NOT out to get you. We know our stuff. If your map doesn't make it through even though it is "finished", and maps that have dev textures do get through (and are "unfinished") then it should be obvious, that those maps just play much better than yours, and that you might still have huge flaws in the map. OR your map might play well, but the others just play better. This is a fun contest, don't worry about taking it seriously.

This is your friend. (http://feedback.tf2maps.net/map/pl_curve_b8/)

[A Boojum Snark]

On the subject of finished vs unfinished, dev textures, and the like, I would like to point out one thing...

PLR Nightfall's final submission to the PLR/ADCTF contest had an unfinished largely dev-texture third stage. It then took first place in the PLR category anyway. It was then bought by valve after being finished up.

[stevethepocket]

ics does kind of have a point: How many of us over in the preliminary voting thread just looked over the full list and picked out the maps we could recall playing, rather than making sure we had played through all of them at least once?

[Wilson]

ics does kind of have a point: How many of us over in the preliminary voting thread just looked over the full list and picked out the maps we could recall playing, rather than making sure we had played through all of them at least once?

We did have contest map tests plenty of times before the voting began.

[Sergis]

ics does kind of have a point: How many of us over in the preliminary voting thread just looked over the full list and picked out the maps we could recall playing, rather than making sure we had played through all of them at least once?

i didnt play all of the maps because for quite a few of them it was clear anyway that they will not be in my top 10 so why would i even bother

which is why i wanted 1-10 ratings

i might go over the eliminated ones and try to give some feedbacks but no promises there

[ForbiddenDonut]

ics does kind of have a point: How many of us over in the preliminary voting thread just looked over the full list and picked out the maps we could recall playing, rather than making sure we had played through all of them at least once?

I made sure to at least look at every map on the list. And to be honest, it was a close call for my top ten. I did enjoy ics' map. Unfortunately, we only have ten slots to pick from. And there were many, many high quality maps in this contest.

If I recall correctly, Ravidge noted that, while we are voting for ten maps, that doesn't mean only ten maps will be in the final draw. The whole purpose of the preliminary voting was to cut the work load for the judges; not to single out designers for their contribution.

Regardless, and I say this for all the contest participants: if your map didn't make it past preliminary voting, then go over your feedback page, look at the comments in your thread and think up the reasons why it was eliminated. Look at your competition: what did they do right? Look at Valve's maps: they aren't perfect, but they are always good examples to go off of. Then make changes or be aware of your mistakes for your next map.

That's whole idea of this website, isn't it? To improve.

[Monsterclip]

We did have contest map tests plenty of times before the voting began.

This isn't an excuse. People working on their maps had up to the deadline to fix any of the problems that you surely had already set your prejudice on. If you didn't test and base your map selection solely on what was submitted after the deadline, you did a disservice to everyone in the contest.

I'm not getting worked up over it, it's just the way I feel about that particular subject, especially coming from someone who was so adamant about people listening to the feedback from play tests.

[grazr]

In my experience gameplay always trumps aesthetics, especially in a contest where those contributing and voting understand the creative process and that it is long and arduous. Having aesthetics is and always will be the "cherry on the top".

It is true that there is an issue that many people will vote based on the maps they experienced which is why i'm refraining from voting in the prelims because i've only played 12 of the maps and don't want to vicariously vote out certain maps simply because i have missed them. But then this is still an issue that could have been pro-actively avoided by the map authors.

I knew for a fact that this does play a small role in the voting process and made sure i had several weeks of testing before the deadline just so that people could get to know the map as much as get feedback and improve it. Because maps do tend to gather a following of fans. Escarpment is a prime example of that fact, that fanbase had a chance to grow and it was the authors involvement in the entire development process which allowed that to happen.

People who didn't do this have basically shot themselves in the foot in more ways than one and everything else is just excuses. Hopefully this will encourage you to come back next time stronger and more experienced.

[Wolfen]

I knew for a fact that this does play a small role in the voting process and made sure i had several weeks of testing before the deadline just so that people could get to know the map as much as get feedback and improve it. Because maps do tend to gather a following of fans. Escarpment is a prime example of that fact, that fanbase had a chance to grow and it was the authors involvement in the entire development process which allowed that to happen.

People who didn't do this have basically shot themselves in the foot in more ways than one and everything else is just excuses. Hopefully this will encourage you to come back next time stronger and more experienced.

This is very much true. People you don't even know will become friends or enemies of your maps over your testing period. But make sure you test it on the servers where the competition is being held.

My issue was I tested on the Gamers with Jobs server. Most of them had, and some still have, no clue who I am, but I received favorable feedback there over multiple tests and modified my map to their critiques and thoughts.

When I came here and posted in the last few weeks I was met with polar opposite opinions and thoughts compared to what I had been seeing.

On the plus side, I now have a few fans and interested parties on their server about what I'll do next while here it sounds like I have much more proving and work to do to try and get up to the same level as over there.

Also, I did think about sending in a resume to become a judge but I decided not too since I feel there are much better qualified individuals on here. I have a bad habit of nitpicking others maps while mine have the same issues. I think the judges should be people who know what their talking about.

[LeSwordfish]

And they will be, but I suspect it would take literally months for a small team of expert judges to choose accurately from amongst thirty maps. Unless they were to do a quick check of some and remove them based on first impressions, which would be, surprise surprise, worse than this.

As for the public vote, we'd end up in the exact same situation. No-one could play them all, certainly not enough for a true decision, and people would only rate the ones they'd played.

Personally I'm worried I'll do less well in this stage due to not being here to pimp myself. But eh.

[Wilson]

This isn't an excuse. People working on their maps had up to the deadline to fix any of the problems that you surely had already set your prejudice on. If you didn't test and base your map selection solely on what was submitted after the deadline, you did a disservice to everyone in the contest.

I'm not getting worked up over it, it's just the way I feel about that particular subject, especially coming from someone who was so adamant about people listening to the feedback from play tests.

I think you misunderstood me.
We started testing the maps almost soon as the upload thread closed.
Maps in that thread were gameplay wise finished for this contest, so i see no harm done at all.

[ics]

Note this part here.
The people who are voting are NOT out to get you. We know our stuff. If your map doesn't make it through even though it is "finished", and maps that have dev textures do get through (and are "unfinished") then it should be obvious, that those maps just play much better than yours, and that you might still have huge flaws in the map. OR your map might play well, but the others just play better. This is a fun contest, don't worry about taking it seriously.

This is your friend. (http://feedback.tf2maps.net/map/pl_curve_b8/)

I wouldn't be so sure after reading all the comments, especially on _b4 at feedback (and i did see _b4 and _b8 earlier before contest ended. _b8, barely and no excuses on not having time to fix all the things.). It seems partly like general SPUF trolling instead of contributing. There are regulars that are familiar with eachothers and then some stranger comes in and.. you either like his map or don't. Prejudice is usually set right there and later versions never matter. There *are* usefull posts, there and i would shake hands with everyone who made a valid point that i was able to fix on later release, like you did with the framerate thing which you also posted on the thread so that i would definitely see it. I have also addressed rest of the issues and will run some more tests that i am currently conducting on 2 different size servers, 24 and 30 slots.

ics does kind of have a point: How many of us over in the preliminary voting thread just looked over the full list and picked out the maps we could recall playing, rather than making sure we had played through all of them at least once?

You just said the thing i was aiming for. Those that played previous versions of the maps before the contest one already might have made a judgement on particular map. Someone who sees it for the first time, has different impression. This is what Monsterclip was also talking about in his post.

Also, being an expert isn't requirement for being a judge. All we know the folks could be just regulars (or strangers?) from here that may have never done a finished map. That doesn't really matter does it as it's the first impression that counts. It's like that for me, i either like some map or i don't despite the later versions of it so i do understand if someone hates something i make or doesn't. It's not going to get me lose my sleep.

No, i'm not bitter either. Internet isn't good way to express how people think and feel.

[Tarry H Sruman]

Also, being an expert isn't requirement for being a judge. All we know the folks could be just regulars (or strangers?) from here that may have never done a finished map. That doesn't really matter does it as it's the first impression that counts.

Ravidge isn't going to select anyone who is incapable of giving fair, unbiased, and comprehensive feedback. Every map that makes it past the prelims will get a very extensive judging. You have to understand that the prelim process is necessary, because if there are nearly 30 maps up for judging the judges will either have to a) spend an inordinate amount of time playtesting and writing feedback, something most cannot do on account of school/work/etc, or b) Cut the amount of time given to each map and give everyone incomplete feedback.

[grazr]

...Also, being an expert isn't requirement for being a judge. All we know the folks could be just regulars (or strangers?) from here that may have never done a finished map...

The requirements for being a judge are basically the ability of critical thinking and the capacity to prove that skill with either essays or tutorials or diaries of previous map projects. Along with the understanding that you are making a commitment to the judgement process as the time spent playing maps and writing comprehensive reviews will be very, very time consuming.

The idea of the official judge is to weigh in a more professional view point so that the voting process doesn't get cluttered with purely subjective opinions as there are criteria to anylise and a brief to be met. We all understand that the average players feedback amounts to "don't like this", "this is cool", which is insufficient.

Have a little faith. We've been doing these contests for over 4 years. We've honed our judgement procedure to make everyone happy and the results fair and balanced. Winners will be decided on both popularity from the player base and skill reviewed by novices.

[Wilson]

I wouldn't be so sure after reading all the comments, especially on _b4 at feedback (and i did see _b4 and _b8 earlier before contest ended. _b8, barely and no excuses on not having time to fix all the things.). It seems partly like general SPUF trolling instead of contributing. There are regulars that are familiar with eachothers and then some stranger comes in and.. you either like his map or don't. Prejudice is usually set right there and later versions never matter.


Have you consider that there is something wrong with your map, instead of assuming everyone is trolling?

I just look at both feedback pages, and besides few dumb annotations, i see nothing out of ordinary.

[Ezekel]

ics does kind of have a point: How many of us over in the preliminary voting thread just looked over the full list and picked out the maps we could recall playing, rather than making sure we had played through all of them at least once?

this is/was my fundamental problem with how the ctf contest was handled.

people will vote in prelims usually without heavily considering the maps. i.e. based on either the screenshots or their memories of playing it (or a combo of both).
- this means that maps that were tested on tf2maps gamedays have a distinct advantage over those that were tested elsewhere, as the people who vote in the prelims will be able to look at the name/screenshot and go 'ah, i remember that map, it's layout was like this and when i played it i was able to uber that heavy near cp1' or similar.

although i'll also add that a good non-dev-texture screenshot will help garner attention if the map is one of the ones that people don't know well.


i don't have an outright solution unfortunately. i suppose you could insist that every map receive a 1 sentence review from each voter before their top ten get counted, but then we'd see a lot fewer voters.

- prelims did work well in the 72 hr contest last year, but i put that down to how everyone was on even footing pretty much with the pre-submission testing and the post-submission playthroughs. the testing period seemed to be quite high spirited in my opinion, and a good number of voters gave feedback for all the maps in the contest. perhaps it's partly to do with the time of year the the submission is set at (i.e. people right now are a lil burnt out from the major mapping content + tis the season to be sitting exams for most).