What was your online test score with an audition

This is where all of the games are discussed.

Moderators: alietr, trainman, econgator, dhkendall

User avatar
BigDaddyMatty
Hoping not to get pruney this time
Posts: 3300
Joined: Wed Jan 21, 2015 11:05 am
Location: Anderson, IN

Re: What was your online test score with an audition

Post by BigDaddyMatty »

CFParrinello wrote:
BobF wrote:
DysonSphere wrote:Well put, Trivia Wayne. I always assumed all the above was true. I can't imagine they actually go through tests by hand, and obviously some contestants will be selected for over others for TV considerations.
Why would they ever go through ONLINE tests by HAND? It's all stored in a database, name, number of correct answers. A click of a button and they can change the threshold from 35 to 40 or 30.
Well, not by hand, but they explicitly say not to worry about minor misspellings and partial answers -- is that a red herring to get people to relax, or does that just mean that they have a list of likely mispellings and alternate answers pre-logged and parsed by software? I agree that it wouldn't be cost effective to have human beings spending time looking at each individual's answers, but I wonder how reliably those alternate acceptable answers can be predicted. I suppose that doesn't matter that much when you have more applicants than you need.
I have no idea how TPTB deal with spelling, but it wouldn't be particularly costly to hand-grade only the answers that were marked incorrect for only the tests that are on the borderline.
Sprinkles are for winners.
bpmod
Rank
Posts: 5424
Joined: Fri Jul 01, 2011 12:26 pm
Location: Hamilton Ontario

Re: What was your online test score with an audition

Post by bpmod »

It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.

Brian
...but the senator, while insisting he was not intoxicated, could not explain his nudity.

If I had 50 cents for every math question I got right, I'd have $6.30 by now.
User avatar
triviawayne
Hoping I don’t drown in this contestant pool
Posts: 2677
Joined: Tue Feb 17, 2015 9:50 pm

Re: What was your online test score with an audition

Post by triviawayne »

bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.

Brian
With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process.

Again, it comes down to time/work = cost, and that is money that they don't need to spend.
Total game show career losings = $171,522
davey
Watches Jeopardy! Way Too Much
Posts: 6030
Joined: Thu Jan 12, 2012 10:55 pm

Re: What was your online test score with an audition

Post by davey »

triviawayne wrote:
bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.

Brian
With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process.

Again, it comes down to time/work = cost, and that is money that they don't need to spend.
I agree - and this is why I always think of the online test only as a screener. A far better one than "Hey, if you think you'd be good at J! send us a postcard!"
bpmod
Rank
Posts: 5424
Joined: Fri Jul 01, 2011 12:26 pm
Location: Hamilton Ontario

Re: What was your online test score with an audition

Post by bpmod »

triviawayne wrote:
bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.

Brian
With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process.

Again, it comes down to time/work = cost, and that is money that they don't need to spend.
I am using a couple of assumptions here:
1. The folks at Jeopardy! claim that spelling errors and incomplete answers will be considered. If what you say is true, then they would not need to consider any of those.
2. Assuming (1) is correct, it would be far less work (and therefore far cheaper) to check spelling errors etc. against a finite list (of actual responses) than to try to compile a list of every possible wrong but acceptable answer ahead of time.

If they don't do something like that, then a question such as the 3 & 27, which has at least 3 responses as correct as each other, several people who came up with responses they didn't anticipate would be scoring less than they should have.

Brian
...but the senator, while insisting he was not intoxicated, could not explain his nudity.

If I had 50 cents for every math question I got right, I'd have $6.30 by now.
CFParrinello
Newbie
Posts: 9
Joined: Fri Jan 29, 2016 2:44 am

Re: What was your online test score with an audition

Post by CFParrinello »

triviawayne wrote:
bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.

Brian
With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process..
It's mostly because (depending on how the string is read) "milton" is considered a different answer than "Milton" which is considered a different answer than "john milton" or "John Milton" even if Jeopardy all considers them the same answer. The software reading the answers doesn't know which is "correct," as it doesn't know English literature or history. It simply knows which string to compare it to and what to return.

So it's not that I think TPTB are super eager to give freebies to aspiring contestants or whatever, just that the nature of a typed (rather than multiple choice) test requires either an amount of foresight about the way answers are going to be input, or a way to review answers given. The more complex the question, or the murkier its wording, the more correct answers there could be (even if all of those answers are actually the same thing). So even if we're following your logic of "TPTB will act as cost-effectively as possible for these online tests," I don't think bpmod's suggestion is unreasonable.
User avatar
BobF
Watches Jeopardy! Way Too Much
Posts: 2180
Joined: Thu Jul 07, 2011 3:03 pm
Location: All over the east coast
Contact:

Re: What was your online test score with an audition

Post by BobF »

CFParrinello wrote:
triviawayne wrote:
bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.

Brian
With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process..
It's mostly because (depending on how the string is read) "milton" is considered a different answer than "Milton" which is considered a different answer than "john milton" or "John Milton" even if Jeopardy all considers them the same answer. The software reading the answers doesn't know which is "correct," as it doesn't know English literature or history. It simply knows which string to compare it to and what to return.

So it's not that I think TPTB are super eager to give freebies to aspiring contestants or whatever, just that the nature of a typed (rather than multiple choice) test requires either an amount of foresight about the way answers are going to be input, or a way to review answers given. The more complex the question, or the murkier its wording, the more correct answers there could be (even if all of those answers are actually the same thing). So even if we're following your logic of "TPTB will act as cost-effectively as possible for these online tests," I don't think bpmod's suggestion is unreasonable.
For anybody familiar with SQL

SELECT * FROM Answers WHERE Answer LIKE '%Milton%'

They could even do LIKE '%m%i%l%t%o%n' so if somebody has a reasonably close spelling they could be counted right.

Computers are pretty darn powerful.
Was once hugged by Maggie Speak!
bpmod
Rank
Posts: 5424
Joined: Fri Jul 01, 2011 12:26 pm
Location: Hamilton Ontario

Re: What was your online test score with an audition

Post by bpmod »

BobF wrote:For anybody familiar with SQL

SELECT * FROM Answers WHERE Answer LIKE '%Milton%'

They could even do LIKE '%m%i%l%t%o%n' so if somebody has a reasonably close spelling they could be counted right.

Computers are pretty darn powerful.
Yes, that is indeed possible. But I highly doubt they'd want to be accepting a response of "Alexander Hamilton Ontario" for a clue that wanted John Milton. And your second example opens up an even wider range of garbage.

Brian
...but the senator, while insisting he was not intoxicated, could not explain his nudity.

If I had 50 cents for every math question I got right, I'd have $6.30 by now.
CFParrinello
Newbie
Posts: 9
Joined: Fri Jan 29, 2016 2:44 am

Re: What was your online test score with an audition

Post by CFParrinello »

BobF wrote:
CFParrinello wrote:
triviawayne wrote:
bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.

Brian
With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process..
It's mostly because (depending on how the string is read) "milton" is considered a different answer than "Milton" which is considered a different answer than "john milton" or "John Milton" even if Jeopardy all considers them the same answer. The software reading the answers doesn't know which is "correct," as it doesn't know English literature or history. It simply knows which string to compare it to and what to return.

So it's not that I think TPTB are super eager to give freebies to aspiring contestants or whatever, just that the nature of a typed (rather than multiple choice) test requires either an amount of foresight about the way answers are going to be input, or a way to review answers given. The more complex the question, or the murkier its wording, the more correct answers there could be (even if all of those answers are actually the same thing). So even if we're following your logic of "TPTB will act as cost-effectively as possible for these online tests," I don't think bpmod's suggestion is unreasonable.
For anybody familiar with SQL

SELECT * FROM Answers WHERE Answer LIKE '%Milton%'

They could even do LIKE '%m%i%l%t%o%n' so if somebody has a reasonably close spelling they could be counted right.

Computers are pretty darn powerful.
I'm far from the most knowledgeable programmer here, but I may have been unclear. The point wasn't just that capitalization matters or that some strings could contain parts of other strings, but that the database (even one using SQL) doesn't know what is correct, just what matches. Sometimes that's going to be fine, but the format of the test ultimately requires the presence of discretion, even if in a very small amount.
User avatar
BulgarianBeast
Valued Contributor
Posts: 96
Joined: Tue Nov 17, 2015 10:25 am

Re: What was your online test score with an audition

Post by BulgarianBeast »

CFParrinello wrote:
BobF wrote:
CFParrinello wrote:
triviawayne wrote:
bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.

Brian
With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process..
It's mostly because (depending on how the string is read) "milton" is considered a different answer than "Milton" which is considered a different answer than "john milton" or "John Milton" even if Jeopardy all considers them the same answer. The software reading the answers doesn't know which is "correct," as it doesn't know English literature or history. It simply knows which string to compare it to and what to return.

So it's not that I think TPTB are super eager to give freebies to aspiring contestants or whatever, just that the nature of a typed (rather than multiple choice) test requires either an amount of foresight about the way answers are going to be input, or a way to review answers given. The more complex the question, or the murkier its wording, the more correct answers there could be (even if all of those answers are actually the same thing). So even if we're following your logic of "TPTB will act as cost-effectively as possible for these online tests," I don't think bpmod's suggestion is unreasonable.
For anybody familiar with SQL

SELECT * FROM Answers WHERE Answer LIKE '%Milton%'

They could even do LIKE '%m%i%l%t%o%n' so if somebody has a reasonably close spelling they could be counted right.

Computers are pretty darn powerful.
I'm far from the most knowledgeable programmer here, but I may have been unclear. The point wasn't just that capitalization matters or that some strings could contain parts of other strings, but that the database (even one using SQL) doesn't know what is correct, just what matches. Sometimes that's going to be fine, but the format of the test ultimately requires the presence of discretion, even if in a very small amount.
It wouldn't be too hard(or too time consuming) to declare an array with a couple of different strings for questions where there could be a few anticipated responses, and seeing if the given answer matches one of those.
bpmod
Rank
Posts: 5424
Joined: Fri Jul 01, 2011 12:26 pm
Location: Hamilton Ontario

Re: What was your online test score with an audition

Post by bpmod »

BulgarianBeast wrote:It wouldn't be too hard(or too time consuming) to declare an array with a couple of different strings for questions where there could be a few anticipated responses, and seeing if the given answer matches one of those.
See point two in my post 5 or 6 back. And also CFP's subsequent post.

Brian
...but the senator, while insisting he was not intoxicated, could not explain his nudity.

If I had 50 cents for every math question I got right, I'd have $6.30 by now.
User avatar
richardt
Contributor
Posts: 11
Joined: Wed Jan 27, 2016 9:51 pm
Location: Tennessee

Re: What was your online test score with an audition

Post by richardt »

Here is my take on this discussion.

1. It may not matter how many correct responses we get, because we do not know how it is scored. What if each question is worth a set amount of points like J6! is currently scored and although 2 people get 40 questions correct only one may have the points necessary to pass.

2. This year they may have to lower their standards because many smart Canadians were excluded from the test. Maybe Canada should have their own version of Jeopardy.

3. With the high number of passers it is still subject to the luck of the draw, and in that regard I wish everyone good luck.

4. What is WKRP in Cincinnati? (... and now the hog reports from Les Nessman.)
rjaguar3
Watches Jeopardy! Way Too Much
Posts: 888
Joined: Wed Jan 04, 2012 5:54 pm

Re: What was your online test score with an audition

Post by rjaguar3 »

I think TPTB could easily do something like I do with the Summer Hiatus Challenge.

The Summer Hiatus Challenge scoring software uses regular expressions. For each clue, there are a set of regular expressions corresponding to correct answers and another set of regular expressions corresponding to incorrect answers. For example, a regular expression might be "(John )?Milton", indicating that either "Milton" or "John Milton" is acceptable.

What I do when grading the SHC is as follows:
  1. I enter the clue and a regular expression that should capture most common variations of the correct answer.
  2. I then run the script to gather all the answers (the online test equivalent would be everyone takes the test).
  3. The grading page is then automatically populated with answers that do not match the regular expression I wrote for the correct answer, and buttons for marking right or wrong answers. Each answer only appears once, and it is case insensitive.
  4. Then (after consultation, if necessary), I go through each answer and click the right or wrong button; this adds a new regular expression (matching only the exact text of that answer) for that answer to be marked right or wrong.
  5. All of the scoring is done using SQL.
As far as scale goes, the primary difference between the SHC and the online test is that the SHC has about 100 players and 420 clues, while the online test has about 100,000 players (if I understand correctly) and 50 clues. But despite the thousand-fold increase in responses to grade (maybe more since wrong answers are not penalized in the online test), there still shouldn't be 1000 times more different unexpected responses to look at. I think this kind of approach could straightforwardly be done by a team of graders.
bpmod
Rank
Posts: 5424
Joined: Fri Jul 01, 2011 12:26 pm
Location: Hamilton Ontario

Re: What was your online test score with an audition

Post by bpmod »

rjaguar3 wrote:For example, a regular expression might be "(John )?Milton", indicating that either "Milton" or "John Milton" is acceptable.
But not "john milton" or "John Mliton".

Yes, it is possible that TPTB reject all responses that don't fit a very finite pre-compiled list of acceptable responses. Why, then, would they state that minor spelling & typographical errors and incomplete responses will be considered?

Or am I just naively and blindly trusting that they would not reject my response of "john milton" or "prime factor".

Brian
...but the senator, while insisting he was not intoxicated, could not explain his nudity.

If I had 50 cents for every math question I got right, I'd have $6.30 by now.
rjaguar3
Watches Jeopardy! Way Too Much
Posts: 888
Joined: Wed Jan 04, 2012 5:54 pm

Re: What was your online test score with an audition

Post by rjaguar3 »

bpmod wrote:
rjaguar3 wrote:For example, a regular expression might be "(John )?Milton", indicating that either "Milton" or "John Milton" is acceptable.
But not "john milton" or "John Mliton".
The regular expression code I use is case-insensitive. I can't see why TPTB would treat responses differently based on capitalization.
bpmod wrote: Yes, it is possible that TPTB reject all responses that don't fit a very finite pre-compiled list of acceptable responses. Why, then, would they state that minor spelling & typographical errors and incomplete responses will be considered?
As I refer to later in the post, that's just the beginning of the grading process for the SHC; non-matching responses are displayed to mark as correct or incorrect. (Sometimes I will make a new regular expression that covers multiple correct or incorrect responses.)
User avatar
Volante
Harbinger of the Doomed Lemur
Posts: 9254
Joined: Thu Jul 14, 2011 11:42 pm

Re: What was your online test score with an audition

Post by Volante »

Something like this is probably involved too: https://en.wikipedia.org/wiki/Levenshtein_distance
The best thing that Neil Armstrong ever did, was to let us all imagine we were him.
Latest movies (1-10): Everything Everywhere All at Once (10), Ruby Gillman: Teenage Kraken (6), Black Sunday /1960/ (6), Marcel the Shell with Shoes On (7)
User avatar
Leopardy!
Contributor
Posts: 18
Joined: Sun Apr 05, 2015 12:13 pm

Re: What was your online test score with an audition

Post by Leopardy! »

I know there's plenty to do in most large cities, but it could be that (depending on their itinerary--if they come east first) TPTB get more days in NYC to allow the testers to get over jet lag and to see more Broadway shows--a huge plus for most visitors, especially those in show biz.
User avatar
TheSunWillComeOut
Two-Morrow
Posts: 344
Joined: Fri Jun 21, 2013 10:12 pm

Re: What was your online test score with an audition

Post by TheSunWillComeOut »

Leopardy! wrote:I know there's plenty to do in most large cities, but it could be that (depending on their itinerary--if they come east first) TPTB get more days in NYC to allow the testers to get over jet lag and to see more Broadway shows--a huge plus for most visitors, especially those in show biz.
Based on speaking with other future and would-be contestants at my NYC audition in 2013, I think that NYC is a smart place for TPTB to spend several days. Two words: Northeast Corridor. NYC is a transportation hub for the entire East Coast, and it's extremely easy to get to by bus and train for out-of-towners ranging from D.C. to Boston and everywhere in between. A lot of people there had started their journey at 4/5 a.m., with the plan to audition in the morning, spend an afternoon doing something fun in the city, and then head home in the evening.
User avatar
OSXpert
Watches Jeopardy! Way Too Much
Posts: 881
Joined: Thu Jun 05, 2014 7:29 pm

Re: What was your online test score with an audition

Post by OSXpert »

And here I am sitting in Manhattan, waiting to get an invite to Detroit.
User avatar
morbeedo
Loyal Jeopardista
Posts: 3065
Joined: Wed Jan 18, 2012 12:58 pm

Re: What was your online test score with an audition

Post by morbeedo »

OSXpert wrote:And here I am sitting in Manhattan, waiting to get an invite to Detroit.
LMFAO
Post Reply