I have no idea how TPTB deal with spelling, but it wouldn't be particularly costly to hand-grade only the answers that were marked incorrect for only the tests that are on the borderline.CFParrinello wrote:Well, not by hand, but they explicitly say not to worry about minor misspellings and partial answers -- is that a red herring to get people to relax, or does that just mean that they have a list of likely mispellings and alternate answers pre-logged and parsed by software? I agree that it wouldn't be cost effective to have human beings spending time looking at each individual's answers, but I wonder how reliably those alternate acceptable answers can be predicted. I suppose that doesn't matter that much when you have more applicants than you need.BobF wrote:Why would they ever go through ONLINE tests by HAND? It's all stored in a database, name, number of correct answers. A click of a button and they can change the threshold from 35 to 40 or 30.DysonSphere wrote:Well put, Trivia Wayne. I always assumed all the above was true. I can't imagine they actually go through tests by hand, and obviously some contestants will be selected for over others for TV considerations.
What was your online test score with an audition
Moderators: alietr, trainman, econgator, dhkendall
- BigDaddyMatty
- Hoping not to get pruney this time
- Posts: 3300
- Joined: Wed Jan 21, 2015 11:05 am
- Location: Anderson, IN
Re: What was your online test score with an audition
Sprinkles are for winners.
-
- Rank
- Posts: 5424
- Joined: Fri Jul 01, 2011 12:26 pm
- Location: Hamilton Ontario
Re: What was your online test score with an audition
It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.
Brian
Brian
...but the senator, while insisting he was not intoxicated, could not explain his nudity.
If I had 50 cents for every math question I got right, I'd have $6.30 by now.
If I had 50 cents for every math question I got right, I'd have $6.30 by now.
- triviawayne
- Hoping I don’t drown in this contestant pool
- Posts: 2677
- Joined: Tue Feb 17, 2015 9:50 pm
Re: What was your online test score with an audition
With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process.bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.
Brian
Again, it comes down to time/work = cost, and that is money that they don't need to spend.
Total game show career losings = $171,522
-
- Watches Jeopardy! Way Too Much
- Posts: 6030
- Joined: Thu Jan 12, 2012 10:55 pm
Re: What was your online test score with an audition
I agree - and this is why I always think of the online test only as a screener. A far better one than "Hey, if you think you'd be good at J! send us a postcard!"triviawayne wrote:With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process.bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.
Brian
Again, it comes down to time/work = cost, and that is money that they don't need to spend.
-
- Rank
- Posts: 5424
- Joined: Fri Jul 01, 2011 12:26 pm
- Location: Hamilton Ontario
Re: What was your online test score with an audition
I am using a couple of assumptions here:triviawayne wrote:With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process.bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.
Brian
Again, it comes down to time/work = cost, and that is money that they don't need to spend.
1. The folks at Jeopardy! claim that spelling errors and incomplete answers will be considered. If what you say is true, then they would not need to consider any of those.
2. Assuming (1) is correct, it would be far less work (and therefore far cheaper) to check spelling errors etc. against a finite list (of actual responses) than to try to compile a list of every possible wrong but acceptable answer ahead of time.
If they don't do something like that, then a question such as the 3 & 27, which has at least 3 responses as correct as each other, several people who came up with responses they didn't anticipate would be scoring less than they should have.
Brian
...but the senator, while insisting he was not intoxicated, could not explain his nudity.
If I had 50 cents for every math question I got right, I'd have $6.30 by now.
If I had 50 cents for every math question I got right, I'd have $6.30 by now.
-
- Newbie
- Posts: 9
- Joined: Fri Jan 29, 2016 2:44 am
Re: What was your online test score with an audition
It's mostly because (depending on how the string is read) "milton" is considered a different answer than "Milton" which is considered a different answer than "john milton" or "John Milton" even if Jeopardy all considers them the same answer. The software reading the answers doesn't know which is "correct," as it doesn't know English literature or history. It simply knows which string to compare it to and what to return.triviawayne wrote:With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process..bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.
Brian
So it's not that I think TPTB are super eager to give freebies to aspiring contestants or whatever, just that the nature of a typed (rather than multiple choice) test requires either an amount of foresight about the way answers are going to be input, or a way to review answers given. The more complex the question, or the murkier its wording, the more correct answers there could be (even if all of those answers are actually the same thing). So even if we're following your logic of "TPTB will act as cost-effectively as possible for these online tests," I don't think bpmod's suggestion is unreasonable.
- BobF
- Watches Jeopardy! Way Too Much
- Posts: 2180
- Joined: Thu Jul 07, 2011 3:03 pm
- Location: All over the east coast
- Contact:
Re: What was your online test score with an audition
For anybody familiar with SQLCFParrinello wrote:It's mostly because (depending on how the string is read) "milton" is considered a different answer than "Milton" which is considered a different answer than "john milton" or "John Milton" even if Jeopardy all considers them the same answer. The software reading the answers doesn't know which is "correct," as it doesn't know English literature or history. It simply knows which string to compare it to and what to return.triviawayne wrote:With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process..bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.
Brian
So it's not that I think TPTB are super eager to give freebies to aspiring contestants or whatever, just that the nature of a typed (rather than multiple choice) test requires either an amount of foresight about the way answers are going to be input, or a way to review answers given. The more complex the question, or the murkier its wording, the more correct answers there could be (even if all of those answers are actually the same thing). So even if we're following your logic of "TPTB will act as cost-effectively as possible for these online tests," I don't think bpmod's suggestion is unreasonable.
SELECT * FROM Answers WHERE Answer LIKE '%Milton%'
They could even do LIKE '%m%i%l%t%o%n' so if somebody has a reasonably close spelling they could be counted right.
Computers are pretty darn powerful.
Was once hugged by Maggie Speak!
-
- Rank
- Posts: 5424
- Joined: Fri Jul 01, 2011 12:26 pm
- Location: Hamilton Ontario
Re: What was your online test score with an audition
Yes, that is indeed possible. But I highly doubt they'd want to be accepting a response of "Alexander Hamilton Ontario" for a clue that wanted John Milton. And your second example opens up an even wider range of garbage.BobF wrote:For anybody familiar with SQL
SELECT * FROM Answers WHERE Answer LIKE '%Milton%'
They could even do LIKE '%m%i%l%t%o%n' so if somebody has a reasonably close spelling they could be counted right.
Computers are pretty darn powerful.
Brian
...but the senator, while insisting he was not intoxicated, could not explain his nudity.
If I had 50 cents for every math question I got right, I'd have $6.30 by now.
If I had 50 cents for every math question I got right, I'd have $6.30 by now.
-
- Newbie
- Posts: 9
- Joined: Fri Jan 29, 2016 2:44 am
Re: What was your online test score with an audition
I'm far from the most knowledgeable programmer here, but I may have been unclear. The point wasn't just that capitalization matters or that some strings could contain parts of other strings, but that the database (even one using SQL) doesn't know what is correct, just what matches. Sometimes that's going to be fine, but the format of the test ultimately requires the presence of discretion, even if in a very small amount.BobF wrote:For anybody familiar with SQLCFParrinello wrote:It's mostly because (depending on how the string is read) "milton" is considered a different answer than "Milton" which is considered a different answer than "john milton" or "John Milton" even if Jeopardy all considers them the same answer. The software reading the answers doesn't know which is "correct," as it doesn't know English literature or history. It simply knows which string to compare it to and what to return.triviawayne wrote:With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process..bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.
Brian
So it's not that I think TPTB are super eager to give freebies to aspiring contestants or whatever, just that the nature of a typed (rather than multiple choice) test requires either an amount of foresight about the way answers are going to be input, or a way to review answers given. The more complex the question, or the murkier its wording, the more correct answers there could be (even if all of those answers are actually the same thing). So even if we're following your logic of "TPTB will act as cost-effectively as possible for these online tests," I don't think bpmod's suggestion is unreasonable.
SELECT * FROM Answers WHERE Answer LIKE '%Milton%'
They could even do LIKE '%m%i%l%t%o%n' so if somebody has a reasonably close spelling they could be counted right.
Computers are pretty darn powerful.
- BulgarianBeast
- Valued Contributor
- Posts: 96
- Joined: Tue Nov 17, 2015 10:25 am
Re: What was your online test score with an audition
It wouldn't be too hard(or too time consuming) to declare an array with a couple of different strings for questions where there could be a few anticipated responses, and seeing if the given answer matches one of those.CFParrinello wrote:I'm far from the most knowledgeable programmer here, but I may have been unclear. The point wasn't just that capitalization matters or that some strings could contain parts of other strings, but that the database (even one using SQL) doesn't know what is correct, just what matches. Sometimes that's going to be fine, but the format of the test ultimately requires the presence of discretion, even if in a very small amount.BobF wrote:For anybody familiar with SQLCFParrinello wrote:It's mostly because (depending on how the string is read) "milton" is considered a different answer than "Milton" which is considered a different answer than "john milton" or "John Milton" even if Jeopardy all considers them the same answer. The software reading the answers doesn't know which is "correct," as it doesn't know English literature or history. It simply knows which string to compare it to and what to return.triviawayne wrote:With 100,000 taking the test and 90,000 of them not passing; why would they bother to do that when they have enough people to move on to the next step in the process..bpmod wrote:It would be almost nothing to write a script that compiles all wrong answers for each question. The list of wrong answers could be graded as being close enough or not, and the tests could be run through again with the "close enough"s added as correct.
Brian
So it's not that I think TPTB are super eager to give freebies to aspiring contestants or whatever, just that the nature of a typed (rather than multiple choice) test requires either an amount of foresight about the way answers are going to be input, or a way to review answers given. The more complex the question, or the murkier its wording, the more correct answers there could be (even if all of those answers are actually the same thing). So even if we're following your logic of "TPTB will act as cost-effectively as possible for these online tests," I don't think bpmod's suggestion is unreasonable.
SELECT * FROM Answers WHERE Answer LIKE '%Milton%'
They could even do LIKE '%m%i%l%t%o%n' so if somebody has a reasonably close spelling they could be counted right.
Computers are pretty darn powerful.
-
- Rank
- Posts: 5424
- Joined: Fri Jul 01, 2011 12:26 pm
- Location: Hamilton Ontario
Re: What was your online test score with an audition
See point two in my post 5 or 6 back. And also CFP's subsequent post.BulgarianBeast wrote:It wouldn't be too hard(or too time consuming) to declare an array with a couple of different strings for questions where there could be a few anticipated responses, and seeing if the given answer matches one of those.
Brian
...but the senator, while insisting he was not intoxicated, could not explain his nudity.
If I had 50 cents for every math question I got right, I'd have $6.30 by now.
If I had 50 cents for every math question I got right, I'd have $6.30 by now.
- richardt
- Contributor
- Posts: 11
- Joined: Wed Jan 27, 2016 9:51 pm
- Location: Tennessee
Re: What was your online test score with an audition
Here is my take on this discussion.
1. It may not matter how many correct responses we get, because we do not know how it is scored. What if each question is worth a set amount of points like J6! is currently scored and although 2 people get 40 questions correct only one may have the points necessary to pass.
2. This year they may have to lower their standards because many smart Canadians were excluded from the test. Maybe Canada should have their own version of Jeopardy.
3. With the high number of passers it is still subject to the luck of the draw, and in that regard I wish everyone good luck.
4. What is WKRP in Cincinnati? (... and now the hog reports from Les Nessman.)
1. It may not matter how many correct responses we get, because we do not know how it is scored. What if each question is worth a set amount of points like J6! is currently scored and although 2 people get 40 questions correct only one may have the points necessary to pass.
2. This year they may have to lower their standards because many smart Canadians were excluded from the test. Maybe Canada should have their own version of Jeopardy.
3. With the high number of passers it is still subject to the luck of the draw, and in that regard I wish everyone good luck.
4. What is WKRP in Cincinnati? (... and now the hog reports from Les Nessman.)
-
- Watches Jeopardy! Way Too Much
- Posts: 888
- Joined: Wed Jan 04, 2012 5:54 pm
Re: What was your online test score with an audition
I think TPTB could easily do something like I do with the Summer Hiatus Challenge.
The Summer Hiatus Challenge scoring software uses regular expressions. For each clue, there are a set of regular expressions corresponding to correct answers and another set of regular expressions corresponding to incorrect answers. For example, a regular expression might be "(John )?Milton", indicating that either "Milton" or "John Milton" is acceptable.
What I do when grading the SHC is as follows:
The Summer Hiatus Challenge scoring software uses regular expressions. For each clue, there are a set of regular expressions corresponding to correct answers and another set of regular expressions corresponding to incorrect answers. For example, a regular expression might be "(John )?Milton", indicating that either "Milton" or "John Milton" is acceptable.
What I do when grading the SHC is as follows:
- I enter the clue and a regular expression that should capture most common variations of the correct answer.
- I then run the script to gather all the answers (the online test equivalent would be everyone takes the test).
- The grading page is then automatically populated with answers that do not match the regular expression I wrote for the correct answer, and buttons for marking right or wrong answers. Each answer only appears once, and it is case insensitive.
- Then (after consultation, if necessary), I go through each answer and click the right or wrong button; this adds a new regular expression (matching only the exact text of that answer) for that answer to be marked right or wrong.
- All of the scoring is done using SQL.
-
- Rank
- Posts: 5424
- Joined: Fri Jul 01, 2011 12:26 pm
- Location: Hamilton Ontario
Re: What was your online test score with an audition
But not "john milton" or "John Mliton".rjaguar3 wrote:For example, a regular expression might be "(John )?Milton", indicating that either "Milton" or "John Milton" is acceptable.
Yes, it is possible that TPTB reject all responses that don't fit a very finite pre-compiled list of acceptable responses. Why, then, would they state that minor spelling & typographical errors and incomplete responses will be considered?
Or am I just naively and blindly trusting that they would not reject my response of "john milton" or "prime factor".
Brian
...but the senator, while insisting he was not intoxicated, could not explain his nudity.
If I had 50 cents for every math question I got right, I'd have $6.30 by now.
If I had 50 cents for every math question I got right, I'd have $6.30 by now.
-
- Watches Jeopardy! Way Too Much
- Posts: 888
- Joined: Wed Jan 04, 2012 5:54 pm
Re: What was your online test score with an audition
The regular expression code I use is case-insensitive. I can't see why TPTB would treat responses differently based on capitalization.bpmod wrote:But not "john milton" or "John Mliton".rjaguar3 wrote:For example, a regular expression might be "(John )?Milton", indicating that either "Milton" or "John Milton" is acceptable.
As I refer to later in the post, that's just the beginning of the grading process for the SHC; non-matching responses are displayed to mark as correct or incorrect. (Sometimes I will make a new regular expression that covers multiple correct or incorrect responses.)bpmod wrote: Yes, it is possible that TPTB reject all responses that don't fit a very finite pre-compiled list of acceptable responses. Why, then, would they state that minor spelling & typographical errors and incomplete responses will be considered?
- Volante
- Harbinger of the Doomed Lemur
- Posts: 9254
- Joined: Thu Jul 14, 2011 11:42 pm
Re: What was your online test score with an audition
Something like this is probably involved too: https://en.wikipedia.org/wiki/Levenshtein_distance
The best thing that Neil Armstrong ever did, was to let us all imagine we were him.
Latest movies (1-10): Everything Everywhere All at Once (10), Ruby Gillman: Teenage Kraken (6), Black Sunday /1960/ (6), Marcel the Shell with Shoes On (7)
Latest movies (1-10): Everything Everywhere All at Once (10), Ruby Gillman: Teenage Kraken (6), Black Sunday /1960/ (6), Marcel the Shell with Shoes On (7)
- Leopardy!
- Contributor
- Posts: 18
- Joined: Sun Apr 05, 2015 12:13 pm
Re: What was your online test score with an audition
I know there's plenty to do in most large cities, but it could be that (depending on their itinerary--if they come east first) TPTB get more days in NYC to allow the testers to get over jet lag and to see more Broadway shows--a huge plus for most visitors, especially those in show biz.
- TheSunWillComeOut
- Two-Morrow
- Posts: 344
- Joined: Fri Jun 21, 2013 10:12 pm
Re: What was your online test score with an audition
Based on speaking with other future and would-be contestants at my NYC audition in 2013, I think that NYC is a smart place for TPTB to spend several days. Two words: Northeast Corridor. NYC is a transportation hub for the entire East Coast, and it's extremely easy to get to by bus and train for out-of-towners ranging from D.C. to Boston and everywhere in between. A lot of people there had started their journey at 4/5 a.m., with the plan to audition in the morning, spend an afternoon doing something fun in the city, and then head home in the evening.Leopardy! wrote:I know there's plenty to do in most large cities, but it could be that (depending on their itinerary--if they come east first) TPTB get more days in NYC to allow the testers to get over jet lag and to see more Broadway shows--a huge plus for most visitors, especially those in show biz.
- OSXpert
- Watches Jeopardy! Way Too Much
- Posts: 881
- Joined: Thu Jun 05, 2014 7:29 pm
Re: What was your online test score with an audition
And here I am sitting in Manhattan, waiting to get an invite to Detroit.
- morbeedo
- Loyal Jeopardista
- Posts: 3065
- Joined: Wed Jan 18, 2012 12:58 pm
Re: What was your online test score with an audition
LMFAOOSXpert wrote:And here I am sitting in Manhattan, waiting to get an invite to Detroit.