J-Archive Web Scraper (DELETED)

This is where all of the games are discussed.

Moderators: alietr, trainman, econgator, dhkendall

Post Reply
User avatar
SweepingDeveloper
Loyal Jeopardista
Posts: 127
Joined: Fri Jan 10, 2014 12:11 pm
Contact:

J-Archive Web Scraper (DELETED)

Post by SweepingDeveloper »

Web scraper has been deleted.
Last edited by SweepingDeveloper on Mon Nov 26, 2018 1:27 pm, edited 2 times in total.
User avatar
opusthepenguin
The Best Darn Penguin on the Whole JBoard
Posts: 10319
Joined: Thu Aug 11, 2011 2:33 pm
Location: Shawnee, KS
Contact:

Re: J-Archive Web Scraper

Post by opusthepenguin »

SweepingDeveloper wrote: Sun Nov 25, 2018 3:54 pm I'm learning some Python for fun,
Well you can't go wrong with the Dead Parrot sketch or Argument Clinic. After that, something completely different?
User avatar
MinnesotaMyron
JBOARDIE OF THE MONTH!
Posts: 3422
Joined: Fri Jul 01, 2011 12:53 pm

Re: J-Archive Web Scraper

Post by MinnesotaMyron »

SweepingDeveloper wrote: Sun Nov 25, 2018 3:54 pm I'm learning some Python for fun, and one of the things I stumbled upon was the Beautiful Soup web scraper. So, I decided to give it a try on the J-Archive page, and this is what I came up with:

https://github.com/SweepingDeveloper/j- ... eb_scraper

It randomly chooses a game, then lists the categories, clues, and correct responses from that day's game.

What do you think?
The Archive does that automatically. You pick a game, click on the link, and all that stuff comes right up!
User avatar
SweepingDeveloper
Loyal Jeopardista
Posts: 127
Joined: Fri Jan 10, 2014 12:11 pm
Contact:

Re: J-Archive Web Scraper

Post by SweepingDeveloper »

MinnesotaMyron wrote: Sun Nov 25, 2018 6:47 pm
The Archive does that automatically. You pick a game, click on the link, and all that stuff comes right up!
Right. What my program does is find the game information from the HTML tags, puts them in lists, then displays them. It's basically copying from the generated HTML page. I'm not accessing any SQL databases or PHP scripts.
User avatar
MinnesotaMyron
JBOARDIE OF THE MONTH!
Posts: 3422
Joined: Fri Jul 01, 2011 12:53 pm

Re: J-Archive Web Scraper

Post by MinnesotaMyron »

SweepingDeveloper wrote: Sun Nov 25, 2018 7:02 pm
MinnesotaMyron wrote: Sun Nov 25, 2018 6:47 pm
The Archive does that automatically. You pick a game, click on the link, and all that stuff comes right up!
Right. What my program does is find the game information from the HTML tags, puts them in lists, then displays them. It's basically copying from the generated HTML page. I'm not accessing any SQL databases or PHP scripts.
Why not just click on the J-Archive? Boom, there it is!
User avatar
SenseiCAY
Jeopardy Participant
Posts: 617
Joined: Mon Oct 30, 2017 3:56 pm
Location: Silver Spring, MD
Contact:

Re: J-Archive Web Scraper

Post by SenseiCAY »

I dunno if you're actually wondering or if you're just messing with SweepingDeveloper, but I'll assume the former.

If you have other applications that use the Jeopardy clues, this could be useful, so that you can get formatted text files or something that would be easy for your app to read. I scraped J-Archive for a Jeopardy game that I built a few years back - it put the clues and responses into text files that would be input into an HTML page that simulated the buzzers and lecterns - one person could play "Alex" and read the clues and control the game, and the other three players could play the game with a buzzer system that I built.
User avatar
MinnesotaMyron
JBOARDIE OF THE MONTH!
Posts: 3422
Joined: Fri Jul 01, 2011 12:53 pm

Re: J-Archive Web Scraper

Post by MinnesotaMyron »

SenseiCAY wrote: Mon Nov 26, 2018 10:35 am I dunno if you're actually wondering or if you're just messing with SweepingDeveloper, but I'll assume the former.

If you have other applications that use the Jeopardy clues, this could be useful, so that you can get formatted text files or something that would be easy for your app to read. I scraped J-Archive for a Jeopardy game that I built a few years back - it put the clues and responses into text files that would be input into an HTML page that simulated the buzzers and lecterns - one person could play "Alex" and read the clues and control the game, and the other three players could play the game with a buzzer system that I built.
Oh.
User avatar
SenseiCAY
Jeopardy Participant
Posts: 617
Joined: Mon Oct 30, 2017 3:56 pm
Location: Silver Spring, MD
Contact:

Re: J-Archive Web Scraper

Post by SenseiCAY »

MinnesotaMyron wrote: Mon Nov 26, 2018 11:50 am
SenseiCAY wrote: Mon Nov 26, 2018 10:35 am I dunno if you're actually wondering or if you're just messing with SweepingDeveloper, but I'll assume the former.

If you have other applications that use the Jeopardy clues, this could be useful, so that you can get formatted text files or something that would be easy for your app to read. I scraped J-Archive for a Jeopardy game that I built a few years back - it put the clues and responses into text files that would be input into an HTML page that simulated the buzzers and lecterns - one person could play "Alex" and read the clues and control the game, and the other three players could play the game with a buzzer system that I built.
Wouldn’t it have been easier just to get copies of the games and type in the clues yourself?
My scraper probably took a day or so to write and test, and then 45 minutes to scrape the archive of a few thousand games. I don't think I could have beaten that by typing them in myself.
User avatar
MinnesotaMyron
JBOARDIE OF THE MONTH!
Posts: 3422
Joined: Fri Jul 01, 2011 12:53 pm

Re: J-Archive Web Scraper

Post by MinnesotaMyron »

SenseiCAY wrote: Mon Nov 26, 2018 11:52 am
MinnesotaMyron wrote: Mon Nov 26, 2018 11:50 am
SenseiCAY wrote: Mon Nov 26, 2018 10:35 am I dunno if you're actually wondering or if you're just messing with SweepingDeveloper, but I'll assume the former.

If you have other applications that use the Jeopardy clues, this could be useful, so that you can get formatted text files or something that would be easy for your app to read. I scraped J-Archive for a Jeopardy game that I built a few years back - it put the clues and responses into text files that would be input into an HTML page that simulated the buzzers and lecterns - one person could play "Alex" and read the clues and control the game, and the other three players could play the game with a buzzer system that I built.
Wouldn’t it have been easier just to get copies of the games and type in the clues yourself?
My scraper probably took a day or so to write and test, and then 45 minutes to scrape the archive of a few thousand games. I don't think I could have beaten that by typing them in myself.
Me either. It took, like, years to type them all in.
User avatar
Robert K S
Jeopardy! Champion
Posts: 5239
Joined: Thu Jun 30, 2011 1:26 pm
Location: Cleveland, Ohio
Contact:

Re: J-Archive Web Scraper

Post by Robert K S »

^ Like

Please don't scrape the Archive. Aside from the simple matter of being respectful to the archivists who spend a lot of time to input the data, there are controls built into the server that I have no control over that may blacklist you. I then have to go through the rigmarole to address blacklist removal requests. I don't like to spend time doing this.
User avatar
SweepingDeveloper
Loyal Jeopardista
Posts: 127
Joined: Fri Jan 10, 2014 12:11 pm
Contact:

Re: J-Archive Web Scraper

Post by SweepingDeveloper »

My apologies. It's been deleted.
Post Reply