J-Archive Web Scraper (DELETED)
Moderators: alietr, trainman, econgator, dhkendall
- SweepingDeveloper
- Loyal Jeopardista
- Posts: 127
- Joined: Fri Jan 10, 2014 12:11 pm
- Contact:
J-Archive Web Scraper (DELETED)
Web scraper has been deleted.
Last edited by SweepingDeveloper on Mon Nov 26, 2018 1:27 pm, edited 2 times in total.
- opusthepenguin
- The Best Darn Penguin on the Whole JBoard
- Posts: 10319
- Joined: Thu Aug 11, 2011 2:33 pm
- Location: Shawnee, KS
- Contact:
Re: J-Archive Web Scraper
Well you can't go wrong with the Dead Parrot sketch or Argument Clinic. After that, something completely different?
- MinnesotaMyron
- JBOARDIE OF THE MONTH!
- Posts: 3422
- Joined: Fri Jul 01, 2011 12:53 pm
Re: J-Archive Web Scraper
The Archive does that automatically. You pick a game, click on the link, and all that stuff comes right up!SweepingDeveloper wrote: ↑Sun Nov 25, 2018 3:54 pm I'm learning some Python for fun, and one of the things I stumbled upon was the Beautiful Soup web scraper. So, I decided to give it a try on the J-Archive page, and this is what I came up with:
https://github.com/SweepingDeveloper/j- ... eb_scraper
It randomly chooses a game, then lists the categories, clues, and correct responses from that day's game.
What do you think?
- SweepingDeveloper
- Loyal Jeopardista
- Posts: 127
- Joined: Fri Jan 10, 2014 12:11 pm
- Contact:
Re: J-Archive Web Scraper
Right. What my program does is find the game information from the HTML tags, puts them in lists, then displays them. It's basically copying from the generated HTML page. I'm not accessing any SQL databases or PHP scripts.MinnesotaMyron wrote: ↑Sun Nov 25, 2018 6:47 pm
The Archive does that automatically. You pick a game, click on the link, and all that stuff comes right up!
- MinnesotaMyron
- JBOARDIE OF THE MONTH!
- Posts: 3422
- Joined: Fri Jul 01, 2011 12:53 pm
Re: J-Archive Web Scraper
Why not just click on the J-Archive? Boom, there it is!SweepingDeveloper wrote: ↑Sun Nov 25, 2018 7:02 pmRight. What my program does is find the game information from the HTML tags, puts them in lists, then displays them. It's basically copying from the generated HTML page. I'm not accessing any SQL databases or PHP scripts.MinnesotaMyron wrote: ↑Sun Nov 25, 2018 6:47 pm
The Archive does that automatically. You pick a game, click on the link, and all that stuff comes right up!
- SenseiCAY
- Jeopardy Participant
- Posts: 617
- Joined: Mon Oct 30, 2017 3:56 pm
- Location: Silver Spring, MD
- Contact:
Re: J-Archive Web Scraper
I dunno if you're actually wondering or if you're just messing with SweepingDeveloper, but I'll assume the former.
If you have other applications that use the Jeopardy clues, this could be useful, so that you can get formatted text files or something that would be easy for your app to read. I scraped J-Archive for a Jeopardy game that I built a few years back - it put the clues and responses into text files that would be input into an HTML page that simulated the buzzers and lecterns - one person could play "Alex" and read the clues and control the game, and the other three players could play the game with a buzzer system that I built.
If you have other applications that use the Jeopardy clues, this could be useful, so that you can get formatted text files or something that would be easy for your app to read. I scraped J-Archive for a Jeopardy game that I built a few years back - it put the clues and responses into text files that would be input into an HTML page that simulated the buzzers and lecterns - one person could play "Alex" and read the clues and control the game, and the other three players could play the game with a buzzer system that I built.
- MinnesotaMyron
- JBOARDIE OF THE MONTH!
- Posts: 3422
- Joined: Fri Jul 01, 2011 12:53 pm
Re: J-Archive Web Scraper
Oh.SenseiCAY wrote: ↑Mon Nov 26, 2018 10:35 am I dunno if you're actually wondering or if you're just messing with SweepingDeveloper, but I'll assume the former.
If you have other applications that use the Jeopardy clues, this could be useful, so that you can get formatted text files or something that would be easy for your app to read. I scraped J-Archive for a Jeopardy game that I built a few years back - it put the clues and responses into text files that would be input into an HTML page that simulated the buzzers and lecterns - one person could play "Alex" and read the clues and control the game, and the other three players could play the game with a buzzer system that I built.
- SenseiCAY
- Jeopardy Participant
- Posts: 617
- Joined: Mon Oct 30, 2017 3:56 pm
- Location: Silver Spring, MD
- Contact:
Re: J-Archive Web Scraper
My scraper probably took a day or so to write and test, and then 45 minutes to scrape the archive of a few thousand games. I don't think I could have beaten that by typing them in myself.MinnesotaMyron wrote: ↑Mon Nov 26, 2018 11:50 amWouldn’t it have been easier just to get copies of the games and type in the clues yourself?SenseiCAY wrote: ↑Mon Nov 26, 2018 10:35 am I dunno if you're actually wondering or if you're just messing with SweepingDeveloper, but I'll assume the former.
If you have other applications that use the Jeopardy clues, this could be useful, so that you can get formatted text files or something that would be easy for your app to read. I scraped J-Archive for a Jeopardy game that I built a few years back - it put the clues and responses into text files that would be input into an HTML page that simulated the buzzers and lecterns - one person could play "Alex" and read the clues and control the game, and the other three players could play the game with a buzzer system that I built.
- MinnesotaMyron
- JBOARDIE OF THE MONTH!
- Posts: 3422
- Joined: Fri Jul 01, 2011 12:53 pm
Re: J-Archive Web Scraper
Me either. It took, like, years to type them all in.SenseiCAY wrote: ↑Mon Nov 26, 2018 11:52 amMy scraper probably took a day or so to write and test, and then 45 minutes to scrape the archive of a few thousand games. I don't think I could have beaten that by typing them in myself.MinnesotaMyron wrote: ↑Mon Nov 26, 2018 11:50 amWouldn’t it have been easier just to get copies of the games and type in the clues yourself?SenseiCAY wrote: ↑Mon Nov 26, 2018 10:35 am I dunno if you're actually wondering or if you're just messing with SweepingDeveloper, but I'll assume the former.
If you have other applications that use the Jeopardy clues, this could be useful, so that you can get formatted text files or something that would be easy for your app to read. I scraped J-Archive for a Jeopardy game that I built a few years back - it put the clues and responses into text files that would be input into an HTML page that simulated the buzzers and lecterns - one person could play "Alex" and read the clues and control the game, and the other three players could play the game with a buzzer system that I built.
- Robert K S
- Jeopardy! Champion
- Posts: 5239
- Joined: Thu Jun 30, 2011 1:26 pm
- Location: Cleveland, Ohio
- Contact:
Re: J-Archive Web Scraper
^ Like
Please don't scrape the Archive. Aside from the simple matter of being respectful to the archivists who spend a lot of time to input the data, there are controls built into the server that I have no control over that may blacklist you. I then have to go through the rigmarole to address blacklist removal requests. I don't like to spend time doing this.
Please don't scrape the Archive. Aside from the simple matter of being respectful to the archivists who spend a lot of time to input the data, there are controls built into the server that I have no control over that may blacklist you. I then have to go through the rigmarole to address blacklist removal requests. I don't like to spend time doing this.
- SweepingDeveloper
- Loyal Jeopardista
- Posts: 127
- Joined: Fri Jan 10, 2014 12:11 pm
- Contact: