asked    Archer     2018-10-22       javascript       81 view        2 Answers

[SOLVED] Scrape a text that was written by javascript from website

I am using BeautifulSoup to scrape character info from a website. When trying to get the win rate of a character, BeautifulSoup can't find it.

When I inspect the text, it is listed as under . All I can find in the sites source code and all that BeautifulSoup finds is "ranking-stats-placeholder".

This is the code I am currently using.

import bs4
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = "https://u.gg/lol/champions/darius/build/?role=top"

#opening up connection, grabbing the page
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()

#html parsing
page_soup = soup(page_html, "html.parser")

#champion name
champ_name = page_soup.findAll("span", {"class":"champion-name"})[0].text

#champion win rate
champ_wr = page.soup.findAll("div", {"class":"win-rate okay-tier"})

I believe that the win rate text is added by javascript, but I have no idea how I can get the text. The code I currently have returns "None" for champ_wr

  2 Answers  

        answered    Coral     2018-10-22      

Although this text technically could be in the javascript itself, my first guess is the JS is pulling it in via an ajax request. Have your program simulate that and you'll probably get all the data you need handed right to you without any scraping involved!

It will take a little detective work though. I suggest turning on your network traffic logger (such as "Web Developer Toolbar" in Firefox) and then visiting the site. Focus your attention attention on any/all XmlHTTPRequests.

Best of luck!



        answered    Giles     2018-10-22      

not sure how tied to BeautifulSoup you are, but I can get selenium doing useful things with:

# load code from selenium package
from selenium.webdriver import Remote
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

# start an instance of Chrome up
chrome = Service('/usr/local/bin/chromedriver')
chrome.start()
driver = Remote(chrome.service_url)

# get the page loading
driver.get("https://u.gg/lol/champions/darius/build/?role=top")

# wait for the win rate to be populated
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "win-rate")))

# get the values you wanted
name = driver.find_element_by_class_name("champion-name").text
winrate = driver.find_element_by_class_name("win-rate").text

# display them
print(f"name: {repr(name)}, winrate: {winrate.split()[0]}")

# clean up a bit
driver.quit()




Your Answer





 2018-10-22         Leo

Rotate transformation on matplotlib axis in subplot

I am trying to create a four-panel figure where the bottom-left panel contains a scatter plot, and the other three panels contain histograms. The top-left will be a standard histogram across the x-dimension of the scatter, the bottom-right will be a 90° rotated histogram for the y-dimension. Both of these are easy to do in matplotlib.I am running into problems with the third histogram, which is to be a 45° rotated plot in the top-right of the figure giving the distribution of the differences between the x and y points. I have made such figures before by manually rotating an...
 python                     1 answers                     62 view
 2018-10-22         Jill

How to `update_all` based on a subquery in Rails

I'm trying to achieve something pretty simple in PostgreSQL the Rails way.Say you have a User model with 3 columns id, cached_segment, cached_step.Say you already have a complex query that calculates segment and query on the fly, encapsulated in a scope User.decorate_with_segment_and_step. This returns an ActiveRecord relation, same as User, but with 3 additional columns:id cached_segment cached_step segment step cache_invalid1 NULL NULL segment_1 step_1 TRUE2 segment_1 step_2 segment_1 step_2 FALSE3 ...The SQL I would like to gen...
 ruby-on-rails                     1 answers                     63 view
 2018-10-22         Emmanuel

GTest: fixture required when TYPE_TESTING global functions?

I want to unit-test some global templated functions using TYPED_TEST. The following code works, I just wonder whether there is way I can get rid of the test fixture, as it does not seem to be needed..#include <gtest/gtest.h>#include <base/mathfunctions.h>template <class T>class MinTest : public testing::Test {};// The list of types we want to test.typedef ::testing::Types<int, float> Implementations;TYPED_TEST_CASE(MinTest, Implementations);TYPED_TEST(MinTest, ReturnsMinimumValue) { EXPECT_EQ(Base::Min<TypeParam>(-5, 5), -5);} ... as it ...
 unit-testing                     1 answers                     61 view