How to scrape vue data?

Question

What

I want to scrape vueData below, and I'm trying to resolve this, but still don't discover the way.

<script>
    var Item = function(val, errorMsg, required) {
      return {
        val: val,
        hasError: !!errorMsg,
        errorMsg: errorMsg,
        required: !!required
      }
    };
    window.vueData = {
      isError: 'false' === 'true',
      breadcrumb: [{"name":"Top","url":"/"},{"name":"List","url":"/search/companylist/"},{"name":"company1","url":"/search/companydetail/1/"}],
      pageNo: "1",
      pageList: [],
      tRagCorporate : {
  code:                "1",
  name:                "company1",
  establishDateText:   "2017/11/1",
  ceo:                 "Smith",
  employeeCount:       "200",
  capital:             "100,000,000",
  stockOpening:        "not opening",
  salesAmt:            "1,000,000,000,000",
  zipNo:               "xxx-yyyy",
  pref:                "Tokyo",
  city:                "Shinjuku",
},
      jobofferList : [
    {
      companyname: "company1",
      jobName: "Machine Learning Engineer",
      ability: "machine learning experience",
      salary: "5,000,000 - 12,000,000",
      bizPlace: "Tokyo",
    }
  ]
</script>

My processing source code is:

from selenium import webdriver

TARGET_SITE = 'https://www.hoge.com/search/companydetail/1'

def main():
  companies[1] = 'url': TARGET_SITE

  browser = webdriver.Firefox(executable_path='/path/to/geckodriver')
  browser.get(companies[1]['url'])
  r = browser.find_element_by_class_name('a-table')
  print(r)
end

if __name__ == '__main__':
    main()

tRagCorporate will be set at the table class of a-table.
I tried to scrape after composed this table, but I can't.

After this, searching for scraping vue data, but can't get info...
Please give me solution, or some hints if you have.

Supplement

python 3.7

Answer 1

I tried to scrape after composed this table, but I can't.

What does "can't" mean concretely? Can you share the actual error message you got?

In general, when using selenium, we need to wait for JavaScript code to be executed to get DOM elements built with JavaScript. I guess it's the cause of the problem.

Comments

[tmk-hsn]

Thanks for your advise. Finally, I got vue binding data by waiting until DOM elements built.
Here's fixed code.

```

import time
from selenium import webdriver

TARGET_SITE = 'https://www.hoge.com/search/companydetail/1'

def main():
companies[1] = 'url': TARGET_SITE

browser = webdriver.Firefox(executable_path='/path/to/geckodriver')
browser.get(companies[1]['url'])
time.sleep(2)
r = browser.find_element_by_class_name('a-table')
print(r.get_attribute('outerHTML'))
end

if __name__ == '__main__':
main()

```

[gh640]

Good to hear that. Thanks for sharing the code.

FYI, we can also optimize the wait time with `WebDriverWait`:

https://selenium-python.readthedocs.io/waits.html#explicit-waits

I'm not sure if this works for your case but I believe you're interested. Please take a look :)