Beautifulsoup Does Not Read 'full' Html Obtained By Requests

I am trying to scrape URL's from a website presented as HTML using the BeautifulSoup and requests libraries. I am running both of them on Python 3.5. It seems I am succesfully gett

Solution 1:

This is because the page you're scraping is dynamic. Meaning that its content is generated with JavaScript and it takes some times to fully render it (not initially present statically).

You should use something like Selenium or Puppeteer to load the page, wait for it to fully render, then scrape the content you need to extract.