Pandas read_html():

• If we find data of interest in web pages, they may well be stored in tables. And pandas provides a way to read tables from a web page into pandas DataFrames.
  • This lets us write very conice Python code to collect tabular data from web pages.
• To read an HTML page’s tables into a list of DataFrames, pandas provides the read_html() function.
  • This function accepts an argument that can be the contents of an HTML page.
  • pd.read_html() uses the lxml parser so you may need to install it using pip install lxml.

The Requests Library:

• The requests library is a popular Python library for making HTTP requests to web servers. It allows you to send HTTP GET requests and handle responses easily.
  • The requests library should be used in combination with pandas’ read_html() to avoid SSL certificate verification errors and HTTP errors when screenscraping web pages.
  • To install the requests library, you can use pip install requests.

1%pip install lxml requests
2import pandas as pd
3import requests
4from io import StringIO # Wraps text as a file-like object
5
6url = 'https://en.wikipedia.org/wiki/List_of_S%26P_500_companies'
7
8headers = {
9    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
10                  "AppleWebKit/537.36 (KHTML, like Gecko) "
11                  "Chrome/120.0.0.0 Safari/537.36"
12}
13
14response = requests.get(url, headers = headers)
15dfs = pd.read_html(StringIO(response.text))

• To see what was returned in the DataFrame list dfs, we can use the type() and len() functions.

16print(type(dfs))
17print(len(dfs))

Output:

1<class 'list'>
22

• Since there are two tables returned by read_html(), we can interactively explore what was returned in dfs using the DataFrame head() method.

18print(dfs[0].head())

Output: