Question

I m trying to build a simple web scrapping tool. Right now I m having an issue extracting data from each row because <tr> header is missing. (Only <tr> header is missing, and < r> header is still there)

下面是我的法典

from bs4 import BeautifulSoup
import requests

url = "https://companiesmarketcap.com/dow-jones/largest-companies-by-market-cap/"
data = requests.get(url).text
print(data)

它缺少一个头盔,只有每个行人都有。

<tbody>
((THERES SUPPOSED TO BE A <tr> TAG HERE))))!!!
<td class="fav"><img alt="favorite icon" src="/img/fav.svg?v2" data-id="2"></td>
</td><td class="rank-td td-right" data-sort="1">1
</td><td class="name-td">
<div class="logo-container"><img loading="lazy" class="company-logo" alt="Apple logo" src="/img/company-logos/64/AAPL.png" data-img-path="/img/company-logos/64/AAPL.png" data-img-dark-path="/img/company-logos/64/AAPL.D.png"></div>
<div class="name-div"><a href="/apple/marketcap/"><div class="company-name">Apple</div>
<div class="company-code"><span class="rank d-none"></span>AAPL</div>
</a></div></td><td class="td-right" data-sort="2891576508416">$2.891 T</td><td class="td-right" data-sort="18592">$185.92</td><td data-sort="18" class="rh-sm"><span class="percentage-green"><svg class="a" viewBox="0 0 12 12"><path d="M10 8H2l4-4 4 4z"></path></svg>0.18%</span></td><td class="p-0 sparkline-td red"><svg><path d="M0,21 5,18 10,22 15,14 20,16 25,12 30,8 35,14 40,11 45,3 50,3 55,4 60,8 65,6 70,10 75,11 80,13 85,13 90,14 95,14 100,13 105,16 110,16 115,31 120,34 125,39 130,41 135,31 140,32 145,30 150,31 155,30" /></svg></td><td>?? <span class="responsive-hidden">USA</span></td>
</tr>

谢谢!

+ I tried following

soup = BeautifulSoup(data, "lxml")
table = soup.find("table")
# print(table)
rows = table.find_all("tr")

但它只做少量工作,因为再一次,<tr>。头盔失踪

Answer 1

问题是该网页的超文本已经过时。因此,将其同浏览器一样使用html5lib。 parser:

import requests
from bs4 import BeautifulSoup

url =  https://companiesmarketcap.com/dow-jones/largest-companies-by-market-cap/ 

soup = BeautifulSoup(requests.get(url).content,  html5lib )

for tr in soup.table.select( tr ):
    tds = [t for td in tr.select( td ) if (t:=td.get_text(strip=True, separator=   ))]
    if len(tds) == 6:
        print(*tds, sep= 	 )

印刷:

1       Apple AAPL      $2.891 T        $185.92 0.18%   ?? USA
2       Microsoft MSFT  $2.887 T        $388.47 1.00%   ?? USA
3       Visa V  $542.91 B       $264.17 0.05%   ?? USA
4       JPMorgan Chase JPM      $488.72 B       $169.05 0.73%   ?? USA
5       UnitedHealth UNH        $482.35 B       $521.51 3.37%   ?? USA
6       Walmart WMT     $434.31 B       $161.32 0.13%   ?? USA
7       Johnson & Johnson JNJ   $390.91 B       $162.39 0.77%   ?? USA
8       Procter & Gamble PG     $354.94 B       $150.60 0.06%   ?? USA

...

Answer 2

There is header above tbody (table > thead). You don t need header if you want to extract data in the table. Just refer to the "loc", which is CSS selector and can be used in BS4 select, in the following xml to extract the data in the table:

  <actions>
    <action_goto url="https://companiesmarketcap.com/dow-jones/largest-companies-by-market-cap/" />
    <action_loopineles>
      <element loc="table.dataTable > tbody > tr:not([class*=sort])" />
        <action_extract tabname="dat_00000000000012ab">
          <column_element colname="c01" nickname="rank">
            <element loc="td.rank-td" />
            <elecontent_text top="true" />
          </column_element>
          <column_element colname="c02" nickname="name">
            <element loc="td.name-td div.company-name" />
          </column_element>
          <column_element colname="c03" nickname="marketCap">
            <element loc="tr > td:nth-child(4)" />
          </column_element>
          <column_element colname="c04" nickname="price">
            <element loc="tr > td:nth-child(5)" />
          </column_element>
          <column_element colname="c05" nickname="today">
            <element loc="tr > td:nth-child(5)" />
          </column_element>
          <column_element colname="c06" nickname="country">
            <element loc="tr > td:nth-child(8)" />
          </column_element>
        </action_extract>
    </action_loopineles>
  </actions>

Sample of extracted data:

友情链接