English 中文(简体)
Bulk Scraping Tik Tok View,来自最近20个员额
原标题:Bulk Scraping TikTok View Count from 20 most recent posts

我正试图制作一本书,以输入Tik Tok账户的链接,每个账户在最近20个录像中分别有20个。 随附的法典是无所作为的。 任何帮助都得到真正的赞赏:

from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
import time
import json

#reload etc
def scroll_load(driver, scrolls=3):
    for _ in range(scrolls):
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(2) # pause between scrolls

#scrape view count
def get_view_counts(driver, url):
    driver.get(url)
    time.sleep(4)
    scroll_load(driver, scrolls=5)

    view_counts_elements = driver.find_elements(By.XPATH,  //*[@id="main-content-others_homepage"]/div/div[2]/div[2]/div/div[1]/div[1]/div/div/a/div/div[2]/strong )[:20] # update  view-count-class 
    view_counts = [el.text for el in view_counts_elements]

    return view_counts

def save_view_counts(urls, filename):
    data = {}
    service = Service(ChromeDriverManager().install())
    driver = webdriver.Chrome(service=service)

    for url in urls:
        data[url] = get_view_counts(driver, url)
        time.sleep(3) # delay between requests

    driver.quit()

    # save data
    with open(filename,  w ) as f:
        f.write(json.dumps(data, indent=4))

# urls 2 scrape
urls = [
     https://www.tiktok.com/@netflix ,
    # ...
]

save_view_counts(urls,  views.txt )

我预计,每个账户的数据将予以保存,指出每个账户的“URL”账户,并将计算在其最近的职位上。 然而,该产出只是一种ur,没有数据。

问题回答

我认为,该网页有至少20个最近的录像。 因此,如果你只想收看20个最近的录像,我就没有必要进行滚动。

在这方面,你可以尝试:

from selenium.webdriver import Chrome
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
import selenium.webdriver.support.expected_conditions as EC
import time
import json

def save_view_counts(urls, filename):
    data = {}
    driver = Chrome(service=Service(ChromeDriverManager().install()))

    for url in urls:

        driver.get(url)
        recent_videos = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,  strong[data-e2e="video-views"] )))
        print(f"number of recent videos: {len(recent_videos)}")
        data[url] = [i.get_attribute( innerHTML ) for i in recent_videos]

        time.sleep(3) # delay between requests

    driver.quit()
    print(data)

    # save data
    with open(filename,  w ) as f:
        f.write(json.dumps(data, indent=4))

# urls 2 scrape
urls = [
     https://www.tiktok.com/@netflix ,
     https://www.tiktok.com/@twitter 
]

save_view_counts(urls,  views.txt )

产出:

number of recent videos: 34
number of recent videos: 23
{ https://www.tiktok.com/@netflix : [ 99.7K ,  136.7K ,  27.6K ,  18.1K ,  12.8K ,  7670 ,  87K ,  15.8K ,  14.5K ,  102.1K ,  25.7K ,  203.2K ,  4.1M ,  43K ,  32.9K ,  101.5K ,  2.3M ,  233K ,  440.9K ,  92.4K ,  25.9K ,  53.3K ,  33.3K ,  449.5K ,  92K ,  53.2K ,  215.5K ,  32.1K ,  1.6M ,  415K ,  224K ,  319.1K ,  469.8K ,  420.1K ],  https://www.tiktok.com/@twitter : [ 361.4K ,  138.5K ,  54.4K ,  169.3K ,  67.6K ,  90.4K ,  4.6M ,  115.4K ,  48.4K ,  45.6K ,  73K ,  223.8K ,  107K ,  11.8M ,  155.7K ,  100K ,  1.4M ,  94.6K ,  55.3K ,  67.4K ,  48K ,  40.7K ,  40.4K ]}




相关问题
Can Django models use MySQL functions?

Is there a way to force Django models to pass a field to a MySQL function every time the model data is read or loaded? To clarify what I mean in SQL, I want the Django model to produce something like ...

An enterprise scheduler for python (like quartz)

I am looking for an enterprise tasks scheduler for python, like quartz is for Java. Requirements: Persistent: if the process restarts or the machine restarts, then all the jobs must stay there and ...

How to remove unique, then duplicate dictionaries in a list?

Given the following list that contains some duplicate and some unique dictionaries, what is the best method to remove unique dictionaries first, then reduce the duplicate dictionaries to single ...

What is suggested seed value to use with random.seed()?

Simple enough question: I m using python random module to generate random integers. I want to know what is the suggested value to use with the random.seed() function? Currently I am letting this ...

How can I make the PyDev editor selectively ignore errors?

I m using PyDev under Eclipse to write some Jython code. I ve got numerous instances where I need to do something like this: import com.work.project.component.client.Interface.ISubInterface as ...

How do I profile `paster serve` s startup time?

Python s paster serve app.ini is taking longer than I would like to be ready for the first request. I know how to profile requests with middleware, but how do I profile the initialization time? I ...

Pragmatically adding give-aways/freebies to an online store

Our business currently has an online store and recently we ve been offering free specials to our customers. Right now, we simply display the special and give the buyer a notice stating we will add the ...

Converting Dictionary to List? [duplicate]

I m trying to convert a Python dictionary into a Python list, in order to perform some calculations. #My dictionary dict = {} dict[ Capital ]="London" dict[ Food ]="Fish&Chips" dict[ 2012 ]="...

热门标签