Question

I made a script to scrape data from a website. Everything was fine until today when I ran my script and didn t get any html output from Beautiful Soup. The code is:

import requests
from bs4 import BeautifulSoup

url =  https://www.immobiliare.it 
headers = {     "User-Agent" : "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"   }

response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content,  html.parser )
soup.encode( utf-8 )

print(soup)

我甚至利用灯塔进行审判:

soup = BeautifulSoup(response.content,  lxml )

但是,产出中的所有Im都只是一个数字,总是在45,105,116,34,125,60,47,115,99,114,105,112,116,62,60,47,98,111,100,121,62,60,47,104,116,109,108,62。

I even tried changing headers and IP but nothing changes. When I print the STATUS code it shows 200 though.

It doesn t show html body and tags, it just shows numbers and commas. It doesn t even have <html> and <body> tags. This problem happens only one this website even though the same script was working few days ago.

Answer 1

I believe the problem is the user-agent.
Without user-agent, I was able to get proper result.
Also, you should use

soup = soup.encode( utf-8 )

而不是

soup.encode( utf-8 )

to set the encoded html into soup.
Here is the final code.

import requests
from bs4 import BeautifulSoup

url =  https://www.immobiliare.it 

response = requests.get(url)
soup = BeautifulSoup(response.content,  html.parser )
soup = soup.encode( utf-8 )

print(soup)

结果

友情链接