Learning notes ： Cat's Eye top100 Movie information crawling

Get web source

Check it casually on the Internet User-agent As a browser proxy

try:
        headers = {
    
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36'
        }
        response = requests.get(url, headers=headers)

Regular expression parsing

pattern = re.compile('<dd>.*?board-index.*?>(\d+)</i>.*?data-src="(.*?)".*?name"><a'
                         + '.*?>(.*?)</a>.*?star">(.*?)</p>.*?releasetime">(.*?)</p>'
                         + '.*?integer">(.*?)</i>.*?fraction">(.*?)</i>.*?</dd>', re.S)

1. Parentheses are used in re.findall Get the required information , Pay attention to using non greedy matching .*? Avoid missing information caused by excessive matching .
2. yield function ： Instead of return, Make the returned data more tidy .

write file

json.dumps Convert dictionary to string .（ Add ：loads Convert string to dictionary ）

Cross page crawling

Observed cat's eye website top Each page of the interface has only 10 A movie message , And turn the page Url The information changes as follows ：

http://maoyan.com/board/4?
 Page turning ->
http://maoyan.com/board/4?offset=10

in other words ,offset Every addition 10 Control the page to jump back one side , Use this law to crawl ：

for i in range(10):
        main(offset=i * 10)
        print(str(i*10 + 1) + " to " + str(i*10 +10) + " got!")
        time.sleep(1) # Avoid access restrictions caused by high access frequency 

Although most of them are direct copy Examples in the book , But still full of sense of achievement ！