多线程爬取某贴吧帖子

Lobolang

由于刚学没多久,技术有限,有不少错误望各位师兄师姐多多指点..

[Python] 纯文本查看 复制代码

?

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61

import requests import threading from lxml import etree from queue import Queue   headers={'User-Agent':'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Mobile Safari/537.36'}   class TiebaSpider:     def __init__(self):         self.url='http://tieba.baidu.com/f?&kw=lol&pn={}'     #本代码以lol为例子,lol可更换为其他贴吧的吧名         self.url_queen = Queue()         self.response_queen = Queue()       def get_urllist(self):         for i in range(0,5):             self.url_queen.put(self.url.format(i*30))       def get_response(self):         while True:             url = self.url_queen.get()             response=requests.get(url,headers=headers)             self.response_queen.put(response.content.decode())             self.url_queen.task_done()       def get_info(self):         while True:             html = etree.HTML(self.response_queen.get())             title=html.xpath('//div[@class="ti_title"]/span[not(contains(text(),"置顶"))][not(contains(@class,"icon"))]/text()')             title_url=html.xpath('//ul[@class="threads_list"]//a[not(contains(@class,"ti_item j_click_stats"))][not(contains(text(),"立即查看"))]/@href')             str='http://tieba.baidu.com'             url=[str+i for i in title_url]             count=0             while count<len(title):                 print('标题:%s 地址:%s' %(title[count],url[count]))                 count+=1             self.response_queen.task_done()       def run(self):         thread_list=[]         for i in range(1):                   #获取页码的线程数:1     线程数可根据自己喜好更改             t_list=threading.Thread(target=self.get_urllist)             thread_list.append(t_list)         for i in range(10):                #获取响应的线程数:10             t_response=threading.Thread(target=self.get_response)             thread_list.append(t_response)         for i in range(3):                  #采集数据的线程数:3             t_info=threading.Thread(target=self.get_info)             thread_list.append(t_info)           for t in thread_list:             t.setDaemon(True)             t.start()           for p in (self.url_queen,self.response_queen):             p.join()                                         #设置主线程等待其他线程结束再结束           print("\33[36m获取结束")   if __name__ == '__main__':     q=TiebaSpider()     q.run()

Lobolang

xpath使用不太了解,有时候遇到只有图片没有title内容的帖子就获取不到标题,然后后面标题和网址就乱了,或者就列表溢出,,,,经排查发现是获取title的xpath中的span标签为空,,,我也不知道这个怎么写,望有师兄师姐指点指点

JavaSB

谢谢分享

逆向学习

感谢分享

夏橙M兮

谢谢分享。

天气敷和

感谢热心分享！