看大家发的小说爬虫，本人爬虫小白写的

nianboy

我是爬虫小白，之前搞得给我老爹下小说的，大佬看了不要喷

[Python] 纯文本查看 复制代码

?

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57

# -*- coding: utf-8 -*- import requests import re import os from lxml import etree from urllib.parse import quote   if __name__ == '__main__':     keyword = input("请输入书籍名：").encode("gb2312")     url = "https://www.tingchina.com/search1.asp?keyword=" + quote(keyword)     headers = {         "user-agent": "Mozilla/7.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"     }     page_text = requests.get(url=url, headers=headers).text.encode('iso-8859-1').decode('gbk')     tree = etree.HTML(page_text)     booksname = tree.xpath('/html/body/div[2]/div[2]/dl/dd/ul/li/a/text()')     booknum = tree.xpath('/html/body/div[2]/div[2]/dl/dd/ul/li/a/@href')     result = ''.join(booknum)     booknumber = re.findall(r'yousheng/disp_(.*?).htm', result)     for name,num in zip(booksname, booknumber):         print("书名：" + name)         print("编号：" + num)     pagenum=input("请输入书籍编号:")     page_text1 = requests.get(url="https://www.tingchina.com/yousheng/"+str(pagenum)+"/play_"+str(pagenum)+"_0.htm", headers=headers).text.encode('iso-8859-1').decode('gbk')     pagenum1 = re.findall(r'play+_+\d+_(\d+)', page_text1)     m = (int(pagenum1[len(pagenum1)-2]) + 1)     print("总章节数："+str(m))     firstnum=input("请输入开始下载章数")     endnum=input("请输入结束下载章数") for page in range(int(firstnum)-1,int(endnum)):     page=str(page)     indexurl="https://www.tingchina.com/yousheng/"+str(pagenum)+"/play_"+str(pagenum)+"_"+page+".htm"     url="https://img.tingchina.com/play/h5_jsonp.asp?0.9091809774033375"     headers={         "User-Agent": "Mozilla/7.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36",         "Referer": "https://www.tingchina.com/yousheng/"+str(pagenum)+"/play_"+str(pagenum)+"_5.htm",     }     getbook=requests.get(url=indexurl,headers=headers,verify=False).text.encode('iso-8859-1').decode('gbk')     sonurl=re.findall(r'fileUrl= "(.*?)"',getbook)[0]     name=re.findall(r'fileUrl= "/yousheng/(.*?).mp3"',getbook)[0]     bookname=re.findall(r'如果您喜欢的话，请为(.*?).mp3投一票',getbook)[0]     book=re.findall(r';"><strong>(.*?)</strong>',getbook)[0]     bookshu=re.findall(r'.htm">(.*?)</ul>',getbook)[0]     getshu=requests.get(url=url,headers=headers,verify=False).text     son=re.findall(r'"(.*?)";',getshu)[0].replace('" +  "',"")     bookurl="https://t33.tingchina.com"+sonurl+son     headers1={         "Referer": "http://www.23ts.com/",         "User-Agent": "Mozilla/6.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36",     }     getdj=requests.get(url=bookurl,headers=headers,verify=False).content     file_name="./"+book+"/"     if not os.path.exists(file_name):         os.mkdir(file_name)     with open(file_name+"{}.mp3".format(bookname),"wb") as f:         f.write(getdj)         print(bookname+"-----------下载成功！")

jiatao

一提到爬虫大家都用python哈，看来应该考虑封装一个通用的爬虫函数了

ericzhao666

新人报道，之前见过一个人开发的网页版的，自动爬取小说加入书架，然后会做本地缓存，存储书和书签和阅读到哪，支持kindle，可惜后来地址无法访问了

nianboy

昨天刚注册的吾爱，之前就经常看吾爱的帖子，本人技术有限，写的东西也是乱七八糟，大佬们看了不要嫌我菜

xiaoming123456

经常看吾爱的帖子

lzn223568

新人报道，好好学习

lye123456

你写的这个爬虫如何用？

zkz6969

加个多线程就不错了

wsz12312

新人报道

testc0de

厉害。我还是几年前写过，为了方便看天涯的小说。

52pojie66

这是用来下哪个网站的呀，某点可以吗