一个简单的demo，只爬取了标题信息，主要是连接数据库遇到的坑
执行数据库语句之后需要主动使用commit()方法提交：
Commit any pending transaction to the database.
Note that if the database supports an auto-commit feature, this must be >initially off. An interface method may be provided to turn it back on.
Database modules that do not support transactions should implement this method with void functionality.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
|
'''
@文件 :spider.py
@时间 :2020-12-18 08:57:05
@作者 :Karcyril
'''
import urllib.request
import urllib.parse
import urllib.error
from bs4 import BeautifulSoup
import pymysql
class MysqlClass:
def __init__(self, host, root, passwd, db) -> None:
self.conn = pymysql.connect(host, root, passwd, db)
self.cursor = self.conn.cursor()
def execute(self, sql):
self.cursor.execute(sql)
def commit(self):
self.conn.commit()
def close(self):
self.cursor.close()
self.conn.close()
def main():
baseUrl = 'https://movie.douban.com/top250?start='
dataList = getData(baseUrl)
mysqlClass = MysqlClass('主机', '帐号', '密码', '数据库')
for item in dataList:
mysqlClass.execute('INSERT INTO wp_posts (post_title,post_content) VALUES("%s","%s");' % (
item, '随便写点'))
mysqlClass.commit()
mysqlClass.close()
def getData(baseUrl):
dataList = []
for i in range(0, 10):
url = baseUrl+str(i*25)
html = askURL(url)
bs = BeautifulSoup(html, 'html.parser')
mList = bs.select(
'#wrapper #content .grid_view li .hd a span:nth-of-type(1)')
dataList.append('第%d页---------------' % (i+1))
for item in mList:
dataList.append(item.string)
return dataList
def askURL(baseUrl):
head = {
"User-Agent": "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Mobile Safari/537.36"}
res = urllib.request.Request(url=baseUrl, headers=head)
html = ""
try:
response = urllib.request.urlopen(res)
html = response.read().decode('utf-8')
except urllib.error.URLError as e:
if hasattr(e, "code"):
print(e.code)
if hasattr(e, 'reason'):
print(e.reason)
return html
if __name__ == "__main__":
main()
|
微信