Python学习基础笔记三十四——爬虫小例子
创始人
2024-03-29 15:14:09
0次
一个爬虫小例子：
import requestsimport re
import jsondef getPage(url):response=requests.get(url)return response.textdef parsePage(s):com=re.compile('.*?.*?(?P\d+).*?(?P.*?)</span>''.*?<span class="rating_num" .*?>(?P<rating_num>.*?)</span>.*?<span>(?P<comment_num>.*?)评价</span>',re.S)ret=com.finditer(s)for i in ret:yield {"id":i.group("id"),"title":i.group("title"),"rating_num":i.group("rating_num"),"comment_num":i.group("comment_num"),}def main(num):url='https://movie.douban.com/top250?start=%s&filter='%numresponse_html=getPage(url)ret=parsePage(response_html)print(ret)f=open("move_info7","a",encoding="utf8")for obj in ret:print(obj)data=json.dumps(obj,ensure_ascii=False)f.write(data+"\n")f.close()if __name__ == '__main__':count=0for i in range(10):main(count)count+=25</code></pre>
<p>但是这个例子我跑结果的时候出现问题，没有得到返回结果，我进行了单步调试：</p>
<p><img alt="" height="202" src="https://img.pic99.top/linuxoffice369/202403/cda24c00261564b.png" width="1031" /></p>
<p>看到reponse的返回值是418，百度下这个应该是网站的反爬程序返回的。所以，这个程序要进行下修改：</p>
<pre><code class="language-python">import requests
import urllib.request
import redef getPage(url):herders = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1;WOW64) AppleWebKit/537.36 (KHTML,like GeCKO) Chrome/45.0.2454.85 Safari/537.36 115Broswer/6.0.3','Referer': 'https://movie.douban.com/','Connection': 'keep-alive'}req = urllib.request.Request(url, headers=herders)response = urllib.request.urlopen(req)html = response.read().decode('utf-8')return htmldef parsePage(s):com = re.compile('<div class="item">.*?<div class="pic">.*?<em .*?>(?P<id>\d+).*?<span class="title">(?P<title>.*?)</span>''.*?<span class="rating_num" .*?>(?P<rating_num>.*?)</span>.*?<span>(?P<comment_num>.*?)评价</span>', re.S)ret = com.finditer(s)for i in ret:yield {"id": i.group("id"),"title": i.group("title"),"rating_num": i.group("rating_num"),"comment_num": i.group("comment_num"),}def main(num):url = 'https://movie.douban.com/top250?start=%s&filter=' % numresponse_html = getPage(url)ret = parsePage(response_html)for obj in ret:print(str(obj))count = 0
for i in range(10):main(count)count += 25</code></pre>
<p>我们再来分析下正则表达式这块代码：</p>
<pre><code class="language-python">'<div class="item">.*?<div class="pic">.*?<em .*?>(?P<id>\d+).*?<span class="title">(?P<title>.*?)</span>''.*?<span class="rating_num" .*?>(?P<rating_num>.*?)</span>.*?<span>(?P<comment_num>.*?)评价</span>'</code></pre>
<p> 上面这段是正则代码：</p>
<p>1）</p>
<p>.*? ：这是一个惰性匹配，只匹配到之前标签的这个位置。</p>
<p><img alt="" height="652" src="https://img.pic99.top/linuxoffice369/202403/9983be6605b8412.png" width="1200" /></p>
<p> 2）然后再看正则表达式：</p>
<p>(?P<id>\d+): 这是一个分组，然后里面\d是表示的数字，后面的加号，说明是多个数字。</p>
<p>前面的?P<id>是给这个分组一个名称。那么通过group(n)就可以获得这个值。</p>
<p>3）compile返回的com是一个正则表达式对象。然后该对象执行finditer，由于找的内容比较多，我们就使用迭代器。</p>
<p>4）然后我们就返回每个分组，每个分组都有自己的名称。我们没有使用return，而是使用的yield，那说明这个函数是生成器。这样就不会一下子占用很多内存，而是你边生成边获取。节省了内存空间。</p>
<p>5）再来看下爬虫程序的整个过程:</p>
<p>1. url从网页上将代码搞下来；</p>
<p>2、bytes code ->utf-8 网页内容就是我们的待匹配字符串；</p>
<p>3、ret是所有匹配的内容组成的列表；</p>
<p>6）理解下正则的用法。</p> <!--end::Text-->
</div>
<!--end::Description-->
<div class="mt-5">
<!--关键词搜索-->
<a href="/index.php?s=news&c=search&keyword=%E8%AF%8D%E5%BA%93%E5%8A%A0%E8%BD%BD%E9%94%99%E8%AF%AF%3A%E6%9C%AA%E8%83%BD%E6%89%BE%E5%88%B0%E6%96%87%E4%BB%B6%E2%80%9CE%3A%5Chighferrum_mysql%5CConfiguration%5CDict_Stopwords.txt%E2%80%9D%E3%80%82" class="badge badge-light-primary fw-bold my-2" target="_blank">词库加载错误:未能找到文件“E:\highferrum_mysql\Configuration\Dict_Stopwords.txt”。</a>
</div>
<div class="mt-5">
<p class="fc-show-prev-next">
<strong>上一篇：</strong><a href="/code/14053.html">Java集合 超详细版+面试题</a><br>
</p>
<p class="fc-show-prev-next">
<strong>下一篇：</strong><a href="/code/14055.html">Postman 的简单使用</a> </p>
</div>
<!--begin::Block-->
<div class="d-flex flex-stack mb-2 mt-10">
<!--begin::Title-->
<h3 class="text-dark fs-5 fw-bold text-gray-800">相关内容</h3>
<!--end::Title-->
</div>
<div class="separator separator-dashed mb-9"></div>
<!--end::Block-->
<div class="row g-10">
</div>
</div>
<!--end::Table widget 14-->
</div>
<!--end::Col-->
<!--begin::Col-->
<div class="col-xl-4 mt-0">
<!--begin::Chart Widget 35-->
<div class="card card-flush h-md-100">
<!--begin::Header-->
<div class="card-header pt-5 ">
<!--begin::Title-->
<h3 class="card-title align-items-start flex-column">
<!--begin::Statistics-->
<div class="d-flex align-items-center mb-2">
<!--begin::Currency-->
<span class="fs-5 fw-bold text-gray-800 ">热门资讯</span>
<!--end::Currency-->
</div>
<!--end::Statistics-->
</h3>
<!--end::Title-->
</div>
<!--end::Header-->
<!--begin::Body-->
<div class="card-body pt-3">
<!--begin::Item-->
<div class="d-flex flex-stack mb-7">
<!--begin::Symbol-->
<div class="symbol symbol-60px symbol-2by3 me-4">
<div class="symbol-label" style="background-image: url('/uploadfile/202506/479a03325f01a.png')"></div>
</div>
<!--end::Symbol-->
<!--begin::Title-->
<div class="m-0">
<a href="/code/203145.html" class="text-dark fw-bold text-hover-primary fs-6">前端-session、jwt</a>
<span class="text-gray-600 fw-semibold d-block pt-1 fs-7">目录： （1）session （2&#x...</span>
</div>
<!--end::Title-->
</div>
<!--begin::Item-->
<div class="d-flex flex-stack mb-7">
<!--begin::Symbol-->
<div class="symbol symbol-60px symbol-2by3 me-4">
<div class="symbol-label" style="background-image: url('/uploadfile/202506/41e94318013874a.png')"></div>
</div>
<!--end::Symbol-->
<!--begin::Title-->
<div class="m-0">
<a href="/code/203144.html" class="text-dark fw-bold text-hover-primary fs-6">linux入门---制作进度条</a>
<span class="text-gray-600 fw-semibold d-block pt-1 fs-7">了解缓冲区 我们首先来看看下面的操作： 我们首先创建了一个文件并在这个文件里面添加了...</span>
</div>
<!--end::Title-->
</div>
<!--begin::Item-->
<div class="d-flex flex-stack mb-7">
<!--begin::Symbol-->
<div class="symbol symbol-60px symbol-2by3 me-4">
<div class="symbol-label" style="background-image: url('/uploadfile/202506/f46035502278.png')"></div>
</div>
<!--end::Symbol-->
<!--begin::Title-->
<div class="m-0">
<a href="/code/203142.html" class="text-dark fw-bold text-hover-primary fs-6">关于测试，我发现了哪些新大陆</a>
<span class="text-gray-600 fw-semibold d-block pt-1 fs-7">关于测试 平常也只是听说过一些关于测试的术语，但并没有使用过测试工具。偶然看到编程老师...</span>
</div>
<!--end::Title-->
</div>
<!--begin::Item-->
<div class="d-flex flex-stack mb-7">
<!--begin::Symbol-->
<div class="symbol symbol-60px symbol-2by3 me-4">
<div class="symbol-label" style="background-image: url('/uploadfile/202506/144325901c5d8a4.jpeg')"></div>
</div>
<!--end::Symbol-->
<!--begin::Title-->
<div class="m-0">
<a href="/code/203141.html" class="text-dark fw-bold text-hover-primary fs-6">前缀和与对数器与二分法</a>
<span class="text-gray-600 fw-semibold d-block pt-1 fs-7">1. 前缀和 假设有一个数组，我们想大量频繁的去访问L到R这个区间的和，...</span>
</div>
<!--end::Title-->
</div>
<!--begin::Item-->
<div class="d-flex flex-stack mb-7">
<!--begin::Symbol-->
<div class="symbol symbol-60px symbol-2by3 me-4">
<div class="symbol-label" style="background-image: url('/uploadfile/202506/34f4814491043.png')"></div>
</div>
<!--end::Symbol-->
<!--begin::Title-->
<div class="m-0">
<a href="/code/203140.html" class="text-dark fw-bold text-hover-primary fs-6">nodejs：本地安装nvm实...</a>
<span class="text-gray-600 fw-semibold d-block pt-1 fs-7">一、背景-使用不同版本node的原因 vue3+ts、nuxt3版本，node...</span>
</div>
<!--end::Title-->
</div>
<!--begin::Item-->
<div class="d-flex flex-stack mb-7">
<!--begin::Symbol-->
<div class="symbol symbol-60px symbol-2by3 me-4">
<div class="symbol-label" style="background-image: url('/uploadfile/202506/df3e979d44113a1.png')"></div>
</div>
<!--end::Symbol-->
<!--begin::Title-->
<div class="m-0">
<a href="/code/203139.html" class="text-dark fw-bold text-hover-primary fs-6">JAVA集合知识整理</a>
<span class="text-gray-600 fw-semibold d-block pt-1 fs-7">Java集合知识整理 HashMap相关 HashMap的底层数据结构：jdk1.8之...</span>
</div>
<!--end::Title-->
</div>
<!--begin::Item-->
<div class="d-flex flex-stack mb-7">
<!--begin::Symbol-->
<div class="symbol symbol-60px symbol-2by3 me-4">
<div class="symbol-label" style="background-image: url('/uploadfile/202506/1be2323205d7790.png')"></div>
</div>
<!--end::Symbol-->
<!--begin::Title-->
<div class="m-0">
<a href="/code/203138.html" class="text-dark fw-bold text-hover-primary fs-6">无刷直流电机介绍及单片机控制实...</a>
<span class="text-gray-600 fw-semibold d-block pt-1 fs-7">无刷直流电机介绍及单片机控制实例前言基本概念优势与劣势使用寿命基本结构使用单片机控制实例电子调速器&...</span>
</div>
<!--end::Title-->
</div>
<!--begin::Item-->
<div class="d-flex flex-stack mb-7">
<!--begin::Symbol-->
<div class="symbol symbol-60px symbol-2by3 me-4">
<div class="symbol-label" style="background-image: url('/uploadfile/202506/425e93d5bdcf.gif')"></div>
</div>
<!--end::Symbol-->
<!--begin::Title-->
<div class="m-0">
<a href="/code/203137.html" class="text-dark fw-bold text-hover-primary fs-6">fwdiary(2) dp2</a>
<span class="text-gray-600 fw-semibold d-block pt-1 fs-7">1.传纸条 AcWing 275. 传纸条 - AcWing 走两条路，走一条最大的...</span>
</div>
<!--end::Title-->
</div>
<!--begin::Item-->
<div class="d-flex flex-stack mb-7">
<!--begin::Symbol-->
<div class="symbol symbol-60px symbol-2by3 me-4">
<div class="symbol-label" style="background-image: url('/uploadfile/202506/841a0a68fd0d5.png')"></div>
</div>
<!--end::Symbol-->
<!--begin::Title-->
<div class="m-0">
<a href="/code/203136.html" class="text-dark fw-bold text-hover-primary fs-6">常用的DOS命令</a>
<span class="text-gray-600 fw-semibold d-block pt-1 fs-7">常用的DOS命令 DOS（Disk Operating System，磁...</span>
</div>
<!--end::Title-->
</div>
<!--begin::Item-->
<div class="d-flex flex-stack mb-7">
<!--begin::Symbol-->
<div class="symbol symbol-60px symbol-2by3 me-4">
<div class="symbol-label" style="background-image: url('/uploadfile/202506/3caf744315af4a2.png')"></div>
</div>
<!--end::Symbol-->
<!--begin::Title-->
<div class="m-0">
<a href="/code/203135.html" class="text-dark fw-bold text-hover-primary fs-6">＜C++＞ 类和对象(下)</a>
<span class="text-gray-600 fw-semibold d-block pt-1 fs-7">1.const成员函数将const修饰的“成员函数”称之为const成员函数，cons...</span>
</div>
<!--end::Title-->
</div>
</div>
<!--end::Body-->
</div>
<!--end::Chart Widget 35-->
</div>
<!--end::Col-->
</div>
</div>
<!--end::Content container-->
</div>
<!--end::Content-->
</div>
<!--end::Content wrapper-->
<!--begin::Footer-->
<div id="kt_app_footer" class="app-footer">
<!--begin::Footer container-->
<div class="app-container container-xxl d-flex flex-column flex-md-row flex-center flex-md-stack py-3">
<!--begin::Copyright-->
<div class="text-dark order-2 order-md-1">
<span class="text-muted fw-semibold me-1">2026 ©</span>
<a href="/" target="_blank" class="text-gray-800 text-hover-primary">linux办公网</a>
</div>
<!--end::Copyright-->
<!--begin::Menu-->
<ul class="menu menu-gray-600 menu-hover-primary fw-semibold order-1">
<li class="menu-item">
<a href="/news/" target="_blank" class="menu-link px-2">linux资讯</a>
</li>
<li class="menu-item">
<a href="/yingyong/" target="_blank" class="menu-link px-2">linux应用</a>
</li>
<li class="menu-item">
<a href="/code/" target="_blank" class="menu-link px-2">编程开发</a>
</li>
<li class="menu-item">
<a href="/linuxzg/" target="_blank" class="menu-link px-2">Linux中国 </a>
</li>
</ul>
<!--end::Menu-->
</div>
<!--end::Footer container-->
</div>
<!--end::Footer-->
</div>
<!--end:::Main-->
</div>
<!--end::Wrapper-->
</div>
<!--end::Page-->
</div>
<!--end::App-->
<div id="kt_scrolltop" class="scrolltop" data-kt-scrolltop="true">
<!--begin::Svg Icon | path: icons/duotune/arrows/arr066.svg-->
<span class="svg-icon">
<svg width="24" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<rect opacity="0.5" x="13" y="6" width="13" height="2" rx="1" transform="rotate(90 13 6)" fill="currentColor"></rect>
<path d="M12.5657 8.56569L16.75 12.75C17.1642 13.1642 17.8358 13.1642 18.25 12.75C18.6642 12.3358 18.6642 11.6642 18.25 11.25L12.7071 5.70711C12.3166 5.31658 11.6834 5.31658 11.2929 5.70711L5.75 11.25C5.33579 11.6642 5.33579 12.3358 5.75 12.75C6.16421 13.1642 6.83579 13.1642 7.25 12.75L11.4343 8.56569C11.7467 8.25327 12.2533 8.25327 12.5657 8.56569Z" fill="currentColor"></path>
</svg>
</span>
<!--end::Svg Icon-->
</div>
<!--begin::Javascript-->
<script>var hostUrl = "/static/default/pc/";</script>
<!--begin::Global Javascript Bundle(mandatory for all pages)-->
<script src="/static/default/pc/plugins/global/plugins.bundle.js"></script>
<script src="/static/default/pc/js/scripts.bundle.js"></script>
<!--end::Global Javascript Bundle-->
<!--end::Javascript-->
</body>
<!--end::Body-->
</html>