{"id":170577,"date":"2023-09-11T03:28:44","date_gmt":"2023-09-11T02:28:44","guid":{"rendered":"https:\/\/datascientest.com\/en\/?p=170577"},"modified":"2023-10-30T20:03:50","modified_gmt":"2023-10-30T19:03:50","slug":"scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool","status":"publish","type":"post","link":"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool","title":{"rendered":"Scrapy: Everything you need to know about this Python web scraping tool"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"170577\" class=\"elementor elementor-170577\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-989c3cc e-flex e-con-boxed e-con\" data-id=\"989c3cc\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-03d2619 elementor-widget elementor-widget-heading\" data-id=\"03d2619\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-heading-title{padding:0;margin:0;line-height:1}.elementor-widget-heading .elementor-heading-title[class*=elementor-size-]>a{color:inherit;font-size:inherit;line-height:inherit}.elementor-widget-heading .elementor-heading-title.elementor-size-small{font-size:15px}.elementor-widget-heading .elementor-heading-title.elementor-size-medium{font-size:19px}.elementor-widget-heading .elementor-heading-title.elementor-size-large{font-size:29px}.elementor-widget-heading .elementor-heading-title.elementor-size-xl{font-size:39px}.elementor-widget-heading .elementor-heading-title.elementor-size-xxl{font-size:59px}<\/style><h2 class=\"elementor-heading-title elementor-size-default\">During internet browsing, many websites do not allow direct saving of data for personal use. The simplest solution in this case is to manually copy and paste the data, which can quickly become tedious and time-consuming. That's why web scraping techniques are often used to extract data from websites.<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7828223 elementor-widget elementor-widget-text-editor\" data-id=\"7828223\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t<p><strong>Web Scraping<\/strong> is the automation of the quasi-automatic process of <a href=\"https:\/\/datascientest.com\/en\/machine-learning-python-where-to-start\">extracting data from websites.<\/a> This operation is performed using scraping tools often known as web scrapers. These tools allow you to load and extract specific data from websites based on users&#8217; needs. They are most often custom-designed for a single site and then configured to work with other websites that have the same structure.<\/p><p>With the<a href=\"https:\/\/datascientest.com\/en\/python-the-most-popular-programming-language\"> Python programming language, the most commonly used tools<\/a> in the field of Web Scraping are BeautifulSoup and Scrapy Crawler. In this article, we will present some differences between these two tools and focus on Scrapy later on.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-31098ab e-flex e-con-boxed e-con\" data-id=\"31098ab\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-dd687fa elementor-widget elementor-widget-heading\" data-id=\"dd687fa\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Web Scraping vs Web Crawling<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-294270d elementor-widget elementor-widget-text-editor\" data-id=\"294270d\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Before delving into the subject, it&#8217;s quite interesting to understand the difference between Web Scraping and Web Crawling techniques:<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7634bf7 elementor-widget elementor-widget-heading\" data-id=\"7634bf7\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">- Web Scraping<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f870902 elementor-widget elementor-widget-text-editor\" data-id=\"f870902\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Web Scraping uses bots to programmatically analyze a web page and extract content from it. With Web Scraping, it&#8217;s necessary to target and extract specific data.<\/p><p>Example of web data extraction: Extracting prices of various specific products on Amazon or any other e-commerce website.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b76ed29 elementor-widget elementor-widget-heading\" data-id=\"b76ed29\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">- Web Crawling<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8cbc9e9 elementor-widget elementor-widget-text-editor\" data-id=\"8cbc9e9\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>The term crawling is used as an analogy to how a spider crawls (that&#8217;s also why web crawlers are often called spiders). Web Crawling tools also use <strong>robots (bots called crawlers)<\/strong> to systematically browse the World Wide Web, typically for the purpose of indexing it.<\/p><p>This involves looking at a page in its entirety and cataloging all elements, including the last letter and period on the page. The bots used will then, during their navigation through heaps of data and information, locate and retrieve information that resides in the deepest layers.<\/p><p>As examples of <strong>Web Crawling<\/strong> tools, you can consider all search engines like Google, Yahoo, or Bing. They crawl web pages and use the extracted information to index them.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-9cd0dc8 e-flex e-con-boxed e-con\" data-id=\"9cd0dc8\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-d6fe64f elementor-widget elementor-widget-heading\" data-id=\"d6fe64f\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">BeautifulSoup vs Scrapy<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e093d65 elementor-widget elementor-widget-text-editor\" data-id=\"e093d65\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Let&#8217;s continue with a quick comparison between BeautifulSoup and Scrapy, the two most widely used Web scraping libraries.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-86e2aee elementor-widget elementor-widget-heading\" data-id=\"86e2aee\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">- BeautifulSoup<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8da0684 elementor-widget elementor-widget-text-editor\" data-id=\"8da0684\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>BeautifulSoup is a very popular Python library that can be used to analyse HTML or XML documents in order to describe them using a tree or dictionary structure. This makes it easy to find and extract specific data from web pages. BeautifulSoup is fairly easy to learn and has good, comprehensive documentation that makes it easy to learn.<\/p><p>The advantages of BeautifulSoup :<\/p><p>&#8211; Very good documentation (very useful when you&#8217;re just starting out).<br \/>&#8211; Large community of users.<br \/>&#8211; Easy for beginners to learn and master.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b801d14 elementor-widget elementor-widget-text-editor\" data-id=\"b801d14\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Disadvantages:<\/p><p>&#8211; Dependence on other external Python libraries.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c9d8828 elementor-widget elementor-widget-heading\" data-id=\"c9d8828\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">- Scrapy<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f05189b elementor-widget elementor-widget-text-editor\" data-id=\"f05189b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Scrapy is a <a href=\"https:\/\/datascientest.com\/en\/the-new-champion-of-open-source-llm-falcon\">comprehensive open-source framework<\/a> and is among the most powerful libraries used for web data extraction. Scrapy natively integrates functions for extracting data from HTML or XML sources using CSS and XPath expressions.<\/p><p>Some advantages of Scrapy:<\/p><ul><li>Efficient in terms of memory and CPU.<\/li><li>Built-in functions for data extraction.<\/li><li>Easily extendable for large-scale projects.<\/li><li>Relatively high performance and speed compared to other libraries.<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-050226b elementor-widget elementor-widget-text-editor\" data-id=\"050226b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>As disadvantages, we can mention the limited documentation, which can be discouraging for beginners.<\/p><p>To summarize all the points mentioned above:<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-367150c elementor-widget__width-initial elementor-widget elementor-widget-html\" data-id=\"367150c\" data-element_type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style type=\"text\/css\">\n.tg  {border-collapse:collapse;border-spacing:0;}\n.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;\n  overflow:hidden;padding:10px 5px;word-break:normal;}\n.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;\n  font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}\n.tg .tg-8xxg{background-color:#9b9b9b;border-color:inherit;font-size:22px;font-weight:bold;text-align:center;vertical-align:top}\n.tg .tg-0pky{border-color:inherit;text-align:left;vertical-align:top}\n<\/style>\n<table class=\"tg\" style=\"undefined;table-layout: fixed; width: 800px\">\n<colgroup>\n<col style=\"width: 400px\">\n<col style=\"width: 400px\">\n<\/colgroup>\n<thead>\n  <tr>\n    <th class=\"tg-8xxg\">Scrapy<\/th>\n    <th class=\"tg-8xxg\">BeautifulSoup<\/th>\n  <\/tr>\n<\/thead>\n<tbody>\n  <tr>\n    <td class=\"tg-0pky\">\n        <ul>\n        <li>A framework<br><\/li>\n        <li>High processing speed due to its built-in functions<br><\/li>\n        <li>Best choice for complex projects and tasks<br><\/li>\n        <li>Scrapy is more complex than BeautifulSoup<br><\/li>\n        <li>Works more like a web crawler (spider = Python classes defining what to crawl, how to do it, and how to extract information)<br><\/li>\n        <li>Integrates features for creating pipelines<\/li><br>\n        <\/ul>\n        <\/td>\n    <td class=\"tg-0pky\">\n        <ul>\n        <li>Python library\/module<br><\/li>\n        <li>Can become slow depending on task complexity<br><\/li>\n        <li>Ideal for small projects<br><\/li>\n        <li>Ideal for beginners<br><\/li>\n        <li>Considered as a syntax analyzer (parser)<br><\/li>\n        <\/ul>\n        <\/td>\n  <\/tr>\n<\/tbody>\n<\/table>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7c7641e elementor-widget elementor-widget-image\" data-id=\"7c7641e\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-image{text-align:center}.elementor-widget-image a{display:inline-block}.elementor-widget-image a img[src$=\".svg\"]{width:48px}.elementor-widget-image img{vertical-align:middle;display:inline-block}<\/style>\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image3-2.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-efbf2c6 elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"efbf2c6\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Discover our courses<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-45a5a24 e-flex e-con-boxed e-con\" data-id=\"45a5a24\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-46a37850 elementor-widget elementor-widget-heading\" data-id=\"46a37850\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Scrapy architecture<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6f5a0618 elementor-widget elementor-widget-text-editor\" data-id=\"6f5a0618\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>When a project is created, various files are used to interact with Scrapy&#8217;s main components. Scrapy&#8217;s architecture, as described in the official documentation, is shown below:<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b74a49d elementor-widget elementor-widget-image\" data-id=\"b74a49d\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1042\" height=\"621\" src=\"https:\/\/datascientest.com\/en\/files\/2023\/09\/image9.png\" class=\"attachment-full size-full wp-image-170582\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2023\/09\/image9.png 1042w, https:\/\/datascientest.com\/en\/files\/2023\/09\/image9-300x179.png 300w, https:\/\/datascientest.com\/en\/files\/2023\/09\/image9-1024x610.png 1024w, https:\/\/datascientest.com\/en\/files\/2023\/09\/image9-768x458.png 768w\" sizes=\"(max-width: 1042px) 100vw, 1042px\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7d8b16bf elementor-widget elementor-widget-text-editor\" data-id=\"7d8b16bf\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>If we analyse Scrapy&#8217;s architecture diagram, we can see that its central element, the engine, controls four executive components:<\/p><ul><li>Spiders<\/li><li>Item Pipelines<\/li><li>The Downloader<\/li><li>The Scheduler<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-62ec37cf elementor-widget elementor-widget-text-editor\" data-id=\"62ec37cf\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>At the beginning of the process, communication occurs through Spiders, which allow the transmission of requests (containing the URLs to be scraped and the information to be extracted) to the engine.<\/p><p>The engine then forwards the request to the Scheduler to enqueue it<strong> (if multiple URLs are provided).<\/strong><\/p><p>The engine also receives requests from the Scheduler, which has ordered tasks previously, and forwards them to the Downloader module, which downloads the HTML code of the page and transforms it into a Response object. The Response object is then passed to the Spider and subsequently to the ItemPipeline module. This process repeats for different URL links of web pages.<\/p><p>Therefore, we can now better define the roles of the components:<\/p><ul><li><strong>Spiders:<\/strong> Classes defining different scraping methods by users. The methods are invoked by Scrapy when necessary.<\/li><li><strong>Scrapy Engine<\/strong>: Controls the data flow and triggers all events.<br \/>Scheduler: Communicates with the Engine regarding the order of tasks to be performed.<\/li><li><strong>Downloader:<\/strong> Receives requests from the Engine to download the content of web pages.<\/li><li><strong>ItemPipeline:<\/strong> Successive transformation steps<a href=\"https:\/\/datascientest.com\/en\/data-cleaning-definition-methods-and-relevance-in-data-science\"> (for cleaning, data validation, or database insertion)<\/a> applied to the raw extracted data.<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-f068e4c e-flex e-con-boxed e-con\" data-id=\"f068e4c\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-7a20f90 elementor-widget elementor-widget-heading\" data-id=\"7a20f90\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Scrapy Installation <\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-02a2ae1 elementor-widget elementor-widget-text-editor\" data-id=\"02a2ae1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Scrapy is fairly easy to install. Simply run the command below in an Ubuntu terminal. You can easily find the equivalent commands for other operating systems:<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-03c80c9 elementor-widget elementor-widget-text-editor\" data-id=\"03c80c9\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p># creation of a virtual environment (OPTIONAL)<br \/>virtualenv scrapy_env<\/p><p># environment activation (OPTIONAL)<br \/>source scrapy_env\/bin\/active<\/p><p># install Scrapy<br \/>pip install scrapy<\/p><p># installation verification test<br \/>scrapy<\/p><p># Run a quick benchmark to see how Scrappy works on your hardware.<br \/>scrapy bench<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-abdd108 e-flex e-con-boxed e-con\" data-id=\"abdd108\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-e7e5099 elementor-widget elementor-widget-heading\" data-id=\"e7e5099\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">The Scrapy command prompt<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4c77111 elementor-widget elementor-widget-text-editor\" data-id=\"4c77111\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>During the experimentation phase, when you are searching for the code syntax to extract information from web pages, Scrapy has a dedicated command-line interface for interactive interaction with the Engine: the Scrapy Shell.<\/p><p>The Scrapy Shell is built on Python, so you can import any modules you need.<\/p><p>To access this command-line interface (once Scrapy is installed), simply execute the following command:<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7de503a elementor-widget elementor-widget-text-editor\" data-id=\"7de503a\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<pre style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\"># Open the shell scrapy<\/span><br \/><span style=\"font-weight: 400;\">scrapy shell \"URL-de-la-page-internet\"<\/span><br \/><span style=\"font-weight: 400;\"># exemple: scrapy shell \"https:\/\/www.ville-ideale.fr\/abries_5001\"<\/span><\/pre>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-abf56f6 elementor-widget elementor-widget-text-editor\" data-id=\"abf56f6\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Once launched, it is within the Shell that you can execute commands to actually extract information from the specified web page. You can interactively test different commands and extraction approaches.<\/p><p>After various experiments, the extraction code lines will be grouped into a Spider class for automation.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-05c39be elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"05c39be\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Learn Scrapy<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-761389a e-flex e-con-boxed e-con\" data-id=\"761389a\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-33a96d1 elementor-widget elementor-widget-heading\" data-id=\"33a96d1\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">CSS and XPATH selectors<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d8ef785 elementor-widget elementor-widget-text-editor\" data-id=\"d8ef785\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>During the creation of a <strong>Spider class<\/strong>, the most important step is to create the code responsible for data extraction (code determined in the previous step from the Scrapy Shell).<\/p><p>To indicate which data from the website should be downloaded by <strong>Scrapy<\/strong>, you can use:<\/p><p>XPath Selectors:<\/p><p>XPath selectors are frequently used in web scraping due to their extensive capabilities. For example, you can use them to:<\/p><p>&#8211; Specify the exact element to extract from the page.<br \/>&#8211; Retrieve the text associated with an element.<br \/>&#8211; Download the parent or child element.<br \/>&#8211; Fetch adjacent elements.<br \/>&#8211; Retrieve elements that start\/end with keywords.<br \/>&#8211; Obtain elements whose attributes satisfy a mathematical condition.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-293b575 elementor-widget elementor-widget-text-editor\" data-id=\"293b575\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<ul><li aria-level=\"1\"><p>CSS Selectors:<\/p><p>CSS selectors provide an easier alternative, especially for beginners who are familiar with CSS commands. CSS selectors have slightly fewer capabilities than XPath, but in the case of Scrapy, they have been extended with additional syntax for retrieving a specific element&#8217;s attribute.<\/p><\/li><li aria-level=\"1\"><p>BeautifulSoup Library:<\/p><p>Since Scrapy is written in Python, it&#8217;s entirely possible to import other libraries for specific tasks if needed. This includes the BeautifulSoup library, which can be used (and imported) when defining data extraction classes (Spiders).<\/p><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-579c74e e-flex e-con-boxed e-con\" data-id=\"579c74e\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-0c334aa elementor-widget elementor-widget-heading\" data-id=\"0c334aa\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Example of data extraction with Scrapy<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-70a9ea4 elementor-widget elementor-widget-text-editor\" data-id=\"70a9ea4\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>To give you a concrete idea of what Scrapy can do, we&#8217;re going to extract some data about DataScientest from the <a href=\"https:\/\/fr.trustpilot.com\/review\/datascientest.com website.\" data-wplink-url-error=\"true\">https:\/\/fr.trustpilot.com\/review\/datascientest.com website.<\/a><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f0ab80e elementor-widget elementor-widget-image\" data-id=\"f0ab80e\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image2-3.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5e016e0 elementor-widget elementor-widget-text-editor\" data-id=\"5e016e0\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>The idea is to gather in a single CSV file all the comments and ratings of learners available on trustpilot. For the sake of simplicity, we will limit ourselves to reviews given in French.<\/p><p>Recommended methodology for Web Scraping with Scrapy :<\/p><ul><li>Analyse and locate the information to be extracted on the web page<\/li><li>Prototype in the Scrapy Shell the various commands for extracting each of the elements identified in the previous step<\/li><li>Create a Scrapy project and create the Spider (to define how to extract information from all the pages)<\/li><li>Test the Spider on one page<\/li><li>Apply the spider to all the pages to retrieve all the information.<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-bc5b24e elementor-widget elementor-widget-heading\" data-id=\"bc5b24e\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Step 1: Analysing and locating the information to be extracted<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-627ec35 elementor-widget elementor-widget-text-editor\" data-id=\"627ec35\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The aim of this fairly manual stage is simply to locate the useful information and identify the associated HTML tags. If we focus on a review (below) :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-31cc8c9 elementor-widget elementor-widget-image\" data-id=\"31cc8c9\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image12.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-83d30b9 elementor-widget elementor-widget-text-editor\" data-id=\"83d30b9\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>We&#8217;ll look at the information below:<\/p><ul><li>The comment<\/li><li>The date of the comment<\/li><li>Training date<\/li><li>The title of the comment<\/li><li>The note<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0b12406 elementor-widget elementor-widget-text-editor\" data-id=\"0b12406\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">To access the HTML code of the page, simply (in the Edge or Firefox browser) right-click on the page and click on the inspect option. The different tags associated with the different elements of the page will appear on the right.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b232521 elementor-widget elementor-widget-image\" data-id=\"b232521\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image6-2.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-dc43973 elementor-widget elementor-widget-text-editor\" data-id=\"dc43973\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">To access the tag associated with a specific element directly, simply select the element and repeat the previous operation (right-click + inspect).<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-37162be elementor-widget elementor-widget-image\" data-id=\"37162be\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image8.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ba0b349 elementor-widget elementor-widget-text-editor\" data-id=\"ba0b349\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>For example, to obtain the tag associated with the title and after manipulation, we can read the main tag associated with the title of the comment opposite: .typography_heading-s__f7029. This is a CSS tag (also known as a CSS selector) from which only the text can be extracted.<\/p><p>This operation will be repeated for all the elements you wish to extract, so that the corresponding CSS tags are associated with each of them.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7f3049a elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"7f3049a\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Discover our courses<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7914fc4 elementor-widget elementor-widget-heading\" data-id=\"7914fc4\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Step 2: Experimenting with the Scrapy Shell<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-277e06e elementor-widget elementor-widget-text-editor\" data-id=\"277e06e\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Once the tags are clearly identified, you can enter the Scrapy command prompt to define the extraction commands completely.<\/p><p>To enter the Scrapy Shell, you will use the following command (after activating the virtual environment):<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1246a33 elementor-widget elementor-widget-text-editor\" data-id=\"1246a33\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<pre style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\"># Open the Scrapy shell on the website trustpilot<\/span><br \/><span style=\"font-weight: 400;\">scrapy shell \"https:\/\/fr.trustpilot.com\/review\/datascientest.com\"<\/span><\/pre>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-56d7351 elementor-widget elementor-widget-text-editor\" data-id=\"56d7351\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>La commande ci-dessus permet de :<\/p><p>1. R\u00e9cup\u00e9rer l&#8217;ensemble des \u00e9l\u00e9ments de la page sp\u00e9cifi\u00e9e \u00e0 l&#8217;aide de l&#8217;API de Scrapy. Ces \u00e9l\u00e9ments seront stock\u00e9s dans une variable &#8220;response&#8221;.<br \/>2. Ouvrir le Scrapy Shell pour interagir avec la page web et tester des commandes d&#8217;extraction.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-dbe26bb elementor-widget elementor-widget-text-editor\" data-id=\"dbe26bb\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Using the CSS selectors identified in the previous step, we can use the &#8220;response&#8221; variable to extract the precise information we are looking for.<\/p><p>To extract the information in a simple and iterative way (and to ensure that the information associated with each comment is simply retrieved), the first element to be extracted is the list of all the information block selectors on a page.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-01dafd1 elementor-widget elementor-widget-image\" data-id=\"01dafd1\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image5-1.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-477dedd elementor-widget elementor-widget-text-editor\" data-id=\"477dedd\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>The CSS selector associated with a block is as follows: .styles_reviewContentwrapper__zH_9M&#8217;.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2b3edce elementor-widget elementor-widget-image\" data-id=\"2b3edce\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image4-3.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d35a550 elementor-widget elementor-widget-text-editor\" data-id=\"d35a550\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">You can then run the following command in the Scrapy shell to extract the list of all the blocks.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3c6dceb elementor-widget elementor-widget-text-editor\" data-id=\"3c6dceb\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p># Extraction of all blocks<br \/>response.css(&#8216;.styles_reviewContentwrapper__zH_9M&#8217;)<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-dd24d75 elementor-widget elementor-widget-text-editor\" data-id=\"dd24d75\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The output result is as follows:<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7ad4d63 elementor-widget elementor-widget-image\" data-id=\"7ad4d63\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image11.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ec0ee78 elementor-widget elementor-widget-text-editor\" data-id=\"ec0ee78\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>This is a list of selectors enabling the data contained in each block to be retrieved using each of the elements in the list. We can then extract everything we need with the following lines of code (each line must be executed separately in the Scrapy shell):<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7855b0c elementor-widget elementor-widget-text-editor\" data-id=\"7855b0c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p># recovery of all information groups<br \/>selectors = response.css(&#8216;.styles_reviewContentwrapper__zH_9M&#8217;)<\/p><p># extract the note on an element<br \/>selectors[0].css(&#8216;img::attr(alt)&#8217;).extract()<\/p><p># extracting the title from an element<br \/>selectors[0].css(&#8216;.typography_heading-s__f7029::text&#8217;).extract()<\/p><p># comment and date on an element<br \/>elt = selectors[0].css(&#8216;.typography_color-black__5LYEn::text&#8217;).extract()<br \/>exp_date = elt[-1]<br \/>comment = &#8221;.join([word for word in elt[:-1]])<\/p><p># extract the date of the comment (one day later than what is displayed)<br \/>selectors[0].css(&#8216;div.typography_body-m__xgxZ_.typography_appearance-subtle__8_H2l.styles_datesWrapper__RCEKH &gt; time::text&#8217;).extract()<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c93536e elementor-widget elementor-widget-heading\" data-id=\"c93536e\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Step 3: Creating a Scrapy project<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-306f091 elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"306f091\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Learn Scrapy<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d891859 elementor-widget elementor-widget-text-editor\" data-id=\"d891859\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Once the prototyping of the<strong> Scrapy code lines<\/strong> is completed, you can easily create a Spider class that will be the consolidation of all the code lines above within a single Python file.<\/p><p>Scrapy provides native functions for initializing a Scrapy project (and thus initializing the Spider class files). To create a Scrapy project, simply run the following command in the Shell:<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-35bde0c elementor-widget elementor-widget-text-editor\" data-id=\"35bde0c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p># Creating a project (example: trustdst project)<br \/>scrapy startproject trustdst<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-247dc8f elementor-widget elementor-widget-text-editor\" data-id=\"247dc8f\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">When you have finished, you will see :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4c3c747 elementor-widget elementor-widget-image\" data-id=\"4c3c747\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image7-1.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a8b3b6c elementor-widget elementor-widget-text-editor\" data-id=\"a8b3b6c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The command we have just run has created a folder with initialised Python files. The architecture can be seen below:<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-640b682 elementor-widget elementor-widget-text-editor\" data-id=\"640b682\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The output result is as follows:<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1bff9d7 elementor-widget elementor-widget-image\" data-id=\"1bff9d7\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image10.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f7a5363 elementor-widget elementor-widget-heading\" data-id=\"f7a5363\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Step 4 : Creating the Spider<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-138fb63 elementor-widget elementor-widget-text-editor\" data-id=\"138fb63\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">We&#8217;re going to use the architecture created in the previous step to create the Python class file that will extract all the information from a page at once. Here again, Scrapy will allow us to initialise the file in question with the command below:<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-32be587 elementor-widget elementor-widget-text-editor\" data-id=\"32be587\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p># creation of the Spider class for scraping<br \/>scrapy genspider trustpilotspider en.trustpilot.com\/review\/datascientest.com<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f81f3c6 elementor-widget elementor-widget-text-editor\" data-id=\"f81f3c6\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>This command will create the trustpilotspider.py file that we will modify and use for data scraping.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-75f41c7 elementor-widget elementor-widget-image\" data-id=\"75f41c7\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image1-2.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-49924f7 elementor-widget elementor-widget-text-editor\" data-id=\"49924f7\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">You can then modify the file as follows:<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ff7cc12 elementor-widget elementor-widget-text-editor\" data-id=\"ff7cc12\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p># Import scrapy<br \/>import scrapy<\/p><p># definition of the sipder<br \/>TrustpilotspiderSpider(scrapy.Spider) class :<br \/>&#8220;&#8221;&#8221;<br \/>name : a class attribute which gives a name to the spider.<br \/>We&#8217;ll use it when we run our scrapy crawl spider later.<br \/>&lt;spider_name&gt;.<\/p><p>allowed_domains: a class attribute that tells Scrapy that it<br \/>should only ever fetch pages from the chocolate.co.uk domain.<br \/>This prevents the spider from fetching pages in the chocolate domain.<\/p><p>star_urls : a class attribute which tells Scrapy the first url<br \/>that it should explore. We&#8217;ll modify it in a moment.<\/p><p>parse: the parse function is called after a response has been<br \/>received from the target website.<br \/>&#8220;&#8221;&#8221;<br \/>name = &#8220;trustpilotspider<br \/>allowed_domains = [&#8220;en.trustpilot.com&#8221;]<br \/>start_urls = [&#8220;https:\/\/fr.trustpilot.com\/review\/datascientest.com&#8221;] # Shift URL<\/p><p>def parse(self, response) :<br \/>&#8220;&#8221;&#8221;<br \/>Module for extracting information<br \/>&#8220;&#8221;&#8221;<\/p><p># Loop over all the information blocks<br \/>selectors = response.css(&#8216;.styles_reviewContentwrapper__zH_9M&#8217;)<\/p><p># Iterative extraction of information<br \/>for selector in selectors :<br \/># Information to be returned<br \/>yield{<br \/>&#8216;notes&#8217;: selector.css(&#8216;img::attr(alt)&#8217;).extract(),<br \/>&#8216;title&#8217;: selector.css(&#8216;.typography_heading-s__f7029::text&#8217;).get(), # .extract()[0]<br \/>exp_date&#8217; : selector.css(&#8216;.typography_color-black__5LYEn::text&#8217;).extract()[-1],<br \/>&#8216;comments&#8217; : &#8221;.join([text for text in selector.css(&#8216;.typography_color-black__5LYEn::text&#8217;).extract()[:-1]),<br \/>comment_date&#8217; : selector.css(&#8216;div.typography_body-m__xgxZ_.typography_appearance-subtle__8_H2l.styles_datesWrapper__RCEKH &gt; time::text&#8217;).get()<br \/>}<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f3e2945 elementor-widget elementor-widget-text-editor\" data-id=\"f3e2945\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">To run the file, simply execute the command below:<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8320dee elementor-widget elementor-widget-text-editor\" data-id=\"8320dee\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>scrapy crawl trustpilotspider ou scrapy crawl trustpilotspider -O myonepagescrapeddata.json<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6cbb13a elementor-widget elementor-widget-text-editor\" data-id=\"6cbb13a\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">(if you want to save the result in a JSON file)<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-2aec0f8 e-flex e-con-boxed e-con\" data-id=\"2aec0f8\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-8b6fc4e elementor-widget elementor-widget-heading\" data-id=\"8b6fc4e\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Conclusion<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5a0b340 elementor-widget elementor-widget-text-editor\" data-id=\"5a0b340\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><a href=\"https:\/\/datascientest.com\/en\/data-mining-everything-you-need-to-know-about-data-mining\">Data is one of the most valuable assets that a company can possess.<\/a> It is at the heart of Data Science and Data Analysis. Companies actively collecting data can gain a competitive advantage over those that do not. With enough data, organizations can better determine the root causes of problems and make informed decisions.<\/p><p>There are scenarios where an organization may not have enough data to draw the necessary insights. This is often the case for startups that typically begin with little to no data. One solution in such cases is to employ a data acquisition technique like <strong>Web Scraping.<\/strong><\/p><p>Scrapy is an <a href=\"https:\/\/datascientest.com\/en\/framework-what-is-it\">open-source framework that efficiently extracts data from the web<\/a> and has a large community of users. It is well-suited for large-scale Web Scraping projects as it provides a clear structure and tools for processing the retrieved information.<\/p><p>This article was intended to introduce Scrapy with some of its basic features used by <a href=\"https:\/\/datascientest.com\/en\/data-engineer-role-skills-salary\">Data Engineers,<\/a> Data Scientists, or Data Analysts for information extraction.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-43bf8b0 elementor-widget elementor-widget-text-editor\" data-id=\"43bf8b0\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<span style=\"font-weight: 400;\">If you want to go further with Scrapy feel free to consult the official documentation: <a href=\"https:\/\/docs.scrapy.org\/en\/latest\/intro\/tutorial.html\">Scrapy Tutorial &#8211; Scrapy 2.8.0 documentation<\/a><\/span><span style=\"font-weight: 400;\">. To learn more about information acquisition technologies and methodologies, <strong>feel free to check out DataScientest&#8217;s training courses<\/strong>.<\/span>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ec21e89 elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"ec21e89\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Discover our courses<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-bf83af3 e-flex e-con-boxed e-con\" data-id=\"bf83af3\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-c2d6a9c elementor-widget elementor-widget-html\" data-id=\"c2d6a9c\" data-element_type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<script type=\"application\/ld+json\">\n{\n  \"@context\": \"https:\/\/schema.org\",\n  \"@type\": \"FAQPage\",\n  \"mainEntity\": [{\n    \"@type\": \"Question\",\n    \"name\": \"Scrapy Architecture\",\n    \"acceptedAnswer\": {\n      \"@type\": \"Answer\",\n      \"text\": \"When a project is created, various files are used to interact with the main Scrapy components. The architecture of Scrapy as described in the official documentation can be seen below\".\n    }\n  },{\n    \"@type\": \"Question\",\n    \"name\": \"Scrapy installation\",\n    \"acceptedAnswer\": {\n      \"@type\": \"Answer\",\n      \"text\": \"Scrapy is quite simple to install. Just run the command below in an Ubuntu terminal. You can easily find the equivalent commands for other operating systems\".\n    }\n  },{\n    \"@type\": \"Question\",\n    \"name\": \"Web Scraping Vs Web Crawling\",\n    \"acceptedAnswer\": {\n      \"@type\": \"Answer\",\n      \"text\": \"Web scraping uses robots to programmatically analyse a web page in order to extract content. With Web Scraping it is therefore necessary to search for data in a precise manner. The term crawling is used as an analogy with the way a spider crawls (which is also why web crawlers are often called spiders). Web crawling tools will also use robots (bots called crawlers) to systematically crawl the World Wide Web, usually with the aim of indexing it.\"\n    }\n  }]\n}\n<\/script\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>During internet browsing, many websites do not allow direct saving of data for personal use. The simplest solution in this case is to manually copy and paste the data, which can quickly become tedious and time-consuming. That&#8217;s why web scraping techniques are often used to extract data from websites. Web Scraping is the automation of [&hellip;]<\/p>\n","protected":false},"author":67,"featured_media":170579,"comment_status":"open","ping_status":"open","sticky":false,"template":"elementor_theme","format":"standard","meta":{"content-type":"","inline_featured_image":false},"categories":[2425],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.7 (Yoast SEO v21.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Scrapy: Everything you need to know about this Python web scraping tool<\/title>\n<meta name=\"description\" content=\"Web Scraping in Python with Scrapy is a good alternative to some of the well-known tools in the industry to achieve...\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Scrapy: Everything you need to know about this Python web scraping tool\" \/>\n<meta property=\"og:description\" content=\"During internet browsing, many websites do not allow direct saving of data for personal use. The simplest solution in this case is to manually copy and\" \/>\n<meta property=\"og:url\" content=\"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool\" \/>\n<meta property=\"og:site_name\" content=\"Data Science Courses | DataScientest\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataScientestEN\" \/>\n<meta property=\"article:published_time\" content=\"2023-09-11T02:28:44+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-10-30T19:03:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/datascientest.com\/en\/files\/2023\/09\/scrapy.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1025\" \/>\n\t<meta property=\"og:image:height\" content=\"563\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"Melanie\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:site\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Melanie\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"14 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool#article\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool\"},\"author\":{\"name\":\"Melanie\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d\"},\"headline\":\"Scrapy: Everything you need to know about this Python web scraping tool\",\"datePublished\":\"2023-09-11T02:28:44+00:00\",\"dateModified\":\"2023-10-30T19:03:50+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool\"},\"wordCount\":2612,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"articleSection\":[\"Data Science\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool#respond\"]}],\"copyrightYear\":\"2023\",\"copyrightHolder\":{\"@id\":\"https:\/\/datascientest.com\/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool\",\"url\":\"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool\",\"name\":\"Scrapy: Everything you need to know about this Python web scraping tool\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/#website\"},\"datePublished\":\"2023-09-11T02:28:44+00:00\",\"dateModified\":\"2023-10-30T19:03:50+00:00\",\"description\":\"Web Scraping in Python with Scrapy is a good alternative to some of the well-known tools in the industry to achieve...\",\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/datascientest.com\/en\/#website\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"name\":\"Data Science Courses | DataScientest\",\"description\":\"Leader in Data Science Courses\",\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/datascientest.com\/en\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/datascientest.com\/en\/#organization\",\"name\":\"DataScientest\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"contentUrl\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"width\":301,\"height\":301,\"caption\":\"DataScientest\"},\"image\":{\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/DataScientestEN\",\"https:\/\/twitter.com\/DataScientest__\",\"https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/\",\"https:\/\/www.tiktok.com\/@datascientest\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d\",\"name\":\"Melanie\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g\",\"caption\":\"Melanie\"},\"url\":\"https:\/\/datascientest.com\/en\/author\/melaniekrause\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Scrapy: Everything you need to know about this Python web scraping tool","description":"Web Scraping in Python with Scrapy is a good alternative to some of the well-known tools in the industry to achieve...","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool","og_locale":"en_GB","og_type":"article","og_title":"Scrapy: Everything you need to know about this Python web scraping tool","og_description":"During internet browsing, many websites do not allow direct saving of data for personal use. The simplest solution in this case is to manually copy and","og_url":"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool","og_site_name":"Data Science Courses | DataScientest","article_publisher":"https:\/\/www.facebook.com\/DataScientestEN","article_published_time":"2023-09-11T02:28:44+00:00","article_modified_time":"2023-10-30T19:03:50+00:00","og_image":[{"width":1025,"height":563,"url":"https:\/\/datascientest.com\/en\/files\/2023\/09\/scrapy.webp","type":"image\/webp"}],"author":"Melanie","twitter_card":"summary_large_image","twitter_creator":"@DataScientest__","twitter_site":"@DataScientest__","twitter_misc":{"Written by":"Melanie","Estimated reading time":"14 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool#article","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool"},"author":{"name":"Melanie","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d"},"headline":"Scrapy: Everything you need to know about this Python web scraping tool","datePublished":"2023-09-11T02:28:44+00:00","dateModified":"2023-10-30T19:03:50+00:00","mainEntityOfPage":{"@id":"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool"},"wordCount":2612,"commentCount":0,"publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"articleSection":["Data Science"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool#respond"]}],"copyrightYear":"2023","copyrightHolder":{"@id":"https:\/\/datascientest.com\/#organization"}},{"@type":"WebPage","@id":"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool","url":"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool","name":"Scrapy: Everything you need to know about this Python web scraping tool","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/#website"},"datePublished":"2023-09-11T02:28:44+00:00","dateModified":"2023-10-30T19:03:50+00:00","description":"Web Scraping in Python with Scrapy is a good alternative to some of the well-known tools in the industry to achieve...","inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool"]}]},{"@type":"WebSite","@id":"https:\/\/datascientest.com\/en\/#website","url":"https:\/\/datascientest.com\/en\/","name":"Data Science Courses | DataScientest","description":"Leader in Data Science Courses","publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/datascientest.com\/en\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/datascientest.com\/en\/#organization","name":"DataScientest","url":"https:\/\/datascientest.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","contentUrl":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","width":301,"height":301,"caption":"DataScientest"},"image":{"@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataScientestEN","https:\/\/twitter.com\/DataScientest__","https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/","https:\/\/www.tiktok.com\/@datascientest"]},{"@type":"Person","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d","name":"Melanie","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g","caption":"Melanie"},"url":"https:\/\/datascientest.com\/en\/author\/melaniekrause"}]}},"_links":{"self":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/170577"}],"collection":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/users\/67"}],"replies":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/comments?post=170577"}],"version-history":[{"count":0,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/170577\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media\/170579"}],"wp:attachment":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media?parent=170577"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/categories?post=170577"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/tags?post=170577"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}