{"id":196695,"date":"2023-04-13T14:53:21","date_gmt":"2023-04-13T14:53:21","guid":{"rendered":"https:\/\/datascientest.com\/?p=196695"},"modified":"2023-11-09T13:53:27","modified_gmt":"2023-11-09T13:53:27","slug":"scrapy-web-scraping-tout-savoir","status":"publish","type":"post","link":"https:\/\/datascientest.com\/scrapy-web-scraping-tout-savoir","title":{"rendered":"Scrapy : tout savoir sur cet outil Python de web scraping"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"196695\" class=\"elementor elementor-196695\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-0b2e0fb e-flex e-con-boxed e-con\" data-id=\"0b2e0fb\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-419fa43 elementor-widget elementor-widget-heading\" data-id=\"419fa43\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-heading-title{padding:0;margin:0;line-height:1}.elementor-widget-heading .elementor-heading-title[class*=elementor-size-]>a{color:inherit;font-size:inherit;line-height:inherit}.elementor-widget-heading .elementor-heading-title.elementor-size-small{font-size:15px}.elementor-widget-heading .elementor-heading-title.elementor-size-medium{font-size:19px}.elementor-widget-heading .elementor-heading-title.elementor-size-large{font-size:29px}.elementor-widget-heading .elementor-heading-title.elementor-size-xl{font-size:39px}.elementor-widget-heading .elementor-heading-title.elementor-size-xxl{font-size:59px}<\/style><h2 class=\"elementor-heading-title elementor-size-default\">Pendant la navigation sur internet, de nombreux sites ne permettent pas de directement sauvegarder des donn\u00e9es pour un usage personnel. La solution la plus simple dans ce cas de figure est simplement de copier-coller manuellement les donn\u00e9es, ce qui peut vite devenir fastidieux et chronophage. C\u2019est la raison pour laquelle on a souvent recours aux techniques de Web Scraping pour extraire des donn\u00e9es de sites web. <\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-00ddc39 elementor-widget elementor-widget-text-editor\" data-id=\"00ddc39\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t<p><span style=\"font-weight: 400;\">Le Web Scraping est l&rsquo;automatisation du processus d&rsquo;extraction quasi-automatique des donn\u00e9es des sites Web. Cette op\u00e9ration se fait \u00e0 l&rsquo;aide d\u2019outils de scraping souvent connus sous le nom de web scrapers. Ces derniers permettent alors de charger et d\u2019extraire des donn\u00e9es sp\u00e9cifiques des sites Web en fonction des besoins des utilisateurs. Ils sont le plus souvent con\u00e7us sur mesure pour un seul site et configur\u00e9s ensuite pour fonctionner avec d&rsquo;autres sites Web ayant la m\u00eame structure.<\/span><\/p><p><span style=\"font-weight: 400;\">Avec le langage de programmation Python, les outils les plus utilis\u00e9s dans le domaine du Web Scraping sont <\/span><strong>BeautifulSoup<\/strong> <span style=\"font-weight: 400;\">et <\/span><b>Scrapy Crawler<\/b><span style=\"font-weight: 400;\">. Dans cet article, nous allons pr\u00e9senter quelques diff\u00e9rences entre ces deux outils et nous concentrer sur Scrapy par la suite.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-28c9bd1 e-flex e-con-boxed e-con\" data-id=\"28c9bd1\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-ff31e09 elementor-widget elementor-widget-heading\" data-id=\"ff31e09\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Web Scraping vs Web Crawling<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3b123c0 elementor-widget elementor-widget-text-editor\" data-id=\"3b123c0\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Avant de rentrer dans le vif du sujet il est assez int\u00e9ressant de comprendre la diff\u00e9rence entre les techniques de Web Scraping et Web Crawling :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4de1a3b elementor-widget elementor-widget-heading\" data-id=\"4de1a3b\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">- Le Web Scraping<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f0d6150 elementor-widget elementor-widget-text-editor\" data-id=\"f0d6150\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Le <a href=\"https:\/\/datascientest.com\/web-scraping-theorie-et-application-accessible-a-tous\">Web Scraping<\/a> utilise des robots pour analyser de mani\u00e8re programmatique une page web afin d&rsquo;en extraire du contenu. Avec le Web Scraping il est donc n\u00e9cessaire de chercher les donn\u00e9es <\/span><b>de mani\u00e8re pr\u00e9cise<\/b><span style=\"font-weight: 400;\">.<\/span><\/p><p><span style=\"font-weight: 400;\">Exemple d&rsquo;extraction de donn\u00e9es sur le Web : Extraction des prix de divers produits sp\u00e9cifiques sur le site Amazon ou tout autre site d\u2019e-commerce.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d73fc06 elementor-widget elementor-widget-heading\" data-id=\"d73fc06\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">- Le Web Crawling<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6bfb74e elementor-widget elementor-widget-text-editor\" data-id=\"6bfb74e\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Le terme <\/span><b>crawling<\/b><span style=\"font-weight: 400;\"> est utilis\u00e9 comme une analogie avec la fa\u00e7on dont une araign\u00e9e rampe (c\u2019est aussi la raison pour laquelle les \u00ab\u00a0web crawlers\u00a0\u00bb sont souvent appel\u00e9s des <\/span><b>spiders<\/b><span style=\"font-weight: 400;\">). Les outils de Web Crawling vont \u00e9galement utiliser des robots (bots appel\u00e9s crawlers) pour parcourir syst\u00e9matiquement le <strong>World Wide Web<\/strong>, g\u00e9n\u00e9ralement dans le but de l&rsquo;indexer. Il s&rsquo;agit de regarder une page dans son int\u00e9gralit\u00e9 et de r\u00e9f\u00e9rencer tous les \u00e9l\u00e9ments s\u2019y trouvant, y compris la derni\u00e8re lettre et le dernier point de la page. Les bots utilis\u00e9s vont alors, pendant leur navigation \u00e0 travers des tas de donn\u00e9es et d\u2019informations, localiser et r\u00e9cup\u00e9rer les informations qui se trouvent dans les couches les plus profondes.\u00a0<\/span><\/p><p><span style=\"font-weight: 400;\">Comme exemple d\u2019outils de Web Crawling on peut citer tous les moteurs de recherche tels que <strong>Google<\/strong>, <strong>Yahoo<\/strong> ou <strong>Bing<\/strong>. Ces derniers explorent les pages web et utilisent les informations extraites pour indexer ces derni\u00e8res.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-aabce4f e-flex e-con-boxed e-con\" data-id=\"aabce4f\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-2a9ac7e elementor-widget elementor-widget-heading\" data-id=\"2a9ac7e\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">BeautifulSoup vs Scrapy<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d7ffe55 elementor-widget elementor-widget-text-editor\" data-id=\"d7ffe55\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Poursuivons avec une rapide comparaison entre <a href=\"https:\/\/datascientest.com\/decouvrir-le-web-scraping-avec-beautiful-soup\">BeautifulSoup<\/a> et Scrapy, les deux biblioth\u00e8ques de Web Scraping les plus utilis\u00e9es.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3e33f80 elementor-widget elementor-widget-heading\" data-id=\"3e33f80\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">- BeautifulSoup<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a2ff9f5 elementor-widget elementor-widget-text-editor\" data-id=\"a2ff9f5\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">BeautifulSoup est une librairie Python tr\u00e8s populaire qui permet d\u2019analyser les documents HTML ou XML de mani\u00e8re \u00e0 les d\u00e9crire en utilisant une structure arborescente ou sous forme de dictionnaire. Cette derni\u00e8re permet alors de retrouver et d\u2019extraire facilement des <\/span><b>donn\u00e9es sp\u00e9cifiques<\/b><span style=\"font-weight: 400;\"> des pages Web. BeautifulSoup est assez facile \u00e0 prendre en main et dispose d&rsquo;une bonne documentation compl\u00e8te qui permet d&rsquo;apprendre facilement.<\/span><\/p><p style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\">Les avantages de BeautifulSoup :<\/span><\/p><ul><li style=\"list-style-type: none;\"><ul><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Tr\u00e8s bonne documentation (tr\u00e8s utile lorsqu\u2019on d\u00e9bute).<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Grande communaut\u00e9 d\u2019utilisateurs.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Facile \u00e0 apprendre et \u00e0 ma\u00eetriser pour les d\u00e9butants.<\/span><\/li><\/ul><\/li><\/ul><ul><li style=\"list-style-type: none;\">\u00a0<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0f3cc77 elementor-widget elementor-widget-text-editor\" data-id=\"0f3cc77\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\">Inconv\u00e9nients :<\/span><\/p><ul><li style=\"list-style-type: none;\"><ul><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">D\u00e9pendance d\u2019autres librairies Python externes.<\/span><\/li><\/ul><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7c0c489 elementor-widget elementor-widget-heading\" data-id=\"7c0c489\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">- Scrapy<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-17966fd elementor-widget elementor-widget-text-editor\" data-id=\"17966fd\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Scrapy est un framework complet open-source et est parmi les biblioth\u00e8ques les plus puissantes utilis\u00e9es pour l\u2019extraction de donn\u00e9es sur internet. Scrapy int\u00e8gre de mani\u00e8re native des fonctions pour extraire des donn\u00e9es de sources HTML ou XML en utilisant des expressions <a href=\"https:\/\/en.wikipedia.org\/wiki\/CSS\">CSS<\/a> <\/span><span style=\"font-weight: 400;\">et <a href=\"https:\/\/fr.wikipedia.org\/wiki\/XPath\">XPath<\/a><\/span><span style=\"font-weight: 400;\">.<\/span><\/p><p><span style=\"font-weight: 400;\">Quelques avantages de Scrapy :<\/span><\/p><ul><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Efficace en termes de m\u00e9moire et de CPU.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Fonctions int\u00e9gr\u00e9es pour l&rsquo;extraction de donn\u00e9es.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Facilement extensible pour des projets de grande envergure.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Assez performant et rapide par rapport aux autres biblioth\u00e8ques.<\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3cf9728 elementor-widget elementor-widget-text-editor\" data-id=\"3cf9728\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Comme inconv\u00e9nients, on peut citer la faible documentation qui peut d\u00e9courager les d\u00e9butants.<\/span><\/p><p><span style=\"font-weight: 400;\">Pour r\u00e9sumer tous les points \u00e9voqu\u00e9s ci-dessus :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0ed0432 elementor-widget__width-initial elementor-widget elementor-widget-html\" data-id=\"0ed0432\" data-element_type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style type=\"text\/css\">\r\n.tg  {border-collapse:collapse;border-spacing:0;}\r\n.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;\r\n  overflow:hidden;padding:10px 5px;word-break:normal;}\r\n.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;\r\n  font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}\r\n.tg .tg-8xxg{background-color:#9b9b9b;border-color:inherit;font-size:22px;font-weight:bold;text-align:center;vertical-align:top}\r\n.tg .tg-0pky{border-color:inherit;text-align:left;vertical-align:top}\r\n<\/style>\r\n<table class=\"tg\" style=\"undefined;table-layout: fixed; width: 800px\">\r\n<colgroup>\r\n<col style=\"width: 400px\">\r\n<col style=\"width: 400px\">\r\n<\/colgroup>\r\n<thead>\r\n  <tr>\r\n    <th class=\"tg-8xxg\">Scrapy<\/th>\r\n    <th class=\"tg-8xxg\">BeautifulSoup<\/th>\r\n  <\/tr>\r\n<\/thead>\r\n<tbody>\r\n  <tr>\r\n    <td class=\"tg-0pky\">\r\n        <ul>\r\n        <li>Un framework<br><\/li>\r\n        <li>Grande vitesse de traitement gr\u00e2ce \u00e0 ses fonctions int\u00e9gr\u00e9es<br><\/li>\r\n        <li>Meilleur choix pour les projets et t\u00e2ches complexes<br><\/li>\r\n        <li>Scrapy est plus complexe que BeautifulSoup<br><\/li>\r\n        <li>Fonctionne beaucoup plus comme un r\u00e9f\u00e9renceur (spider = Classes Python permettant de d\u00e9finir ce qu'il faut r\u00e9f\u00e9rencer, comme il faut le faire et comment extraire les informations)<br><\/li>\r\n        <li>Int\u00e8gre des fonctionnalit\u00e9s pour la cr\u00e9ation de pipelines<\/li><br>\r\n        <\/ul>\r\n        <\/td>\r\n    <td class=\"tg-0pky\">\r\n        <ul>\r\n        <li>Librairie \/ module Python<br><\/li>\r\n        <li>Peut devenir lent suivant la complexit\u00e9 de la t\u00e2che<br><\/li>\r\n        <li>Id\u00e9al pour les petits projets<br><\/li>\r\n        <li>Id\u00e9al pour les d\u00e9butant<br><\/li>\r\n        <li>Consid\u00e9r\u00e9 comme un analyseur synthaxique (parser)<br><\/li>\r\n        <\/ul>\r\n        <\/td>\r\n  <\/tr>\r\n<\/tbody>\r\n<\/table>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-723746e elementor-widget elementor-widget-image\" data-id=\"723746e\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-image{text-align:center}.elementor-widget-image a{display:inline-block}.elementor-widget-image a img[src$=\".svg\"]{width:48px}.elementor-widget-image img{vertical-align:middle;display:inline-block}<\/style>\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"804\" height=\"350\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image3-2.png\" class=\"attachment-full size-full wp-image-196712\" alt=\"\" loading=\"lazy\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image3-2.png 804w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image3-2-300x131.png 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image3-2-768x334.png 768w\" sizes=\"(max-width: 804px) 100vw, 804px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2efd9ae elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"2efd9ae\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/formations-data\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Apprendre \u00e0 utiliser Scrapy et BeautifulSoup<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-0158786 e-flex e-con-boxed e-con\" data-id=\"0158786\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-b9a2128 elementor-widget elementor-widget-heading\" data-id=\"b9a2128\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Architecture de Scrapy<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ceafbf0 elementor-widget elementor-widget-text-editor\" data-id=\"ceafbf0\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">\u00c0 la cr\u00e9ation d\u2019un projet, diff\u00e9rents fichiers vont permettre d\u2019interagir avec les principaux composants de Scrapy. On peut voir ci-dessous l\u2019architecture de Scrapy telle que d\u00e9crite dans la <a href=\"https:\/\/docs.scrapy.org\/en\/latest\/topics\/architecture.html\">documentation officielle<\/a> <\/span><span style=\"font-weight: 400;\">:<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e5cd591 elementor-widget elementor-widget-image\" data-id=\"e5cd591\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1042\" height=\"621\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image9.png\" class=\"attachment-full size-full wp-image-196713\" alt=\"\" loading=\"lazy\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image9.png 1042w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image9-300x179.png 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image9-1024x610.png 1024w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image9-768x458.png 768w\" sizes=\"(max-width: 1042px) 100vw, 1042px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d53c5c6 elementor-widget elementor-widget-text-editor\" data-id=\"d53c5c6\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">En analysant le diagramme de l&rsquo;architecture de Scrapy, nous voyons que son \u00e9l\u00e9ment central, l\u2019<\/span><i><span style=\"font-weight: 400;\">engine <\/span><\/i><span style=\"font-weight: 400;\">ou <\/span><i><span style=\"font-weight: 400;\">moteur<\/span><\/i><span style=\"font-weight: 400;\">, qui contr\u00f4le quatre composants ex\u00e9cutifs :<\/span><\/p><ul><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Les Spiders<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Les Item Pipelines<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Le Downloader<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Le Scheduler<\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-21b9444 elementor-widget elementor-widget-text-editor\" data-id=\"21b9444\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Au d\u00e9but du processus, la communication s\u2019effectue \u00e0 travers les Spiders qui vont permettre de transmettre des requ\u00eates (contenant les URL \u00e0 scrapper et les informations \u00e0 extraire) \u00e0 l\u2019<\/span><b>engine<\/b><span style=\"font-weight: 400;\">. C\u2019est ce dernier qui va transmettre la requ\u00eate au planificateur (<\/span><b>Scheduler<\/b><span style=\"font-weight: 400;\">) pour l\u2019enregistrer dans la file d\u2019attente (si plusieurs URLs sont transmises). Le moteur (Engine) va \u00e9galement recevoir des requ\u00eates du Scheduler qui a ordonn\u00e9 les t\u00e2ches pr\u00e9c\u00e9demment pour les transmettre au module <\/span><b>Downloader<\/b><span style=\"font-weight: 400;\">, qui t\u00e9l\u00e9charge le code HTML de la page et le transforme en un objet <\/span><b>Response<\/b><span style=\"font-weight: 400;\">. L&rsquo;objet response est ensuite transmis au Spider, puis au module <\/span><b>ItemPipeline<\/b><span style=\"font-weight: 400;\">. Le processus va ainsi se r\u00e9p\u00e9ter pour les diff\u00e9rents liens URL des pages web.<\/span><\/p><p><span style=\"font-weight: 400;\">On peut donc d\u00e9sormais mieux d\u00e9finir les r\u00f4les des\u00a0 composantes:<\/span><\/p><ul><li style=\"font-weight: 400;\" aria-level=\"1\"><b>Spiders <\/b><span style=\"font-weight: 400;\">: Les classes d\u00e9finissant les diff\u00e9rentes m\u00e9thodes de scrapping par les utilisateurs. Les m\u00e9thodes sont alors invoqu\u00e9es par Scrapy lorsque cela est n\u00e9cessaire<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><b>Scrapy Engine <\/b><span style=\"font-weight: 400;\">: Permet de contr\u00f4ler le flux de donn\u00e9es et d\u00e9clencher tous les \u00e9v\u00e9nements<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><b>Scheduler <\/b><span style=\"font-weight: 400;\">: communique avec l\u2019Engine sur l\u2019ordre des t\u00e2ches \u00e0 r\u00e9aliser<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><b>Downloader <\/b><span style=\"font-weight: 400;\">: Re\u00e7oit les demandes de l\u2019Engine pour t\u00e9l\u00e9charger le contenu des pages web<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><b>ItemPipeline <\/b><span style=\"font-weight: 400;\">: Successions d\u2019\u00e9tapes de transformations (pour le nettoyage, la validation de donn\u00e9es ou l\u2019insertion en base de donn\u00e9es) appliqu\u00e9es \u00e0 la donn\u00e9e brute extraite<\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-0f369bf e-flex e-con-boxed e-con\" data-id=\"0f369bf\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-27ce7dd elementor-widget elementor-widget-heading\" data-id=\"27ce7dd\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Installation de Scrapy<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d19b8b9 elementor-widget elementor-widget-text-editor\" data-id=\"d19b8b9\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Scrapy est assez simple d\u2019installation. Il suffit d&rsquo;ex\u00e9cuter la commande ci-dessous dans un terminal Ubuntu. Vous pourrez trouver assez facilement les \u00e9quivalents de ces commandes pour les autre syst\u00e8mes d\u2019exploitation :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b23e7ce elementor-widget elementor-widget-text-editor\" data-id=\"b23e7ce\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<pre style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\"># cr\u00e9ation d\u2019un environnement virtuel (OPTIONNEL)<\/span><br \/><span style=\"font-weight: 400;\">virtualenv scrapy_env<\/span><br \/><br \/><span style=\"font-weight: 400;\"># Activation de l\u2019environnement (OPTIONNEL)<\/span><br \/><span style=\"font-weight: 400;\">source scrapy_env\/bin\/active<\/span><br \/><br \/><span style=\"font-weight: 400;\"># installation de Scrapy<\/span><br \/><span style=\"font-weight: 400;\">pip install scrapy<\/span><br \/><br \/><span style=\"font-weight: 400;\"># test de v\u00e9rification de l\u2019installation<\/span><br \/><span style=\"font-weight: 400;\">scrapy<\/span><br \/><br \/><span style=\"font-weight: 400;\"># Effectuez un test d'\u00e9valuation rapide pour voir comment Scrappy fonctionne sur votre mat\u00e9riel.<\/span><br \/><span style=\"font-weight: 400;\">scrapy bench<\/span><\/pre>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-0a04f05 e-flex e-con-boxed e-con\" data-id=\"0a04f05\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-6e14ee0 elementor-widget elementor-widget-heading\" data-id=\"6e14ee0\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">L\u2019invite de commande Scrapy<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a8ca5b9 elementor-widget elementor-widget-text-editor\" data-id=\"a8ca5b9\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">En phase d\u2019exp\u00e9rimentation, c\u2019est-\u00e0-dire lorsque l\u2019on recherche la syntaxe du code \u00e0 ex\u00e9cuter pour extraire des informations des pages internet, Scrapy poss\u00e8de un invite de commandes d\u00e9di\u00e9 permettant d\u2019interagir avec l\u2019Engine de mani\u00e8re interactive : le Shell Scrapy.<\/span><\/p><p><span style=\"font-weight: 400;\">Le Shell Scrapy est construit sur python, nous pouvons donc importer n&rsquo;importe quel module dont nous avons besoin.<\/span><\/p><p><span style=\"font-weight: 400;\">Pour acc\u00e9der \u00e0 cet invite de commande (une fois Scrapy install\u00e9), il suffit d&rsquo;ex\u00e9cuter la commande ci-dessous :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1e9d523 elementor-widget elementor-widget-text-editor\" data-id=\"1e9d523\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<pre style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\"># Ouvrir le shell scrapy<\/span><br \/><span style=\"font-weight: 400;\">scrapy shell \"URL-de-la-page-internet\"<\/span><br \/><span style=\"font-weight: 400;\"># exemple: scrapy shell \"https:\/\/www.ville-ideale.fr\/abries_5001\"<\/span><\/pre>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8b7e07c elementor-widget elementor-widget-text-editor\" data-id=\"8b7e07c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Une\u00a0 fois lanc\u00e9, c\u2019est dans le Shell qu\u2019on va pouvoir ex\u00e9cuter les commandes qui vont concr\u00e8tement extraire les informations sur la page sp\u00e9cifi\u00e9e. On pourra alors <\/span><b>de mani\u00e8re interactive<\/b><span style=\"font-weight: 400;\"> tester diff\u00e9rentes commandes et approches d\u2019extraction.<\/span><\/p><p><span style=\"font-weight: 400;\">\u00c0 l\u2019issue des diff\u00e9rentes exp\u00e9rimentations les lignes de code d\u2019extraction seront regroup\u00e9es au sein d\u2019une classe Spider pour automatisation.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2718a45 elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"2718a45\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/formations-data\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Ma\u00eetriser Scrapy<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-f3c8e01 e-flex e-con-boxed e-con\" data-id=\"f3c8e01\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-1cde6ed elementor-widget elementor-widget-heading\" data-id=\"1cde6ed\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Les s\u00e9lecteurs CSS et XPATH<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7cca1f1 elementor-widget elementor-widget-text-editor\" data-id=\"7cca1f1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Lors de la cr\u00e9ation d&rsquo;une classe Spider, l&rsquo;\u00e9tape la plus importante consiste \u00e0 cr\u00e9er le code responsable de l&rsquo;extraction des donn\u00e9es (code d\u00e9termin\u00e9 \u00e0 l\u2019\u00e9tape pr\u00e9c\u00e9dente depuis le Shell Scrapy).<\/span><\/p><p><span style=\"font-weight: 400;\">Pour indiquer quelles donn\u00e9es du site doivent \u00eatre t\u00e9l\u00e9charg\u00e9es par Scrapy, nous pouvons utiliser :<\/span><\/p><ul><li aria-level=\"1\"><b>Les s\u00e9lecteurs XPath:<\/b><\/li><\/ul><p><span style=\"font-weight: 400;\">Les s\u00e9lecteurs XPath sont tr\u00e8s souvent utilis\u00e9s en Web Scraping en raison de leurs vastes possibilit\u00e9s. Par exemple :\u00a0<\/span><\/p><ul><li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Indiquer l&rsquo;\u00e9l\u00e9ment exact \u00e0 extraire de la page.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Obtenir le texte associ\u00e9 \u00e0 un \u00e9l\u00e9ment l&rsquo;\u00e9l\u00e9ment.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">T\u00e9l\u00e9charger l&rsquo;\u00e9l\u00e9ment Parent ou enfant.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">T\u00e9l\u00e9charger les \u00e9l\u00e9ments adjacents.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">T\u00e9l\u00e9charger les \u00e9l\u00e9ments qui commencent\/finissent par des mots cl\u00e9s.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Obtenir les \u00e9l\u00e9ments dont les attributs satisfont une condition math\u00e9matique<\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9337b86 elementor-widget elementor-widget-text-editor\" data-id=\"9337b86\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<ul><li aria-level=\"1\"><b>Les s\u00e9lecteurs CSS:<\/b><\/li><\/ul><p><span style=\"font-weight: 400;\">Les s\u00e9lecteurs CSS constituent une alternative plus facile pour les d\u00e9butants, en particulier ceux qui sont familiers avec les commandes CSS. Les s\u00e9lecteurs CSS ont un peu moins de capacit\u00e9s que XPath, mais dans le cas de Scrapy, ils ont \u00e9t\u00e9 \u00e9tendus avec une syntaxe suppl\u00e9mentaire qui permet de r\u00e9cup\u00e9rer un attribut d&rsquo;\u00e9l\u00e9ment sp\u00e9cifique.<\/span><\/p><ul><li aria-level=\"1\"><b>Biblioth\u00e8que BeautifulSoup:<\/b><\/li><\/ul><p><span style=\"font-weight: 400;\">Scrapy \u00e9tant \u00e9crit en <a href=\"https:\/\/datascientest.com\/formation-langage-python\">Python<\/a>, il est tout \u00e0 fait possible si besoin d\u2019importer d\u2019autres librairies pour la r\u00e9alisation de certaines t\u00e2ches. C\u2019est le cas avec la librairie BeautifulSoup qui peut \u00eatre utilis\u00e9e (et donc import\u00e9e) lors de la d\u00e9finition des classes d\u2019extraction de donn\u00e9es (les Spiders).<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-7d2e3d7 e-flex e-con-boxed e-con\" data-id=\"7d2e3d7\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-e7baec8 elementor-widget elementor-widget-heading\" data-id=\"e7baec8\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Exemple d\u2019extraction de donn\u00e9es avec Scrapy<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-45cac38 elementor-widget elementor-widget-text-editor\" data-id=\"45cac38\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Histoire de vous donner un aper\u00e7u concret des possibilit\u00e9s de Scrapy, nous allons extraire quelques donn\u00e9es au sujet de DataScientest depuis le site <a href=\"https:\/\/fr.trustpilot.com\/review\/datascientest.com\">https:\/\/fr.trustpilot.com\/review\/datascientest.com<\/a><\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fea68f5 elementor-widget elementor-widget-image\" data-id=\"fea68f5\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1240\" height=\"664\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image2-3.png\" class=\"attachment-full size-full wp-image-196717\" alt=\"\" loading=\"lazy\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image2-3.png 1240w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image2-3-300x161.png 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image2-3-1024x548.png 1024w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image2-3-768x411.png 768w\" sizes=\"(max-width: 1240px) 100vw, 1240px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2eeb5c9 elementor-widget elementor-widget-text-editor\" data-id=\"2eeb5c9\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">L\u2019id\u00e9e est de rassembler dans un seul <a href=\"https:\/\/datascientest.com\/importer-un-csv-sur-power-bi\">fichier CSV<\/a> tous les commentaires et les notes des apprenants disponibles sur trustpilot. On va se limiter aux avis donn\u00e9s en Fran\u00e7ais par souci de simplicit\u00e9.<\/span><\/p><p><span style=\"font-weight: 400;\">La m\u00e9thodologie recommand\u00e9e en mati\u00e8re de Web Scraping avec Scrapy :<\/span><\/p><ol><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Analyser et localiser sur la page internet les diff\u00e9rentes informations \u00e0 extraire<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Prototyper dans le Shell Scrapy les diff\u00e9rentes commandes pour extraire chacun des \u00e9l\u00e9ments \u00e0 extraire identifi\u00e9s \u00e0 l\u2019\u00e9tape pr\u00e9c\u00e9dente<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Cr\u00e9er un projet Scrapy et cr\u00e9er le Spider (pour d\u00e9finir comment extraire les informations de toutes les pages)<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Tester le Spider sur une page\u00a0<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Appliquer le Spider \u00e0 toutes les pages pour la r\u00e9cup\u00e9ration de toutes les informations.<\/span><\/li><\/ol>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-47fdf73 elementor-widget elementor-widget-heading\" data-id=\"47fdf73\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Step 1 : Analyse et localisation des informations \u00e0 extraire<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-78b8d53 elementor-widget elementor-widget-text-editor\" data-id=\"78b8d53\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">L\u2019objectif de cette \u00e9tape assez manuelle est simplement de rep\u00e9rer les informations utiles et d\u2019identifier les balises HTML associ\u00e9es. Si l\u2019on se focalise sur un avis (ci-dessous) :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c321b04 elementor-widget elementor-widget-image\" data-id=\"c321b04\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"930\" height=\"379\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image12.png\" class=\"attachment-full size-full wp-image-196718\" alt=\"\" loading=\"lazy\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image12.png 930w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image12-300x122.png 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image12-768x313.png 768w\" sizes=\"(max-width: 930px) 100vw, 930px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2121db5 elementor-widget elementor-widget-text-editor\" data-id=\"2121db5\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">On va s\u2019int\u00e9resser aux informations ci-dessous:<\/span><\/p><ul><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Le commentaire<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">La date du commentaire<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">La date de formation<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Le titre du commentaire<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">La note<\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-35e4e15 elementor-widget elementor-widget-text-editor\" data-id=\"35e4e15\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Pour acc\u00e9der au code HTML de la page il suffit (dans le navigateur Edge ou Firefox) de faire un clic droit sur la page et de cliquer sur l\u2019option inspecter. On pourra voir appara\u00eetre sur la droite les diff\u00e9rentes balises associ\u00e9es aux diff\u00e9rents \u00e9l\u00e9ments de la page.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7a36fe4 elementor-widget elementor-widget-image\" data-id=\"7a36fe4\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1999\" height=\"750\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image6-2.png\" class=\"attachment-full size-full wp-image-196719\" alt=\"\" loading=\"lazy\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image6-2.png 1999w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image6-2-300x113.png 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image6-2-1024x384.png 1024w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image6-2-768x288.png 768w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image6-2-1536x576.png 1536w\" sizes=\"(max-width: 1999px) 100vw, 1999px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2e266c5 elementor-widget elementor-widget-text-editor\" data-id=\"2e266c5\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Pour acc\u00e9der directement \u00e0 la balise associ\u00e9e \u00e0 un \u00e9l\u00e9ment sp\u00e9cifique, il suffit de s\u00e9lectionner ce dernier et de refaire la manipulation pr\u00e9c\u00e9dente (clic droit + inspecter).<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-28953d2 elementor-widget elementor-widget-image\" data-id=\"28953d2\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1999\" height=\"1333\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image8.png\" class=\"attachment-full size-full wp-image-196720\" alt=\"\" loading=\"lazy\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image8.png 1999w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image8-300x200.png 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image8-1024x683.png 1024w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image8-768x512.png 768w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image8-1536x1024.png 1536w\" sizes=\"(max-width: 1999px) 100vw, 1999px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-83e642d elementor-widget elementor-widget-text-editor\" data-id=\"83e642d\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Par exemple, pour obtenir la balise associ\u00e9e au titre et apr\u00e8s la manipulation, on peut lire la balise principale associ\u00e9e au titre du commentaire ci-contre: <\/span><b>.typography_heading-s__f7029<\/b><span style=\"font-weight: 400;\">. Il s\u2019agit d\u2019une balise CSS (on parle aussi de <\/span><b>s\u00e9lecteur CSS<\/b><span style=\"font-weight: 400;\">) \u00e0 partir de laquelle on peut extraire uniquement le texte.<\/span><\/p><p><span style=\"font-weight: 400;\">Cette op\u00e9ration sera r\u00e9p\u00e9t\u00e9e pour tous les \u00e9l\u00e9ments que l\u2019on souhaite extraire de mani\u00e8re \u00e0 associer \u00e0 chacun de ces derniers les balises CSS correspondantes.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-16dcce2 elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"16dcce2\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/formations-data\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Se former \u00e0 Scrapy<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6f8802b elementor-widget elementor-widget-heading\" data-id=\"6f8802b\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Step 2 : Exp\u00e9rimentation avec le Scrapy Shell<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6fa1caf elementor-widget elementor-widget-text-editor\" data-id=\"6fa1caf\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Une fois les balises clairement identifi\u00e9es, on peut entrer dans l\u2019invite de commande Scrapy pour d\u00e9finir compl\u00e8tement les commandes d\u2019extraction.<\/span><\/p><p><span style=\"font-weight: 400;\">Pour entrer dans le Scrapy Shell on va entrer la commande suivante (apr\u00e8s avoir activ\u00e9 l\u2019environnement virtuel) :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-08b1fe7 elementor-widget elementor-widget-text-editor\" data-id=\"08b1fe7\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<pre style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\"># Ouverture du Scrapy shell sur le site trustpilot<\/span><br \/><span style=\"font-weight: 400;\">scrapy shell \"https:\/\/fr.trustpilot.com\/review\/datascientest.com\"<\/span><\/pre>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7c00bc8 elementor-widget elementor-widget-text-editor\" data-id=\"7c00bc8\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">La commande ci-dessus permet de::<\/span><\/p><ol><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">R\u00e9cup\u00e9rer l\u2019ensemble des \u00e9l\u00e9ments de la page sp\u00e9cifi\u00e9e \u00e0 l\u2019aide de l\u2019API de Scrapy. Ces \u00e9l\u00e9ments seront stock\u00e9s dans une variable \u00ab\u00a0response\u00a0\u00bb<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Ouvrir le Scrapy Shell<\/span><\/li><\/ol>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-28e6f5b elementor-widget elementor-widget-text-editor\" data-id=\"28e6f5b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">C\u2019est sur la variable <strong>\u00ab\u00a0<\/strong><\/span><b>response\u00a0\u00bb<\/b><span style=\"font-weight: 400;\">\u00a0que l\u2019on va pouvoir, \u00e0 l\u2019aide des s\u00e9lecteurs CSS identifi\u00e9s \u00e0 l\u2019\u00e9tape pr\u00e9c\u00e9dente, extraire avec pr\u00e9cision les informations que l\u2019on cherche.<\/span><\/p><p><span style=\"font-weight: 400;\">Pour extraire les informations de mani\u00e8re simple et<\/span><b> it\u00e9rative<\/b><span style=\"font-weight: 400;\"> (et s\u2019assurer de r\u00e9cup\u00e9rer simplement les informations associ\u00e9es \u00e0 chaque commentaire), le premier \u00e9l\u00e9ment que l\u2019on va extraire est la liste de tous les s\u00e9lecteurs de blocs d\u2019information sur une page.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a66ad6f elementor-widget elementor-widget-image\" data-id=\"a66ad6f\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"601\" height=\"415\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image5-1.png\" class=\"attachment-full size-full wp-image-196724\" alt=\"\" loading=\"lazy\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image5-1.png 601w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image5-1-300x207.png 300w\" sizes=\"(max-width: 601px) 100vw, 601px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-55b1832 elementor-widget elementor-widget-text-editor\" data-id=\"55b1832\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Le s\u00e9lecteur CSS associ\u00e9 \u00e0 un bloc est le suivant : <\/span><b>.styles_reviewContentwrapper__zH_9M&rsquo;<\/b><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-eb2abd6 elementor-widget elementor-widget-image\" data-id=\"eb2abd6\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1999\" height=\"1333\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image4-3.png\" class=\"attachment-full size-full wp-image-196725\" alt=\"\" loading=\"lazy\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image4-3.png 1999w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image4-3-300x200.png 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image4-3-1024x683.png 1024w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image4-3-768x512.png 768w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image4-3-1536x1024.png 1536w\" sizes=\"(max-width: 1999px) 100vw, 1999px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-33afe12 elementor-widget elementor-widget-text-editor\" data-id=\"33afe12\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">On peut alors ex\u00e9cuter la commande suivante dans le shell Scrapy pour extraire la liste de tous les blocs.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-dc22ba0 elementor-widget elementor-widget-text-editor\" data-id=\"dc22ba0\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<pre style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\"># Extraction de tous blocs<\/span><br \/><span style=\"font-weight: 400;\">response.css('.styles_reviewContentwrapper__zH_9M')<\/span><\/pre>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2830d4c elementor-widget elementor-widget-text-editor\" data-id=\"2830d4c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">On obtient en sortie le r\u00e9sultat ci-dessous :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ad94d22 elementor-widget elementor-widget-image\" data-id=\"ad94d22\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1728\" height=\"651\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image11.png\" class=\"attachment-full size-full wp-image-196726\" alt=\"\" loading=\"lazy\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image11.png 1728w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image11-300x113.png 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image11-1024x386.png 1024w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image11-768x289.png 768w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image11-1536x579.png 1536w\" sizes=\"(max-width: 1728px) 100vw, 1728px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f845a01 elementor-widget elementor-widget-text-editor\" data-id=\"f845a01\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Il s\u2019agit d\u2019une liste de s\u00e9lecteurs permettant la r\u00e9cup\u00e9ration des donn\u00e9es contenues dans chaque bloc \u00e0 l\u2019aide de chacun des \u00e9l\u00e9ments de la liste. On va alors pouvoir extraire tout ce dont on a besoin avec les lignes code suivantes (<\/span><b>chaque ligne doit \u00eatre ex\u00e9cut\u00e9e s\u00e9par\u00e9ment dans le shell Scrapy<\/b><span style=\"font-weight: 400;\">) :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4aea866 elementor-widget elementor-widget-text-editor\" data-id=\"4aea866\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<pre style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\"># r\u00e9cup\u00e9ration de tous les groupes d'information<\/span><br \/><span style=\"font-weight: 400;\">selectors = response.css('.styles_reviewContentwrapper__zH_9M')<\/span><br \/><br \/><span style=\"font-weight: 400;\"># extraction de la note sur un \u00e9l\u00e9ment<\/span><br \/><span style=\"font-weight: 400;\">selectors[0].css('img::attr(alt)').extract()<\/span><br \/><br \/><span style=\"font-weight: 400;\"># extraction du titre sur un \u00e9l\u00e9ment<\/span><br \/><span style=\"font-weight: 400;\">selectors[0].css('.typography_heading-s__f7029::text').extract()<\/span><br \/><br \/><span style=\"font-weight: 400;\"># commentaire et date sur un \u00e9l\u00e9ment<\/span><br \/><span style=\"font-weight: 400;\">elt = selectors[0].css('.typography_color-black__5LYEn::text').extract()<\/span><br \/><span style=\"font-weight: 400;\">exp_date = elt[-1]<\/span><br \/><span style=\"font-weight: 400;\">comment = ''.join([word for word in elt[:-1]])<\/span><br \/><br \/><span style=\"font-weight: 400;\"># extraction de la date du commentaire (un jour de d\u00e9calage avec ce qui est affich\u00e9)<\/span><br \/><span style=\"font-weight: 400;\">selectors[0].css('div.typography_body-m__xgxZ_.typography_appearance-subtle__8_H2l.styles_datesWrapper__RCEKH &gt; time::text').extract()<\/span><\/pre>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5c4fa03 elementor-widget elementor-widget-heading\" data-id=\"5c4fa03\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Step 3 : Cr\u00e9ation d\u2019un projet Scrapy<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d7d4738 elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"d7d4738\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/formations-data\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Apprendre \u00e0 cr\u00e9er un projet Scrapy<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d994c74 elementor-widget elementor-widget-text-editor\" data-id=\"d994c74\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Une fois le prototypage des lignes de code Scrapy termin\u00e9, on peut ais\u00e9ment cr\u00e9er une classe Spider qui ne sera que la r\u00e9union de toutes les lignes de code ci-dessus au sein d\u2019un seul et m\u00eame fichier Python.<\/span><\/p><p><span style=\"font-weight: 400;\">Scrapy offre de mani\u00e8re native des fonctions permettant d\u2019initialiser un projet Scrapy (et donc d\u2019initialiser les fichiers de classes Spider). Pour cr\u00e9er un projet Scrapy, il suffit d&rsquo;ex\u00e9cuter dans le Shell la commande ci-dessous :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3fac9bf elementor-widget elementor-widget-text-editor\" data-id=\"3fac9bf\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<pre style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\"># Cr\u00e9ation d\u2019un projet (exemple projet trustdst)<\/span><br \/><span style=\"font-weight: 400;\">scrapy startproject trustdst<\/span><\/pre>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-910002f elementor-widget elementor-widget-text-editor\" data-id=\"910002f\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Apr\u00e8s ex\u00e9cution, vous verrez alors appara\u00eetre :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-11a4a06 elementor-widget elementor-widget-image\" data-id=\"11a4a06\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1473\" height=\"232\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image7-1.png\" class=\"attachment-full size-full wp-image-196727\" alt=\"\" loading=\"lazy\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image7-1.png 1473w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image7-1-300x47.png 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image7-1-1024x161.png 1024w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image7-1-768x121.png 768w\" sizes=\"(max-width: 1473px) 100vw, 1473px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b04ab6b elementor-widget elementor-widget-text-editor\" data-id=\"b04ab6b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">La commande que nous venons d&rsquo;ex\u00e9cuter a permis de cr\u00e9er un dossier avec des fichiers Python initialis\u00e9s. On peut alors voir l&rsquo;architecture ci-dessous :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-de488fd elementor-widget elementor-widget-image\" data-id=\"de488fd\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"370\" height=\"373\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image10.png\" class=\"attachment-full size-full wp-image-196728\" alt=\"\" loading=\"lazy\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image10.png 370w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image10-298x300.png 298w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image10-150x150.png 150w\" sizes=\"(max-width: 370px) 100vw, 370px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f9b5dcd elementor-widget elementor-widget-heading\" data-id=\"f9b5dcd\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Step 4 : Cr\u00e9ation du Spider<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f69f34e elementor-widget elementor-widget-text-editor\" data-id=\"f69f34e\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">On va s\u2019appuyer sur l\u2019architecture cr\u00e9\u00e9e \u00e0 l\u2019\u00e9tape pr\u00e9c\u00e9dente pour cr\u00e9er le fichier de <a href=\"https:\/\/datascientest.com\/python-class\">class Python<\/a> qui va permettre d\u2019extraire toutes les informations d\u2019une page en une seule fois. Encore ici, Scrapy va nous permettre d\u2019initialiser le fichier en question avec la commande ci-dessous :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d89b46e elementor-widget elementor-widget-text-editor\" data-id=\"d89b46e\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<pre style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\"># cr\u00e9ation de la classe Spider de scraping<\/span><br \/><span style=\"font-weight: 400;\">scrapy genspider trustpilotspider fr.trustpilot.com\/review\/datascientest.com<\/span><\/pre>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-56df8b0 elementor-widget elementor-widget-text-editor\" data-id=\"56df8b0\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Cette commande va permettre de cr\u00e9er le fichier <\/span><b>trustpilotspider.py <\/b><span style=\"font-weight: 400;\">que nous allons modifier et utiliser pour le scraping de donn\u00e9es.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b516f29 elementor-widget elementor-widget-image\" data-id=\"b516f29\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"780\" height=\"604\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image1-2.png\" class=\"attachment-full size-full wp-image-196735\" alt=\"\" loading=\"lazy\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image1-2.png 780w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image1-2-300x232.png 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/image1-2-768x595.png 768w\" sizes=\"(max-width: 780px) 100vw, 780px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c33bf78 elementor-widget elementor-widget-text-editor\" data-id=\"c33bf78\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">On peut alors modifier le fichier de mani\u00e8re suivante :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-be26d89 elementor-widget elementor-widget-text-editor\" data-id=\"be26d89\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<pre style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\"># import de scrapy<\/span><br \/><span style=\"font-weight: 400;\">import scrapy<\/span><br \/><br \/><span style=\"font-weight: 400;\"># d\u00e9finition du sipder<\/span><br \/><span style=\"font-weight: 400;\">class TrustpilotspiderSpider(scrapy.Spider):<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\"\"\"<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0name: a class attribute that gives a name to the spider.<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0We will use this when running our spider later scrapy crawl<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0&lt;spider_name&gt;.<\/span><br \/><br \/><span style=\"font-weight: 400;\">\u00a0\u00a0\u00a0\u00a0allowed_domains: a class attribute that tells Scrapy that it<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0should only ever scrape pages of the chocolate.co.uk domain.<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0This prevents the spider going<\/span><br \/><br \/><span style=\"font-weight: 400;\">\u00a0\u00a0\u00a0\u00a0star_urls: a class attribute that tells Scrapy the first url<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0it should scrape. We will be changing this in a bit.<\/span><br \/><br \/><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0parse: the parse function is called after a response has been<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0recieved from the target website.<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\"\"\"<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0name = \"trustpilotspider\"<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0allowed_domains = [\"fr.trustpilot.com\"]<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0start_urls = [\"https:\/\/fr.trustpilot.com\/review\/datascientest.com\"] # Maj de l'URL<\/span><br \/><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0def parse(self, response):<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"\"\"<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Module permettant d'extraire les informations<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"\"\"<\/span><br \/><br \/><span style=\"font-weight: 400;\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Boucle sur tous les blocs d'information<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0selectors = response.css('.styles_reviewContentwrapper__zH_9M')<\/span><br \/><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Extraction it\u00e9rative des informations<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0for selector in selectors:<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Les informations \u00e0 retourner<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0yield{<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'notes' : selector.css('img::attr(alt)').extract(),<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'titre' : selector.css('.typography_heading-s__f7029::text').get(), # .extract()[0]<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'exp_date' : selector.css('.typography_color-black__5LYEn::text').extract()[-1],<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'comments' : ''.join([text for text in selector.css('.typography_color-black__5LYEn::text').extract()[:-1]]),<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'comment_date' : selector.css('div.typography_body-m__xgxZ_.typography_appearance-subtle__8_H2l.styles_datesWrapper__RCEKH &gt; time::text').get()<\/span><br \/><span style=\"font-weight: 400;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}<\/span><\/pre>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-af516be elementor-widget elementor-widget-text-editor\" data-id=\"af516be\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Pour ex\u00e9cuter le fichier, il suffit d&rsquo;ex\u00e9cuter la commande\u00a0 ci-dessous :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-edff679 elementor-widget elementor-widget-text-editor\" data-id=\"edff679\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<pre><span style=\"font-weight: 400;\">scrapy crawl trustpilotspider<\/span><span style=\"font-weight: 400;\"> ou <\/span><span style=\"font-weight: 400;\">scrapy crawl trustpilotspider -O myonepagescrapeddata.jso<\/span><span style=\"font-weight: 400;\">n<br \/><\/span><\/pre>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-642197a elementor-widget elementor-widget-text-editor\" data-id=\"642197a\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">(si l\u2019on souhaite sauvegarder le r\u00e9sultat dans un fichier JSON)<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-1b07e84 e-flex e-con-boxed e-con\" data-id=\"1b07e84\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-49baeb3 elementor-widget elementor-widget-heading\" data-id=\"49baeb3\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Conclusion<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-892ea7c elementor-widget elementor-widget-text-editor\" data-id=\"892ea7c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Les donn\u00e9es font partie des actifs les plus pr\u00e9cieux qu&rsquo;une entreprise puisse poss\u00e9der. Elles sont au c\u0153ur de la <a href=\"https:\/\/datascientest.com\/les-tendances-de-data-science-a-surveiller-pour-2023\">Science des donn\u00e9es<\/a> et de l&rsquo;analyse des donn\u00e9es. Les entreprises qui collectent activement des donn\u00e9es peuvent d\u00e9velopper un <strong>avantage concurrentiel<\/strong> sur celles qui ne le font pas. Avec suffisamment de donn\u00e9es, les organisations peuvent mieux d\u00e9terminer la cause des probl\u00e8mes et <strong>prendre des d\u00e9cisions \u00e9clair\u00e9es<\/strong>.<\/span><\/p><p><span style=\"font-weight: 400;\">Il existe des sc\u00e9narios dans lesquels une organisation peut ne pas disposer de suffisamment de donn\u00e9es pour en tirer les enseignements n\u00e9cessaires. C\u2019est le cas par exemple de certaines start-up qui commencent presque toujours sans donn\u00e9es. Une solution dans ce cas de figure consiste \u00e0 employer une technique d&rsquo;acquisition de donn\u00e9es telle que le Web Scraping.<\/span><\/p><p><span style=\"font-weight: 400;\">Scrapy est un <strong>framework open-source<\/strong> qui permet d\u2019extraire des donn\u00e9es sur le web de mani\u00e8re efficace et qui b\u00e9n\u00e9ficie d\u2019une large communaut\u00e9. Il est donc totalement adapt\u00e9 aux <strong>projets de Web Scraping de grande envergure<\/strong>, car il donne une structure claire et des instruments pour traiter les informations r\u00e9cup\u00e9r\u00e9es.<\/span><\/p><p><span style=\"font-weight: 400;\">Cet article n\u2019avait pour vocation que d\u2019introduire Scrapy avec quelques-unes de ses fonctionnalit\u00e9s de base utilis\u00e9es par des <a href=\"https:\/\/datascientest.com\/data-engineer-une-carriere-en-forte-demande\">Data Engineers<\/a>, <a href=\"https:\/\/datascientest.com\/lead-data-scientist\">Data Scientists<\/a> ou <a href=\"https:\/\/datascientest.com\/emploi-data-analyst\">Data Analysts<\/a> pour l\u2019extraction d&rsquo;information.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-07cd64a elementor-widget elementor-widget-text-editor\" data-id=\"07cd64a\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Si vous souhaitez aller plus loin avec Scrapy n\u2019h\u00e9sitez pas \u00e0 consulter la documentation officielle : <a href=\"https:\/\/docs.scrapy.org\/en\/latest\/intro\/tutorial.html\">Scrapy Tutorial \u2014 Scrapy 2.8.0 documentation<\/a><\/span><span style=\"font-weight: 400;\">. Pour en apprendre davantage sur les technologies et les m\u00e9thodologies d\u2019acquisition de l&rsquo;information, <strong>n&rsquo;h\u00e9sitez pas \u00e0 consulter les formations de DataScientest<\/strong>.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b7ed41d elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"b7ed41d\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/formations-data\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">D\u00e9couvrir nos formations Data<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-7a07000 e-flex e-con-boxed e-con\" data-id=\"7a07000\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-0a8695f elementor-widget elementor-widget-html\" data-id=\"0a8695f\" data-element_type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<script type=\"application\/ld+json\">\n{\n  \"@context\": \"https:\/\/schema.org\",\n  \"@type\": \"FAQPage\",\n  \"mainEntity\": [{\n    \"@type\": \"Question\",\n    \"name\": \"Architecture de Scrapy\",\n    \"acceptedAnswer\": {\n      \"@type\": \"Answer\",\n      \"text\": \"\u00c0 la cr\u00e9ation d\u2019un projet, diff\u00e9rents fichiers vont permettre d\u2019interagir avec les principaux composants de Scrapy. On peut voir ci-dessous l\u2019architecture de Scrapy telle que d\u00e9crite dans la documentation officielle\"\n    }\n  },{\n    \"@type\": \"Question\",\n    \"name\": \"Installation de Scrapy\",\n    \"acceptedAnswer\": {\n      \"@type\": \"Answer\",\n      \"text\": \"Scrapy est assez simple d\u2019installation. Il suffit d'ex\u00e9cuter la commande ci-dessous dans un terminal Ubuntu. Vous pourrez trouver assez facilement les \u00e9quivalents de ces commandes pour les autre syst\u00e8mes d\u2019exploitation\"\n    }\n  },{\n    \"@type\": \"Question\",\n    \"name\": \"Web Scraping Vs Web Crawling\",\n    \"acceptedAnswer\": {\n      \"@type\": \"Answer\",\n      \"text\": \"Le Web Scraping utilise des robots pour analyser de mani\u00e8re programmatique une page web afin d'en extraire du contenu. Avec le Web Scraping il est donc n\u00e9cessaire de chercher les donn\u00e9es de mani\u00e8re pr\u00e9cise. Le terme crawling est utilis\u00e9 comme une analogie avec la fa\u00e7on dont une araign\u00e9e rampe (c\u2019est aussi la raison pour laquelle les \\\"web crawlers\\\" sont souvent appel\u00e9s des spiders). Les outils de Web Crawling vont \u00e9galement utiliser des robots (bots appel\u00e9s crawlers) pour parcourir syst\u00e9matiquement le World Wide Web, g\u00e9n\u00e9ralement dans le but de l'indexer.\"\n    }\n  }]\n}\n<\/script>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Pendant la navigation sur internet, de nombreux sites ne permettent pas de directement sauvegarder des donn\u00e9es pour un usage personnel. La solution la plus simple dans ce cas de figure est simplement de copier-coller manuellement les donn\u00e9es, ce qui peut vite devenir fastidieux et chronophage. C\u2019est la raison pour laquelle on a souvent recours aux [&hellip;]<\/p>\n","protected":false},"author":78,"featured_media":196757,"comment_status":"open","ping_status":"open","sticky":false,"template":"elementor_theme","format":"standard","meta":{"content-type":"","inline_featured_image":false},"categories":[2497],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.7 (Yoast SEO v21.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Scrapy : tout savoir sur cet outil Python de web scraping<\/title>\n<meta name=\"description\" content=\"Avec le langage de programmation Python, les outils les plus utilis\u00e9s dans le domaine du web scraping sont BeautifulSoup et Scrapy Crawler. Dans cet article, nous allons pr\u00e9senter quelques diff\u00e9rences entre ces deux outils et nous concentrer sur Scrapy par la suite.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/datascientest.com\/scrapy-web-scraping-tout-savoir\" \/>\n<meta property=\"og:locale\" content=\"fr_FR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Scrapy : tout savoir sur cet outil Python de web scraping\" \/>\n<meta property=\"og:url\" content=\"https:\/\/datascientest.com\/scrapy-web-scraping-tout-savoir\" \/>\n<meta property=\"og:site_name\" content=\"Formation Data Science | DataScientest.com\" \/>\n<meta property=\"article:published_time\" content=\"2023-04-13T14:53:21+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-11-09T13:53:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/Illu_BLOG_scrapy.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1025\" \/>\n\t<meta property=\"og:image:height\" content=\"563\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"J\u00e9r\u00e9my Robert\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u00c9crit par\" \/>\n\t<meta name=\"twitter:data1\" content=\"J\u00e9r\u00e9my Robert\" \/>\n\t<meta name=\"twitter:label2\" content=\"Dur\u00e9e de lecture estim\u00e9e\" \/>\n\t<meta name=\"twitter:data2\" content=\"17 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/datascientest.com\/scrapy-web-scraping-tout-savoir\",\"url\":\"https:\/\/datascientest.com\/scrapy-web-scraping-tout-savoir\",\"name\":\"Scrapy : tout savoir sur cet outil Python de web scraping\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/#website\"},\"datePublished\":\"2023-04-13T14:53:21+00:00\",\"dateModified\":\"2023-11-09T13:53:27+00:00\",\"author\":{\"@id\":\"https:\/\/datascientest.com\/#\/schema\/person\/7af2361c48e24e79a8aa2360e6e74925\"},\"description\":\"Avec le langage de programmation Python, les outils les plus utilis\u00e9s dans le domaine du web scraping sont BeautifulSoup et Scrapy Crawler. Dans cet article, nous allons pr\u00e9senter quelques diff\u00e9rences entre ces deux outils et nous concentrer sur Scrapy par la suite.\",\"inLanguage\":\"fr-FR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/datascientest.com\/scrapy-web-scraping-tout-savoir\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/datascientest.com\/#website\",\"url\":\"https:\/\/datascientest.com\/\",\"name\":\"Formation Data Science | DataScientest.com\",\"description\":\"Leader de la formation en Data Science\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/datascientest.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"fr-FR\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/datascientest.com\/#\/schema\/person\/7af2361c48e24e79a8aa2360e6e74925\",\"name\":\"J\u00e9r\u00e9my Robert\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"fr-FR\",\"@id\":\"https:\/\/datascientest.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/11\/jeremyrobert.jpeg\",\"contentUrl\":\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/11\/jeremyrobert.jpeg\",\"caption\":\"J\u00e9r\u00e9my Robert\"},\"description\":\"J\u00e9r\u00e9my Robert est actuellement Directeur p\u00e9dagogique B2B chez DataScientest.com. Avant de rejoindre DataScientest, il a travaill\u00e9 en tant que Consultant Business Intelligence chez DEAPDATA, o\u00f9 il a g\u00e9r\u00e9 des projets dans les domaines de l'assurance, bancaire et industrie. Il a \u00e9galement \u00e9t\u00e9 Co-fondateur et pr\u00e9sident de Tripzer. En tant que consultant ind\u00e9pendant, J\u00e9r\u00e9my a propos\u00e9 des formations sur MicroStrategy pour divers niveaux professionnels. Il est dipl\u00f4m\u00e9 de l'Universit\u00e9 Paris Cit\u00e9 avec un Master MIAGE, combinant comp\u00e9tences en informatique et gestion.\",\"sameAs\":[\"https:\/\/www.linkedin.com\/in\/jeremyrobert1\/\",\"Data Science\"],\"url\":\"https:\/\/datascientest.com\/author\/jeremyrobert\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Scrapy : tout savoir sur cet outil Python de web scraping","description":"Avec le langage de programmation Python, les outils les plus utilis\u00e9s dans le domaine du web scraping sont BeautifulSoup et Scrapy Crawler. Dans cet article, nous allons pr\u00e9senter quelques diff\u00e9rences entre ces deux outils et nous concentrer sur Scrapy par la suite.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/datascientest.com\/scrapy-web-scraping-tout-savoir","og_locale":"fr_FR","og_type":"article","og_title":"Scrapy : tout savoir sur cet outil Python de web scraping","og_url":"https:\/\/datascientest.com\/scrapy-web-scraping-tout-savoir","og_site_name":"Formation Data Science | DataScientest.com","article_published_time":"2023-04-13T14:53:21+00:00","article_modified_time":"2023-11-09T13:53:27+00:00","og_image":[{"width":1025,"height":563,"url":"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/04\/Illu_BLOG_scrapy.png","type":"image\/png"}],"author":"J\u00e9r\u00e9my Robert","twitter_card":"summary_large_image","twitter_misc":{"\u00c9crit par":"J\u00e9r\u00e9my Robert","Dur\u00e9e de lecture estim\u00e9e":"17 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/datascientest.com\/scrapy-web-scraping-tout-savoir","url":"https:\/\/datascientest.com\/scrapy-web-scraping-tout-savoir","name":"Scrapy : tout savoir sur cet outil Python de web scraping","isPartOf":{"@id":"https:\/\/datascientest.com\/#website"},"datePublished":"2023-04-13T14:53:21+00:00","dateModified":"2023-11-09T13:53:27+00:00","author":{"@id":"https:\/\/datascientest.com\/#\/schema\/person\/7af2361c48e24e79a8aa2360e6e74925"},"description":"Avec le langage de programmation Python, les outils les plus utilis\u00e9s dans le domaine du web scraping sont BeautifulSoup et Scrapy Crawler. Dans cet article, nous allons pr\u00e9senter quelques diff\u00e9rences entre ces deux outils et nous concentrer sur Scrapy par la suite.","inLanguage":"fr-FR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/datascientest.com\/scrapy-web-scraping-tout-savoir"]}]},{"@type":"WebSite","@id":"https:\/\/datascientest.com\/#website","url":"https:\/\/datascientest.com\/","name":"Formation Data Science | DataScientest.com","description":"Leader de la formation en Data Science","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/datascientest.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"fr-FR"},{"@type":"Person","@id":"https:\/\/datascientest.com\/#\/schema\/person\/7af2361c48e24e79a8aa2360e6e74925","name":"J\u00e9r\u00e9my Robert","image":{"@type":"ImageObject","inLanguage":"fr-FR","@id":"https:\/\/datascientest.com\/#\/schema\/person\/image\/","url":"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/11\/jeremyrobert.jpeg","contentUrl":"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/11\/jeremyrobert.jpeg","caption":"J\u00e9r\u00e9my Robert"},"description":"J\u00e9r\u00e9my Robert est actuellement Directeur p\u00e9dagogique B2B chez DataScientest.com. Avant de rejoindre DataScientest, il a travaill\u00e9 en tant que Consultant Business Intelligence chez DEAPDATA, o\u00f9 il a g\u00e9r\u00e9 des projets dans les domaines de l'assurance, bancaire et industrie. Il a \u00e9galement \u00e9t\u00e9 Co-fondateur et pr\u00e9sident de Tripzer. En tant que consultant ind\u00e9pendant, J\u00e9r\u00e9my a propos\u00e9 des formations sur MicroStrategy pour divers niveaux professionnels. Il est dipl\u00f4m\u00e9 de l'Universit\u00e9 Paris Cit\u00e9 avec un Master MIAGE, combinant comp\u00e9tences en informatique et gestion.","sameAs":["https:\/\/www.linkedin.com\/in\/jeremyrobert1\/","Data Science"],"url":"https:\/\/datascientest.com\/author\/jeremyrobert"}]}},"_links":{"self":[{"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/posts\/196695"}],"collection":[{"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/users\/78"}],"replies":[{"embeddable":true,"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/comments?post=196695"}],"version-history":[{"count":0,"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/posts\/196695\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/media\/196757"}],"wp:attachment":[{"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/media?parent=196695"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/categories?post=196695"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/tags?post=196695"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}