{"id":174070,"date":"2023-11-27T10:26:18","date_gmt":"2023-11-27T09:26:18","guid":{"rendered":"https:\/\/datascientest.com\/en\/?p=174070"},"modified":"2023-11-27T10:32:04","modified_gmt":"2023-11-27T09:32:04","slug":"selenium-python-euronews-web-scraping-case-study","status":"publish","type":"post","link":"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study","title":{"rendered":"Selenium Python: Euronews web scraping case study"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"174070\" class=\"elementor elementor-174070\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-2649d56 e-flex e-con-boxed e-con\" data-id=\"2649d56\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-f24d07d elementor-widget elementor-widget-heading\" data-id=\"f24d07d\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-heading-title{padding:0;margin:0;line-height:1}.elementor-widget-heading .elementor-heading-title[class*=elementor-size-]>a{color:inherit;font-size:inherit;line-height:inherit}.elementor-widget-heading .elementor-heading-title.elementor-size-small{font-size:15px}.elementor-widget-heading .elementor-heading-title.elementor-size-medium{font-size:19px}.elementor-widget-heading .elementor-heading-title.elementor-size-large{font-size:29px}.elementor-widget-heading .elementor-heading-title.elementor-size-xl{font-size:39px}.elementor-widget-heading .elementor-heading-title.elementor-size-xxl{font-size:59px}<\/style><h3 class=\"elementor-heading-title elementor-size-default\">What is web scraping?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b3614fb elementor-widget elementor-widget-text-editor\" data-id=\"b3614fb\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t<p>As a Data Scientist, <a href=\"https:\/\/datascientest.com\/en\/data-engineer-course\">Data Engineer<\/a>, or Data Analyst, you often find yourself handling datasets that will feed into an application, a <a href=\"https:\/\/datascientest.com\/en\/cracking-the-code-on-underfitting-in-machine-learning\">Machine Learning algorithm, or data analysis.<\/a><\/p><p>In some cases, you may use pre-constructed datasets, but there are instances where you need to create your own dataset or supplement an existing one by collecting information from the internet.<\/p><p>Manually retrieving information, such as copying and pasting the content you need from each webpage, is impractical when dealing with a substantial amount of data. Fortunately, automating the collection of online content is possible <a href=\"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool\">through web scraping.<\/a><\/p><p><a href=\"https:\/\/datascientest.com\/en\/web-scraping-unveiled-your-comprehensive-guide-to-data-extraction\">Web scraping refers to the automatic retrieval of data and content from the internet.<\/a> Python offers several libraries that enable web scraping.<\/p><p>Among them, Beautiful Soup is perhaps the most well-known and has the advantage of being user-friendly.<\/p><p>However, it is not capable of retrieving dynamically inserted information on a webpage via JavaScript.<\/p><p>Today, we will focus on the Selenium <a href=\"https:\/\/datascientest.com\/en\/flask-one-of-the-most-popular-python-frameworks\">framework<\/a>, which, among other benefits, allows for this capability.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-076c43e e-flex e-con-boxed e-con\" data-id=\"076c43e\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-0ae6d19 elementor-widget elementor-widget-heading\" data-id=\"0ae6d19\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Why use Selenium for web scrapping?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fb34be5 elementor-widget elementor-widget-text-editor\" data-id=\"fb34be5\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><strong>Selenium<\/strong> is an <a href=\"https:\/\/datascientest.com\/en\/open-source-definition-use-cases-and-more\">open-source framework<\/a> originally developed for running automated tests on various web browsers (Chrome, Internet Explorer, Firefox, Safari, etc.). This library is also used for web scraping because it enables navigation between web pages and interaction with page elements as a real user would. This makes it possible to scrape dynamic websites, which are sites that return specific results to the user based on their actions.<\/p><p>In the rest of this article, we will see how to use <strong>Selenium<\/strong> to retrieve news articles from the Euronews website and create a dataframe ready for use in a text mining project.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-9134e08 e-flex e-con-boxed e-con\" data-id=\"9134e08\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-791945b elementor-widget elementor-widget-heading\" data-id=\"791945b\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Case study: web scraping of press articles from Euronews<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a92d0a4 elementor-widget elementor-widget-text-editor\" data-id=\"a92d0a4\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>To implement our web scraping script and analysis, we will use Jupyter Notebook. First, it will be necessary to install the various libraries required for using Selenium via a pip install in your Python environment.<\/p><p><em>pip install selenium<\/em><br \/><em>pip install webdriver_manager<\/em><\/p><p>The Selenium module provides access to:<\/p><ul><li>The Selenium webdriver, an essential component that interprets our code, here in the <a href=\"https:\/\/datascientest.com\/en\/scikit-learn-discover-the-python-library-dedicated-to-machine-learning\">Python language<\/a>, and interacts with the browser. In this tutorial, we will use a web driver that controls the Chrome browser.<\/li><li>The By method, which allows us to interact with the DOM and find elements on the web page.<\/li><\/ul><p>The <strong>WebDriver Manager<\/strong> module ensures the management (download, configuration, and maintenance) of the drivers required by Selenium WebDriver.<\/p><p>Now let&#8217;s import the libraries &#8211; we&#8217;ll need:<\/p><ul><li>import time<\/li><li>import pandas as pd<\/li><li>from selenium import webdriver<\/li><li>from selenium.webdriver.common.by import By<\/li><li>from selenium.webdriver.common.keys import Keys<\/li><li>from webdriver_manager.chrome import<\/li><li>ChromeDriverManager<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-735c734 elementor-widget elementor-widget-shortcode\" data-id=\"735c734\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/maelysBB\/dc5da1873f08e24214890f9cc813fcce.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a7b804f elementor-widget elementor-widget-text-editor\" data-id=\"a7b804f\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>We&#8217;re ready to dive into the subject. We will scrape articles from <strong>Euronews published yesterday.<\/strong><\/p><p>Here&#8217;s an overview of the different steps we will perform:<\/p><p>1. Navigate to the archive page of articles from the previous day.<\/p><p><br \/>2. Retrieve the links to all articles published yesterday and store them in a list.<\/p><p><br \/>3. Iterate over the links in our list and create another list containing a dictionary for each article with:<\/p><p><br \/>&#8211; The article&#8217;s title<br \/>&#8211; The date it was published<br \/>&#8211; The authors<br \/>&#8211; The paragraphs of the article<br \/>&#8211; The category<br \/>&#8211; The link to the article.<\/p><p>4. Conduct a quick visual analysis of our text corpus.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-cf37424 e-flex e-con-boxed e-con\" data-id=\"cf37424\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-206e9b1 elementor-widget elementor-widget-heading\" data-id=\"206e9b1\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Step 1: Use Selenium to access previous day's articles<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d4acabb elementor-widget elementor-widget-text-editor\" data-id=\"d4acabb\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>First, let&#8217;s take a look at the structure of Euronews&#8217; archive URLs. To retrieve articles from a specific day, the URL follows this format: https:\/\/www.euronews.com\/{year}\/{month}\/{day}.<\/p><p>We also notice that there is pagination with 30 articles per page. When the number of articles exceeds 30, you can navigate to other pages by adding a query parameter to the URL: https:\/\/www.euronews.com\/{year}\/{month}\/{day}?p={page}<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-106875c elementor-widget elementor-widget-image\" data-id=\"106875c\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-image{text-align:center}.elementor-widget-image a{display:inline-block}.elementor-widget-image a img[src$=\".svg\"]{width:48px}.elementor-widget-image img{vertical-align:middle;display:inline-block}<\/style>\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2022\/12\/image9.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-36acdde elementor-widget elementor-widget-text-editor\" data-id=\"36acdde\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>To retrieve the date of the previous day, we use the datetime library and create three variables containing the year, month, and day.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-76efeea elementor-widget elementor-widget-shortcode\" data-id=\"76efeea\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/maelysBB\/888cad7dab9cecbfa281b354e8ec52be.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0afc449 elementor-widget elementor-widget-text-editor\" data-id=\"0afc449\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Let&#8217;s test the<strong> Selenium library<\/strong> by initializing a Chrome webdriver. The code below allows us to automate with Selenium everything we would have done manually: opening the browser, accessing the Euronews page containing articles from the previous day, clicking the cookie acceptance button, and scrolling on the page.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e63c913 elementor-widget elementor-widget-shortcode\" data-id=\"e63c913\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/maelysBB\/88df6c9ce52adcb1aa96f0e2afbc7a7b.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6fd11d0 elementor-widget elementor-widget-text-editor\" data-id=\"6fd11d0\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"flex flex-grow flex-col max-w-full gap-3 gizmo:gap-0\"><div class=\"min-h-[20px] text-message flex flex-col items-start gap-3 whitespace-pre-wrap break-words [.text-message+&amp;]:mt-5 overflow-x-auto\" data-message-author-role=\"assistant\" data-message-id=\"f54420f2-d2f5-4e4d-bf14-14da4a86535b\"><div class=\"markdown prose w-full break-words dark:prose-invert light\"><p>It&#8217;s worth noting that the &#8220;scroll&#8221; function has been defined in a cell located higher up in the notebook.<\/p><\/div><\/div><\/div>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-21fda75 elementor-widget elementor-widget-shortcode\" data-id=\"21fda75\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/maelysBB\/d02d6ab0f3d82d13ee7629c5c66a873f.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-21b22ad elementor-widget elementor-widget-text-editor\" data-id=\"21b22ad\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Here is a video showing the result of our code:<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-673bb8c elementor-widget__width-initial elementor-widget elementor-widget-video\" data-id=\"673bb8c\" data-element_type=\"widget\" data-settings=\"{&quot;youtube_url&quot;:&quot;https:\\\/\\\/youtu.be\\\/EVs2QuMydoA&quot;,&quot;video_type&quot;:&quot;youtube&quot;,&quot;controls&quot;:&quot;yes&quot;}\" data-widget_type=\"video.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-video .elementor-widget-container{overflow:hidden;transform:translateZ(0)}.elementor-widget-video .elementor-wrapper{aspect-ratio:var(--video-aspect-ratio)}.elementor-widget-video .elementor-wrapper iframe,.elementor-widget-video .elementor-wrapper video{height:100%;width:100%;display:flex;border:none;background-color:#000}@supports not (aspect-ratio:1\/1){.elementor-widget-video .elementor-wrapper{position:relative;overflow:hidden;height:0;padding-bottom:calc(100% \/ var(--video-aspect-ratio))}.elementor-widget-video .elementor-wrapper iframe,.elementor-widget-video .elementor-wrapper video{position:absolute;top:0;right:0;bottom:0;left:0}}.elementor-widget-video .elementor-open-inline .elementor-custom-embed-image-overlay{position:absolute;top:0;right:0;bottom:0;left:0;background-size:cover;background-position:50%}.elementor-widget-video .elementor-custom-embed-image-overlay{cursor:pointer;text-align:center}.elementor-widget-video .elementor-custom-embed-image-overlay:hover .elementor-custom-embed-play i{opacity:1}.elementor-widget-video .elementor-custom-embed-image-overlay img{display:block;width:100%;aspect-ratio:var(--video-aspect-ratio);-o-object-fit:cover;object-fit:cover;-o-object-position:center center;object-position:center center}@supports not (aspect-ratio:1\/1){.elementor-widget-video .elementor-custom-embed-image-overlay{position:relative;overflow:hidden;height:0;padding-bottom:calc(100% \/ var(--video-aspect-ratio))}.elementor-widget-video .elementor-custom-embed-image-overlay img{position:absolute;top:0;right:0;bottom:0;left:0}}.elementor-widget-video .e-hosted-video .elementor-video{-o-object-fit:cover;object-fit:cover}.e-con-inner>.elementor-widget-video,.e-con>.elementor-widget-video{width:var(--container-widget-width);--flex-grow:var(--container-widget-flex-grow)}<\/style>\t\t<div class=\"elementor-wrapper elementor-open-inline\">\n\t\t\t<div class=\"elementor-video\"><\/div>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-88c7579 elementor-widget elementor-widget-text-editor\" data-id=\"88c7579\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p style=\"text-align: center;\">Taking control of a Chrome browser with Selenium<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-ed93adf e-flex e-con-boxed e-con\" data-id=\"ed93adf\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-dd0e599 elementor-widget elementor-widget-heading\" data-id=\"dd0e599\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Step 2: Retrieve article links<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1c72228 elementor-widget elementor-widget-text-editor\" data-id=\"1c72228\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>We notice that the number of articles for a given day is written on the web page, which will tell us how many pages we need to <strong>retrieve the articles from:<\/strong><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5a27d53 elementor-widget elementor-widget-image\" data-id=\"5a27d53\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2022\/12\/image4-1.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e55b894 elementor-widget elementor-widget-text-editor\" data-id=\"e55b894\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Regardless of the web scraping libraries used, it&#8217;s essential to inspect the page to identify the elements of the web page that interest us. We can access them via <strong>CSS selectors.<\/strong><\/p><p>To inspect a page, right-click and then click &#8220;Inspect,&#8221; or use the shortcut Ctrl + Cmd + C on macOS or Ctrl + Shift + C on Windows and Linux.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-85209c7 elementor-widget__width-initial elementor-widget elementor-widget-video\" data-id=\"85209c7\" data-element_type=\"widget\" data-settings=\"{&quot;youtube_url&quot;:&quot;https:\\\/\\\/youtu.be\\\/AAWuDSM20Kc&quot;,&quot;video_type&quot;:&quot;youtube&quot;,&quot;controls&quot;:&quot;yes&quot;}\" data-widget_type=\"video.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-wrapper elementor-open-inline\">\n\t\t\t<div class=\"elementor-video\"><\/div>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d8b47f7 elementor-widget elementor-widget-text-editor\" data-id=\"d8b47f7\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p style=\"text-align: center;\">How do I inspect a web page?<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-95e4799 elementor-widget elementor-widget-text-editor\" data-id=\"95e4799\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Thanks to <strong>Selenium&#8217;s BY class<\/strong> (which we&#8217;ll use instead of the .find_elements_by method), we can choose between several types of properties to locate an element on a web page, including :<\/p><ul><li>Element ID<\/li><li>Element name<\/li><li>Tag name<\/li><li>Xpath<\/li><li>Its class<\/li><\/ul><p>After inspection, we see that the element counter is located in the first strong tag of the &#8220;c-block-listing__results&#8221; class.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7bbd9a6 elementor-widget elementor-widget-image\" data-id=\"7bbd9a6\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2022\/12\/image3-1.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f8c85a2 elementor-widget elementor-widget-text-editor\" data-id=\"f8c85a2\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>To retrieve the element, we will use its XPath. XPath is an XML path that allows you to navigate the <strong>HTML structure<\/strong> of a page and find any element on it. In web scraping, it is particularly useful to use the XPath path of an element when the element&#8217;s class alone is not sufficient to identify it, for example, when we want to retrieve a subtag of an element with a certain class.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-33198f6 elementor-widget elementor-widget-shortcode\" data-id=\"33198f6\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/maelysBB\/212c46f5fbee135913332791a3856107.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-76ef6b7 elementor-widget elementor-widget-text-editor\" data-id=\"76ef6b7\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>The Xpath of an HTML element is structured as follows:<\/p><pre>\/\/tag_name_of_major_element[@attribute=\"Value\"]\/tag_name_of_sub_element[index_of_the_sub_element]<\/pre><p>Where:<\/p><ul><li>tag_name_of_major_element corresponds to the main tag (div, img, p, b etc.)<\/li><li>attribute corresponds to the element&#8217;s class or id and value to its value<\/li><li>tag_name_of_sub_element corresponds to the tag of the sub tag<\/li><li>index_of_the_sub_element starts at 1 and corresponds to the number of the sub-tag. This index is only used if there are several sub-elements contained in the main tag.<\/li><\/ul><p>The <strong>method .text<\/strong>, allows us to access the text value of the element, here &#8220;57&#8221;.<\/p><p>Using the number of items retrieved and the pagination at 30 we&#8217;ve observed, we calculate the number of pages we&#8217;ll need to scrape, in this case 2.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b86b41f elementor-widget elementor-widget-shortcode\" data-id=\"b86b41f\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/maelysBB\/73b301de563edf363c9f4c941709eff1.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e2b6425 elementor-widget elementor-widget-text-editor\" data-id=\"e2b6425\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>We&#8217;re now going to retrieve the links for the various articles. On inspection, we notice that the article links have the object__title__link class. We retrieve all elements with this class and create a list containing, for each element, the value of the href attribute (which contains the link) retrieved using the get attribute method.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f07a935 elementor-widget elementor-widget-shortcode\" data-id=\"f07a935\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/maelysBB\/97f046499ed29c2a0db6c99474eafcfb.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-f92af99 e-flex e-con-boxed e-con\" data-id=\"f92af99\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-b592b93 elementor-widget elementor-widget-heading\" data-id=\"b592b93\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Step 3: Scraping the content of each article<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-de160ce elementor-widget elementor-widget-text-editor\" data-id=\"de160ce\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>We&#8217;re now entering the crucial part of this article: scraping each article from the list created earlier. Let&#8217;s inspect the structure of the<strong> HTML page:<\/strong><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e5d25d1 elementor-widget elementor-widget-image\" data-id=\"e5d25d1\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2022\/12\/image18.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fe95b18 elementor-widget elementor-widget-text-editor\" data-id=\"fe95b18\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"flex flex-grow flex-col max-w-full gap-3 gizmo:gap-0\"><div class=\"min-h-[20px] text-message flex flex-col items-start gap-3 whitespace-pre-wrap break-words [.text-message+&amp;]:mt-5 overflow-x-auto\" data-message-author-role=\"assistant\" data-message-id=\"5166f4e6-33d4-474e-91d8-7d96c8df645b\"><div class=\"markdown prose w-full break-words dark:prose-invert light\"><p>We can observe that the contextual data we&#8217;re interested in, such as the title, author, category, etc., are consistently in the second position among the located classes. Indeed, in the first position, the HTML contains information about the previous article, which is not displayed on the page.<\/p><p>The code below retrieves the content we&#8217;re interested in on the page:<\/p><\/div><\/div><\/div>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-84c4753 elementor-widget elementor-widget-shortcode\" data-id=\"84c4753\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/maelysBB\/9de95e92205f6de93bca7ac99bc5f5c5.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b46e319 elementor-widget elementor-widget-text-editor\" data-id=\"b46e319\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Here are the <strong>different steps of the previous code:<\/strong><\/p><p>1. Access each article.<br \/>2. Retrieve the text from the contextual data of the article (title, authors, category, publication date).<br \/>3. Retrieve the paragraphs of the article.<br \/>4. Create a dictionary with the retrieved data.<br \/>5. Add this dictionary to a list named &#8220;list_of_articles.&#8221;<br \/>6. In case of an error, record the link of the article in an &#8220;errors&#8221; list.<\/p><p><strong>Note:<\/strong> It&#8217;s noticeable that the majority of articles in error are video articles with a different HTML structure. For the purpose of this article, we decide not to handle these few error cases.<\/p><p>Here is a video showing the result of our code for the first scraped pages.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-27c5302 elementor-widget__width-initial elementor-widget elementor-widget-video\" data-id=\"27c5302\" data-element_type=\"widget\" data-settings=\"{&quot;youtube_url&quot;:&quot;https:\\\/\\\/youtu.be\\\/OC5UYTrKkJc&quot;,&quot;video_type&quot;:&quot;youtube&quot;,&quot;controls&quot;:&quot;yes&quot;}\" data-widget_type=\"video.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-wrapper elementor-open-inline\">\n\t\t\t<div class=\"elementor-video\"><\/div>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1278510 elementor-widget elementor-widget-text-editor\" data-id=\"1278510\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p style=\"text-align: center;\">Automatic page navigation with Selenium<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-0c63e02 e-flex e-con-boxed e-con\" data-id=\"0c63e02\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-c60833b elementor-widget elementor-widget-heading\" data-id=\"c60833b\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Step 4: Quick view<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cf7759b elementor-widget elementor-widget-text-editor\" data-id=\"cf7759b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Before concluding, we&#8217;ll take a quick look at the scraped articles. This is one of the first steps in a <a href=\"https:\/\/datascientest.com\/en\/chatgpt-how-does-this-nlp-algorithm-work\">Natural Processing Language (NLP) project.<\/a><\/p><p>First, we import the libraries we&#8217;ll need for the rest of the process, then build a dataframe using the list_of_articles dictionary list.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d44f9db elementor-widget elementor-widget-shortcode\" data-id=\"d44f9db\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/maelysBB\/c37ce213912ef028f781a23d9bef25ad.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-12cb9a1 elementor-widget elementor-widget-text-editor\" data-id=\"12cb9a1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Here are the first lines of the dataframe:<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b1c8cce elementor-widget elementor-widget-image\" data-id=\"b1c8cce\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2022\/12\/image19.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0d4ff15 elementor-widget elementor-widget-text-editor\" data-id=\"0d4ff15\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>First, we&#8217;ll look at the different article categories, displaying the number of articles per category using the Seaborn library.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0aac3f6 elementor-widget elementor-widget-shortcode\" data-id=\"0aac3f6\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/maelysBB\/08fe126f1ef2e5e422b409fd5f20ae27.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-29e1505 elementor-widget elementor-widget-image\" data-id=\"29e1505\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2022\/12\/image2-2.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7acdb2b elementor-widget elementor-widget-text-editor\" data-id=\"7acdb2b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Before displaying the word cloud, we remove stopwords using the stopwords class in the nltk.corpus package.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e9b3357 elementor-widget elementor-widget-shortcode\" data-id=\"e9b3357\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/maelysBB\/e187fc36689503fd9a4c3eb8cbf16ae4.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9dba8a1 elementor-widget elementor-widget-text-editor\" data-id=\"9dba8a1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Finally, we define a function to build a word cloud using the wordcloud module, which takes our article series as an argument.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-351abc8 elementor-widget elementor-widget-shortcode\" data-id=\"351abc8\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/maelysBB\/f87f301a2dfa3bad480d27bb0c5b1b09.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9108387 elementor-widget elementor-widget-image\" data-id=\"9108387\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2022\/12\/image7.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7aa685b elementor-widget elementor-widget-text-editor\" data-id=\"7aa685b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>That&#8217;s it, we&#8217;ve come to the end of this tutorial, which has shown how Selenium can help you scrape data from the Internet, and given us the opportunity to perform a <strong>quick visual analysis on textual data.<\/strong><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b881f0c elementor-widget elementor-widget-text-editor\" data-id=\"b881f0c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>If you&#8217;d like to find out more about how web scraping is used in the various data professions, <a href=\"https:\/\/datascientest.com\/en\/how-to-become-a-data-scientist-the-the-best-kept-secrets\">take a look at our article on the subject<\/a>, or find out more about the content of our various courses.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-bf485b1 e-flex e-con-boxed e-con\" data-id=\"bf485b1\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-2b5d169 elementor-widget elementor-widget-html\" data-id=\"2b5d169\" data-element_type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<script type=\"application\/ld+json\">\n{\n  \"@context\": \"https:\/\/schema.org\",\n  \"@type\": \"FAQPage\",\n  \"mainEntity\": [{\n    \"@type\": \"Question\",\n    \"name\": \"What is web scraping?\",\n    \"acceptedAnswer\": {\n      \"@type\": \"Answer\",\n      \"text\": \"Manually retrieving information, for example by copying and pasting the content you're interested in on each page, is unthinkable if the amount of information to be retrieved is large. Fortunately, it's possible to automate the collection of content on the Internet thanks to web scraping. Web scraping is the automatic retrieval of data and content from the Internet. Python has several libraries for web scraping.\"\n    }\n  },{\n    \"@type\": \"Question\",\n    \"name\": \"Why use Selenium to web scrape data?\",\n    \"acceptedAnswer\": {\n      \"@type\": \"Answer\",\n      \"text\": \"Selenium is an open-source framework initially developed to run automated tests on various browsers (Chrome, Internet Explorer, Firefox, Safari,...). This library is also used for web scraping, as it enables users to navigate between web pages and interact with page elements like a real user. In this way, it becomes possible to scrape dynamic web sites, i.e. sites that return a certain result to the user depending on his or her actions.\"\n    }\n  }]\n}\n<\/script>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-0fdb068 e-flex e-con-boxed e-con\" data-id=\"0fdb068\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-9f43911 elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"9f43911\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/en\/data-scientist-course\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Discover our Data Science training courses<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>What is web scraping? As a Data Scientist, Data Engineer, or Data Analyst, you often find yourself handling datasets that will feed into an application, a Machine Learning algorithm, or data analysis. In some cases, you may use pre-constructed datasets, but there are instances where you need to create your own dataset or supplement an [&hellip;]<\/p>\n","protected":false},"author":67,"featured_media":174072,"comment_status":"open","ping_status":"open","sticky":false,"template":"elementor_theme","format":"standard","meta":{"content-type":"","inline_featured_image":false},"categories":[2425],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.7 (Yoast SEO v21.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Selenium Python: Euronews web scraping case study<\/title>\n<meta name=\"description\" content=\"Selenium is a powerful tool for automating web browser interactions, and Python is a popular programming language for web scraping and automation tasks\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Selenium Python: Euronews web scraping case study\" \/>\n<meta property=\"og:description\" content=\"What is web scraping? As a Data Scientist, Data Engineer, or Data Analyst, you often find yourself handling datasets that will feed into an application, a\" \/>\n<meta property=\"og:url\" content=\"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study\" \/>\n<meta property=\"og:site_name\" content=\"Data Science Courses | DataScientest\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataScientestEN\" \/>\n<meta property=\"article:published_time\" content=\"2023-11-27T09:26:18+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-11-27T09:32:04+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/datascientest.com\/en\/files\/2023\/11\/selenium-webscraping-1.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"421\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"Melanie\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:site\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Melanie\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study#article\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study\"},\"author\":{\"name\":\"Melanie\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d\"},\"headline\":\"Selenium Python: Euronews web scraping case study\",\"datePublished\":\"2023-11-27T09:26:18+00:00\",\"dateModified\":\"2023-11-27T09:32:04+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study\"},\"wordCount\":1565,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"articleSection\":[\"Data Science\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study#respond\"]}],\"copyrightYear\":\"2023\",\"copyrightHolder\":{\"@id\":\"https:\/\/datascientest.com\/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study\",\"url\":\"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study\",\"name\":\"Selenium Python: Euronews web scraping case study\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/#website\"},\"datePublished\":\"2023-11-27T09:26:18+00:00\",\"dateModified\":\"2023-11-27T09:32:04+00:00\",\"description\":\"Selenium is a powerful tool for automating web browser interactions, and Python is a popular programming language for web scraping and automation tasks\",\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/datascientest.com\/en\/#website\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"name\":\"Data Science Courses | DataScientest\",\"description\":\"Leader in Data Science Courses\",\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/datascientest.com\/en\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/datascientest.com\/en\/#organization\",\"name\":\"DataScientest\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"contentUrl\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"width\":301,\"height\":301,\"caption\":\"DataScientest\"},\"image\":{\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/DataScientestEN\",\"https:\/\/twitter.com\/DataScientest__\",\"https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/\",\"https:\/\/www.tiktok.com\/@datascientest\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d\",\"name\":\"Melanie\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g\",\"caption\":\"Melanie\"},\"url\":\"https:\/\/datascientest.com\/en\/author\/melaniekrause\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Selenium Python: Euronews web scraping case study","description":"Selenium is a powerful tool for automating web browser interactions, and Python is a popular programming language for web scraping and automation tasks","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study","og_locale":"en_GB","og_type":"article","og_title":"Selenium Python: Euronews web scraping case study","og_description":"What is web scraping? As a Data Scientist, Data Engineer, or Data Analyst, you often find yourself handling datasets that will feed into an application, a","og_url":"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study","og_site_name":"Data Science Courses | DataScientest","article_publisher":"https:\/\/www.facebook.com\/DataScientestEN","article_published_time":"2023-11-27T09:26:18+00:00","article_modified_time":"2023-11-27T09:32:04+00:00","og_image":[{"width":1024,"height":421,"url":"https:\/\/datascientest.com\/en\/files\/2023\/11\/selenium-webscraping-1.webp","type":"image\/webp"}],"author":"Melanie","twitter_card":"summary_large_image","twitter_creator":"@DataScientest__","twitter_site":"@DataScientest__","twitter_misc":{"Written by":"Melanie","Estimated reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study#article","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study"},"author":{"name":"Melanie","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d"},"headline":"Selenium Python: Euronews web scraping case study","datePublished":"2023-11-27T09:26:18+00:00","dateModified":"2023-11-27T09:32:04+00:00","mainEntityOfPage":{"@id":"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study"},"wordCount":1565,"commentCount":0,"publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"articleSection":["Data Science"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study#respond"]}],"copyrightYear":"2023","copyrightHolder":{"@id":"https:\/\/datascientest.com\/#organization"}},{"@type":"WebPage","@id":"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study","url":"https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study","name":"Selenium Python: Euronews web scraping case study","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/#website"},"datePublished":"2023-11-27T09:26:18+00:00","dateModified":"2023-11-27T09:32:04+00:00","description":"Selenium is a powerful tool for automating web browser interactions, and Python is a popular programming language for web scraping and automation tasks","inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/datascientest.com\/en\/selenium-python-euronews-web-scraping-case-study"]}]},{"@type":"WebSite","@id":"https:\/\/datascientest.com\/en\/#website","url":"https:\/\/datascientest.com\/en\/","name":"Data Science Courses | DataScientest","description":"Leader in Data Science Courses","publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/datascientest.com\/en\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/datascientest.com\/en\/#organization","name":"DataScientest","url":"https:\/\/datascientest.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","contentUrl":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","width":301,"height":301,"caption":"DataScientest"},"image":{"@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataScientestEN","https:\/\/twitter.com\/DataScientest__","https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/","https:\/\/www.tiktok.com\/@datascientest"]},{"@type":"Person","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d","name":"Melanie","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g","caption":"Melanie"},"url":"https:\/\/datascientest.com\/en\/author\/melaniekrause"}]}},"_links":{"self":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/174070"}],"collection":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/users\/67"}],"replies":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/comments?post=174070"}],"version-history":[{"count":0,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/174070\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media\/174072"}],"wp:attachment":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media?parent=174070"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/categories?post=174070"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/tags?post=174070"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}