{"id":177150,"date":"2024-01-20T16:43:09","date_gmt":"2024-01-20T15:43:09","guid":{"rendered":"https:\/\/datascientest.com\/en\/?p=177150"},"modified":"2024-01-21T09:17:39","modified_gmt":"2024-01-21T08:17:39","slug":"beautiful-soup-introduction-to-web-scraping-with-python","status":"publish","type":"post","link":"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python","title":{"rendered":"Beautiful Soup: Introduction to web scraping with Python"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"177150\" class=\"elementor elementor-177150\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-ec58ff2 e-flex e-con-boxed e-con\" data-id=\"ec58ff2\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-87a4f0d e-con-full e-flex e-con\" data-id=\"87a4f0d\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;full&quot;}\">\n\t\t\t\t<div class=\"elementor-element elementor-element-fdeb8a3 elementor-widget elementor-widget-heading\" data-id=\"fdeb8a3\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-heading-title{padding:0;margin:0;line-height:1}.elementor-widget-heading .elementor-heading-title[class*=elementor-size-]>a{color:inherit;font-size:inherit;line-height:inherit}.elementor-widget-heading .elementor-heading-title.elementor-size-small{font-size:15px}.elementor-widget-heading .elementor-heading-title.elementor-size-medium{font-size:19px}.elementor-widget-heading .elementor-heading-title.elementor-size-large{font-size:29px}.elementor-widget-heading .elementor-heading-title.elementor-size-xl{font-size:39px}.elementor-widget-heading .elementor-heading-title.elementor-size-xxl{font-size:59px}<\/style><h2 class=\"elementor-heading-title elementor-size-default\">As an Internet browser, we have access to a lot of information about customers, offers, share prices, physical phenomena and so on. This data can be read by users, but we'd like to be able to exploit it by transforming it into an operational format so that we can analyze it and make the most of it. Web scraping is the technique that enables this information to be extracted in a format that can be used by computer programs. In this article, we'll find out how to do it with Beautiful Soup.<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-c392005 e-flex e-con-boxed e-con\" data-id=\"c392005\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-2796458 elementor-widget elementor-widget-heading\" data-id=\"2796458\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">What is Beautiful Soup?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-775a2ae elementor-widget elementor-widget-text-editor\" data-id=\"775a2ae\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t<p>For example, you might want to have access to all the reviews of a Pack of HP Black Ink Cartridges on Amazon, so you can perform syntactic, semantic and sentimental analysis and form your own opinion. <a href=\"https:\/\/datascientest.com\/en\/web-scraping-unveiled-your-comprehensive-guide-to-data-extraction\">Web scraping from a store locator (such as a map)<\/a> can be used to create a list of commercial locations. You can also obtain stock prices to help you make better investment decisions.<\/p><p>As far as data analysis is concerned, there are specific techniques for each type of data and each objective. The following diagram illustrates the &#8220;logistical&#8221; process involved in making informed decisions:<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-4802583 e-flex e-con-boxed e-con\" data-id=\"4802583\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-a2de633 elementor-widget elementor-widget-image\" data-id=\"a2de633\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-image{text-align:center}.elementor-widget-image a{display:inline-block}.elementor-widget-image a img[src$=\".svg\"]{width:48px}.elementor-widget-image img{vertical-align:middle;display:inline-block}<\/style>\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2022\/03\/schema_webscraping.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\">Logistical process for decision-making<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-5057181 e-flex e-con-boxed e-con\" data-id=\"5057181\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-688cdb6 elementor-widget elementor-widget-text-editor\" data-id=\"688cdb6\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><a href=\"https:\/\/datascientest.com\/en\/scrapy-everything-you-need-to-know-about-this-python-web-scraping-tool\">If we&#8217;re in the data retrieval phase<\/a>, we&#8217;d like to have access to all the information present on a web page, so that we can then carry out the desired study.<\/p><p>This can be done by copying it &#8220;by hand&#8221; into another document. But this can be a time-consuming process, not to mention the typing errors that can occur during input. As mentioned in the introduction, web scraping provides access to this information in a usable format.<\/p><p>In the second phase, we call on the technical skills of data analysts, data engineers or data scientists to implement algorithms and relevant statistical studies. For example, when analyzing product reviews, an <a href=\"https:\/\/datascientest.com\/en\/chatgpt-how-does-this-nlp-algorithm-work\">NLP algorithm can be used<\/a>, which enables machines to understand human language.<\/p><p>Data interpretation is often carried out within a team, taking into account the opinions of specialists in the field (for example, taking into account the doctor&#8217;s opinion if working on a project with medical data), to finally arrive at an optimal decision.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9391bbb elementor-widget elementor-widget-heading\" data-id=\"9391bbb\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Introducing Beautiful Soup<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-37704ec elementor-widget elementor-widget-text-editor\" data-id=\"37704ec\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>In this article, we&#8217;ll concentrate on Phase 1, since we&#8217;re interested in data retrieval.<\/p><p>In the next paragraph, we&#8217;ll take a look at an application of the Beautiful Soup library for <a href=\"https:\/\/datascientest.com\/en\/top-10-native-python-functions\">web scraping in Python.<\/a><\/p><p>In the next paragraph, we&#8217;ll take a look at a web scraping library available in Python that provides an excellent (and easy-to-handle) tool for extracting information from unstructured data: <strong>Beautiful Soup.<\/strong><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b1e0262 elementor-widget elementor-widget-text-editor\" data-id=\"b1e0262\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>The <strong>Beautiful Soup<\/strong><a href=\"https:\/\/datascientest.com\/en\/tpot-all-about-this-machine-learning-python-library\"> Python library<\/a> extracts content and transforms it into a Python list, array or dictionary.<\/p><p>This library is very popular because of its comprehensive documentation and well-structured functionality. What&#8217;s more, there&#8217;s a large community offering various solutions for using this library.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d96bf2b elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"d96bf2b\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/en\/data-analyst-course\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Learn how to use Beautiful Soup <\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f3763f3 elementor-widget elementor-widget-heading\" data-id=\"f3763f3\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Why is it called \"Beautiful Soup\"?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b4e5b73 elementor-widget elementor-widget-text-editor\" data-id=\"b4e5b73\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Websites are written in HTML and CSS, the computer languages used to lay out web pages. HTML is used to manage and organize content. CSS is used to manage the appearance of the web page (colors, text size, etc.).<\/p><p>In web development, &#8220;tag soup&#8221; is a derogatory term for syntactically or structurally incorrect <strong>HTML written for a web page.<\/strong><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-34f456e elementor-widget elementor-widget-heading\" data-id=\"34f456e\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">An example of web scraping with Beautiful Soup<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d1e1c0f elementor-widget elementor-widget-text-editor\" data-id=\"d1e1c0f\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Let&#8217;s take a simple example to familiarize ourselves with these notions. The following example has been <a href=\"https:\/\/datascientest.com\/en\/kaggle-everything-you-need-to-know-about-this-platform\">taken from Kaggle<\/a> and the aim is to scrape data on the world&#8217;s population. The data is available on the Worldometer site, an open source managed by an international team of volunteer developers and researchers, whose aim is to make global statistics available to a wide audience worldwide.<\/p><p>Here&#8217;s a sneak preview of the page we&#8217;ll be scraping:<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8a8aa78 elementor-widget elementor-widget-image\" data-id=\"8a8aa78\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"https:\/\/www.worldometers.info\/world-population\/population-by-country\/\">\n\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2022\/03\/beautifulsoup.png\" title=\"\" alt=\"\" loading=\"lazy\" \/>\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\"><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d154e71 elementor-widget elementor-widget-text-editor\" data-id=\"d154e71\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Our aim is to retrieve this table and transform it into a DataFrame without having to copy all the data by hand.<\/p><p>In a <a href=\"https:\/\/datascientest.com\/en\/ipython-discover-the-python-shell-at-the-heart-of-jupyter-notebook\">Jupyter Notebook<\/a>, we start by importing the necessary libraries.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8cdcc3a elementor-widget elementor-widget-shortcode\" data-id=\"8cdcc3a\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/adrianalazureanu\/028f62976cd9c15305db8e81fe3336bd.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6857466 elementor-widget elementor-widget-text-editor\" data-id=\"6857466\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<ul><li style=\"font-weight: 400;\" aria-level=\"1\">Then we create a url variable in string format (text) containing the link to the page in question.<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-69805b1 elementor-widget elementor-widget-shortcode\" data-id=\"69805b1\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/adrianalazureanu\/79b06296735683b0fd58da47330ee73f.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-23ed164 elementor-widget elementor-widget-text-editor\" data-id=\"23ed164\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>To prepare the data, use the requests.get() function:<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b983b5e elementor-widget elementor-widget-shortcode\" data-id=\"b983b5e\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/adrianalazureanu\/60d8ef58df7c62bd619d19b677ec6b4e.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ef6ee0e elementor-widget elementor-widget-text-editor\" data-id=\"ef6ee0e\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<ul><li style=\"font-weight: 400;\" aria-level=\"1\">Now that the data has been prepared, the BeautifulSoup() function is used to extract the HTML code from this page. In the argument to this function, we select the .text object.<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8491e1a elementor-widget elementor-widget-shortcode\" data-id=\"8491e1a\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/adrianalazureanu\/0641ca684077c53207aab3dd95b01611.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-eb0e7d0 elementor-widget elementor-widget-text-editor\" data-id=\"eb0e7d0\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<ul><li style=\"font-weight: 400;\" aria-level=\"1\">In the variable data we store the HTML code we search for the keyword &#8220;table&#8221; with the function .find_all() :<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2cc6e68 elementor-widget elementor-widget-shortcode\" data-id=\"2cc6e68\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/adrianalazureanu\/7c34c036fdb59f7c14edbc7bcac59b49.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-bb90faf elementor-widget elementor-widget-text-editor\" data-id=\"bb90faf\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<ul><li style=\"font-weight: 400;\" aria-level=\"1\">We use the .read_html(str()) command to have the machine read the HTML code and then retrieve the first and only element of this object (the array).<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1f99391 elementor-widget elementor-widget-shortcode\" data-id=\"1f99391\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/adrianalazureanu\/369ee11d330f4b308fd4bf44cbf273c0.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-bd63203 elementor-widget elementor-widget-text-editor\" data-id=\"bd63203\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Now we display the first elements, DataFrame .head() command:<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-26e62f0 elementor-widget elementor-widget-shortcode\" data-id=\"26e62f0\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/adrianalazureanu\/ff26184e4b98ff81b0311cee24c16c57.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7dbaedc elementor-widget elementor-widget-text-editor\" data-id=\"7dbaedc\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<ul><li style=\"font-weight: 400;\" aria-level=\"1\">You can also export the database in csv format with the following command:<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b4a4a7d elementor-widget elementor-widget-shortcode\" data-id=\"b4a4a7d\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\"><script src=\"https:\/\/gist.github.com\/adrianalazureanu\/33a12e7740b90c907b2bc1792a661296.js\"><\/script><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-119c31b elementor-widget elementor-widget-text-editor\" data-id=\"119c31b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>This command creates a file in csv format, located at the path indicated.<\/p><p>To make it easier to manipulate the DataFrame, consider changing the column names or eliminating some of them if you don&#8217;t use them.<\/p><p>Now that we&#8217;ve managed to obtain the data made available on the Worldometer site, and that it&#8217;s in <strong>DataFrame format,<\/strong> we can move on to the next steps and carry out our studies. We can then embark on the rest of the process (Phases 2, 3 and 4, see diagram in introduction). Depending on the nature of the data and the objectives to be achieved, different studies can be carried out: exploratory analysis, <a href=\"https:\/\/datascientest.com\/en\/adversarial-examples-definition-and-importance-in-machine-learning\">proposing a machine learning model<\/a>, time series modeling, etc.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ce66a8d elementor-widget elementor-widget-heading\" data-id=\"ce66a8d\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Beautiful Soup - Conclusion<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-bd037ac elementor-widget elementor-widget-text-editor\" data-id=\"bd037ac\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>We&#8217;ve just seen an example of how to retrieve data stored in a table, but it&#8217;s important to remember that different libraries and functions are used depending on the structure of the web page you want to scrape data from. The good news is that there are many examples available on the Internet, depending on the format of the page and the configuration of the data you want to scrape.<\/p><p>To sum up, web scraping enables you to navigate the Internet &#8220;intelligently&#8221;, making it a rich resource for any area of research or personal interest.<\/p><p><strong>Datascientest&#8217;s Data Analyst training course<\/strong> enables you to familiarize yourself with and put into practice web scraping skills &#8211; and not just web scraping skills. If you&#8217;d like to learn more about data analysis and gain technical skills related to the subject, don&#8217;t hesitate to sign up for the Data Analyst training course.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e16e872 elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"e16e872\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/en\/data-analyst-course\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Discover the Data Analyst career path<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-00d9354 e-flex e-con-boxed e-con\" data-id=\"00d9354\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-6e4f49b elementor-widget elementor-widget-html\" data-id=\"6e4f49b\" data-element_type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<script type=\"application\/ld+json\">\n{\n  \"@context\": \"https:\/\/schema.org\",\n  \"@type\": \"FAQPage\",\n  \"mainEntity\": [{\n    \"@type\": \"Question\",\n    \"name\": \"What is Beautiful Soup for?\",\n    \"acceptedAnswer\": {\n      \"@type\": \"Answer\",\n      \"text\": \"For example, you might want to have access to all the reviews of a Pack of HP Black Ink Cartridges on Amazon so you can do syntactic, semantic and sentimental analysis and make up your own mind. Web scraping from a store locator (such as a map) can be used to create a list of commercial locations. You can also obtain stock prices to help you make better investment decisions.\"\n    }\n  },{\n    \"@type\": \"Question\",\n    \"name\": \"Beautiful Soup presentation\",\n    \"acceptedAnswer\": {\n      \"@type\": \"Answer\",\n      \"text\": \"In this article, we'll concentrate on Phase 1, since we're interested in data retrieval. In the next paragraph, we'll look at an application of the Beautiful Soup library, which enables web scraping in Python. In the next paragraph, we'll take a look at a web scraping library available in Python that provides an excellent (and easy-to-handle) tool for extracting information from unstructured data: Beautiful Soup.\"\n    }\n  },{\n    \"@type\": \"Question\",\n    \"name\": \"Why is it called \"Beautiful Soup\"?\",\n    \"acceptedAnswer\": {\n      \"@type\": \"Answer\",\n      \"text\": \"Websites are written in HTML and CSS, the computer languages used to lay out web pages. HTML is used to manage and organize content. CSS is used to manage the appearance of the web page (colors, text size, etc.). In the field of web development, \"tag soup\" is a deprecatory term for syntactically or structurally incorrect HTML written for a web page.\"\n    }\n  }]\n}\n<\/script>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>As an Internet browser, we have access to a lot of information about customers, offers, share prices, physical phenomena and so on. This data can be read by users, but we&#8217;d like to be able to exploit it by transforming it into an operational format so that we can analyze it and make the most [&hellip;]<\/p>\n","protected":false},"author":67,"featured_media":177160,"comment_status":"open","ping_status":"open","sticky":false,"template":"elementor_theme","format":"standard","meta":{"content-type":"","inline_featured_image":false},"categories":[2425],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.7 (Yoast SEO v21.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Beautiful Soup: Introduction to web scraping with Python<\/title>\n<meta name=\"description\" content=\"Unleash the power of web scraping with Beautiful Soup. Dive into the essentials of Beautiful Soup, a Python library that simplifies the extraction of data from HTML and XML files. Learn how to navigate and parse web content effortlessly, making web scraping projects a breeze.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Beautiful Soup: Introduction to web scraping with Python\" \/>\n<meta property=\"og:description\" content=\"As an Internet browser, we have access to a lot of information about customers, offers, share prices, physical phenomena and so on. This data can be read\" \/>\n<meta property=\"og:url\" content=\"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python\" \/>\n<meta property=\"og:site_name\" content=\"Data Science Courses | DataScientest\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataScientestEN\" \/>\n<meta property=\"article:published_time\" content=\"2024-01-20T15:43:09+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-01-21T08:17:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/datascientest.com\/en\/files\/2024\/01\/beautiful-soup.png\" \/>\n\t<meta property=\"og:image:width\" content=\"343\" \/>\n\t<meta property=\"og:image:height\" content=\"147\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Melanie\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:site\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Melanie\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python#article\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python\"},\"author\":{\"name\":\"Melanie\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d\"},\"headline\":\"Beautiful Soup: Introduction to web scraping with Python\",\"datePublished\":\"2024-01-20T15:43:09+00:00\",\"dateModified\":\"2024-01-21T08:17:39+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python\"},\"wordCount\":1069,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"articleSection\":[\"Data Science\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python#respond\"]}],\"copyrightYear\":\"2024\",\"copyrightHolder\":{\"@id\":\"https:\/\/datascientest.com\/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python\",\"url\":\"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python\",\"name\":\"Beautiful Soup: Introduction to web scraping with Python\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/#website\"},\"datePublished\":\"2024-01-20T15:43:09+00:00\",\"dateModified\":\"2024-01-21T08:17:39+00:00\",\"description\":\"Unleash the power of web scraping with Beautiful Soup. Dive into the essentials of Beautiful Soup, a Python library that simplifies the extraction of data from HTML and XML files. Learn how to navigate and parse web content effortlessly, making web scraping projects a breeze.\",\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/datascientest.com\/en\/#website\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"name\":\"Data Science Courses | DataScientest\",\"description\":\"Leader in Data Science Courses\",\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/datascientest.com\/en\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/datascientest.com\/en\/#organization\",\"name\":\"DataScientest\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"contentUrl\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"width\":301,\"height\":301,\"caption\":\"DataScientest\"},\"image\":{\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/DataScientestEN\",\"https:\/\/twitter.com\/DataScientest__\",\"https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/\",\"https:\/\/www.tiktok.com\/@datascientest\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d\",\"name\":\"Melanie\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g\",\"caption\":\"Melanie\"},\"url\":\"https:\/\/datascientest.com\/en\/author\/melaniekrause\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Beautiful Soup: Introduction to web scraping with Python","description":"Unleash the power of web scraping with Beautiful Soup. Dive into the essentials of Beautiful Soup, a Python library that simplifies the extraction of data from HTML and XML files. Learn how to navigate and parse web content effortlessly, making web scraping projects a breeze.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python","og_locale":"en_GB","og_type":"article","og_title":"Beautiful Soup: Introduction to web scraping with Python","og_description":"As an Internet browser, we have access to a lot of information about customers, offers, share prices, physical phenomena and so on. This data can be read","og_url":"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python","og_site_name":"Data Science Courses | DataScientest","article_publisher":"https:\/\/www.facebook.com\/DataScientestEN","article_published_time":"2024-01-20T15:43:09+00:00","article_modified_time":"2024-01-21T08:17:39+00:00","og_image":[{"width":343,"height":147,"url":"https:\/\/datascientest.com\/en\/files\/2024\/01\/beautiful-soup.png","type":"image\/png"}],"author":"Melanie","twitter_card":"summary_large_image","twitter_creator":"@DataScientest__","twitter_site":"@DataScientest__","twitter_misc":{"Written by":"Melanie","Estimated reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python#article","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python"},"author":{"name":"Melanie","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d"},"headline":"Beautiful Soup: Introduction to web scraping with Python","datePublished":"2024-01-20T15:43:09+00:00","dateModified":"2024-01-21T08:17:39+00:00","mainEntityOfPage":{"@id":"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python"},"wordCount":1069,"commentCount":0,"publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"articleSection":["Data Science"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python#respond"]}],"copyrightYear":"2024","copyrightHolder":{"@id":"https:\/\/datascientest.com\/#organization"}},{"@type":"WebPage","@id":"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python","url":"https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python","name":"Beautiful Soup: Introduction to web scraping with Python","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/#website"},"datePublished":"2024-01-20T15:43:09+00:00","dateModified":"2024-01-21T08:17:39+00:00","description":"Unleash the power of web scraping with Beautiful Soup. Dive into the essentials of Beautiful Soup, a Python library that simplifies the extraction of data from HTML and XML files. Learn how to navigate and parse web content effortlessly, making web scraping projects a breeze.","inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/datascientest.com\/en\/beautiful-soup-introduction-to-web-scraping-with-python"]}]},{"@type":"WebSite","@id":"https:\/\/datascientest.com\/en\/#website","url":"https:\/\/datascientest.com\/en\/","name":"Data Science Courses | DataScientest","description":"Leader in Data Science Courses","publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/datascientest.com\/en\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/datascientest.com\/en\/#organization","name":"DataScientest","url":"https:\/\/datascientest.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","contentUrl":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","width":301,"height":301,"caption":"DataScientest"},"image":{"@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataScientestEN","https:\/\/twitter.com\/DataScientest__","https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/","https:\/\/www.tiktok.com\/@datascientest"]},{"@type":"Person","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d","name":"Melanie","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g","caption":"Melanie"},"url":"https:\/\/datascientest.com\/en\/author\/melaniekrause"}]}},"_links":{"self":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/177150"}],"collection":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/users\/67"}],"replies":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/comments?post=177150"}],"version-history":[{"count":0,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/177150\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media\/177160"}],"wp:attachment":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media?parent=177150"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/categories?post=177150"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/tags?post=177150"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}