{"id":165614,"date":"2023-01-05T14:38:37","date_gmt":"2023-01-05T13:38:37","guid":{"rendered":"https:\/\/datascientest.com\/en\/?p=165614"},"modified":"2023-10-30T18:10:27","modified_gmt":"2023-10-30T17:10:27","slug":"pyspark-the-python-library","status":"publish","type":"post","link":"https:\/\/datascientest.com\/en\/pyspark-the-python-library","title":{"rendered":"PySpark: Everything about the Python library"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"165614\" class=\"elementor elementor-165614\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-1def455 e-flex e-con-boxed e-con\" data-id=\"1def455\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-ba48bc5 elementor-widget elementor-widget-heading\" data-id=\"ba48bc5\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-heading-title{padding:0;margin:0;line-height:1}.elementor-widget-heading .elementor-heading-title[class*=elementor-size-]>a{color:inherit;font-size:inherit;line-height:inherit}.elementor-widget-heading .elementor-heading-title.elementor-size-small{font-size:15px}.elementor-widget-heading .elementor-heading-title.elementor-size-medium{font-size:19px}.elementor-widget-heading .elementor-heading-title.elementor-size-large{font-size:29px}.elementor-widget-heading .elementor-heading-title.elementor-size-xl{font-size:39px}.elementor-widget-heading .elementor-heading-title.elementor-size-xxl{font-size:59px}<\/style><h3 class=\"elementor-heading-title elementor-size-default\">When we talk about database processing in python, we immediately think of the pandas library. However, when dealing with very large databases, calculations become too slow. Fortunately, there is another python library, very similar to pandas, which allows for the processing of very large amounts of data: PySpark.\nIn this article, we will present central elements of Spark, starting with RDDs, the most basic structure of Spark. We will then study the DataFrame type, a richer structure than RDDs that is optimized for machine learning.<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-9be3673 e-flex e-con-boxed e-con\" data-id=\"9be3673\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-47ec497 elementor-widget elementor-widget-heading\" data-id=\"47ec497\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\"><span style=\"font-weight:normal;\" id=\"docs-internal-guid-93c89775-7fff-c8d0-0d0c-cf94ddbf39d1\"><span style=\"font-size: 15pt; font-family: Arial; background-color: transparent; font-weight: 700; font-variant-numeric: normal; font-variant-east-asian: normal; vertical-align: baseline;\">What is Apache Spark ?<\/span><\/span><\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fbf1924 elementor-widget elementor-widget-text-editor\" data-id=\"fbf1924\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t<p><b>Apache Spark<\/b><span style=\"font-weight: 400;\"> is an open-source framework developed by UC Berkeley&#8217;s AMPLab that allows for the processing of large databases using <\/span><b>distributed computing<\/b><span style=\"font-weight: 400;\">, a technique that uses multiple units of computation distributed in clusters for a single project in order to divide the execution time of a query.<\/span><\/p><p><span style=\"font-weight: 400;\">Spark was developed in <\/span><b>Scala<\/b><span style=\"font-weight: 400;\"> and is at its best in its native language. However, the PySpark library allows it to be used with the <a href=\"https:\/\/datascientest.com\/en\/python-the-most-popular-programming-language\">Python language<\/a> while maintaining similar performance to Scala implementations.<\/span><\/p><p><span style=\"font-weight: 400;\">Pyspark is therefore a <\/span><b>good alternative to the pandas library<\/b><span style=\"font-weight: 400;\"> when looking to process data sets that are too large and lead to time-consuming calculations.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-31132f8 e-flex e-con-boxed e-con\" data-id=\"31132f8\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-b1a1f59 elementor-widget elementor-widget-heading\" data-id=\"b1a1f59\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">How structured PySpark is ?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-caef642 elementor-widget elementor-widget-text-editor\" data-id=\"caef642\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">First of all, it is important to understand the basis of how Spark works.<\/span><\/p><p><span style=\"font-weight: 400;\">When you interact with Spark through PySpark, you <\/span><b>send instructions to the Driver<\/b><span style=\"font-weight: 400;\">. The Driver coordinates all operations. The Driver can be communicated by a SparkContext object. This object <\/span><b>coordinates the different calculations<\/b><span style=\"font-weight: 400;\"> on the different clusters.<\/span><\/p><p><span style=\"font-weight: 400;\">The big advantage of Spark is that the code is completely independent of the SparkContext. So you can <\/span><b>develop your code<\/b><span style=\"font-weight: 400;\"> locally on any machine.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-0e0f167 e-flex e-con-boxed e-con\" data-id=\"0e0f167\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-61c49d2 elementor-widget elementor-widget-heading\" data-id=\"61c49d2\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">What\u2019s the definition of Resilient Distributed Data (RDD) ?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-34c11eb elementor-widget elementor-widget-text-editor\" data-id=\"34c11eb\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">An RDD is the Spark <\/span><b>representation of a data table<\/b><span style=\"font-weight: 400;\">. It is a collection of elements that can be used to contain tuples, dictionaries, lists\u2026<\/span><\/p><p><span style=\"font-weight: 400;\">The strength of an RDD<\/span><b> lies in its ability<\/b><span style=\"font-weight: 400;\"> to evaluate the code lazily: the start of the calculations is postponed until absolutely necessary.<\/span><\/p><p><span style=\"font-weight: 400;\">For example, when importing a file, <\/span><b>only a pointer to it is created<\/b><span style=\"font-weight: 400;\">. It is really only at the last moment, when you are looking to display or use a result, that the calculation is done.<\/span><\/p><p><span style=\"font-weight: 400;\">To go further in the handling of an RDD, we can use the documentation available here<\/span><\/p><p><span style=\"font-weight: 400;\">An <\/span><b>RDD reads line by line<\/b><span style=\"font-weight: 400;\">, which makes it effective for <\/span><b>processing text files<\/b><span style=\"font-weight: 400;\"> (counting the number of occurrences of each word in the miserable integral for example), but it is an <\/span><b>unsuitable structure for calculations per column<\/b><span style=\"font-weight: 400;\">.<\/span><\/p><p><span style=\"font-weight: 400;\">To do Machine Learning, we need to introduce a new structure: DataFrames.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-7035425 e-flex e-con-boxed e-con\" data-id=\"7035425\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-8f97225 elementor-widget elementor-widget-heading\" data-id=\"8f97225\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\"><span style=\"font-weight:normal;\" id=\"docs-internal-guid-3c8f0bd8-7fff-c848-acd3-47ce748f968d\"><span style=\"font-size: 15pt; font-family: Arial; background-color: transparent; font-weight: 700; font-variant-numeric: normal; font-variant-east-asian: normal; vertical-align: baseline;\">DataFrame pyspark<\/span><\/span><\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f4f6cc1 elementor-widget elementor-widget-text-editor\" data-id=\"f4f6cc1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The pyspark DataFrame is the <\/span><b>most optimized structure<\/b><span style=\"font-weight: 400;\"> in Machine Learning. It uses the underlying bases of an RDD but has been structured in columns as well as rows in an SQL structure. Its shape is<\/span><b> inspired by the DataFrame<\/b><span style=\"font-weight: 400;\"> of the panda module.<\/span><\/p><p><span style=\"font-weight: 400;\">Thanks to the DataFrame structure, we can make efficient calculations through a familiar language (similar to pandas), avoiding the cost of learning a new functional language: Scala.<\/span><\/p><p><b>Spark SQL<\/b><span style=\"font-weight: 400;\"> is a Spark module that allows you to <\/span><b>work on structured data<\/b><span style=\"font-weight: 400;\">. It is therefore within this module that the Spark DataFrame was developed.<\/span><\/p><p><span style=\"font-weight: 400;\">Spark SQL has a fairly rich one-page documentation, both in examples and explanations. Contrary to what you can find on the internet, this <\/span><b>documentation is the only document perpetually updated <\/b><span style=\"font-weight: 400;\">with the latest version of Spark.<\/span><\/p><p><span style=\"font-weight: 400;\">This article is just an introduction to the <\/span><b>main concepts<\/b><span style=\"font-weight: 400;\"> of Pyspark. Our trainings contains an entire module on learning this essential tool for handling big data. If you want to master this tool, let yourself be tempted by one of our <\/span><b>data science trainings.<\/b><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-02b8c51 e-flex e-con-boxed e-con\" data-id=\"02b8c51\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-6469942 elementor-widget elementor-widget-html\" data-id=\"6469942\" data-element_type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<script type=\"application\/ld+json\">\r\n{\r\n  \"@context\": \"https:\/\/schema.org\",\r\n  \"@type\": \"FAQPage\",\r\n  \"mainEntity\": [{\r\n    \"@type\": \"Question\",\r\n    \"name\": \"How structured PySpark is ?\",\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"When you interact with Spark through PySpark, you send instructions to the Driver. The Driver coordinates all operations. The Driver can be communicated by a SparkContext object. This object coordinates the different calculations on the different clusters. The big advantage of Spark is that the code is completely independent of the SparkContext. So you can develop your code locally on any machine.\"\r\n    }\r\n  },{\r\n    \"@type\": \"Question\",\r\n    \"name\": \"What is Apache Spark ?\",\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"Apache Spark is an open-source framework developed by UC Berkeley's AMPLab that allows for the processing of large databases using distributed computing, a technique that uses multiple units of computation distributed in clusters for a single project in order to divide the execution time of a query.\"\r\n    }\r\n  },{\r\n    \"@type\": \"Question\",\r\n    \"name\": \"What\u2019s the definition of Resilient Distributed Data (RDD) ?\",\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"An RDD is the Spark representation of a data table. It is a collection of elements that can be used to contain tuples, dictionaries, lists\u2026 The strength of an RDD lies in its ability to evaluate the code lazily: the start of the calculations is postponed until absolutely necessary.\"\r\n    }\r\n  }]\r\n}\r\n<\/script>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-d8b5df3 e-flex e-con-boxed e-con\" data-id=\"d8b5df3\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-d92e7b1 elementor-align-center button-principal elementor-widget elementor-widget-button\" data-id=\"d92e7b1\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-md\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Discover our trainings<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>When we talk about database processing in python, we immediately think of the pandas library. However, when dealing with very large databases, calculations become too slow. Fortunately, there is another python library, very similar to pandas, which allows for the processing of very large amounts of data: PySpark. In this article, we will present central [&hellip;]<\/p>\n","protected":false},"author":63,"featured_media":44186,"comment_status":"open","ping_status":"open","sticky":false,"template":"elementor_theme","format":"standard","meta":{"content-type":"","inline_featured_image":false},"categories":[2425],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.7 (Yoast SEO v21.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>PySpark: Everything about the Python library<\/title>\n<meta name=\"description\" content=\"In this article, we will present central elements of Spark, starting with RDDs, the most basic structure of Spark. We will then study the DataFrame type, a richer structure than RDDs that is optimized for machine learning.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/datascientest.com\/en\/pyspark-the-python-library\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"PySpark: Everything about the Python library\" \/>\n<meta property=\"og:description\" content=\"When we talk about database processing in python, we immediately think of the pandas library. However, when dealing with very large databases,\" \/>\n<meta property=\"og:url\" content=\"https:\/\/datascientest.com\/en\/pyspark-the-python-library\" \/>\n<meta property=\"og:site_name\" content=\"Data Science Courses | DataScientest\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataScientestEN\" \/>\n<meta property=\"article:published_time\" content=\"2023-01-05T13:38:37+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-10-30T17:10:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/datascientest.com\/en\/files\/2021\/02\/illu_pyspark_blog-84.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"1405\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:site\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/datascientest.com\/en\/pyspark-the-python-library#article\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/pyspark-the-python-library\"},\"author\":{\"name\":\"\",\"@id\":\"\"},\"headline\":\"PySpark: Everything about the Python library\",\"datePublished\":\"2023-01-05T13:38:37+00:00\",\"dateModified\":\"2023-10-30T17:10:27+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/datascientest.com\/en\/pyspark-the-python-library\"},\"wordCount\":624,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"articleSection\":[\"Data Science\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/datascientest.com\/en\/pyspark-the-python-library#respond\"]}],\"copyrightYear\":\"2023\",\"copyrightHolder\":{\"@id\":\"https:\/\/datascientest.com\/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/datascientest.com\/en\/pyspark-the-python-library\",\"url\":\"https:\/\/datascientest.com\/en\/pyspark-the-python-library\",\"name\":\"PySpark: Everything about the Python library\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/#website\"},\"datePublished\":\"2023-01-05T13:38:37+00:00\",\"dateModified\":\"2023-10-30T17:10:27+00:00\",\"description\":\"In this article, we will present central elements of Spark, starting with RDDs, the most basic structure of Spark. We will then study the DataFrame type, a richer structure than RDDs that is optimized for machine learning.\",\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/datascientest.com\/en\/pyspark-the-python-library\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/datascientest.com\/en\/#website\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"name\":\"Data Science Courses | DataScientest\",\"description\":\"Leader in Data Science Courses\",\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/datascientest.com\/en\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/datascientest.com\/en\/#organization\",\"name\":\"DataScientest\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"contentUrl\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"width\":301,\"height\":301,\"caption\":\"DataScientest\"},\"image\":{\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/DataScientestEN\",\"https:\/\/twitter.com\/DataScientest__\",\"https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/\",\"https:\/\/www.tiktok.com\/@datascientest\"]},{\"@type\":\"Person\",\"@id\":\"\",\"url\":\"https:\/\/datascientest.com\/en\/author\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"PySpark: Everything about the Python library","description":"In this article, we will present central elements of Spark, starting with RDDs, the most basic structure of Spark. We will then study the DataFrame type, a richer structure than RDDs that is optimized for machine learning.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/datascientest.com\/en\/pyspark-the-python-library","og_locale":"en_GB","og_type":"article","og_title":"PySpark: Everything about the Python library","og_description":"When we talk about database processing in python, we immediately think of the pandas library. However, when dealing with very large databases,","og_url":"https:\/\/datascientest.com\/en\/pyspark-the-python-library","og_site_name":"Data Science Courses | DataScientest","article_publisher":"https:\/\/www.facebook.com\/DataScientestEN","article_published_time":"2023-01-05T13:38:37+00:00","article_modified_time":"2023-10-30T17:10:27+00:00","og_image":[{"width":2560,"height":1405,"url":"https:\/\/datascientest.com\/en\/files\/2021\/02\/illu_pyspark_blog-84.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_creator":"@DataScientest__","twitter_site":"@DataScientest__","twitter_misc":{"Written by":"","Estimated reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/datascientest.com\/en\/pyspark-the-python-library#article","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/pyspark-the-python-library"},"author":{"name":"","@id":""},"headline":"PySpark: Everything about the Python library","datePublished":"2023-01-05T13:38:37+00:00","dateModified":"2023-10-30T17:10:27+00:00","mainEntityOfPage":{"@id":"https:\/\/datascientest.com\/en\/pyspark-the-python-library"},"wordCount":624,"commentCount":0,"publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"articleSection":["Data Science"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/datascientest.com\/en\/pyspark-the-python-library#respond"]}],"copyrightYear":"2023","copyrightHolder":{"@id":"https:\/\/datascientest.com\/#organization"}},{"@type":"WebPage","@id":"https:\/\/datascientest.com\/en\/pyspark-the-python-library","url":"https:\/\/datascientest.com\/en\/pyspark-the-python-library","name":"PySpark: Everything about the Python library","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/#website"},"datePublished":"2023-01-05T13:38:37+00:00","dateModified":"2023-10-30T17:10:27+00:00","description":"In this article, we will present central elements of Spark, starting with RDDs, the most basic structure of Spark. We will then study the DataFrame type, a richer structure than RDDs that is optimized for machine learning.","inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/datascientest.com\/en\/pyspark-the-python-library"]}]},{"@type":"WebSite","@id":"https:\/\/datascientest.com\/en\/#website","url":"https:\/\/datascientest.com\/en\/","name":"Data Science Courses | DataScientest","description":"Leader in Data Science Courses","publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/datascientest.com\/en\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/datascientest.com\/en\/#organization","name":"DataScientest","url":"https:\/\/datascientest.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","contentUrl":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","width":301,"height":301,"caption":"DataScientest"},"image":{"@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataScientestEN","https:\/\/twitter.com\/DataScientest__","https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/","https:\/\/www.tiktok.com\/@datascientest"]},{"@type":"Person","@id":"","url":"https:\/\/datascientest.com\/en\/author"}]}},"_links":{"self":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/165614"}],"collection":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/users\/63"}],"replies":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/comments?post=165614"}],"version-history":[{"count":0,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/165614\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media\/44186"}],"wp:attachment":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media?parent=165614"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/categories?post=165614"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/tags?post=165614"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}