{"id":40735,"date":"2023-10-11T16:25:00","date_gmt":"2023-10-11T16:25:00","guid":{"rendered":"https:\/\/datascientest.com\/?p=40735"},"modified":"2023-11-15T10:09:24","modified_gmt":"2023-11-15T10:09:24","slug":"pyspark","status":"publish","type":"post","link":"https:\/\/datascientest.com\/pyspark","title":{"rendered":"PySpark : Tout savoir sur la librairie Python"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"40735\" class=\"elementor elementor-40735\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-c6a2fca elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"c6a2fca\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-37d0b58\" data-id=\"37d0b58\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-a2e2513 elementor-widget elementor-widget-heading\" data-id=\"a2e2513\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-heading-title{padding:0;margin:0;line-height:1}.elementor-widget-heading .elementor-heading-title[class*=elementor-size-]>a{color:inherit;font-size:inherit;line-height:inherit}.elementor-widget-heading .elementor-heading-title.elementor-size-small{font-size:15px}.elementor-widget-heading .elementor-heading-title.elementor-size-medium{font-size:19px}.elementor-widget-heading .elementor-heading-title.elementor-size-large{font-size:29px}.elementor-widget-heading .elementor-heading-title.elementor-size-xl{font-size:39px}.elementor-widget-heading .elementor-heading-title.elementor-size-xxl{font-size:59px}<\/style><h2 class=\"elementor-heading-title elementor-size-default\">Lorsque l\u2019on parle de traitement de bases de donn\u00e9es sur python, on pense imm\u00e9diatement \u00e0 la librairie pandas. Cependant, lorsqu\u2019on a affaire \u00e0 des bases de donn\u00e9es trop massives, les calculs deviennent trop lents.\nHeureusement, il existe une autre librairie python, assez proche de pandas, qui permet de traiter des tr\u00e8s grandes quantit\u00e9s de donn\u00e9es : PySpark.<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-7d5dad7 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"7d5dad7\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-d293462\" data-id=\"d293462\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-9851e63 elementor-widget elementor-widget-heading\" data-id=\"9851e63\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Qu'est-ce que PySpark ?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-08fabd8 elementor-widget elementor-widget-text-editor\" data-id=\"08fabd8\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t<p><b>Apache Spark<\/b><span style=\"font-weight: 400;\"> est un framework <a href=\"https:\/\/datascientest.com\/open-source\">open-source<\/a>\u00a0<\/span><strong><span style=\"color: var( --e-global-color-text ); font-family: var( --e-global-typography-text-font-family ), Sans-serif; text-align: var(--text-align);\">d\u00e9velopp\u00e9 en<\/span><span style=\"color: var( --e-global-color-text ); font-family: var( --e-global-typography-text-font-family ), Sans-serif; text-align: var(--text-align);\">\u00a0<\/span><\/strong><span style=\"color: var( --e-global-color-text ); font-family: var( --e-global-typography-text-font-family ), Sans-serif; text-align: var(--text-align); font-weight: bold;\">Scala <\/span>par<span style=\"color: var( --e-global-color-text ); font-family: var( --e-global-typography-text-font-family ), Sans-serif; text-align: var(--text-align);\">\u00a0l\u2019<\/span><a style=\"font-family: var( --e-global-typography-text-font-family ), Sans-serif; font-weight: var( --e-global-typography-text-font-weight ); background-color: #ffffff; text-align: var(--text-align);\" href=\"https:\/\/amplab.cs.berkeley.edu\/projects\/\" target=\"_blank\" rel=\"noopener\">AMPLab de UC Berkeley<\/a><span style=\"color: var( --e-global-color-text ); font-family: var( --e-global-typography-text-font-family ), Sans-serif; text-align: var(--text-align);\"> permettant de <strong>traiter des bases de donn\u00e9es<\/strong> massives en utilisant le <\/span>calcul distribu\u00e9<span style=\"color: var( --e-global-color-text ); font-family: var( --e-global-typography-text-font-family ), Sans-serif; text-align: var(--text-align);\">, technique qui consiste \u00e0 exploiter plusieurs unit\u00e9s de calcul r\u00e9parties en <\/span>clusters <span style=\"color: var( --e-global-color-text ); font-family: var( --e-global-typography-text-font-family ), Sans-serif; text-align: var(--text-align);\">au profit d\u2019un seul projet afin de <strong>diviser le temps d\u2019ex\u00e9cution d\u2019une requ\u00eate<\/strong>.<\/span><\/p><p><span style=\"font-weight: 400;\">Mais lorsque la collaboration entre Apache Spark et Python se cr\u00e9ent, la librairie <\/span><b>PySpark<\/b><span style=\"font-weight: 400;\"> voit le jour. Elle propose alors \u00e0 ses utilisateurs d\u2019<strong>utiliser le langage <\/strong><\/span><a href=\"https:\/\/datascientest.com\/python-tout-savoir\">Python<\/a><span style=\"font-weight: 400;\">, en gardant des performances similaires \u00e0 des impl\u00e9mentations en Scala.<\/span><span style=\"font-weight: 400;\"><br \/><\/span><span style=\"font-weight: 400;\">Gr\u00e2ce \u00e0 la biblioth\u00e8que <strong>Py4j<\/strong>, Pyspark est donc <strong>une excellente alternative<\/strong> \u00e0 la librairie pandas lorsqu\u2019on cherche \u00e0 traiter <a href=\"https:\/\/datascientest.com\/dataset-definition\">des jeux de donn\u00e9es trop volumineux<\/a> qui entra\u00eenent des calculs trop chronophages.<\/span><span style=\"font-weight: 400;\"><br \/><\/span><b><\/b><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7d8a900 elementor-widget elementor-widget-image\" data-id=\"7d8a900\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-image{text-align:center}.elementor-widget-image a{display:inline-block}.elementor-widget-image a img[src$=\".svg\"]{width:48px}.elementor-widget-image img{vertical-align:middle;display:inline-block}<\/style>\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"800\" height=\"534\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/programming-background-with-person-working-with-codes-computer-1024x683.jpg\" class=\"attachment-large size-large wp-image-222409\" alt=\"\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/programming-background-with-person-working-with-codes-computer-1024x683.jpg 1024w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/programming-background-with-person-working-with-codes-computer-300x200.jpg 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/programming-background-with-person-working-with-codes-computer-768x512.jpg 768w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/programming-background-with-person-working-with-codes-computer.jpg 1500w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-005de39 elementor-widget elementor-widget-heading\" data-id=\"005de39\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Quelle est l'architecture de Spark ?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0658f45 elementor-widget elementor-widget-text-editor\" data-id=\"0658f45\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Lorsque vous interagissez avec Spark gr\u00e2ce \u00e0 PySpark, <strong>vous envoyez des instructions au Driver<\/strong>. C\u2019est lui qui coordonne toutes les op\u00e9rations. On peut communiquer au Driver gr\u00e2ce \u00e0 un objet <\/span><b>SparkContext<\/b><span style=\"font-weight: 400;\">. Cet objet coordonne les diff\u00e9rents calculs sur les diff\u00e9rents clusters. <\/span><span style=\"font-weight: 400;\"><br \/><\/span><span style=\"font-weight: 400;\"><a href=\"https:\/\/datascientest.com\/apache-spark\">Le gros avantage de Spark<\/a> r\u00e9side dans le fait que <strong>le code est totalement ind\u00e9pendant<\/strong> du SparkContext. On peut donc d\u00e9velopper son code en local sur quelconque machine.<\/span><\/p><p><span style=\"font-weight: 400;\">Plusieurs \u00e9l\u00e9ments permettent \u00e0 Spark d&rsquo;\u00eatre une biblioth\u00e8que performante, \u00e0 commencer par <strong>les RDD, la structure la plus \u00e9l\u00e9mentaire de Spark<\/strong>.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-23729cf elementor-align-center button-principal elementor-widget elementor-widget-button\" data-id=\"23729cf\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-md\" href=\"https:\/\/datascientest.com\/formation-data-scientist\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Ma\u00eetriser les logiciels comme PySpark<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-b43fd1c elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"b43fd1c\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-d6f304f\" data-id=\"d6f304f\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-9688feb elementor-widget elementor-widget-heading\" data-id=\"9688feb\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Qu'est-ce qu'un RDD ?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-164e0ba elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"164e0ba\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-80cf13c\" data-id=\"80cf13c\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-4611ed8 elementor-widget elementor-widget-text-editor\" data-id=\"4611ed8\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\"><strong>Un Resilient Distributed Data\u00a0est<\/strong> <strong>la repr\u00e9sentation Spark d\u2019un <\/strong><\/span><b>tableau de donn\u00e9es<\/b><span style=\"font-weight: 400;\">. C\u2019est une collection d\u2019\u00e9l\u00e9ments que l\u2019on peut utiliser pour contenir des tuples, des dictionnaires, des listes\u2026<\/span><span style=\"font-weight: 400;\"><br \/><\/span><span style=\"font-weight: 400;\">La force d\u2019un RDD r\u00e9side dans <strong>sa capacit\u00e9 \u00e0 \u00e9valuer le code de fa\u00e7on paresseuse<\/strong> : le lancement des calculs est report\u00e9 jusqu\u2019\u00e0 ce que ce soit absolument n\u00e9cessaire.<\/span><span style=\"font-weight: 400;\"><br \/><\/span><span style=\"font-weight: 400;\">Par exemple, lorsque l\u2019on importe un fichier, seul un pointeur vers ce dernier est cr\u00e9\u00e9. Ce n&rsquo;est vraiment qu&rsquo;au dernier moment, lorsqu&rsquo;on recherche \u00e0 afficher ou utiliser un r\u00e9sultat, que le calcul est effectu\u00e9.<\/span><span style=\"font-weight: 400;\"><br \/><\/span><span style=\"font-weight: 400;\">Pour aller plus loin dans la <\/span><b>manipulation d\u2019un RDD<\/b><span style=\"font-weight: 400;\">, on peut s\u2019aider de la documentation disponible <a href=\"https:\/\/spark.apache.org\/docs\/2.4.2\/api\/python\/index.html\">ici<\/a><strong>.<\/strong><\/span><\/p><p><span style=\"font-weight: 400;\">Un RDD se lit ligne par ligne, ce qui le rend<\/span><b> efficace pour traiter des fichiers texte<\/b><span style=\"font-weight: 400;\"> (compter le nombre d\u2019occurence de chaque mot dans l\u2019int\u00e9grale des mis\u00e9rables par exemple), mais c\u2019est une <\/span><b>structure inadapt\u00e9e pour des calculs par colonne<\/b><span style=\"font-weight: 400;\">. <\/span><span style=\"font-weight: 400;\"><br \/><\/span><span style=\"font-weight: 400;\">Pour <a href=\"https:\/\/datascientest.com\/machine-learning-tout-savoir\">faire du Machine Learning<\/a>, nous avons donc\u00a0 besoin d\u2019introduire une nouvelle structure : les DataFrames.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-50ea961 elementor-widget elementor-widget-image\" data-id=\"50ea961\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"800\" height=\"534\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/16396122_rm373batch5-19-1024x683.jpg\" class=\"attachment-large size-large wp-image-222410\" alt=\"\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/16396122_rm373batch5-19-1024x683.jpg 1024w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/16396122_rm373batch5-19-300x200.jpg 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/16396122_rm373batch5-19-768x512.jpg 768w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/16396122_rm373batch5-19-1536x1024.jpg 1536w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/16396122_rm373batch5-19-2048x1365.jpg 2048w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-7755d82 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"7755d82\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-d8e302d\" data-id=\"d8e302d\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-1d231c0 elementor-widget elementor-widget-heading\" data-id=\"1d231c0\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Qu'est-ce qu'un DataFrame Pyspark ?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-2a7b61d elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"2a7b61d\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-f42967e\" data-id=\"f42967e\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-9fad949 elementor-widget elementor-widget-text-editor\" data-id=\"9fad949\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Le DataFrame de pyspark est la<\/span><b> structure la plus optimis\u00e9e en Machine Learning<\/b><span style=\"font-weight: 400;\">. Elle utilise de fa\u00e7on sous-jacente les bases d&rsquo;un RDD mais a \u00e9t\u00e9 structur\u00e9e en colonnes autant qu&rsquo;en lignes dans une structure SQL. Sa forme est <strong>inspir\u00e9e des DataFrame du module pandas<\/strong>.<\/span><span style=\"font-weight: 400;\"><br \/><\/span><span style=\"font-weight: 400;\">Gr\u00e2ce \u00e0 la structure DataFrame, nous pouvons donc <strong>faire des calculs performants \u00e0 travers un langage familier<\/strong> (car similaire \u00e0 pandas), en \u00e9vitant le co\u00fbt d&rsquo;entr\u00e9e d&rsquo;apprentissage d&rsquo;un nouveau langage fonctionnel : Scala.<\/span><span style=\"font-weight: 400;\"><br \/><\/span><b>Spark SQL <\/b><span style=\"font-weight: 400;\">est un module de Spark qui permet de travailler sur de la <\/span><b>donn\u00e9e structur\u00e9e<\/b><span style=\"font-weight: 400;\">. C&rsquo;est donc au sein de ce module qu&rsquo;a \u00e9t\u00e9 d\u00e9velopp\u00e9 le Spark DataFrame.<\/span><span style=\"font-weight: 400;\"><br \/><\/span><span style=\"font-weight: 400;\">Spark SQL poss\u00e8de une <\/span><a href=\"https:\/\/spark.apache.org\/docs\/2.4.0\/api\/python\/pyspark.sql.html\">documentation<\/a><span style=\"font-weight: 400;\"> en une seule page assez riche, \u00e0 la fois en exemples et en explications. Contrairement \u00e0 ce que vous pouvez trouver sur internet, cette documentation est <strong>le seul document perp\u00e9tuellement \u00e0 jour<\/strong> avec la derni\u00e8re version de Spark.<br \/>Maintenant que vous connaissez les fondamentaux de Spark, il est temps de parler de Pyspark et ce qui en fait <strong>l\u2019une des meilleures biblioth\u00e8ques de Python<\/strong>.<br \/><\/span><\/p><p><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-341e2fc elementor-align-center button-principal elementor-widget elementor-widget-button\" data-id=\"341e2fc\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-md\" href=\"https:\/\/datascientest.com\/formation-data-scientist\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Commencer une formation de Data Scientist<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t<div class=\"elementor-element elementor-element-35f40f7 e-flex e-con-boxed e-con\" data-id=\"35f40f7\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-4ce906e elementor-widget elementor-widget-heading\" data-id=\"4ce906e\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Qu'est-ce que Py4J ?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-77ad5c2 elementor-widget elementor-widget-text-editor\" data-id=\"77ad5c2\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Pour comprendre la puissance de PySpark, il est essentiel de <strong>conna\u00eetre l&rsquo;importance de Py4J<\/strong>. Cette biblioth\u00e8que int\u00e9gr\u00e9e permet \u00e0 Python d&rsquo;<strong>interagir directement<\/strong> avec les objets de la JVM (Java Virtual Machine). <\/span><\/p><p><span style=\"font-weight: 400;\">Gr\u00e2ce \u00e0 Py4J, les d\u00e9veloppeurs peuvent <strong>utiliser les fonctionnalit\u00e9s de Spark tout en \u00e9crivant du code en Python<\/strong>. Cette interop\u00e9rabilit\u00e9 ouvre la porte \u00e0 une collaboration plus fluide entre les \u00e9cosyst\u00e8mes Python et Java, rendant PySpark particuli\u00e8rement attractif pour les projets de <a href=\"https:\/\/datascientest.com\/data-science-definition\">data science<\/a> et de <a href=\"https:\/\/datascientest.com\/big-data-tout-savoir\">big data<\/a>.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-c6b256d e-flex e-con-boxed e-con\" data-id=\"c6b256d\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-466e6f9 elementor-widget elementor-widget-heading\" data-id=\"466e6f9\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Pourquoi choisir PySpark ?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-32e3156 elementor-widget elementor-widget-text-editor\" data-id=\"32e3156\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>L&rsquo;utilisation de PySpark pr\u00e9sente de multiples avantages. Python est reconnu pour <strong>sa syntaxe claire et sa facilit\u00e9 d&rsquo;apprentissage<\/strong>, ce qui rend PySpark accessible m\u00eame pour ceux qui sont nouveaux dans le domaine du big data.<\/p><p><a href=\"https:\/\/datascientest.com\/api-tout-savoir\">L&rsquo;API de PySpark<\/a> est con\u00e7ue pour \u00eatre intuitive, permettant aux d\u00e9veloppeurs de <strong>se concentrer sur la r\u00e9solution de probl\u00e8mes<\/strong> complexes plut\u00f4t que sur la compr\u00e9hension de l&rsquo;outil lui-m\u00eame. De plus, la communaut\u00e9 Python offre <a href=\"https:\/\/datascientest.com\/avec-la-data-visualisation-donnez-du-sens-a-vos-donnees\">une multitude d&rsquo;options de visualisation des donn\u00e9es<\/a>, rendant les insights plus accessibles et plus compr\u00e9hensibles.<\/p><p>Ces avantages font de PySpark <strong>un choix privil\u00e9gi\u00e9<\/strong> pour les data scientists et les ing\u00e9nieurs de donn\u00e9es qui souhaitent exploiter la puissance de Spark avec la simplicit\u00e9 de Python.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e86f18d elementor-widget elementor-widget-image\" data-id=\"e86f18d\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"800\" height=\"450\" src=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/close-up-hand-holding-futuristic-screen-1024x576.jpg\" class=\"attachment-large size-large wp-image-222411\" alt=\"\" srcset=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/close-up-hand-holding-futuristic-screen-1024x576.jpg 1024w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/close-up-hand-holding-futuristic-screen-300x169.jpg 300w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/close-up-hand-holding-futuristic-screen-768x432.jpg 768w, https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/close-up-hand-holding-futuristic-screen.jpg 1500w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-afdea09 e-flex e-con-boxed e-con\" data-id=\"afdea09\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-c87cf53 elementor-widget elementor-widget-heading\" data-id=\"c87cf53\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Comment am\u00e9liorer PySpark ?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0e5fb4c elementor-widget elementor-widget-text-editor\" data-id=\"0e5fb4c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Vous pouvez enrichir PySpark avec des biblioth\u00e8ques d\u00e9di\u00e9es. PySpark est bien plus qu&rsquo;une simple interface ; il s&rsquo;agit d&rsquo;<strong>un \u00e9cosyst\u00e8me riche en biblioth\u00e8ques sp\u00e9cialis\u00e9es<\/strong>.<\/p><p>Parmi elles, <strong>PySparkSQL<\/strong> se distingue en permettant des analyses de type SQL sur des donn\u00e9es structur\u00e9es ou semi-structur\u00e9es, offrant la possibilit\u00e9 d&rsquo;<a href=\"https:\/\/datascientest.com\/sql-tout-savoir\">ex\u00e9cuter des requ\u00eates SQL<\/a> directement.<\/p><p><strong>MLlib<\/strong>, le wrapper de machine learning pour Spark, <strong>simplifie l&rsquo;impl\u00e9mentation d&rsquo;algorithmes<\/strong> de machine learning en parall\u00e9lisant les donn\u00e9es.<\/p><p><strong>GraphFrames<\/strong> \u00e9tend les capacit\u00e9s de PySpark aux analyses graphiques, permettant <strong>des traitements complexes sur des graphes \u00e0 grande \u00e9chelle<\/strong>.<\/p><p>L&rsquo;int\u00e9gration de ces biblioth\u00e8ques dans PySpark<strong> facilite la manipulation de donn\u00e9es<\/strong> diversifi\u00e9es et ouvre la voie \u00e0 des analyses avanc\u00e9es.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fd3db7d elementor-align-center button-principal elementor-widget elementor-widget-button\" data-id=\"fd3db7d\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-md\" href=\"https:\/\/datascientest.com\/formations-data\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">D\u00e9couvrir nos formations en Big Data<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5c2e4c1 elementor-widget elementor-widget-html\" data-id=\"5c2e4c1\" data-element_type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<script type=\"application\/ld+json\">\r\n{\r\n  \"@context\": \"https:\/\/schema.org\",\r\n  \"@type\": \"FAQPage\",\r\n  \"mainEntity\": [{\r\n    \"@type\": \"Question\",\r\n    \"name\": \"Qu'est-ce que PySpark ?\",\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"PySpark est une collaboration entre Apache Spark et Python. Elle propose alors \u00e0 ses utilisateurs d\u2019utiliser le langage Python, en gardant des performances similaires \u00e0 des impl\u00e9mentations en Scala.\"\r\n    }\r\n  },{\r\n    \"@type\": \"Question\",\r\n    \"name\": \"Pourquoi utiliser PySpark ?\",\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"L'API de PySpark est con\u00e7ue pour \u00eatre intuitive, permettant aux d\u00e9veloppeurs de se concentrer sur la r\u00e9solution de probl\u00e8mes complexes plut\u00f4t que sur la compr\u00e9hension de l'outil lui-m\u00eame.\"\r\n    }\r\n  },{\r\n    \"@type\": \"Question\",\r\n    \"name\": \"Peut-on ajouter des biblioth\u00e8ques \u00e0 PySpark ?\",\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"Vous pouvez enrichir PySpark avec des biblioth\u00e8ques d\u00e9di\u00e9es comme PySparkSQL, MLlib ou encore GraphFrames. L'int\u00e9gration de ces biblioth\u00e8ques dans PySpark facilite la manipulation de donn\u00e9es diversifi\u00e9es.\"\r\n    }\r\n  }]\r\n}\r\n<\/script>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-136a359 e-flex e-con-boxed e-con\" data-id=\"136a359\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Lorsque l\u2019on parle de traitement de bases de donn\u00e9es sur python, on pense imm\u00e9diatement \u00e0 la librairie pandas. Cependant, lorsqu\u2019on a affaire \u00e0 des bases de donn\u00e9es trop massives, les calculs deviennent trop lents. Heureusement, il existe une autre librairie python, assez proche de pandas, qui permet de traiter des tr\u00e8s grandes quantit\u00e9s de donn\u00e9es [&hellip;]<\/p>\n","protected":false},"author":45,"featured_media":223059,"comment_status":"open","ping_status":"open","sticky":false,"template":"elementor_theme","format":"standard","meta":{"content-type":"","inline_featured_image":false},"categories":[2497],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.7 (Yoast SEO v21.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>PySpark : Tout savoir sur la librairie Python<\/title>\n<meta name=\"description\" content=\"D\u00e9couvrez avec nos experts PySpark, la librairie python qui vous permet de traiter tr\u00e8s rapidement un grand volume de donn\u00e9es !\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/datascientest.com\/pyspark\" \/>\n<meta property=\"og:locale\" content=\"fr_FR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"PySpark : Tout savoir sur la librairie Python\" \/>\n<meta property=\"og:url\" content=\"https:\/\/datascientest.com\/pyspark\" \/>\n<meta property=\"og:site_name\" content=\"Formation Data Science | DataScientest.com\" \/>\n<meta property=\"article:published_time\" content=\"2023-10-11T16:25:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-11-15T10:09:24+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/03\/Pyspark.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1366\" \/>\n\t<meta property=\"og:image:height\" content=\"768\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Daniel\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u00c9crit par\" \/>\n\t<meta name=\"twitter:data1\" content=\"Daniel\" \/>\n\t<meta name=\"twitter:label2\" content=\"Dur\u00e9e de lecture estim\u00e9e\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/datascientest.com\/pyspark\",\"url\":\"https:\/\/datascientest.com\/pyspark\",\"name\":\"PySpark : Tout savoir sur la librairie Python\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/#website\"},\"datePublished\":\"2023-10-11T16:25:00+00:00\",\"dateModified\":\"2023-11-15T10:09:24+00:00\",\"author\":{\"@id\":\"https:\/\/datascientest.com\/#\/schema\/person\/5a86e30c5090234fb43dd8baa83c44b0\"},\"description\":\"D\u00e9couvrez avec nos experts PySpark, la librairie python qui vous permet de traiter tr\u00e8s rapidement un grand volume de donn\u00e9es !\",\"inLanguage\":\"fr-FR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/datascientest.com\/pyspark\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/datascientest.com\/#website\",\"url\":\"https:\/\/datascientest.com\/\",\"name\":\"Formation Data Science | DataScientest.com\",\"description\":\"Leader de la formation en Data Science\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/datascientest.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"fr-FR\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/datascientest.com\/#\/schema\/person\/5a86e30c5090234fb43dd8baa83c44b0\",\"name\":\"Daniel\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"fr-FR\",\"@id\":\"https:\/\/datascientest.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/dst.svg\",\"contentUrl\":\"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/dst.svg\",\"caption\":\"Daniel\"},\"sameAs\":[\"https:\/\/www.linkedin.com\/school\/datascientest\/\",\"user\"],\"jobTitle\":\"ing\u00e9nieur logiciel\",\"worksFor\":\"DataScientest\",\"url\":\"https:\/\/datascientest.com\/author\/team-redac\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"PySpark : Tout savoir sur la librairie Python","description":"D\u00e9couvrez avec nos experts PySpark, la librairie python qui vous permet de traiter tr\u00e8s rapidement un grand volume de donn\u00e9es !","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/datascientest.com\/pyspark","og_locale":"fr_FR","og_type":"article","og_title":"PySpark : Tout savoir sur la librairie Python","og_url":"https:\/\/datascientest.com\/pyspark","og_site_name":"Formation Data Science | DataScientest.com","article_published_time":"2023-10-11T16:25:00+00:00","article_modified_time":"2023-11-15T10:09:24+00:00","og_image":[{"width":1366,"height":768,"url":"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/03\/Pyspark.png","type":"image\/png"}],"author":"Daniel","twitter_card":"summary_large_image","twitter_misc":{"\u00c9crit par":"Daniel","Dur\u00e9e de lecture estim\u00e9e":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/datascientest.com\/pyspark","url":"https:\/\/datascientest.com\/pyspark","name":"PySpark : Tout savoir sur la librairie Python","isPartOf":{"@id":"https:\/\/datascientest.com\/#website"},"datePublished":"2023-10-11T16:25:00+00:00","dateModified":"2023-11-15T10:09:24+00:00","author":{"@id":"https:\/\/datascientest.com\/#\/schema\/person\/5a86e30c5090234fb43dd8baa83c44b0"},"description":"D\u00e9couvrez avec nos experts PySpark, la librairie python qui vous permet de traiter tr\u00e8s rapidement un grand volume de donn\u00e9es !","inLanguage":"fr-FR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/datascientest.com\/pyspark"]}]},{"@type":"WebSite","@id":"https:\/\/datascientest.com\/#website","url":"https:\/\/datascientest.com\/","name":"Formation Data Science | DataScientest.com","description":"Leader de la formation en Data Science","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/datascientest.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"fr-FR"},{"@type":"Person","@id":"https:\/\/datascientest.com\/#\/schema\/person\/5a86e30c5090234fb43dd8baa83c44b0","name":"Daniel","image":{"@type":"ImageObject","inLanguage":"fr-FR","@id":"https:\/\/datascientest.com\/#\/schema\/person\/image\/","url":"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/dst.svg","contentUrl":"https:\/\/datascientest.com\/wp-content\/uploads\/2023\/10\/dst.svg","caption":"Daniel"},"sameAs":["https:\/\/www.linkedin.com\/school\/datascientest\/","user"],"jobTitle":"ing\u00e9nieur logiciel","worksFor":"DataScientest","url":"https:\/\/datascientest.com\/author\/team-redac"}]}},"_links":{"self":[{"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/posts\/40735"}],"collection":[{"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/users\/45"}],"replies":[{"embeddable":true,"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/comments?post=40735"}],"version-history":[{"count":0,"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/posts\/40735\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/media\/223059"}],"wp:attachment":[{"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/media?parent=40735"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/categories?post=40735"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datascientest.com\/wp-json\/wp\/v2\/tags?post=40735"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}