Getting started in data science inevitably means embarking on projects that can take a long time. As with any project, you need to know how to organize yourself, prioritize tasks and set milestones so you can monitor progress and make adjustments if necessary.
According to a Chinese proverb, experience is a bald man’s comb. At DataScientest, we use it to provide you with the best tips, like these 5 steps that will guide you step by step through all your Data projects!
1. Understanding the ins and outs
Before you start coding or obtaining data, you need to take the time to understand and grasp the problem at hand.
- What is the goal of this project?
- Has any work already been done on the subject?
- Will I have to work alone, or will I have to involve members of different departments?
- Are my results to be used immediately, or are they part of a larger project?
- Have I made assumptions about my data and its format, and have I verified them?
It’s vital to anticipate these kinds of questions to avoid unpleasant surprises during the course of the project, and to make the best possible estimate of the time needed to complete it.
For example, if you’re going to be working with several teams, you’ll need to think about the best way of coordinating your actions. Also, there may be a specific format expected for the deliverable, so you’ll need to take this into account when modeling your project.
A moment’s reflection beforehand on the nature of the problem and the evaluation method to be chosen is also essential to any good start:
- Am I dealing with a supervised, unsupervised or semi-supervised classification problem, or a regression problem?
- Which metric should I choose? RMSE*? accuracy?
Once again, it’s a question of preparing the ground as well as possible, a crucial step if you are to approach your project from the right angle.
The last thing to bear in mind before getting started is the equipment available.
Which machine for which computing time? There’s no point, for example, in proposing a solution that will take a whole day to run.
2. Retrieve and explore data
When it comes to retrieving the data you’re going to work on, your first priority is to make sure you have the optimum working environment: do you have all the packages you need? You may find yourself working on several projects at once, requiring different environments. If you’re worried about creating conflicts, don’t hesitate to create isolated virtual environments.
Once you’ve checked out your working environment, it’s time to download and explore the data.
Descriptive and visual analysis is crucial to understanding the structure, strengths and weaknesses of your dataset.
Identify the types of variables you have (qualitative, quantitative) and don’t hesitate to look for promising combinations to test for your model.
Finally, don’t forget to analyze the correlations between the different variables, as this will help you to understand your data as a whole.
3. Prepare your work bases
For any data science project, you will generally need to separate your database into two parts: a training database and a test database. This strategy enables you to check the effectiveness of your model.
It’s quite possible that your data in its current state won’t allow you to model it, so it’s up to you to transform it.
To do this, you need to prioritize the management of missing values and define a strategy. Here again, you need to ask yourself the right questions:
Do I have Nans* in the quantitative variables? If so, what proportion for each variable? What is my exclusion threshold? How can I fill in my Nans without jeopardizing my model?
You’ll need to do the same with qualitative variables. You’ll need to transform your categorical variables using discretization methods.
Finally, as Machine Learning algorithms don’t always work properly with numerical variables whose scales are different, you’ll need to recalibrate them using a min-max or normalization transformation.
4. Select and train a model
Once your data is ready, you can start modeling. Scikit-Learn provides a multitude of regression, classification and ensemble methods. Of course, the choice of model depends on the problem at hand.
It will undoubtedly be necessary to go back to the first step and clarify the nature of the problem. Obviously, there is no single regression or classification algorithm.
You have two options:
- Test them all and select the most efficient (probably too costly)
- Decide which one to test on the basis of your data and available resources.
Once you’ve chosen your model, there’s the question of parameterization: how can you optimize the algorithm’s parameters to limit overfitting?
Considering a grid search may be one solution, but it may also prove time-consuming, depending on your resources.
5. Evaluate your results
Once you’ve trained your model, you’ll need to evaluate its effectiveness using your test base and the metric you chose in the first step.
Are you satisfied with the results obtained with your metric? If not, can you improve the results? There are three ways to answer this question:
- The model: it may not be suited to what you want to do. Don’t hesitate to explore other avenues.
- The parameters of your model: they may not be optimized, which may be detrimental to its performance.
- The data: If you’re confident in your choice of algorithm, you may need to enrich your data to improve your model’s performance.
These 5 steps should be seen as benchmarks when you’re working on a project. You’ll need to reconsider certain steps as the situation changes. Don’t hesitate to go back and forth between them.
Would you like to carry out a data project as part of a training course leading to certification? Would you like to improve your Data Science skills with expert guidance? Don’t hesitate, check out our upcoming launch dates or contact us for more information!
