Data Preparation Steps for Machine Learning

Motivation

This note book is supposed to be my reference for different steps required for data wrangling and clean up. This is going to be much larger file, as I gradually copy my learnings and my old records here. Currently it is the bear bone version, but it’s going to be built on it. I hope it would be helpful for others too.

Usage example

This repository is going to be a recording place for typical steps in data preparations, and everyone can benefit from. End of this file would be the beginning of ML exercise, which is not covered here.

Requirements

The major libraries used in these projects are:

1. numpy,
2. pandas,
3. scickitlearn,
4. matplotlib,
5. encodings

File structure

There is a data set called bp2018.csv in this project which contains British Petroleum energy outlook in 2018. Any other data set can be used, and there is nothing special about this particular file.

The notebook file is called DataPreparation4MachineLearning.ipynb and includes the following sections:

Part 0: loading data

Part 1: Getting to know the data

Part 2: null value investigation

Part 3: dropping duplicates

Part 4: Converting the object to appropriate type and parsing dates

Part 5: Imputation

Part 6: Getting dummy variables

Part 7: Finding and removing outliers

Part 8: Scaling and Transforms

References