Role of data scientist in big data

A data scientist is the person who turns raw, large and messy data into useful knowledge that a business can act on. The role combines statistics, programming and domain knowledge.

Main roles and responsibilities

1. Problem definition: work with managers to convert a business problem (e.g. "why are customers leaving?") into a data problem.
2. Data collection and acquisition: gather data from logs, sensors, social media, databases and APIs, and decide which sources are useful.
3. Data cleaning and preparation: remove duplicates, fix missing values, convert formats and integrate sources. This often takes most of the project time.
4. Exploration and analysis: find patterns, correlations and outliers using statistics and visualisation.
5. Model building: apply machine learning (classification, clustering, regression, recommendation) to predict or classify.
6. Using big data tools: write jobs in Hadoop MapReduce, Spark, Hive or Pig, and query NoSQL stores, so the analysis scales over a cluster.
7. Validation: test the model for accuracy, precision, recall and bias before use.
8. Communication: present results through dashboards, charts and reports so non-technical people can decide.
9. Deployment and monitoring: put the model in production and update it as data changes.

Skills required

• Mathematics and statistics, probability
• Programming (Python, R, Java, SQL)
• Big data platforms (Hadoop, Spark, NoSQL)
• Machine learning and data visualisation
• Domain knowledge and communication

Example

In a telecom company, the data scientist collects call records and complaints, cleans them on Hadoop, builds a churn-prediction model, and gives the marketing team a list of customers likely to leave.