---
title: "Big data trends, challenges, characteristics"
subject: "Big Data Technologies"
chapter: 1
key: "semester_8/big_data_technologies:big-data-trends-challenges"
source: https://www.bctengineer.com/q/semester_8/big_data_technologies/big-data-trends-challenges
site: BCT Engineering
---

# Big data trends, challenges, characteristics

Big Data Technologies · Chapter 1: Introduction to Big Data

- Asked in: 2073 Bhadra, 10 marks
- Video: [Challenges and Current Trends of Big Data Technologies: Part 1](https://www.youtube.com/watch?v=ZLRX7LWNOCE) — myAcademic-Scholartica
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_8/big_data_technologies/big-data-trends-challenges
- Whole chapter: https://www.bctengineer.com/md/semester_8/big_data_technologies/chapter-1

## Question

What are the current trends in big data analytics? What are the technical challenges and characteristics of big data?

## Answer

**Big data** means datasets so large, fast-growing and varied that traditional databases and tools cannot capture, store, manage and analyse them efficiently.

#### Current trends in big data analytics
1. **Real-time and streaming analytics:** tools such as Kafka, Spark Streaming and Flink process data as it arrives (fraud detection, live recommendations).
2. **Artificial intelligence and machine learning:** analytics now includes predictive and prescriptive models and deep learning on big data.
3. **Cloud-based big data:** AWS EMR, Google BigQuery and Azure services give pay-as-you-go storage and processing.
4. **In-memory computing:** Apache Spark and in-memory databases give much faster processing than disk-based MapReduce.
5. **Internet of Things (IoT) and edge analytics:** sensors produce huge streams analysed near the source.
6. **NoSQL and NewSQL databases:** flexible, scalable storage for unstructured data.
7. **Data lakes and lakehouses:** keep raw data of all types in one place for later analysis.
8. **Data governance, privacy and security:** laws like GDPR increase focus on protection and compliance.
9. **Self-service analytics and visualisation:** tools such as Tableau and Power BI allow business users to analyse data.
10. **Augmented analytics / AutoML:** automation of data preparation and model building.

#### Technical challenges
1. **Storage and scalability:** storing petabytes cost-effectively and scaling out.
2. **Processing speed:** analysing data fast enough, especially for real time.
3. **Data quality:** incomplete, inconsistent, noisy data.
4. **Data integration:** combining structured, semi-structured and unstructured data from many sources.
5. **Security and privacy:** protecting sensitive data across clusters.
6. **Fault tolerance and consistency:** nodes fail often; replicas must stay consistent (CAP theorem).
7. **Shortage of skills:** few trained data scientists and engineers.
8. **Complex tools and cost:** many tools to learn, deploy and manage.
9. **Data transfer bottleneck:** moving large data over the network is slow.

#### Characteristics of big data (the Vs)
| V | Meaning |
|---|---|
| Volume | Very large size (TB to PB and beyond) |
| Velocity | High speed of data generation and processing |
| Variety | Structured, semi-structured, unstructured data |
| Veracity | Uncertainty and trustworthiness of data |
| Value | Useful insight that can be extracted |
| Variability | Data meaning and flow change over time |

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. Why is Apache Spark's in-memory computing a current trend?
   A. It stores data only on tape
   B. It gives much faster processing than disk-based MapReduce
   C. It removes the need for a cluster
   D. It works only for structured data
   Answer: B. Keeping data in memory avoids repeated disk access, making Spark far faster than disk-based MapReduce. (Key point: In-memory computing trend)

2. Which set of tools is given for real-time and streaming analytics?
   A. Tableau, Power BI, Excel
   B. Sqoop, Hive, Pig
   C. HDFS, GFS, S3
   D. Kafka, Spark Streaming, Flink
   Answer: D. Kafka, Spark Streaming and Flink process data as it arrives, e.g. for fraud detection. (Key point: Real-time analytics tools)

3. Which V of big data refers to the uncertainty and trustworthiness of data?
   A. Veracity
   B. Variability
   C. Velocity
   D. Value
   Answer: A. Veracity is about data quality, noise, bias and missing values; Variability is change of meaning over time. (Key point: Veracity)

4. Which challenge arises because nodes fail often and replicas must agree?
   A. Shortage of skills
   B. Fault tolerance and consistency (CAP theorem)
   C. Self-service visualisation
   D. Data lakes
   Answer: B. With many nodes, failures are common and replicas must stay consistent, which is linked to the CAP theorem. (Key point: Fault tolerance and consistency challenge)

## Related in Big Data Technologies

- [Data analytics process and distributed computing](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Adata-analytics-distributed-role) — Big Data Technologies, chapter 1
- [Role of data scientist in big data](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Arole-of-data-scientist) — Big Data Technologies, chapter 1, asked 2×
- [Why distributed computing for big data](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Adistributed-computing-necessary) — Big Data Technologies, chapter 1
- [Distributed system in big data example](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Adistributed-system-example) — Big Data Technologies, chapter 1
- [Hybrid data in big data warehouse](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Ahybrid-data-warehouse) — Big Data Technologies, chapter 1
- [Attributes of Big Data](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Abig-data-attributes) — Big Data Technologies, chapter 1

## Related in other subjects

- [IS areas using AI / data analytics](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Aai-areas-data-analytics) — Information System, chapter 4, asked 2×
- [Characteristics of technical report writing](https://www.bctengineer.com/question.md?key=semester_7%2Festimating_and_costing%3Atechnical-report-characteristics) — Estimating and Costing, chapter 5
- [Data integration and its challenges](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Adata-integration) — Data Mining, chapter 2
- [Big data and its processing](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Abig-data-processing) — Information System, chapter 8