On Traffic – Aware Partition and Aggregation in MapReduce for Big Data Applications

Abstract

On Traffic – Aware Partition and Aggregation in MapReduce for Big Data Applications big data project report  The MapReduce programming model simplifies large-scale data processing on commodity cluster by exploiting parallel map tasks and reduce tasks. Although many efforts have been made to improve the performance of MapReduce jobs, they ignore the network traffic generated in the shuffle phase, which plays a critical role in performance enhancement. Traditionally, a hash function is used to partition intermediate data among reduce tasks, which, however, is not traffic-efficient because network topology and data size associated with each key are not taken into consideration.

H/W System Configuration:-

System          : Pentium I3 Processor.
Hard Disk       : 500 GB.
Monitor          : Standard LED Monitor
Input Devices : Keyboard
Ram               : 4 GB

S/W System Configuration:-

Operating system              : Windows 7/8/10.
Available Coding Language : Java and Phonegap
Database                          : MYSQL

Conclusion

On Traffic – Aware Partition and Aggregation in MapReduce for Big Data Applications big data project report paper concludes that the network traffic can be reduced in both offline and online cases. This is observed by the proposed methodology where there is an optimization of redundant files in the storage. This is achieved by the IP look up concepts where we are doing a file redirect to reduce the network traffic when requested cloud region is busy.

Also when the files are saving multiple times by the different servers, the replication control concept starts implementing where replicated multiple files cannot be saved in the cloud which is achieved by the MD5 the hash code generation.