Big Data with Cloud computing (Amazon Web Service)

As Big Data projects requires huge amount of resources, cloud computing helps to avoid resource maintenance headache, Already discussed more details in previous post. Here discuss more about big data with cloud provider Amazon Web Service. 

Three major resource requires for any type of computing are Processor (CPU), Memory (RAM), Storage (Hard disk). Amount of each resource requires for projects vary, especially for big data project need more for processing huge amount of data. 

Amazon provides Elastic Compute Cloud (EC2) instance, is similar like a single desktop machine which is cloud, user can compute over network. EC2 are available in different types for user from micro instance to very large instance. user can create any number of EC2 instance. EC2 small instance has memory of 1.7GB, 160GB hard disk and one 32-bit processor. Those who requires more power in single instance can go up-to EC2 Cluster GPU instance, which has 22 GB memory, 33.5 processor(64-bit), 1680GB hard disk.

For big data project, resources need may increase at any time, some time data size will grow, or complexity of process and analysis will increase. For this type of situation, Amazon provide Amazon Cloud Watch which keeps monitoring activity of our resource usage in cloud, when demand arise, user can use Amazon Auto scaling feature to increase the capacity of instance by writing some condition in cloud watch. and decrease it back when demands dulls. Using Amazon Auto scaling service, user can also increase number of instances. In our Big data term ‘user can also increase number of nodes when demand raised’. Auto Scaling real time example - click here (check last paragraph)

Lot of cloud providers offers EC2 type of instance, but the main reason i write about Amazon (AWS)  is, Amazon also provider Amazon Elastic MapReduce instance which pre-configured with hadoop framework. So with Amazon Elastic MapReduce user can create a cluster with any number of nodes in few minutes. This hadoop cluster is well performance tuned, user just need to select type of instance(node) and number of node requires, everything else will be taken care by Amazon. This one answer ‘how to do’ for our previous post question ‘Why should we use Cloud Computing for Big data’ 

Amazon S3 (Simple Storage service) provides storage space to store user data. user can store unlimited amount of data in S3 it will be persist even our instance goes down. For processing huge amount of data, its recommended to store all data in S3 and can refer directly from Amazon Elastic MapReduce to process, and also can store processed result data in S3. It is easy to inter-operate between different amazon web services.