A research paper estimates that the big data market in Japan could reach 1,000 billion Yen by 2020 and lead to smart cities that minimize waste and improve efficiency.
A Critical Application
A typical example of processing such volumes of data in real time is provided by an application in Japan that provides real time earthquake warning to citizens. This kind of application was simply not possible earlier at the scale that cloud computing and big data can provide. As this application improves, it will use more diverse data to improve the quality of its prediction.
How Governments are Getting Involved
With cloud computing approaching maturity (though there is still some distance to go) and companies beginning to get serious mileage out of the technology, government began to get involved to see if there were efficiencies and benefits they could tap. If there is one thing that governments have in plenty - and I don’t mean red tape – it is data. There are mountains of data with the government and till very recently, it simply filed it away without being able to do very much with it. Cloud computing of course has given new capabilities to data management and value extraction. This is where big data comes into its own. Properly harnessed, the information hidden in big data will give citizens –
- Improved healthcare at lower overall cost, with predictive capabilities and cost effective archival of all kinds of medical records, including MRIs, CAT scans, X rays, every ECG and EEG report, every test result and so on. All of it available to any medical practitioner under controlled circumstances.
- Safer living even in the face of increased threats Accurate long and short range weather forecasting – leading to greater efficiency in agriculture and other weather dependent vocations.
- Much greater vehicular densities with much lower accidents
- Far more insurance coverage with highly customized policies – tailored to individuals, much larger claims and much lower fraud rates.
As all of these and other scenarios come true, big data and cloud computing could have an impact on our lives that would be as great as that of the Internet itself.
Unlike private companies where the data driven culture has firmly taken route and the implementation of decisions and directives is rapid, work in the government is far more deliberate and many facets of the problem and the proposed solution are examined and thought through before a decision is taken. Governments also have to contend with privacy and security issues.
Technologies Required
Managing Big Data requires companies to be able to handle
Real Time Analysis and Streaming of data – with massive amounts of data simply flowing in, there has to be a very large I/O capability and parallel processing that uses rules based algorithms to manage the data streams. Typically, this kind of data could be from sensors (think of surveillance cameras), social media sites, video records, communication records. This has to be handled, tagged, classified and stored as it comes in otherwise it would be impossible to locate a data item of interest. This massive, multiprocessor based handling is in reality only possible with cloud based systems.
Map Reduce Frameworks – at this level of processing, a master server maps the data (distributes it) to many different servers, collect their processing results and ‘reduce’ it to a final summarized result. These servers are based on Hadoop and related technology. They provide the capability to process data streams in parallel and move them to distributed servers. They rely on Hadoop Distributed File System to store the data in a manner that is fault tolerant. At this stage, the data can be used to provide output in the form of business analytics, log analysis, web search engine output and so on. Depending on the quantity of data coming in, the map server can call in additional servers or release the ones that are not needed. This capability is native to cloud applications and hence the server is ideally suited to run in the cloud.
In Summary
There are several components that work together to be able to create an effective application cluster. It all starts with a well planned information strategy that then leads to an information infrastructure where the appropriate technology is used to build systems that can handle such applications. There have to be policies that control the management, use and protection of the data a phased execution plan. Obviously, such a complex application cannot be delivered all at once, there is iterative development that continuously refines the application and creates new opportunities. Big data opens up possibilities of applications that were in the realm of fantasy a few years earlier. However, the kind of processing that is needed can only be available in the cloud. Truly, big data is an idea whose time has come.
Be Part of Our Cloud Conversation
Our articles are written to provide you with tools and information to meet your IT and cloud solution needs. Join us on
Facebook and
Twitter.
About the Guest Author:
Sanjay Srivastava has been active in computing infrastructure and has participated in major projects on cloud computing, networking, VoIP and in creation of applications running over distributed databases. Due to a military background, his focus has always been on stability and availability of infrastructure. Sanjay was the Director of Information Technology in a major enterprise and managed the transition from legacy software to fully networked operations using private cloud infrastructure. He now writes extensively on cloud computing and networking and is about to move to his farm in Central India where he plans to use cloud computing and modern technology to improve the lives of rural folk in India.
