Well, you will encounter data in many forms. Sometimes it will already have been "hadooped-down" by someone else, and you can analyze it on a single machine. Don't underestimate what a single machine can do these days, if you have say 16 cores and 32 GB of RAM.
Or you can set up a system that will incrementally summarize the data, and then you could do smaller queries against those summaries. That is the goal of Storm AFAIK.
I think that is better model for a lot of applications. The model of having your production systems save terabytes of raw data and then analyzing it in a big batch job leaves a lot to be desired. It works but it's not very flexible and has this latency problem.
Hadoop is good in that it's the only open source solution I know of that can churn through hundreds of terabytes of data. But I wouldn't say it's a complete solution for "managing big data". It's part of one.
Or you can set up a system that will incrementally summarize the data, and then you could do smaller queries against those summaries. That is the goal of Storm AFAIK.
I think that is better model for a lot of applications. The model of having your production systems save terabytes of raw data and then analyzing it in a big batch job leaves a lot to be desired. It works but it's not very flexible and has this latency problem.
Hadoop is good in that it's the only open source solution I know of that can churn through hundreds of terabytes of data. But I wouldn't say it's a complete solution for "managing big data". It's part of one.