wu :: forums - Print Page


    
      
        wu :: forums
        (http://www.ocf.berkeley.edu/~wwu/cgi-bin/yabb/YaBB.cgi)
      

        riddles >> putnam exam (pure math) >> Information derived from a data set
        
(Message started by: davkrs on Dec 26^th, 2013, 1:25pm)

Title: Information derived from a data set
Post by davkrs on Dec 26^th, 2013, 1:25pm

In this blog post, http://www.evanmiller.org/small-data.html, the author claims that amount of information that can be derived from a data set is proportional to square root of the size of the data set (in the best case).

Is this true. Is there a proof behind this claim. Did you ever notice in your research/job/work/life. What are your thoughts

P.S. Though this question, and blog post linked to this question, are more applied math than pure math, I posted it here because I wasn't sure which category this applied math question should be posted in. Let me know if this question should be moved to a different category

Title: Re: Information derived from a data set
Post by towr on Dec 26^th, 2013, 10:44pm

He seems to be speaking from the viewpoint of regression analysis only.
From basic statistics we also know that e.g. the estimation of the variance (i.e. how certain we are about the true value of a statistical variable) is inversely proportional to the square-root of the size of the data (for approximately normally distributed data). So you get less additional certainty for every extra amount of data.
However, if you want to analyze rare events, then you need to filter a huge amount of data just to find enough of them to be able to say anything about them. And therefore you need to collect a huge amount of data (especially if you don't know beforehand what rare events you want to analyze -- which brings it's own troubles, because you can always find something on a finishing expedition, even if it turns out to be an old shoe). And as he says, if you deal with a lot of parameters, you need a lot of data to estimate them all. And some data is so noisy, you need a lot to average out the noise; it's often not neatly normally distributed data.