I agree with you when it concerns to forecasting the output y. On [1] Candès points out that coherence can be ignored if the goal is to recover Xb and not b.

However, on many cases regression is used to explain the output from the exogenous variables. On that case, I believe X should obey the RIP (or any equivalent condition) in order to preserve the interpretability of the variable selection.

[1] – Candes, Emmanuel J., et al. “Compressed sensing with coherent and redundant dictionaries.” Applied and Computational Harmonic Analysis 31.1 (2011): 59-73.

]]>Thanks for the comment Igor.

Perhaps I focused too much on RIP.

My real point was that ANY conditions needed for recovery

are unrealistic for vanilla regression problems.

To put it another way, recovery is (usually) not an interesting problem

in ordinary regression.

(As a pedantic example, take two covariates to be perfectly

correlated. You can’t separate the two but you don’t need to

if you focus on predictive accuracy.)

In other words, regression and compressed sensing are very

different problems (despite the obvious overlap).

But the difference often gets ignored in some statistics papers.

Thanks for the discussion and references

Larry

]]>Within Compressed Sensing, RIP is not a condition for sparse recovery that is in fashion anymore. In particular since about 2008 and the work of Donoho and Tanner [1], we have empirical proofs that in general this condition is too stringent. The generic Donoho-Tanner phase transition generally shows that l_1 recovery of sparse vectors admit less sparse vectors than what RIP would require.

In a certain sense, the “coup de grace” was given to that argument recently with a phase transition even beyond the traditional Donoho-Tanner PT (see [2]) and featured in the work of Krzakala et al [3]. All this can be easily pictured in this beautiful hand crafted drawing I made for the occasion: http://goo.gl/H3m4C 🙂

The work of Krzakala et al [3] show that by choosing the right design matrix, one can attain very interesting limits. Right, within the community, there is an on-going discussion as to whether we ought to put some constraint on the design matrix or push further the constraint on the vector to be recovered. On the one hand, it means, we need to design different hardware, on the other, it means we need to dig further in the structured sparsity world that the ML folks are investigating. An example of that discussion can be found in [4] and [5].

But let me play the devil’s advocate for RIP, at least within compressive sensing. RIP was for a long while the only mathematically rigorous reasoning at hand for people to show that their empirical finding held even if in most of these papers, the authors acknowledged that their design matrices did not fit the RIP condition. To a certain extent, I can bet you could not even get a paper out unless the first few paragraphs talked about the solid reason as to why sparse recovery was possible (thanks to RIP). We have also seen many authors coming up with different conditions paralleling RIP, so it was a useful tool for a while.

There is also another aspect as to why RIP is interesting: It is a similar tool that is currently used to investigate low rank type of algorithms. Here again, I am sure that most phase transition discovered there will show that RIP is too stringent, but again it may the only theoretical tool around grounded in some theory.

Cheers,

Igor.

[1] Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing by Jared Tanner and David L. Donoho http://www.maths.ed.ac.uk/~tanner/DoTa_Universality.pdf

[2] Compressive Sensing: The Big Picture, https://sites.google.com/site/igorcarron2/cs#measurement

[3] Statistical physics-based reconstruction in compressed sensing by Florent Krzakala, Marc Mézard, François Sausset, Yifan Sun, Lenka Zdeborová, http://arxiv.org/abs/1109.4424

[4] http://nuit-blanche.blogspot.com/2012/08/intra-block-correlation-and-sparse.html

[5] http://nuit-blanche.blogspot.com/2012/08/more-intra-block-correlation-and-sparse.html

You might be right.

I wrote this in a hotel room

and derived it on some scrap paper.

What I wrote might be a bit off

—LW