Monday, February 11, 2013

This Week's Guardians of Science: Zeno Gantner and Peyman Milanfar

In the past week, we have had two public reviews of preprints mentioned here on Nuit Blanche. Is this how post peer review publishing should work. I don't know. Let's examine what happened:


Zeno Gantner provided some thoughtful comment as to why the paper featured in That Netflix RMSE is way too low or is it ? ( Clustering-Based Matrix Factorization - implementation -) was not giving the right result. 

....Found your bug in the evaluation. First of all, that rounding method. No. So wrong in so many ways.
Independently of that, you use long for the variable totalErr. long is an integral type, not a floating point type. You must use double, otherwise you can lose precision. And you do, by rounding to zero, I guess, which makes your RMSEs significantly lower.
If you had written
totalErr = totalErr + errortemp;
instead of
totalErr += errortemp;
there would have been a warning about it:
possible loss of precision
found : float
required: long
After thanking Zeno Gantner several times, the original author did something that I half expected. He removed the preprint from Arxiv (a good thing) and also removed the entire conversation from LinkedIn. I am OK with both actions as long as I was able to salvage the part pointing to the error (see above). With a post publication peer review, we need to have some sort of mechanism by which the lesson learned above remains for future researchers (so they don't make the same mistake).

Other remarks include:

The second blog entry got a large  10 G+, a score showing the interest of people in the potentially ground breaking result of the paper.

I like how the discussion groups on the Google+ Community (212), the CompressiveSensing subreddit (79), the LinkedIn Compressive Sensing group (2043) or the Matrix Factorization (539) helped in getting this conversation going between people who would likely not talk to each other. 

3 comments:

Zeno said...

Hi Igor, thank you for this (quite flattering) follow-up post.

While the LinkedIn discussion is gone, the paper is still on arXiv: http://arxiv.org/abs/1301.6659

Igor said...

Zeno,

He told me he removed it from arxiv. Looks like it is still there....we'll see.

Igor.

Nima said...

Hi Zeno and Igor,

I did withdraw the paper:

http://arxiv.org/abs/1301.6659

Nima

Printfriendly