Take the 2-minute tour ×

I'm trying to write a program that cleans data, using Matlab. This program takes in the max and min that the data can be, and throws out data that is less than the min or greater than the max. There looks like a small issue with the cleaning part. This case ONLY happens when the minimum range of the variable being checked is 0. If this is the case, for one reason or another, the program won't throw away data points that are between 0 and -1. I've been trying to fix this for some time now, and noticed that this is the only case where this happens, and if you try to run a SQL query selecting data that is < 0, it will leave out data between 0 and -1, so effectively the same error as what's happening to me. Wondering if anyone might recognize this and know what it could be.

share|improve this question
you want help with your code ? Show us your code ! And clarify the relationship between the SQL query and the Matlab code. –  High Performance Mark Jul 26 '10 at 17:15
In the section of debugging, under Process of Elimination Andrew Hunt comes with Tip 26: "select" isn't broken", after a programmer that spend two days finding a bug in select because he thought his own code was flawless. It wasn't. The moral: the likelier cause for your error is not the database, or MatLab, but your hand-written code (and to help, we'd like to see it). amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X –  Abel Jul 26 '10 at 17:35
@Abel: Sorry for the lack of code example. New here :) @Amro's suggestion seemed to work. And you were definitely right. I just overlooked something that, now that I see it, have no idea how I did. Thanks for all your time and effort in helping me out. –  Joey Jul 30 '10 at 19:02
Joey, welcome to SO! Make sure to have a look at the short FAQ and have fun asking and answering questions :) –  Abel Jul 30 '10 at 22:15
add comment

5 Answers

up vote 3 down vote accepted

I would write such a function as:

function data = cleanseData(data, limits)
    limits = sort(limits);
    data = data( limits(1) <= data & data <= limits(2) );

an example usage:

a = rand(100,1)*10;

b = cleanseData(a, [-2 5]);
c = cleanseData(a, [0 -1]);
share|improve this answer
add comment

-1 is less than 0, so 0 should be the max value. And if this is the case it will keep points between -1 and 0 by your definition of the cleaning operation:


If you want to throw away (using the above definition)

data points that are between 0 and -1

then you need to set 0 as the min value and -1 as the max value --- which does not make sense.

Also, I think you mean

and throws out data that is less than the min AND greater than the max.

share|improve this answer
I think you misunderstood. When the OP chooses 0 as the minimum value, values in the range -1 to 0 aren't getting removed for some reason. –  gnovice Jul 26 '10 at 17:24
add comment

It may be that the floats are getting casted to ints before the comparison. I don't know matlab, but in python int(-0.5)==0, which could explain the extra data points getting in. You can test this by setting the min to -1, if you then also get values from -1 to -2 then you'll need to make sure casting isn't being done.

share|improve this answer
add comment

If I try to mimic your situation with SQL, and run the following query against a datatable that has 1.00, 0.00, -0.20, -0.80. -1.00, -1.20 and -2.00 in the column SomeVal, it correctly returns -0.20 and -0.80, which is as expected.

FROM    SomeTable
WHERE   (SomeVal < 0) AND (SomeVal > - 1)

The same is true for MatLab. Perhaps there's an error in your code. Dheck the above statement with your own SELECT statement to see if something's amiss.

share|improve this answer
add comment

I can imagine such a bug if you do something like

minimum = 0
if minimum and value < minimum
share|improve this answer
add comment

Your Answer


