Answer by Rustin Bergren:
I think what you’re trying to do is have MATLAB ‘give’ you the curve of the 2D spectrogram plot such that you can compare column evaluations with the proposed spline curve. Then you can say, “yes, the max value in this column is within xx of the spline; therefor, its part of the set.” Is that right?
The problem is, there are literally an infinite number of regressions that will match your data. If you supply the spline algorithm with data points that are part of a curve you’re not really interested in, then they will effect the overall curve and in many cases the new curve will now match those. So what you would need to do is remove those points you don’t want smoothed in to the calculated spline; however, it seems like what you’re trying to do is find out what those points are. In which case, we’re now chasing our own tail.
By example
Let’s take a curve like this:
1 2 3 
x = 0:10;
y = 10.*x.^2+3.*x;
plot(x,y,'o');

Now let’s modify some of the points on the end so it doesn’t match the curve perfectly.
1 2 3 4 5 
y_data = y;
y_data(end) = y_data(end)  888;
y_data(end1) = y_data(end1)  400;
plot(x,y_data,'o');
ylim([0,1200]);

Without adjusting anything let’s spline based on all those points
1 2 3 
yspl = spline(x,y_data,0:.1:10);
plot(x,y_data,'o',[0:.1:10],yspl,'.');
ylim([0,1200]);

The red dashed line is the calculated spline. While our data points are from 0, 1, 2…10; I configured the resultant spline to interpolate 10 times that resolution so it’s defined at 0,.1,.2,.3,…10.
But hey! That’s not what we want. That spline just included the data points we were trying to ignore. So what can we do? Well, let’s exclude some of the points from the calculation.
I’m going to remove the first 3 and last 3 by using range selections but with essentially the same command. Notice, you can keep the same range parameter (the third term in the spline function call)–this will now extrapolate the calculated spline on to that range.
1 2 3 
yspl = spline(x(4:end4),y_data(4:end4),0:.1:10);
plot(x,y_data,'o',[0:.1:10],yspl,'.');

So that calculates the spline using only these 4 points and it doesn’t look to bad for what we were trying to do.
The problem, however, is how do you know from your series which points to include and which to remove? It seems like your trying to do all this programmatically. Can you just assume that the max column values from 30% to 60% of your x range are valid points within your animal call? If so, then gravy–try doing this.
You can also try smoothing, which is essentially the same as the spline but it allows you to supply a smoothing parameter. Notice however that our bad values at the end, if included, still mess us up
1 2 3 
yspl = csaps(x,y_data,.1,0:.1:10);
plot(x,y_data,'o',[0:.1:10],yspl,'.');
ylim([0,1200]);

Not only does smoothing not cause the interpolated curve to not move too much away from our down sloping points at the end, but it also moves the spline curve away from the data points that match our curve.
I hope this answers your question. If you want to know how to extract the points from your FFT, and use the functions I listed, please mention so in the comments. So to review quick, the spline function is called like this:
spline(x_datapoints, y_values_at_x_datapoints, new_x_range)
The new_x_range allows you to both interpolate and extrapolate the calculated regression function.
To be honest, if your trying to isolate the audio sections, while a novel idea, I don’t think this is the best way to go about this. Thanks for the A2A!