Jump to content

Ski Test Retrospect


Horton
 Share

Recommended Posts

  • Administrators


Ski Test Retrospect: Part 1



The question as been asked: Why there is no numerical data in the 2008 ski tests?




For the first two years, it was data driven. My original goal was to generate numbers that would help skiers choose a ski. The math was honest and data collection could not have been better intentioned. There were a number of times where the data said one thing and looking back I believe something else.




Speed is a classic example of why the numerical data was a poor indicator. Skiers were asked to rank the skis biased on speed but speed in an impression that I now believe is far from accurate. If a ski is squirrelly off the ball it gives the impression that it is fast but if a ski is really smooth off the ball it gives the impression of not being fast.




What about the F1? To me it feels very slow off the ball but it gets crazy wide at the next ball. In this case I am just not sure how this ski should be ranked in terms of speed.




In 07 we ranked the MPD and Monza at the bottom of the pack in terms of speed. Looking back that can not be right. That is what the team said so it is what I reported.

 Goode  KD Skis ★ MasterCraft ★ PerfSki ★ Radar ★ Reflex ★ S Lines ★ Stokes

Drop a dime in the can

 

Link to comment
Share on other sites

  • Baller

I found it interesting too that the Monza went from the head of the class in '06 to near the bottom in '07 with basically the same model.

I'm curiou to know...

Do preconceived notions and/or ski reputation play an unintentional play a role in a reviewers rating? (example - Goode has a reputation of being fast, so I'll automatically give it a high speed rating)  Would reviews be any different if every ski had blank graphics?

Were number ratings relative to the testing pool? (example - Goode is the fastest of 10 ski's present, so we give it the highest speed rating possible even though there may be faster skis on the market)

Anthony

Link to comment
Share on other sites

  • Administrators

I would love to do the blank ski thing but most of use could ID the skis even without the graphics.

What happened to the Monza is that we had much better #s one year then the other. That is why we had factory reps come this year to ensure that the skis were set up as well as possible. 

 Goode  KD Skis ★ MasterCraft ★ PerfSki ★ Radar ★ Reflex ★ S Lines ★ Stokes

Drop a dime in the can

 

Link to comment
Share on other sites

I think the Monza was sort of a breakthrough ski when it first hit the market - but now the majority of ski companies are producing high end ski's with the ability to get cross course as fast as the Monza, however I will say that I still believe the Monza is the fastest ski right off the ball on the market today.  But as mentioned on a different thread "speed" can be taken to mean several things in the slalom course - not just mph!

 And I don't think there were any preconceived notions about any of the skis at the tests - everyone just strapped them on and ski'd them.  

Link to comment
Share on other sites

  • Baller
What about scores? Or buoy counts? Or compare the test scores to the skiers' PB or average score. I could never tell if my Nomad was faster than my X5, but I definitely picked up an entire pass in tournaments, but then again, I'm just learning shortline, I might have just gotten better at gates and patience and what-not.
Link to comment
Share on other sites

  • Administrators

I already wrote something like this but I lost it when I rebuilt the forum over the weekend. . . here we go again.

 

The original idea of the “Independent Ski Tests” was to be free of outside influence and gather purely data driven conclusions. After 3 years of the tests I think this data driven part of the idea is clearly flawed.

 

Fins and bindings:

The first year 2006 we set up the skis according to the factory literature; fin and binding setup was not exactly clear. Some of the factories do a great job of spelling out exactly how to set up the skis and some do an awful job.

 

In 2007 we went to great lengths to get explicit instructions from each factory. Ski set up was more even but some skis benefited from the knowledge for test skiers and some did not. We had on hand skiers who had better then factory #s for at least two of the skis as well as familiarity with the skis.

In 2008 I thought I had this problem licked by having each factory send a representative to do all the setup and to effectively do a demo day. The idea is that all the skis would be shown in the best possible light. What happened was that we had different approaches to the demo by each of the factories. Some coached more, some tweaked more and some tried to stay as close to factory #s as possible.

 

In short: in 06 we reported what a group 15 skiers thought of the skis out of the box with a weekend of skiing and uneven setup. In 07 we did the same thing but with better setup. For 08 we had the best setup yet but still not exactly even.

 

Weather:

For and example. In 08 the wind blew the first day. There is no way that the rankings would have been fair when compared to the skis that were reviewed for the rest of the day.

 

Fresh skiers:

With this many skis we have to ski a number of days in a row to get though them all. The first day a skier tests skis they are going to be a lot fresher then the 3rd.

 

Ranking #s mean something different to everyone:

For 06 & 07 we ranked the skis from 1 to 5. A score of 3 was acceptable, a score of 5 was outstanding, and so on. The problem with this is that every skier has a different feel for what each grade feels like. In theory if there were enough test skiers on each ski this would work itself out in the statistics but that would require a LOT of skiing.

 

Test skier score vs normal score:

It has been suggested that I should have reported the best score each skier had on each test ski and their personal best or normal score on their own ski. This sounds good but does not work. Using myself as an example: I ran within one ball of my normal practice score on every ski in the test. On the ski that I have my highest ever score on I spent over a week in it before I clicked and set a new PB.

 

All of the above points are examples of why the data driven review is always somewhat skewed. For that reason, in 08 I chose to write ½ the article as I understood the skiers notes and the other ½  of the article was actual skier notes.

 

When I used the numbers in years past I was stuck with the numbers even if I believed that the score was wrong. If you look at the Monza and F1 scores from 06 to 07 I think you will see example of how reviews can change more then the ski did. I think both of these changes are settings based.

To those who want to criticize, I will say that the 09 Test does not yet have an admin. To those who claim bias all I can say is that I jumped though every hoop for that to not be the case. 

To those who think I have been paid off. I was asked by one manufacturer to supply CarbonFins for the test and I refused on the grounds that aftermarket parts on any ski would be bias.

 Goode  KD Skis ★ MasterCraft ★ PerfSki ★ Radar ★ Reflex ★ S Lines ★ Stokes

Drop a dime in the can

 

Link to comment
Share on other sites

  • Baller

From a test skier perspective, I liked the number rating system. It made me really quantify how I felt about a ski. Maybe a 1 to 10 scale could be better but the 1 to 5 scale meant that there was a lot of leeway in assigning a "3" (or any other number) - close was OK. Skis I liked clearly stood out with high number counts. The numbers were pretty close to my "feel" rankings. Of high value would be buoy count numbers as well.

The writeup from this year's test was a bit too fuzzy. With some numerical data, you could pick a winner in acceleration, stability, carryout, turns, etc and give a bit more useful decision making results. Using numbers to pick a winner overall IS flawed as different skiers have different strengths and weaknesses. Pointing out the strengths of a ski through a numerical ranking is useful. But numerical analysis should not be the prime focus of the tests.

John, your handling of the tests has been fantastic. While nothing is ever perfect, I haven't seen any serious flaws. I am planning to do trick ski tests and will use your tests as a model.

Eric

Link to comment
Share on other sites

  • Administrators

Eric,

One of the biggest problems is that skiers do jump to conclusions. If I have say in next years tests I will push for more detail but no numbers. I guess if my name is not on the tests and I get to ski on all the skis I will not care. (yea like that will happen)

 Goode  KD Skis ★ MasterCraft ★ PerfSki ★ Radar ★ Reflex ★ S Lines ★ Stokes

Drop a dime in the can

 

Link to comment
Share on other sites

John, One possible way to give readers more information is to publish the "raw comments" on this website. It would sure bring out the "independent" aspect of it. Also, testers could simply rank their top 3 with a few reasons why, yet not have a full numerical comparrisson as in last year's test.

 I think in this years reviews, the data is there in comments, but not enough room to publish in the mag.....none the less, it was a great test and quite independent.

 chris carter

 

 

Link to comment
Share on other sites

  • Administrators

For those who were not there this is inside baseball

Chris,

When I looked at publishing all the rawnotes: I saw things like the difference between the 67 & 68 (you know the ski). The 67 we had was the real 2008 ski and the 68 was a 2007 ski. Skiers noted the 68 as not so great and the 67 as really good. I know the 68 has been redone to be just like the 67 we testsed. I think the data on the 68 needs to be filtered out as it is no longer true.

I assume that you saw and heard the what the deal with that ski is. The last thing I want is for someone to read the raw notes and jump to a wrong conclusion.

I am reconsidering releasing the notes but will have to really look hard. It will have to be all or nothing and I need to make sure that nothing will be missleading.

 

 Goode  KD Skis ★ MasterCraft ★ PerfSki ★ Radar ★ Reflex ★ S Lines ★ Stokes

Drop a dime in the can

 

Link to comment
Share on other sites

  • Baller

This is not intended to slight the 2008 test team but I believe the group of skiers we assembled for the 2006 and 2007 test were critical to the success of the test.  These were individuals who were currently on the latest offerings from the manufacturers, who competed at the highest levels of our sport, were involved either directly or indirectly with manufacturers on ski design and set up and were highly skilled at ski tuning.  In addition, these skiers knew the cause and effect of changes in binding / fin adjustment and had the ability to compare each ski against a standard based on their vast experience.  THIS is what made the ski tests work and the "numbers" credible.   I would put the knowledge base of the 2006/2007 test teams we assembled up against any group in the world and that includes the manufacturer's reps themselves.  As an example, I've heard from several of the test team that the rep that Goode sent knew less about the ski set up than some of the testers.  His calipers were "pathetic" and his tool box looked like a carpenters.   Dave Goode sends his prototype skis to one of the 2006 test team members for feedback.  Those skis get passed around to several other members of the 2006/2007 test team.  These are individuals whom I would rely on for feedback and ski set up.   If you want "numbers", these are the guys to get them from.

Link to comment
Share on other sites

  • Administrators


I have reflected on some of the criticism and my attitude is: When you run something like this you have to define a plan and stick with it.



If I were in charge of the 09 test (and I will NOT) it would be an evolution of this years test and would still not include ranking or rating numbers. You have to weigh all the factors when you are publishing and I think a pursuit of fairness has to trump other factors.




I am more or less done with this topic.

 Goode  KD Skis ★ MasterCraft ★ PerfSki ★ Radar ★ Reflex ★ S Lines ★ Stokes

Drop a dime in the can

 

Link to comment
Share on other sites

John I hope you'll reconsider.

 

jdarwin - I don't know whether to be impressed, offended, or just disappointed. Lots of talent on the 06/07 team, that's a fact, very impressive. Problem is it doesn't represent the cross section of the skiing public that routinely get off their wallets and buy new equipment. Those people are in the -22 to -35 range. If I'm working hard on my -28 pass I need constructive comments from a tester who is at the same point in their development. Someone running into -39 telling me a ski is stable and forgiving presents an interpolation challenge for me. John assembled an excellect cross section of talent from -28 to -39. The example of using all those people who pass around DG's prototypes is equally impressive and exactly what John tried to avoid: ANY signs of brand bias. Again, I thought he had a great mix of people who ride different skis. Everyone I know that routinely clears -38 is getting their skis either free or deeply discounted through a rep, dealer or directly from the manufacturer. Ski companies are financially supported by folks in the -22 to -35 range who are inspired by people holding their products on a podium. We all need eachother. the -38/-39 folks aren't using the WSM ski test as much or as frequently as a -22 to -35 skier is to make a critical decision.

The only way this could have been better is if it were twice as many pages.

Bill

Link to comment
Share on other sites

  • Administrators


I do not mind the mostly honest conversation about the test but am a bit annoyed by people reading what I say here and then going elsewhere to comment. I wonder why I go though the trouble to have a forum.

 Goode  KD Skis ★ MasterCraft ★ PerfSki ★ Radar ★ Reflex ★ S Lines ★ Stokes

Drop a dime in the can

 

Link to comment
Share on other sites

  • Baller
Bill - I did not mean to imply that all of the 2006/2007 were "high-end" skiers.  Not my point.  In fact, we had several 28/32/35 off skiers among the group.  They were VERY representative of the ski buying public.  In fact, most of the testers purchase at least one new ski a year.  But ultimately, they were chosen for their expertise in ski set up and design.  The team was chosen for their analytical expertise in addition to their on-the-water skills.  The combination of these two assets is what made the tests successful and the association of numerical scoring possible.  I don't buy the argument that a 39-off skier can not relate critical data to a 22 off skier.  The basics are the same. As a consumer, I'm not looking for someone to tell me what to buy.  I'm interested in skilled individuals who can articulate the performance characteristics of various skis and allow me to reach my own conclusions.  That was what the tests were all about.  The main way the tests could have been better is by providing some critical scoring so those of us outside the loop could have some insight into the performance of the skis.  Twice as many pages may have accomplished that.
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...