PDA

View Full Version : Playtest Spreadsheet


GreyOwl
February 27th, 2008, 05:15 PM
Sign-up Sheet
Raelin the Kyrie Warrior (SotM)

1-on-1 Match #1: vs. Kaemon Awa (GreyOwl) - COMPLETE
1-on-1 Match #2: vs. Krug (GreyOwl) - COMPLETE
1-on-1 Match #3:
1-on-1 Match #4:
1-on-1 Match #5:

1-on-squad Match #1: vs. Nakita Agents (GreyOwl) - COMPLETE
1-on-squad Match #2: vs. Tagawa Samurai (GreyOwl) - COMPLETE
1-on-squad Match #3: vs. The Einar Imperium (allskulls) - COMPLETE
1-on-squad Match #4:
1-on-squad Match #5:

team-on-team Match #1: Raelin (SotM), Sgt. Drak (SotM),3x 4th Massachusetts vs. Kee-Mo-Shi, Ne-Gok-Sa, Isamu, 2x Marro Stingers, 3x Marro Drones (GreyOwl, ZBeeblebrox) - COMPLETE
team-on-team Match #2: Raelin (SotM), 2x Deathreavers, 5x Marro Stingers vs. Laglor, Krav Maga Agents, Major Q9, Agent Carr, Isamu (ZBeeblebrox) - PENDING
team-on-team Match #3:
team-on-team Match #4:
team-on-team Match #5:

Captain America

1-on-1 Match #1: vs. Jotun (IAmBatman) - PENDING
1-on-1 Match #2:
1-on-1 Match #3:
1-on-1 Match #4:
1-on-1 Match #5:

1-on-squad Match #1:
1-on-squad Match #2:
1-on-squad Match #3:
1-on-squad Match #4:
1-on-squad Match #5:

team-on-team Match #1:
team-on-team Match #2:
team-on-team Match #3:
team-on-team Match #4:
team-on-team Match #5:


Krav Maga Agents

1-on-1 Match #1:
1-on-1 Match #2:
1-on-1 Match #3:
1-on-1 Match #4:
1-on-1 Match #5:

1-on-squad Match #1:
1-on-squad Match #2:
1-on-squad Match #3:
1-on-squad Match #4:
1-on-squad Match #5:

team-on-team Match #1: Raelin (SotM), 2x Deathreavers, 5x Marro Stingers vs. Laglor, Krav Maga Agents, Major Q9, Agent Carr, Isamu (ZBeeblebrox) - PENDING
team-on-team Match #2:
team-on-team Match #3:
team-on-team Match #4:
team-on-team Match #5:


The playtest spreadsheet that allskulls and I have been working on is mostly done. The only thing remaining at this point is tuning and calibrating the formulas, and to do that we need real data. That's where you guys come in! :P

Using the attached scorecard, we need data from playtests of official figures (not customs) so we can calibrate everything. There are 3 types of matches: 1-on-1, 1-on-squad, and team-on-team. For each one, we need 5 different matchups, with 10 games each. So that works out to 150 games total, which is obviously too much for one person to do alone. Volunteers? :) :)

The official figures we need tested number around 10-15, due to the need to test different types of figures. But to start with, we need to do:

Raelin the Kyrie Warrior (SotM version)
Captain America
Krav Maga Agents

If people could "sign-up" for what tests they'd like to do and then send me the data it would help tremendously.

For Raelin, some of the test are alredy complete:
1-on-1 Matches
vs. Kaemon Awa - all 10 games complete
vs. Krug - all 10 games complete
need 3 more matches for 1-on-1

1-on-Squad
vs. Nakita Agents - all 10 games complete
vs. Tagawa Samurai - all 10 games complete
vs. The Einar Imperium - all 10 games complete
need 2 more matches for 1-on-Squad

Team-on-Team
Raelin (SotM), Sgt. Drake (SotM), 3x 4th Massachusetts vs. Kee-Mo-Shi, Ne-Gok-Sa, Isamu, 2x Marro Stingers, 3x Marro Drones - 6 games complete, 4 more needed
need 4 more matches for Team-on-Team

for Captain America and Krav Maga Agents, nothing has been done yet.


Instructions
Download and print out the PDF file of the scoresheet: PlaytestScoresheet.pdf (http://www.hiddendragon.com/hs_customs/PlaytestScoresheet.pdf)

Fill it out as you play, then PM me the results.

The scoresheet looks like this:
http://www.hiddendragon.com/hs_customs/PlaytestScoresheet.jpg

Most of it is pretty self-explanatory, but here are some guidelines:

1. Play against figures/teams that are as close as possible in point cost. However, the spreadsheet will scale if they aren't identical. This only works within a limited range, so don't stray too far. For example, Raelin is 120 points. For opponents, try to stay in the 100-140 point range. Even 90-150 might be good. But Raelin vs. Hulk wouldn't give useful results. :)

2. Maps - only use official maps or Battlefields of Valhalla certified maps. Keeps the maps normal size - nothing tiny and nothing huge. Use maps that are "fair" (no long climbs if the test figure can't fly, etc.). The goal is for the test figure to get plenty of play, not just sit on the sidelines doing nothing.

3. Rounds - total # of round the game lasted. Partial rounds count as 1. When the test figure is destroyed, end the game. No need to go on further.

4. Attack Delivered - the # of times the test figure attempted to attack (via attack dice or d20 ability). Count all attempts, so "Double Attack" would count as 2.

5. Wounds Inflicted - total # of wounds markers inflicted.

6. Kill Points - # of kill points delivered by the test figure ( (cost/life)*(# wounds) ). For 1-on-1 and 1-on-squad, no need to fill this out as the spreadsheet can auto-calculate it. For team-on-team, you'll have to figure it out since the spreadsheet can't know which figure(s) received the wounds.

7. # attacks received - count all attempts to attack the test figure (via attack dice or d20)

8. # wounds received - total # of wounds received by the test figure. Obviously, can't exceed the figure's life.

9. Assists - these can be many things. Basically, anytime the test figure helps another figure. Raelin's aura is a great example. When rolling the extra die for defense, just count how many times that extra die was rolled, regardless of the outcome. In Captain America's case, it works similarly. Each extra die rolled for attack or defense counts as 1 assits. For d20 boosts, again the same thing - just count how many times you "boosted" a d20. Other non-die based assists are things like Carry. Everytime you carry a figure, count it as 1 assist.

10. As questions come up for other things, please ask rather than guessing. If you can't ask because you're in the middle of a game, try to records as much information as possible so we can have the data we need later. There will always be weird situations when you get into abilities like bonding and swapping control of figures.


That's it! As people (hopefully) volunteer, I'll keep a sign-up sheet in this first post of who will be testing what, so we don't duplicate effort. Thanks in advance for all the help! :) :) :)

allskulls
February 27th, 2008, 06:03 PM
Let me just say that the suggested costs we are getting from the spreadsheet are looking really good so far.

Also, hopefully there's classic custom guys out there that want to help too. THis is designed to help custom costing across the board, not just Marvel.

GreyOwl
February 27th, 2008, 06:22 PM
I updated step 9 above regarding how to collect data for assists. So if you already read it, please re-read it. If you haven't yet, then you're good. :)

hi1hi1hi1hi1
February 27th, 2008, 08:26 PM
Wow! This is incredible. Great job to both of you. Tomorrow is my brother's birthday, but this weekend I can maybe get some testing in later.

Just one question. How do you plan on working this for customs? Not only are therenot many squads in the point values many customs float around, but 150 tests is a huge amount for one TNT trial.

allskulls
February 27th, 2008, 08:50 PM
Just one question. How do you plan on working this for customs? Not only are therenot many squads in the point values many customs float around, but 150 tests is a huge amount for one TNT trial.
Using multiple squads if needed should not be a problem for the 1-on-squad test.
150 does seem daunting but with enough testers it is easily broken up. The 1-on-1 and 1-on-squad tests go rather quickly. The team tests is what will take time but with preset teams, running games back to back shouldn't be too bad since there's not a lot of set up between games. The key is to get many different testers on these. The team games are the funnest so I'm hoping it will be easy to get players to run those. We may even be able to cut down on the games required to play for these.

GreyOwl
February 27th, 2008, 09:45 PM
For one-on-one games, I've done 10 games in under an hour, depending on the figures. The reason there are so many is that there is wild variation in suggested cost with fewer games. With a single one-on-one matchup, the cost doesn't stabilize until 8-10 games, and the overall one doesn't stabilize until 4-5 different matchups. 3-4 individual games sometimes seem like they're giving consistent results, but that's not always the case. But just like allskulls said, this would have to be done with a lot of testers.

ZBeeblebrox
February 27th, 2008, 10:25 PM
Wow this is an awesome idea! I wish I had more time to help out with this endevor. I may be able to get a game or two in each week, that is if i put other custom playtesting on hold. Which I think is worth it, to get this idea up and running.

Thanks to all who have been the brain trust behins this.

whitestuff
February 28th, 2008, 09:03 AM
Very nice idea lads. :thumbsup:

Are the 1-on-1 matches still played on a full map?

GreyOwl
February 28th, 2008, 10:04 AM
Very nice idea lads. :thumbsup:

Are the 1-on-1 matches still played on a full map?

Yes, but I find a very small map works for those. You need a reasonable map to take advantage of abilities like Flying.

allskulls
February 28th, 2008, 10:57 AM
Very nice idea lads. :thumbsup:

Are the 1-on-1 matches still played on a full map?

Yes, but I find a very small map works for those. You need a reasonable map to take advantage of abilities like Flying.
Yeah, the key for 1-on-1 maps is to ensure attacking will happen by OM 2. All the 1-on-1's or 1-on-squads I have ran, there was attacking happening by at least OM 2 and sometimes OM 1.

IAmBatman
February 28th, 2008, 11:02 PM
So, are you guys handing out assignments, or are we supposed to volunteer or what? I'd like to help, since I have a big interest in this (though I'm not sure how soon I'll be able to get in any tests).

GreyOwl
February 28th, 2008, 11:24 PM
The plan is to have volunteers. Unless you want us to start drafting people? :)

IAmBatman
February 29th, 2008, 12:40 AM
Well, I'm volunteering, but I was wondering if you want me to pick out my own stuff, or if you want to assign stuff to volunteers?

GreyOwl
February 29th, 2008, 08:12 AM
Oh, you can pick your own stuff. I'd prefer to finish up all of Raelin's testing first, but since we need all of them anyway it won't hurt anything if decided to start with something else.

GreyOwl
February 29th, 2008, 09:53 AM
Since we didn't post the actual spreadsheet, I thought I'd put up an explanation of how it works, without getting into the details of all the formulas.

Setup
The first tab has setup information that you define about the figure you want to test. This includes:

figure's name
stated cost (the best guess to start with)
life
names of any assist-related abilities
rarity (unique vs. common)
whether it has any bonding abilities.


If there are bonding abilities, there is a Bonding Calculator that figures out what portion of the cost is due to that. This can be calculated statically because the worth of bonding depends on the figure(s) you bond with and not the figure with the bonding ability. This means that any figures that can be activated via bonding must already be tested and had their costs established. The Bonding Calculator requires the following information:


# of Bondable cards (how many possible cards can the figure bond with)
Average cost of bondable cards
Bonding Type - take an additional turn or instead of this figure's turn
Bonding Scope - move, attack, or both
# of Bonds per turn - the number of cards per turn that can be activated


Once these are filled in, a static number is generated to be added to the suggested cost.


Ratings
After setup is done, the only other things that need to be filled in are the test data from the scoresheet above. Based on the results of playtesting, a figure is rated in 4 basic areas, called Efficiencies, which are displayed as a percentage:


Attack Efficiency - how good the figure is at attacking
Defend Efficiency - how good the figure is at defending
Survival Efficiency - how good the figure is at surviving (lasting to the end of the game)
Assists Efficiency - how good the figure is at supporting other figures


The idea behind these efficiencies is that almost all special abilities will have their effects show up in them, so there won't be any reason to model each ability separately. Only a few kinds of abilities require special consideration, and those are outlined in the testing guidelines (another tab in the spreadsheet).

These four efficiencies are combined to form a Total Efficiency. The Total Efficiency is then compared to the stated cost of the figure to create a Performance Efficiency. This is the most accurate rating of how well the figure performs for their cost. The Performance and Total Efficiencies are compared to the stated cost to come up with a Suggested Cost.

This process is done for all 10 games in a match and all efficiencies are displayed per match. Then all matches are averaged together to show the efficiency per type of match (1-on-1, 1-on-squad, team-on-team). Then all of those are combined to show overall ratings for the figure.

Now for some details about the efficiencies and what is taken into account in them...

Details of Efficiencies


Attack Efficiency - accounts for wounds delivered per attack and kill points delivered per stated cost of figure
Defend Efficiency - accounts for wounds received per attack received
Survival Efficiency - accounts for survival points per game and percentage of games won
Assists Efficiency - accounts for total assists per round



Please ask if there are any questions! We believe we accounted for all important aspects, but if we seem to have left something out let us know.

IAmBatman
February 29th, 2008, 10:57 AM
So, if I want to volunteer to do some one on one tests for Captain America, would it be against those figures listed in the first post (Kaemon Awa, Krug, etc)? And ten times against each?

GreyOwl
February 29th, 2008, 11:01 AM
Not necessarily against those figures. Try to pick figures as close as possible to Captain America in cost, and a variety of figures (as far as abilities). And yes, 10 games for each.

IAmBatman
February 29th, 2008, 11:16 AM
Ah, OK, I was confused on that part - I didn't see that it was the new version of Raelin (thought she was battling a bit out of her league!). So for one on one's, I'm probably looking more at Jotun, Charos, and Braxas ... maybe Iron Man?

edit: I forgot about TKN. He'd probably make a better choice to round out the Jotun/Charos selections.

GreyOwl
February 29th, 2008, 01:11 PM
Any and all of those would be fine. We need 5 different one-on-one matchups anyway. Just let me know which one you want to start with and I'll "sign you up". :)

IAmBatman
February 29th, 2008, 01:26 PM
Guess I'll volunteer for the 10 tests of Captain America against Jotun to start. I don't play Jotun nearly enough.

GreyOwl
February 29th, 2008, 02:19 PM
Guess I'll volunteer for the 10 tests of Captain America against Jotun to start. I don't play Jotun nearly enough.

Thanks! :) Just a reminder, please make sure to do the tests on an official or BoV map.

I've updated the first post to show the sign-up sheet.

UranusPChicago
February 29th, 2008, 02:59 PM
Just kind of thinking out loud here, but isn't Captain America along with Major Q9 thought of as great deals for their cost (ie slightly undercosted)?

If that is the case, why would we be using Captain America as a figure with which to base playtesting off of? Would it not, as a result, slightly under value all the figures that are playtested against the resultant dataset?

I personally have not tested Captain America enough to say one way or another if he is slightly undervalued. At this point, I am just going on what I have seen in some other people's views.

If it is not the case, then please continue on with the regularly scheduled program...

IAmBatman
February 29th, 2008, 04:10 PM
Guess I'll volunteer for the 10 tests of Captain America against Jotun to start. I don't play Jotun nearly enough.

Thanks! :) Just a reminder, please make sure to do the tests on an official or BoV map.

I've updated the first post to show the sign-up sheet.

It might be worth it to put a link to those maps up here ...

GreyOwl
February 29th, 2008, 04:56 PM
Good idea. Is there a link somewhere to official maps? I normally just look them up in the rulebook...

hi1hi1hi1hi1
February 29th, 2008, 08:40 PM
I think they are on the Heroscape.com website under battlefields. Hold on a sec...

Linkage: Official Scenarios (Maps included :wink: ) (http://www.hasbro.com/games/kid-games/heroscape/default.cfm?page=Resources/Scenarios)

ZBeeblebrox
March 1st, 2008, 05:40 PM
If no one else is tesing this:

team-on-team Match #1: Ralein (Sotm), Sgt. Drak (SotM),3x 4th Massachusetts vs. Kee-Mo-Shi, Ne-Gok-Sa, Isamu, 2x Marro Stingers, 3x Marro Drones - PENDING


I will volunteer.

GreyOwl
March 2nd, 2008, 08:12 PM
If no one else is tesing this:

team-on-team Match #1: Ralein (Sotm), Sgt. Drak (SotM),3x 4th Massachusetts vs. Kee-Mo-Shi, Ne-Gok-Sa, Isamu, 2x Marro Stingers, 3x Marro Drones - PENDING


I will volunteer.

Okay, I'll sign you up. 6 games of this have already been completed, so we only need 4 more. And please make sure to use an official or BoV map, and tell me which one it was.

GreyOwl
March 2nd, 2008, 08:15 PM
Note that step 9 (Assists) in the intial post was updated again. This new method will not only give more accurate results, but will be easier to track while playtesting as well.

ZBeeblebrox
March 3rd, 2008, 12:27 PM
If no one else is tesing this:

team-on-team Match #1: Ralein (Sotm), Sgt. Drak (SotM),3x 4th Massachusetts vs. Kee-Mo-Shi, Ne-Gok-Sa, Isamu, 2x Marro Stingers, 3x Marro Drones - PENDING


I will volunteer.

Okay, I'll sign you up. 6 games of this have already been completed, so we only need 4 more. And please make sure to use an official or BoV map, and tell me which one it was.

Great GreyOwl! I've already got two games played, so I'll be done soon.
i've been playing on the BOV map : Embattled Fen by GeameBear.

IAmBatman
March 7th, 2008, 12:05 AM
Sorry, I know I've been aloof for a while, but I'll be trying to get in those Cap tests this weekend.

ZBeeblebrox
March 8th, 2008, 11:05 PM
If no one else is tesing this:

team-on-team Match #1: Ralein (Sotm), Sgt. Drak (SotM),3x 4th Massachusetts vs. Kee-Mo-Shi, Ne-Gok-Sa, Isamu, 2x Marro Stingers, 3x Marro Drones - PENDING


I will volunteer.

Okay, I'll sign you up. 6 games of this have already been completed, so we only need 4 more. And please make sure to use an official or BoV map, and tell me which one it was.

Finished the last four matches...sent a PM.

GreyOwl
March 11th, 2008, 05:06 PM
Thank you! I've added the data to the spread sheet. You recorded every attempt at the assist, right? (the # of times you rolled the extra die, regardless of the outcome)? Just asking because your numbers for that were lower than in my playtests by about half. That could be due to the different maps, too.

So any other volunteers? Lot's of blanks in the signup sheet! :) I'd really like to finish all the tests for Raelin so we can at least begin calibrating the spreadsheet.

ZBeeblebrox
March 13th, 2008, 10:49 PM
Thank you! I've added the data to the spread sheet. You recorded every attempt at the assist, right? (the # of times you rolled the extra die, regardless of the outcome)? Just asking because your numbers for that were lower than in my playtests by about half. That could be due to the different maps, too.

So any other volunteers? Lot's of blanks in the signup sheet! :) I'd really like to finish all the tests for Raelin so we can at least begin calibrating the spreadsheet.

Sorry GreyOwl...I only recorded each time a shield was rolled regardless of the outcome...sorry I did not see your change to the original rules.

I printed a copy to keep close by, so I would not forget anything...and still messed it up.

GreyOwl
March 13th, 2008, 11:43 PM
Did you happen to write down or do you remember the total number of attempts? If not, I can still use the rest of your data unless you'd rather replay the games (or if someone else wants to)?

I think the rules are set for now, though. Sorry about the churn and the confusion around those. :?

ZBeeblebrox
March 15th, 2008, 12:11 AM
Did you happen to write down or do you remember the total number of attempts? If not, I can still use the rest of your data unless you'd rather replay the games (or if someone else wants to)?

I think the rules are set for now, though. Sorry about the churn and the confusion around those. :?

It would have been almost evry attack, because each game ended upo with the Marro charging at the Soldiers and Raelin stayed in range of every figure most of the time. I did not record all the attcks on other figures.

If you really need me to replay these I will, but it will take some time...

GreyOwl
March 15th, 2008, 12:16 AM
I think I can use the data as is. If you have time to replay, I'd rather you play some of the other ones needed instead, like team-on-team for Raelin with a different team on both sides. :)

GreyOwl
March 17th, 2008, 12:40 PM
Updated the first post with new sign-ups.

IAmBatman
March 24th, 2008, 11:54 PM
I apologize for my absence here and elsewhere in this forum. I've been away on Spring break. Hopefully I'll get back to this soon.

Flame Gryphon
April 16th, 2008, 12:12 AM
What new figures do you want me to test?

UranusPChicago
April 16th, 2008, 09:35 AM
What new figures do you want me to test?

You might also look at some of the TNT trials going on and give us some assistance there. We have definitely hit a wall in those. Maybe some new blood will get things moving again.

GreyOwl
April 16th, 2008, 02:41 PM
What new figures do you want me to test?


Just look at the first post and sign up for whatever you'd like. I'd like to finish off all of Raelin's tests first, but it's not a requirement.

Flame Gryphon
April 16th, 2008, 06:15 PM
What new figures do you want me to test?


Just look at the first post and sign up for whatever you'd like. I'd like to finish off all of Raelin's tests first, but it's not a requirement.

I'm only testing new figures for TNT, that's why I made that thread.

Flame Gryphon
April 16th, 2008, 06:16 PM
What new figures do you want me to test?

You might also look at some of the TNT trials going on and give us some assistance there. We have definitely hit a wall in those. Maybe some new blood will get things moving again.

Thanks.