15.11.2007 03:40
|
|
Beginner
|
Benchmarking Tool (released)
In honor of an amazing game (I've been playing my Special Edition)...
Benchmarking Tool for Crysis.
Executive Summary:
The tool provides a robust front end to benchmarking Crysis. Provides ability to queue up many runs and will provide detailed results for each test as well as an overall summary with accurate averages. The first and foremost goals were simplicity and reliability in the tool and in the benchmark scores. Works with retail and demo.
How To Use:
Simply queue up several different runs mixing settings, resolutions, antialiasing levels and click "Begin benchmark Run".
You can add a frame number where screenshots can be made on every frame for comparison. Use Negative numbers to make a screenshot on every N frame (ie: -399). Hit view button to to view them.
For advanced users you can add custom cvars or import a cfg file that will run globally on all tests.
There is no installer/uninstaller, just unzip and double click to use.
Notable Features:
- Run a variety of tests all at one time.
- Choose windowed, 32/64 bit, DX9/DX10 tests.
- Logs setting details for each test run plus provides a summary breakdown (with averages if 3 or more loops).
- Supports custom demos.
- Allows pause/resume by hitting spacebar, between runs.
- Auto-save all benchmark runs in Autosave folder.
- Allows Copy/Paste from results window.
- Save and import a queue of runs for ease of retesting.
- Allows image quality comparisons by taking a screenshot on each run at certain frame(s).
How It Works:
It doesn't change or alter any game files. Except it uses the same log file that the Crysis timedemo does to log the results, that is the only game file altered in any way, although it's just a benchmark log file. There are no batch files used, this is strictly application controlled. The tool will go through the queue one by one. Between test runs you will have an option to pause for 3 seconds (visible with countdown timer) just hit the spacebar to pause and it highlights the test it will run next if resumed. A Summary is provided at the end for reference and details are provided prior. If 3 or more demo loops are set for each run, then the summary will provide an average of all loops except loop 1. For averages the first test loop is ignored since it is cacheing assets on the first loop. If only 2 runs are selected, the averages shown are not 'overall' averages but 'last' average, meaning it will provide the average from loop 2, again ignoring loop 1 wherever possible. All Completed demo runs are saved in the autosave folder by date/time for convenience.
It will detect and notify you if you have any configuration files that may skew benchmark results. Uses same test settings as the batch files, so scores are completely 100% compatible.
The tool provides zero overhead during tests. There is no program logic operating in the tool while a testing run is going, it is only waiting for a signal back from the operating system that the demo loops are done, then it proceeds to start the next run. CPU spikes will be seen between test runs but this doesn't effect the runs at all, since it is essentially asleep during runs. In most cases the tool will find where the game or demo is installed, if you want to specify a path for some reason then go to File/Set Game Path (like if switching between full release and demo).
Limitations:
There are currently a few limitations with the first release that will be added to later.
- Low/Medium/High/VeryHigh global settings can be mixed and matched in the queue as desired but only one custom configuration (mixing individual quality settings) can be mixed into the queue. In other words many custom runs can be added to the queue but it will only use the same custom settings for each run. The custom quality benchmark run will be run with the quality settings that are currently selected at the time that the 'Begin Benchmark' button is pressed.
- For any advanced optional CVARS you add, they will be global to all tests and not on a per test run basis.
- Currently while many resolutions are provided, not all may be supported by your monitor.
- DX10 option will show up in Vista if you don't have DX10 hardware. Choosing DX10 will revert to using DX9, although the log output will show DX10 erroneously. I do not have directx API logic in the tool yet to differentiate HW.
- Vsynch currently disabled as there were issues with it working in demo, will be re-enabled in soon update.
Plans:
I plan to remove the current limitations listed above in a near future release. I also am planning advanced testing options to show video/RAM memory usage as well as per core CPU usage.
Will add DX API support for better resolution detection options.
I also plan to have an advanced tweaking section that I will add to the tool for people that want to test the performance of their tweaks.
Possible reports/graphs and exporting options depending on the community desire for it.
Download and Requirements:
Only tested on XP and Vista 32/64, the tool detects them appropriately.
There is no installer/uninstaller, just unzip and double click to use.
Requires .NET framework. You probably have it installed but if problems arise in running the tool it try these downloads from Microsoft:
x86 .NET framework redist
x64 .NET framework redist
You can download the tool here at these links:
Filefront version 1.05 (1MB)
Rapidshare version 1.05 (1MB)
Crysis Benchmark Tool @Crymod.com
Enjoy!
Change Log
Version 1.05 release 11-20-07
* Changed the way AA initializes for Vista and XP. AA settings work reliably in testing.
* Removed speech audio from playing with the default GPU/CPU benchmarks.
* Modified pause and stop, mostly cosmetic fixes.
* Vsync working and enabled for retail, automatically disabled for demo because doesn't work.
* Complete re-write of SP Demo and Retail detection and pathing. The tool will autodetect what version it is testing and if it's manually pointed to a different version it will detect it appropriately.
* Included build version of game being tested on form title.
* Fixed pointing to the right path for timedemos after a game path is manually changed.
* Added persistent data with a benchmarktool.ini to retain certain user settings between sessions (ie:clear log before run, tooltips checkbox)
* Added better attention grabber to the Add runs to queue button, if begin benchmark is pressed with an empty queue.
* Fixed extension filter issue with save results dialog.
* Reduced working set and private bytes usage of tool by several hundred percent.
* Minor formatting fix on summary under certain conditions.
* Fixed 64 bit testing for Vista 64. XP 64 still unsupported by tool (contact me for testing).
* Included a new benchmark called Assault_Harbor that is available in the timedmo selection drop down box. Instead of a good GPU or CPU test, this focus's on real gameplay from a level of the game with active AI, active physics and stressing particle graphics as well. Each run will be somewhat unique since AI and physics are active during test but is a good representation of real gameplay to test your settings against.
Vesion 1.04 release 11-16-07
* Disabled 64 bit benchmark because of issues.
Feedback would be appreciated to make it a better tool!
Ask questions and post your results here if you like.
Here is one of my benchmark runs:
Q6600 @ 3 GHZ
8800 GTS 640 OC (580/1456/975)
11/13/2007 7:41:55 AM - Vista 64 (DX10 game default)
Run #1- 1024x768 AA=No AA, 32 bit test, Quality: High ~~ Overall Average FPS: 45.23
Run #2- 1024x768 AA=4x, 32 bit test, Quality: High ~~ Overall Average FPS: 36.725
Run #3- 1280x1024 AA=No AA, 32 bit test, Quality: High ~~ Overall Average FPS: 32.65
Run #4- 1280x1024 AA=4x, 32 bit test, Quality: High ~~ Overall Average FPS: 25.98
Run #5- 1600x1200 AA=No AA, 32 bit test, Quality: High ~~ Overall Average FPS: 24.37
Run #6- 1600x1200 AA=4x, 32 bit test, Quality: High ~~ Overall Average FPS: 15.96
Run #7- 1024x768 AA=No AA, 32 bit test, Quality: VeryHigh ~~ Overall Average FPS: 25.07
Run #8- 1280x1024 AA=No AA, 32 bit test, Quality: VeryHigh ~~ Overall Average FPS: 17.465
Run #9- 1600x1200 AA=No AA, 32 bit test, Quality: VeryHigh ~~ Overall Average FPS: 12.705
This post has been edited 6 time(s), it was last edited by Cry-Alex: 23.11.2007 14:53.
|
|
|
|
|
15.11.2007 04:44
|
|
Trainee
|
RE: Benchmarking Tool
works fine ! very usefoul !
thanks m8 !! 
__________________
"So whatever you do, don't be bored.
This is absolutely the most exciting time
we could have possibly hoped to be alive."
|
|
|
|
|
15.11.2007 05:37
|
|
Experienced Modder
|
DAMN! Nice work man, I'll see about getting this on the news 
__________________
Asus M3A AM2+ 770 - AMD Phenom X4 9750 - 4GB DDR2 - eVGA 8800GT 512MiB
|
|
|
|
|
15.11.2007 15:04
|
|
Trainee
|
Woah great stuff! I'll give it a shot this evening!
Are you the same MadBoris of the Supreme Commander forums by the way?
__________________
|
|
|
|
|
15.11.2007 22:17
|
|
Beginner
Thread Starter  |
Same MadBoris, yep that'd be me.
I had went through a few testing revisions with testers, it did take some work to put it together.
I look forward to any feedback good/bad/indifferent for future updates.
Personally, I am not satisfied with the output in simple text format and the summary the way it is, so I will likely work on that some more. But feature lock had to happen a while ago to make it out prior to game release.
|
|
|
|
|
15.11.2007 23:48
|
|
Beginner
|
lol I had the same thoughts... Yep it happened that I also noticed you might be the same MadBoris from the supcom forums. I think your core Maximizer was genious 
|
|
|
|
|
16.11.2007 17:27
|
|
Beginner
Thread Starter |
Bad news for 64 bit users...
Currently the 64 bit option in the benchmark tool is not working due to Securoms measures.
Securom uses a strange method that was not around in the SP demo, with one exe being a securom launcher and the other one the actual 64 bit binary being spawned with a different name and pid that actually runs the game. So the benchmark tool does not understand that strange spawning behaviour of new processes.
Furthermore due to this Securom behaviour the x64 stability seems to have been compromised somewhat in my brief testing. Read '64 bit may be buggy probably because of this securom launcher'. I'm not one to get on the securom bashing bandwagon but the 64 bit binary in the demo showed none of the behaviour I am experiencing without the tool.
Until I figure out a good way to workaround this, the 64 bit option should not be used.
I'm working on it and hopefully I have a new version in the next couple hours, or I'll just grey out the option. It really may take Crytek getting involved down the road with the way Securom is spawning the process to help in the 64 bit version of the game.
|
|
|
|
|
17.11.2007 00:08
|
|
Beginner
|
Hi
Hello all.
Madrid Excellent program performs its tasks. Mint to make comparisons.
Question Where my post result?
result (Crysis SP Demo)
-------------------------
AMD Athlon 64 3800+
Ram 3Gb DD2 800 Dual Channel (Corsair TWIN2X2048-6400C4)
ATI Radeon X1950XT - Ram 256MB GDDR3-256bit, Core Clock: 625 MHz, Mem Speed: 1.8 GHz (driver 7.9)
-------------------------
16/11/2007 06:02:24 p.m. -
Run #1- DX9 1024x768 AA=No AA, 32 bit test, Quality: High ~~ Overall Average FPS: 23.49
Run #2- DX9 1024x768 AA=4x, 32 bit test, Quality: High ~~ Overall Average FPS: 23.58
Run #3- DX9 1280x1024 AA=No AA, 32 bit test, Quality: High ~~ Overall Average FPS: 17.375
Run #4- DX9 1280x1024 AA=4x, 32 bit test, Quality: High ~~ Overall Average FPS: 17.595
Run #5- DX9 1680x1050 AA=No AA, 32 bit test, Quality: High ~~ Overall Average FPS: 12.27
Run #6- DX9 1680x1050 AA=2x, 32 bit test, Quality: High ~~ Overall Average FPS: 12.385
__________________
My English is bad sorry
Nodo
This post has been edited 5 time(s), it was last edited by Nodo: 01.12.2007 03:37.
|
|
|
|
|
17.11.2007 03:00
|
|
Beginner
|
Fantastic tool you have here already,
Much appreciated. keep up the great work
|
|
|
|
|
17.11.2007 03:58
|
|
Trainee
|
thank you again !
|
|
|
|
|
21.11.2007 05:31
|
|
Beginner
Thread Starter |
RE: thank you again !
NEW UPDATE 1.05 11-20-07
This is a highly recommended update as it fixes certain issues with AA testing, enables vsync, x64, better demo/retail detection and switching between. Also includes the first 3rd party timedemo for realworld gameplay testing. See changelog in first post for details.
Thanks for continuing to post any issues or feedback. Barring any minor issues that need quick correction, i'm planning the next update to take a while with focus on new features.
This post has been edited 1 time(s), it was last edited by MadBoris: 21.11.2007 05:32.
|
|
|
|
|
21.11.2007 06:09
|
|
Trainee
|
RE: thank you again !
| quote: |
| includes the first 3rd party timedemo for realworld gameplay testing. |
Could you please explain how to record/playback demos (would be more helpful if in this thread)?
You're the only person I've come across who's had success at doing so.
Anyway, great tool; very useful!
This post has been edited 4 time(s), it was last edited by Martz: 21.11.2007 06:11.
|
|
|
|
|
23.11.2007 17:16
|
|
Beginner
|
no widescreen support??
resolution 1440x900 is missing 
|
|
|
|
|
24.11.2007 10:56
|
|
Beginner
|
Great tool , but having problems here
running on Vista 32 Bit , the program never completes , and crashes with
"Crysis has stopped working "
This means I cant leave it unattended to do more than one run .
The result appears to be there if I select "show last runs" but it never completes without the crash.
any thoughts? Nvidia 8800 ultra with latest beta drivers btw
EDIT :-)
got worse now , Vista responds with "Crysis has stopped working" after 93% of the loading screen during the first run.Always at 93% by the way
In other words the "loading in progress" screen of Crysis.
The crysis screen "whites out" as this happens.
the game runs VERY well on my system, and all crysis benchmarking tools provided with the game run without error..
any ideas?
cheers
Padman
This post has been edited 1 time(s), it was last edited by vsukpadman: 24.11.2007 16:38.
|
|
|
|
|
26.11.2007 08:59
|
|
Beginner
|
Nice thanks
|
|
|
|
|
27.11.2007 16:08
|
|
Beginner
|
Thanks MadBoris
i've been waiting for the fixed version (x64 user)
thanks for the tip on the custom resolutions too
| quote: |
Originally posted by vsukpadman
Great tool , but having problems here
running on Vista 32 Bit , the program never completes , and crashes with
"Crysis has stopped working "
This means I cant leave it unattended to do more than one run .
The result appears to be there if I select "show last runs" but it never completes without the crash.
any thoughts? Nvidia 8800 ultra with latest beta drivers btw
EDIT :-)
got worse now , Vista responds with "Crysis has stopped working" after 93% of the loading screen during the first run.Always at 93% by the way
In other words the "loading in progress" screen of Crysis.
The crysis screen "whites out" as this happens.
the game runs VERY well on my system, and all crysis benchmarking tools provided with the game run without error..
any ideas?
cheers
Padman |
have you modified any of the original files?
i had a similar problem the other day...
until i remembered i had changed the "island" level, after i replaced the original files again, the benchmark worked
This post has been edited 1 time(s), it was last edited by Pyr0: 27.11.2007 16:10.
|
|
|
|
|
27.11.2007 16:10
|
|
Beginner
|
No..
all files are as standard..very odd indeed!
It is the direct download version from EA , and not the dvd install if thats of any significance!
Padman
This post has been edited 1 time(s), it was last edited by vsukpadman: 27.11.2007 16:12.
|
|
|
|
|
27.11.2007 18:20
|
|
Beginner
|
game.cfg
Hi, i am from germany, so i'm sorry for errors
My Problem: The Crysis Benchmark Tool (1.0.5) shows at start the message:
"App warning - game.cfg not found. Can still proceed but current game settings cannot be set in app."
I know that the game.cfg is in my german WinXP at
"c:\Dokumente und Einstellungen\Eigene Dateien\Crysis"
in english:
C:\Documents and Settings\My Files\Crysis"
I have the Full Verision of Crysis, but why i get that message?
Can anybody help me in this?
Tank you very well!
h0m3r
|
|
|
|
|
28.11.2007 11:13
|
|
Beginner
|
RE: game.cfg
Similar issue as Visukpadman.
With a clean install of Crysis on Vista32 meanly I am not able to finish a cue in unattended mode.
-When a loop finishes, I find a steady picture of last frame with the Windows arrow pointer that can be moved around the image (not the big green arrow pointer of Crysis).
To start second item of the cue, I have to press ESC key.
- if I select the Vsync option, I have very fast rendering but with a lot of frames skipped.
- if I chose the Assault demo template and I set to run at hours time different from 5. 6 and 7 (for example at noon (12) or at sunset (17 or 18), the Crysis program starts to load but at end crashes with a Windows message. At 5-6-7 hour time, it works perfectly.
- it seems there is not any way to stop the benchmark during the loops: if I press the ESC key the picture freezes and a big green arrow appears. To exit I have to manually ALT-TAB and close Crysis.
The first issue is surely the most annoying as you can not perform different benches automatically: between any line of cue you have to press ESC key to start following one.
I am available to test new versions.
Anyway, good job MadBoris
P.S. I suggest to put an hint on the label "Time of Day" as it is not selfexplanatory.
This post has been edited 3 time(s), it was last edited by Brama: 28.11.2007 11:22.
|
|
|
|
|
|
|
|
|
|
