About Store Forum Documentation Contact



Post Reply 
Random Performance Issues
Author Message
fatcoder Offline
Member

Post: #1
Random Performance Issues
I'm not sure what is going on, but I have spent the last 6 hours trying to figure out why I'm experiencing what appears to be completely random performance problems. Normally, my scene will run at up to 300 fps, but every now and then (completely random), while I'm moving around, it will drop right down to as low as 18 fps (for a random amount of time, up to 20 seconds) and the game will shudder like it is trying to load something in the background for no reason at all.

I have pretty much tried everything, including stopping unnecessary background processes and running in Release mode. I have also created a very small scene specifically to test this problem, so there shouldn't be any world streaming going on. Scene is 4x4 areas at 64 units/area. It is just a basic terrain (with a few materials applied to it) and some grass, trees and rocks scattered around (from the included Data folder in Esenthel). I have also tried adjusting things like the world mode, background loading and the active area size.

It is not a rendering issue as the video card load will noticeably drop off when it happens, which immediately tells me that it is a CPU bottleneck. Something in Esenthel is randomly consuming a huge amount of CPU cycles on a blocking thread, which is stalling the GPU.

I've noticed that the issue becomes noticeable with objects in the scene and when using vehicle physics. For example, to test the problem, I created a basic scene (as described above). I then created a simple application to load this world and drop a vehicle on it (from the vehicle tutorial). The game will then slow down at random intervals for a random amount of time while I drive around the small world. I can only think of two things that may be causing or contributing to the issue.

1. Esenthel is performing some sort of serious background loading, most likely for the objects in the scene. Perhaps for LOD or something like that. However I find this hard to believe as I've created a very small scene that should fit entirely within the active area. I can play the RPG2 demo without any of these slow down issues.

2. It has something to do with the vehicle physics. Perhaps the wheel colliders are getting stuck in some sort of collision loop with the terrain. I think physics is the most like culprit.

Unfortunately, without the source code, I've come to the end of what I can do. This issue makes vehicle physics pretty much impossible for me to use as it is just not smooth enough to be playable. I would really love to hear from anyone else that has successfully used vehicle physics without any issues like this. Esenthel, if and when you look at this issue, please let me know if there is anything else I can provide to help reproduce the problem.

P.S. Sorry for writing a novel... wink
12-13-2010 04:48 PM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #2
RE: Random Performance Issues
Hi,

You can measure the performances in Physics.simulationTime, Time.stateupdate, Time.stateDraw, Game::World.objUpdateTime (names may be slightly different - im saying from my memory)

Does the same scene without the vehicle runs fine?
12-13-2010 05:19 PM
Find all posts by this user Quote this message in a reply
joacorock Offline
Member

Post: #3
RE: Random Performance Issues
Check this out: http://www.esenthel.com/community/showth...p?tid=2594

I got my problem solved now, there is a video, tell me if you have the same problem
12-13-2010 05:28 PM
Visit this user's website Find all posts by this user Quote this message in a reply
fatcoder Offline
Member

Post: #4
RE: Random Performance Issues
@Esenthel, I'll try those timers out to measure everything and post some results. Didn't know they existed. smile

Also, the scene does seem to run better without the vehicle in it. If I fly the camera around manually, there doesn't seem to be any problem. I also tried disabling hardware physics, but that didn't appear to make much difference. The problem is that every time I run it, I get different results. It is completely random so it is really hard to determine what has an effect on it and what doesn't.

@joacorock, I'm not having the same issue as shown in your video. I'm already updating the world and camera in the correct order. However, my problem does sound similar to what you describe in the last couple of posts of your thread. Not exactly the same, but similar and perhaps related in some way. Did you solve that particular problem?
12-14-2010 01:26 AM
Find all posts by this user Quote this message in a reply
fatcoder Offline
Member

Post: #5
RE: Random Performance Issues
OK, I have run some more tests using the timers and I have some interesting results. It turns out that the problem lies somewhere in the drawing as it is Time.stateDraw that is reporting the massive fluctuations. The other three timers all just sit at zero. I've never seen them move.

To test the issue further, I am using RT_SIMPLE with just the default settings. I've switched off everything, such as the sky, water, etc...

When the game first starts, Time.stateDraw will report 0.001 and I'm sitting on over 1000 fps. Then while I'm moving around, Time.stateDraw will fluctuate randomly and sometimes jump as high as 0.01, which is a 90% performance hit.

Another interesting point is that even if I remove the vehicle I'm still getting the problem now. I know I said before that I wasn't getting the problem if I removed the vehicle, but now it seems I am. This is how random the problem can be.

Despite the issue being reported from the drawing, I still think it is a CPU related problem, possibly to do with terrain or object LOD or some other CPU task that relates to drawing (hard to say though without the source). This is because I notice that the video card load seems to drop off when the problem happens, like it is stalling while waiting for the CPU to do something. Perhaps
12-14-2010 02:47 AM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #6
RE: Random Performance Issues
Does the problem occur also in SDK tutorials, or in World Editor?
Or is it only in your project?
maybe you have incorrect codes?
12-14-2010 02:17 PM
Find all posts by this user Quote this message in a reply
fatcoder Offline
Member

Post: #7
RE: Random Performance Issues
After many more countless hours, I believe I have narrowed the problem down to the terrain texturing.

If I create a small world (4x4 @ 64/area) and use no more than two textures on the terrain, then the problem pretty much disappears. As soon as I add a third texture to the terrain, the problem begins to creep in very slightly. Once I add a fourth texture, then the problem becomes noticeable. With a fifth texture, the problem becomes very bad. With five textures on the terrain it becomes impossible to drive the vehicle smoothly around the world.

My computer has a GTX470 with 1GB video ram, a quad core i7 and 8 GB of ram, so it should have no problem handling this. There shouldn't be any thrashing or paging in/out of ram going on unless there is some really bad cache missing going on, which may be the case as I think it is definitely a CPU issue here.

When I load into the world, the frame rate will sit at a nice steady ~150fps (with pretty much every video setting as high as they go). Once I start moving around, the frame rate will bounce all over the place randomly dropping down very low (under 30fps). I know that it is normal for the frame rate to jump around a bit as the complexity of the scene changes with movement, but this is not right. It is jumping way too much and with such frequency that it is introducing a lot of lag into the physics, making the vehicle movement very jerky.

One interesting point I've noticed is that once I've moved around the small world for a few minutes, the problem seems to stop. It appears as though everything has been loaded into memory and it settles down. I've tried using WORLD_FULL to force everything to be loaded at once, but that doesn't make any difference. Is WORLD_FULL working correctly?

I need to find a solution to this problem as the project I'm working on is a fast paced game where the player only spends a few minutes in a small world at a time, so those few minutes need to be smooth. Any ideas other than only using two textures on the terrain? smile Perhaps a way to force the entire world to be loaded in, so there is no paging going on (like what I thought WORLD_FULL was supposed to do).
(This post was last modified: 12-15-2010 03:29 PM by fatcoder.)
12-15-2010 03:25 PM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #8
RE: Random Performance Issues
Hi,

Does the problem occur also in SDK tutorials, or in World Editor?
Or is it only in your project? This is important.
maybe you have incorrect codes?

Yes WORLD_FULL loads the whole world without any paging.

Can you send me the smallest/simplest case when the error starts occuring (textures+world+model files, + codes)
12-15-2010 05:22 PM
Find all posts by this user Quote this message in a reply
fatcoder Offline
Member

Post: #9
RE: Random Performance Issues
Yes, I can reproduce the problem with the SDK tutorials. I tried using the World with Character tutorial. I modified it to use a third person camera so I can follow the character around and I also increased the player's speed to 10, just so I can run around a bit quicker. You also need to comment out the line that switches v-sync on to get the true frame rate. Finally I added some text in the corner to display the fps and the state draw time.

I then just created a flat 4x4 area world, randomly painted 5 textures across and added the player character to it. I then changed the tutorial to load that world instead of the sample world... and start running around.

Here is the complete code.

Code:
/******************************************************************************/
#include "stdafx.h"
#include "../../../../../data/enum/_enums.h"
/******************************************************************************

   In this tutorial is presented how to combine extending base classes with World Manager usage

/******************************************************************************/
STRUCT(Player , Game::Chr) // extend character structure by defining a player class based on the character
//{
   Memx<Game::Item> items;                                                                   // here is the characters inventory, a container of items
   virtual Memx<Game::Obj>* itemContainer() {Memx<Game::Obj> &items=T.items; return &items;} // override default method of character, to return proper item container

           void updateItems(); // update items actions
   virtual Bool update     (); // here we'll update the player
};
/******************************************************************************/
Game::ObjMemx<Game::Static> Statics; // container for static objects
Game::ObjMemx<Game::Item  > Items  ; // container for item   objects
Game::ObjMemx<      Player> Players; // container for player objects
/******************************************************************************/
void Player::updateItems()
{
   if(Kb.bp(KB_1)) // try to pickup an item
      if(Items.elms()) // if world items container has some elements
         itemPickUp(Items[0]); // pick up the first valid item

   if(Kb.bp(KB_2)) // try to drop down an item
      if(items.elms()) // if inventory has some items
         itemDropDown(items[0]); // drop down the first item

   if(!Kb.alt())grabStop();else // if don't want to grab
   {
      if(grab.is()) // if already grabbing
      {
         Vec pos;
         SinCos(pos.z,pos.x,angle.x+PI_2); // set direction according to player angle
         pos *=ctrl.radius()+0.5;          // set radius    according to player controller radius
         pos.y=ctrl.height()*0.4;          // set vertical  position
         pos +=T.pos();
         grab.pos(pos);                    // set desired grab position
      }else
      if(Items.elms()) // if isn't grabbing anything check for presence of world items
      {
         grabStart(Items[0]); // grab first present
      }
   }
}
/******************************************************************************/
Bool Player::update()
{
   if(action)
   {
      if(Kb.b(KB_W) || Kb.b(KB_S) || Kb.b(KB_A) || Kb.b(KB_D) || Kb.b(KB_Q) || Kb.b(KB_E))actionBreak();
   }

   if(!action)
   {
      // turn & move
      input.anglei.x=Kb.b(KB_Q)-Kb.b(KB_E);
      input.anglei.y=Kb.b(KB_T)-Kb.b(KB_G);
      input.diri  .x=Kb.b(KB_D)-Kb.b(KB_A);
      input.diri  .z=Kb.b(KB_W)-Kb.b(KB_S);
      input.diri  .y=Kb.b(KB_SPACE)-Kb.b(KB_LSHIFT);

      // dodge, crouch, walk, jump
      input.dodge = Kb.bd(KB_D)-Kb.bd(KB_A);
      input.crouch= Kb.b (KB_LSHIFT);
      input.walk  = Kb.b (KB_LCTRL );
      input.jump  =(Kb.bp(KB_SPACE ) ? 3.5 : 0);

      // mouse turn
      Flt max=DegToRad(900)*Time.d(),
          dx =Ms.dir_ds.x*1.7,
          dy =Ms.dir_ds.y*1.7*Ms.inv();
      angle.x-=Mid(dx,-max,max);
      angle.y+=Mid(dy,-max,max);

      // ready stance change
      ready^=Kb.bp(KB_R);
   }

   speed = 10;

   updateItems();
   return __super::update();
}
/******************************************************************************/
void InitPre()
{
   App.name("World with Character");
   App.flag=APP_MS_EXCLUSIVE|APP_FULL_TOGGLE;
   IOPath("../data");
   Paks.add("engine.pak");
}
/******************************************************************************/
Bool Init()
{
   Text_ds.scale*=0.8;

   Physics.create(CSS_NONE,true,"../Installation/PhysX");

   Sun.image=Images("gfx/sky/sun.gfx");
   Sky.atmospheric();

   // create the world
   Game::World.init()
              .setObjType(Statics,OBJ_STATIC)
              .setObjType(Players,OBJ_PLAYER) // please note that here we'll use 'Players' memory container for 'OBJ_PLAYER' objects
              .setObjType(Items  ,OBJ_ITEM  )
              .New("world/test6.world");

   return true;
}
/******************************************************************************/
void Shut()
{
}
/******************************************************************************/
Bool Update()
{
   if(Kb.bp(KB_ESC))return false;

   Game::World.update(Cam.at); // update world to given position

   Cam.dist=Max(1.0f,Cam.dist*ScaleFactor(Ms.wheel()*-0.1)); // update camera distance according to mouse wheel
   Cam.setSpherical(Players[0].pos()+Vec(0,0.5,0), Players[0].angle.x, Players[0].angle.y, 0, Cam.dist); // set spherical camera looking at player position with given player angles

   // after setting camera position and angles:
   Cam.updateVelocities().set(); // update camera velocities and activate it

   return true;
}
/******************************************************************************/
void Render()
{
   Game::World.draw();
}
void Draw()
{
   Renderer(Render);
                     D.text(0,0.9,"Press WSAD keys to move, 1/2 to pick up/drop item, Alt to grab");
   if(Players.elms())D.text(0,0.8,S+"Items in inventory : "+Players[0].items.elms());

   D.text(D.w()-0.2,-D.h()+0.15,S+"Draw "+Time.stateDrawTime());
   D.text(D.w()-0.2,-D.h()+0.05,S+Time.fps());
}
/******************************************************************************/

In testing this further, I've noticed another intersting point. The "random" slow downs appear to be happening when moving through certain spots in the world. I'm wondering if it is happening because some spots have move textures blended across them than others (as a result of me randomly painting the terrain).

My frame rate will fluctuate between 900fps and 80fps. When it drops down low there is some slight stuttering (i.e. character movement is not smooth). This isn't really a problem when running around with a character and if the fps display wasn't there, you probably wouldn't notice the issue. However when trying to drive a vehicle around, controlled by physics, which is very sensitive to these fluctuations, it becomes a real problem.

I remember years ago working on a racing car game and we had a similar problem to this when using Physx. We solved the problem with the help of this article. Specifically the section under "The final touch." You're probably already doing this, but just thought I'd throw it out there just in case.

http://gafferongames.com/game-physics/fi...-timestep/

What helped us massively back then, was taking multiple physx simulation steps per frame at a lower time step/precision, rather than a single simulation step at a higher time step.
12-16-2010 02:54 AM
Find all posts by this user Quote this message in a reply
Driklyn Offline
Member

Post: #10
RE: Random Performance Issues
Now that you mention vSync, I believe this may have been happening to me when I was working on my game demo a while back. I just turned on vSync to fix the problem (I suppose it wasn't really fixed then, just hidden).

However, the problem I was having would have absolutely nothing to do with terrain textures, as I did not use any terrain at all. I would guess that it was related to the number of objects/polygons I had in the scene.
(This post was last modified: 12-16-2010 08:39 AM by Driklyn.)
12-16-2010 08:37 AM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #11
RE: Random Performance Issues
Please attach the same world that you've used for tests. Also any other files that you've used in the test and which aren't present in the SDK.

EE already uses physics steps subdivision, depending on Physics.timestep and precision
12-16-2010 11:57 AM
Find all posts by this user Quote this message in a reply
fatcoder Offline
Member

Post: #12
RE: Random Performance Issues
OK, I will send you the world I created shortly, but it is nothing special. Just a flat 4x4 area world painted with the 5 textures included with the SDK. I haven't used anything other than what is included with the SDK.
12-16-2010 06:05 PM
Find all posts by this user Quote this message in a reply
fatcoder Offline
Member

Post: #13
RE: Random Performance Issues
Here is the test world I am using. Just drop it into the EE SDK Data folder. I haven't used anything that isn't included in the default EE SDK.

As I said before, the frame rate fluctuations aren’t really a problem when just walking a character around. In fact, if you removed the fps counter you probably wouldn't even notice the frame rate jumping around. The fps counter makes it obvious.

The frame rate fluctuations only become a problem when working with sensitive physics simulations like a vehicle. Then it becomes noticeable. None of the EE tutorials have a vehicle driving around a world with 5+ textures used on it., which is why this probably hasn't been noticed by many. In addition, the use of vsync (as previously mentioned) can mask the issue, making it harder to detect.

I'm not sure what is going on under the bonnet of EE with this problem. I think the CPU is doing something that is blocking the GPU. Perhaps swapping textures in/out of VRAM or filling vertex/index buffers for sections of terrain for each texture.... only you can see Esenthel, looking at the source. PIX or PerfHUD can probably help you find the problem.


Attached File(s)
.zip  test6.world.zip (Size: 833.33 KB / Downloads: 3)
12-17-2010 02:54 AM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #14
RE: Random Performance Issues
Hi,

I've tested it, on my Geforce 320M, in World Editor game mode, and in game launched through VS and your codes pasted above, all in DX9.
But in all cases the FPS is pretty much smooth, no fluctuations.

Quote:I think the CPU is doing something that is blocking the GPU. Perhaps swapping textures in/out of VRAM or filling vertex/index buffers for sections of terrain for each texture....
no, for such small world everything is loaded at start, and there's no paging at all.
12-18-2010 01:00 AM
Find all posts by this user Quote this message in a reply
Post Reply