Tuesday, September 29, 2015

Namespaces hell

Welcome in namespace hell. In this place you will find annoying code and the best part: it is on demand. So lets start another topic base on one of my horror stories which ended with hours spend to straight things up.

But let's start from begin. This words probably don't mean a lot for you: s2d, s3d, world2d. world3d, fs, sound, physics, math, navmesh, utility, resources, game, editor, lba, wre, wr.

Personally I know them really well. All of this are namespaces that I used or using in my code. This is collection from 7 years of development and they are probably not all of them. When I start writing engine I wanted to do stuff right. So I created modules which then I encapsulate in namespaces. This was naive and stupid because I ended with annoying code. Who want to write physics::CPhysics for physics class? This is duplicated information. But well with time I got smarter and removed most of them using simple method : Find and replace all.

This was my first sin. After this operation I left with code where you could find stuff like this:

namesace wre
    class A : public wre::class B

The second sin is my mistake of using prefixes for basics types and functions which were outside of namespace. So I had wrVec3, wrQuat, wrMtx44 and functions with similar convention. Why I call it sin ? Because I ended with inconsistent code standard which is not something good and create really a lot of issue. For what you use prefix for what namespace?

So right now I'm decided to clean all of my past mistakes. There will be only one simple convention. Everything will be in one main namespace wr (stands for White Rabbit) and helpers wr::math. I will cleanup naming convention, remove prefixes and put everything related to engine in this namespace.

If somebody think that namespace is useless in he can always use : 

using namespace wr; 

to remove need of using it explicit. I think that this change will only result in simpler and easier to understand code. But well there are no sure things.

What do you think about namespaces ? You would recommend them or rather burn them using fire ?


Sunday, September 27, 2015

Commit comments

Now that I thought about this is interesting topic:

How your comments for commits look like?

In my case you probably think that they are poor because I do project myself. But here I will surprise you (and myself to) because most of the time I do full summary what I done in given change list. So bellow you will find few of my comments from different commits across last 3 years (I use not modified git logs) :

* 13/11/2012

* 25/02/2013

* 30/12/2013

* 05/09/2014

* 21/09/2015

I really like my current form because its simple and easy to understand. Especially with my tendency to submit multi issue change lists. I need to do something about this because this may save me a lot of troubles later. But sometimes you just starting one thing which require something else and before you notice you have overlapping changes. 

What is yours format of check-in comments. ?


Sunday, September 20, 2015

Animation System Changes: Part 1

After weeks of work (probably I could even say months) and hundred resolved tasks and bugs new animation system finally start working ;] Check final effect:

If you think that it is look no different from my old animations then you are almost right :D Currently this new system have even less functionality than old one. Because of that I needed to disable big part of gameplay.

Was it worth it ? 

Short version: I think that: YES. 

Saturday, September 19, 2015

Playing games and Dreams

Normally I don't play too much games but there are days when I just want to relax and play. Sometimes even a little too much for one day but well this have it's charm (till the next day when you need to wake up).

This week I finished playing PS3 Uncharted games and Zone of the Enders. I feel inspired by their style. I really liked Uncharted 2 graphics would it be cool if Little Big Adventure remake had this level of graphics :D Well I need to return to development I still need do few things to make dreams became true.

Tuesday, September 15, 2015

RTTI: Changes for better

As some of you may know White Rabbit Engine use it's own Run Time Type Information system. And well like with everything else sometimes come time when you need to upgrade stuff. Recently I had another occasion to do iteration over this system.

In this post I will try to give you a little bit insight how m RTTI look is engine, what changed and what are plans for future. But well let's start from begin.

Saturday, September 12, 2015

Weird thoughts

Let's start this from some thoughts about who I'm (yes I will be talking about myself).
I'm not seeing myself as somebody really intelligent I would say that I'm event think that I'm stupid. Funny thing is that with time I see that even then I'm more intelligent than a lot of people :|
Question is where this bring us and why I'm in such a position? I thought about this and my answer is: "learning". 

I like to learn new stuff, discover things that I didn't know about. This is for me something incredible and in current time we can do this really simple. Sadly because of that I see how small my knowledge is and how big the sea of possibilities is. There is always ways to growth. 

On other hands we have people who don't really want to learn. They do it only when they are forced by situation. They think that what they know is enough.

I don't know which way of thinking is good or right. The thing that I know is the fact: I don't want to stop learning. I'm where I'm because of things I done right and also all my mistakes. Thanks to all of them I'm different person today than I was yesterday and tomorrow I will be different person too.


Short Term Memory

Funny thing about doing your own tech is that you sometimes need to do some funny stuff. This week I needed done Short Term Memory Manager. My comment in code describe it like this:
Short Term Memory Manager (STM Manager). STM is memory which you want to use for short one-shot tasks:             Create->Execute->[GetResult]->Destroy It is limited by capacity so you never can use more than you pre-allocated for it. Right now it's use to transfer resource data CPU->GPU and in animations tasks.
For me this is temporary memory which I use for example to load texture data and then passing it to GPU upload (which will remove it). There are also other usage where I think using normal allocation would be a little bit overkill.

So my short term memory manager use my new memory ring buffer which look like this:



After creating this code the rest was easy:

    uint8* CShortTermMemoryMgr::cpuAlloc( uint32 a_Nbytes, uint32 a_alignment )
        // memory is HEADER + ALIGNMENT_MEMORY(ALIGNMENT-1) + HEADER_OFFSET(+1) + a_Nbytes
        uint32 memorySize = sizeof(SHeaderCPU) + a_alignment +a_Nbytes;

        uint8* memory = nullptr;

            CMutexAutoLock lock(m_cpuMutex);

            memory = (uint8*)m_cpuMemory.alloc(memorySize);

            if (!memory)
                if (cpuRecoverMemory(memorySize))
                    memory = (uint8*)m_cpuMemory.alloc(memorySize);
                    WR_ERROR(false, "There is not enough memory for this allocation.");
                    return nullptr;

        // Create memory header
        SHeaderCPU* header = (SHeaderCPU*)memory;
        header->isUse = true;
        header->size = a_Nbytes;

        // Fill offset info
        uint8* data = wrAlignAdress(memory + sizeof(SHeaderCPU)+1, a_alignment);
        data[-1] = static_cast<uint8>(data-memory);

        return data; 

    void CShortTermMemoryMgr::cpuFree( uint8* memory )
        if (memory)
            CMutexAutoLock lock(m_cpuMutex);
            SHeaderCPU* header = (SHeaderCPU*)(memory - memory[-1]);
            header->isUse = false;
    bool CShortTermMemoryMgr::cpuRecoverMemory( uint32 a_memoryNeed )
        const void* memory = nullptr;

        while(m_cpuMemory.getBiggestAllocSize() < a_memoryNeed && (memory = m_cpuMemory.peek()))
            const SHeaderCPU* header = (const SHeaderCPU*)memory;

            if (!header->isUse)

        return m_cpuMemory.getBiggestAllocSize() >= a_memoryNeed;
I added also two helper functions:
    template<class t="">
    T* CShortTermMemoryMgr::cpuAllocT( uint32 a_alignment )
        if (uint8* memory = cpuAlloc(sizeof(T), a_alignment))
            return wrNewExt(memory) T;

        return nullptr;

    template<class t="">
    void CShortTermMemoryMgr::cpuFreeT( T* a_memory )
        if (a_memory)

And this finalized my work :] Personally I really like how this came out. Its simple and pretty efficient but well I'm curious what you think about this? Maybe you have some betters idea how to resolve problems like this?


Sunday, September 6, 2015

Hunting of memory corruption

This is the story of memory corruption hunting and what I learned thanks to it.

So the story started when after one of many big changes in component system I needed to re-save all maps and objects templates :] Because I'm lazy bastard some time ago I created processes which loading each of it and re-save. All this after pressing one "magic" button. Great idea and in the same real stress test how object system works.

Sadly after all this time I didn't use it, whole process failed. And in worst possible way: memory corruption. Everybody who have occasion to deal with at leas one not trivial one know that this are not nice bug. Especially in multi-threaded environment:/ In my case normal corruption wasn't enough and sometimes even my callstack was corrupted. Fun :|

So well my hunt begin because I couldn't left this issue without fix.

1st approach: Debugger
The first approach was the most trivial one check what happening when app crashed. Simple and useful but not in this place. The debugger stop when it's already too late. I tried to find some pattern in memory which is corrupted and nothing there :/ I failed.

2nd approach: Enabling all my memory debugging options. 
Yeah in all this years I accumulate some memory debugging tools like: full tracing what memory is allocated, memory guards, validation if memory is not release second time and full allocation/releasing logging. Final result: another defeat :/ 

Non of this tool helped. I only found that my guard sometimes get replaced from "WREGUARD" to "WREGWARD" :/ and it's happening in TiXML node memory (I using XML right now for storing RAW version of map which is use in editor. But let's return to this topic in next approach).

3rd approach: Checks on memory release (Desperation begin somewhere here)
Microsoft _CrtCheckMemory(..) is so nice thing. But well after I done that I can say: I was stupid. XML and full memory check on memory release. Heeeee heeee heee hee he (laugh of crazy person). I started process and started play Uncharted 2 on PS3 and after more than hour later it was still releasing the first map. 

This as bad idea as idea to use XML for project:/ My To-Do task for replacing it with Jason gain higher priority after this.

4th approach: Decoding of callstack from stack memory.
Yep I was so desperate that I learn how to decode callstack from memory for win32 (In the fact it's really easy :D but well. Right now I know how to decode PowerPC, x86 callstack so only x64 left). 

After reading corrupted stack I learned: nothing more than I already know. Releasing of xml again :/

5th approach: Luck
And this one was in the fact funny because I noticed that very often memory which is released is not managed by my memory tools. So after short check I find out sad truth: Allocating memory in one module and releasing it in another is not the best idea :/

Another funny side story:
To be sure that I removed the issue I installed even Valgrind on my Linux computer but it was crashing in initialization of libGlew. First thought: well probably Valgrind do something that glew crash. Because I didn't want to spend too much time into it I just ignored. 

This was mistake. One simple check: run game without Valgrind would show me that the problem was in my changes that I done on Windows not tool itself. But well :/ Day later I figure out this and will soon run full check of editor using Valgrind (This is one of reason why its worth to have Linux build :) )

This one issue shown me so much problems with my code. After it I properly secured my code for releasing of memory from other module, guess what ? I found more of similar problems :/ 

But well right now everything what I found is fixed but this is not the end. I decided that after animations it will be good occasion to spend some time on improving my memory system: make it quicker, allow better control over it and add more tools for checking stuff (just in case). 

From things I will do I will for sure switch on explicit use of dlmalloc and switch on memory tags stack:
 MemStack memstack(EMemTag::System); int32* memory = wrNew int32 [32]
Which will coexist with my existing system:
 int32* memory = wrNewEx(EMemType:System) int32 [32];
Do you have some recommendations what you would want to see in your memory system? How you would trace stuff like that? Do you have maybe some nice secret tool that can help? Or maybe some nice trick in your slave that could help me with future issues ?