Pelles C forum
C language => Expert questions => Topic started by: CommonTater on December 17, 2012, 05:56:16 PM
-
If I do this...
int testfunc(int x, int y)
{
printf("%d %d", x ,y);
{ // start inner scope
int z;
z = x * y;
printf(" = %d \n", z);
} // end inner scope
puts("Thanks for playing!");
}
Is there an actual stack cleanup at the end of the inner scope in Pelles C?
It was my understanding that the { bracket would create a stack frame and the } would clean and remove it when done. But, I've just looked at a couple of dissassemblies of small programs and can't see where it's being done.
-
No Tater, it simply creates different variables, generally temporary allocated in registers or using alloca function.
-
No Tater, it simply creates different variables, generally temporary allocated in registers or using alloca function.
EDIT.... Well that explains the crash.
I just went through a small sample with OllyDbg... it's simply ignoring the braces, far as I can see... It simply allocates everything on function entry and restores the stack on exit. Although it does follow rules of scope in that the variable is not available outside the braces.
-
No Tater, it simply creates different variables, generally temporary allocated in registers or using alloca function.
EDIT.... Well that explains the crash.
I just went through a small sample with OllyDbg... it's simply ignoring the braces, far as I can see... It simply allocates everything on function entry and restores the stack on exit. Although it does follow rules of scope in that the variable is not available outside the braces.
That is what I would have expected in the first place... ;)
The "scope" is a compiler (logic/parsing) thing only, any stack cleanup will only happen on the exit of the function (testfunc in this case) for all local variables declared within, on any level of "scope".
Kind of logical when you look at it from a CPU/assembler point of view, as it would mean to constantly mess with the stack registers/pointers back and forth, in order to ensure proper access to variables on the stack allocated outside of any deeper level scope but used at any level of scope...
Ralf
-
The "scope" is a compiler (logic/parsing) thing only, any stack cleanup will only happen on the exit of the function (testfunc in this case) for all local variables declared within, on any level of "scope".
Kind of logical when you look at it from a CPU/assembler point of view, as it would mean to constantly mess with the stack registers/pointers back and forth, in order to ensure proper access to variables on the stack allocated outside of any deeper level scope but used at any level of scope...
Stack frames are extremely easy to make and break... I believe it's 3 assembler instructions each. The overhead, on modern computers, would be in nanoseconds and the benefit of cleaning the stack at the end of each scope would be enormous.
The image below shows the creation and destruction of a stack frame... It's a total of 4 instructions.
Since at some level all C variables are pointers to memory locations, access beyond current scope is unaffected. Only the variables in the current stack frame would be deleted when the scope closes... which is fine because the compiler blocks access to them anyway.
But like you say... it's not done, so now I have to review every program I've written since 2004 with this in mind.
-
Ok... still trying to get a handle on this mess...
This code ....
#include <stdio.h>
typedef struct t_test
{
char str[256];
int x;
} test;
typedef struct t_test2
{
char str[256];
int x[2];
} test2;
int main (void)
{
puts("Hello");
{
test2 t = {0};
printf("%p\n", &t);
}
{
test u = {0};
printf("%p\n", &u);
}
puts("bye");
return 0;
}
Produces different results based on the Optimization settings...
The first image below show us "optimaztions off" on the left and "Minimize size" (actually any optimization) on the right...
It seems that with optimizations on, it does re-use stack space... so now I'm REALLY confused... The econd image shows the main function... it creates a stack frame then appears to reserve a block big enouogh to hold the largest struct and simply re-uses it? No stack frames... Is that even close to correct?
-
The compiler seems to realise that you are not actually using the structure. Try to assign a string to t, instead of (0).
-
The compiler seems to realise that you are not actually using the structure. Try to assign a string to t, instead of (0).
Same results ...
If you look at the way the code is bracketed... the two structs should not exist at the same time.
It's about C's rules of scope and trying to minimize stack usage in a very large function.
-
Tater, it simply depend on the optimizing strategy adopted in PellesC compiler.
Any compiler can behave differently, i.e. adding variables on local stack using alloca for dynamic allocation, reuse spaces or registers that has been detected to be mutually used in only one scope at time, and so on.
The more aggressive the optimization the more probable a problem :o
Anyway an indepth study of PellesC strategy will only lead to a better knowledge of this compiler behaviour (maybe more usefull if want switch to asm programming), but not strictly related to the C language itself.
-
The compiler seems to realise that you are not actually using the structure. Try to assign a string to t, instead of (0).
Same results ...
Zip the exe and post it. I seriously doubt that the compiler can use overlapping structures for
test2 t = {"I am the first string"};
printf("%p\n", &t);
test u = {"I am the second string"};
printf("%p\n", &u);
printf("%p\n", &t); // use the first string again
-
Tater, it simply depend on the optimizing strategy adopted in PellesC compiler.
Any compiler can behave differently, i.e. adding variables on local stack using alloca for dynamic allocation, reuse spaces or registers that has been detected to be mutually used in only one scope at time, and so on.
The more aggressive the optimization the more probable a problem :o
Anyway an indepth study of PellesC strategy will only lead to a better knowledge of this compiler behaviour (maybe more usefull if want switch to asm programming), but not strictly related to the C language itself.
I hear ya.... I'm just trying to work around infrequent, unexplained crashes in two of my programs (fortunately neither is commercial). As it turns out these are programs with a large structs, where I've used nested scopes in hope of avoiding stack overflows, believing the variables in the inner scopes were released from the stack at the closing brace.
Thing is the books I learned from both said that variables are released from memory at the end of each scope... now, years later, I'm discovering this isn't always true... and the "Why?" question is nagging at me...
I need to understand the problem at least well enough to avoid it, hopefully well enough to fix it.
-
Zip the exe and post it. I seriously doubt that the compiler can use overlapping structures for
test2 t = {"I am the first string"};
printf("%p\n", &t);
test u = {"I am the second string"};
printf("%p\n", &u);
printf("%p\n", &t); // use the first string again
JJ... in all due respect your code is as different from mine as night and day...
In C an opening bracket { signals the beginning of a scope that continues until the matching close bracket } Variables created inside that scope are "encapsulated" by the scope and are unavailable outside the scope... Variables created at larger scope are available inside the smaller scope, except where hidden by a local variable of the same name.
Try this... *with the brackets exactly as I have them*...
int main (void)
{
{
int y = 20;
}
printf("%d", y);
return 0;
}
Unless your compiler is seriously braindead you should get an error at the printf() line saying that y is undefined. This is because it is encapsulated in it's own little scope...
Now back to my original sample:
#include <stdio.h>
typedef struct t_test
{
char str[256];
int x;
} test;
typedef struct t_test2
{
char str[256];
int x[2];
} test2;
int main (void)
{
puts("Hello");
{ <-- begin scope 1
test2 t = {0}; <-- t is encapsulated
printf("%p\n", &t);
} <-- end scope 1 (Is t destroyed?)
{ <-- new scope 2
test u = {0}; <-- u is encapsulated
printf("%p\n", &u);
} <-- end scope 2 (Is u destroyed?)
puts("bye");
return 0;
You cannot access either t or u outside it's own scope. Moreover; since each is in it's own private scope, the two structs should not exist in memory at the same time. The only question here is whether there is a stack cleanup --releasing the space occupied by the structs-- at the end of each scope...
In terms of stack usuage, it should be functionally equivalent to...
int main (void)
{
puts("Hello");
test2* t = calloc(1, sizeof(test2));
printf("%p\n", &t);
free(t);
test* u = calloc(1, sizeof(test));
printf("%p\n", &u);
free(u);
puts("bye");
return 0;
And has the additional benefit of encapsulating large segments of unrelated data.
In the particular problem I'm having, I've written a long function (~ 200 lines) that doesn't seem to want to break up into smaller functions easily; I would have to pass a dozen or more parameters each time... The program crashes near the end of this function in an apparent stack overflow that should not happen if the variables are being removed (released, overwritten, re-used...) at the end of each of the scopes within the function as they should.
-
Tater if you defined large structures local to scopes inside the same function they are created on the stack.
Your problem can be a stack corruption if you write beyond the structure (bad pointer), a stack overflow if data structures are too large or optimization problem.
Knowing you I am sure that you already recompiled everything with no optimization ;D
Remains the other two options.
If you are sure that there is no pointers error, try to increase stack allocation using the switch /STACK on the linker.
It would be good to see a disassembly output of your function to analyze the problem, but if the source is 200 lines.... :P
-
Variables created inside that scope are "encapsulated" by the scope and are unavailable outside the scope...
Point taken.
since each is in it's own private scope, the two structs should not exist in memory at the same time. The only question here is whether there is a stack cleanup --releasing the space occupied by the structs-- at the end of each scope...
That could be easily verified with Olly. Your screenshots seem to show that the stack is OK. However, I could imagine that the optimising compiler does not create the stack frame twice because it "knows" that a single frame for the bigger of the two strucs is enough. A strategy that saves size and cycles, and should not pose any problems under normal conditions. Can you reproduce a situation where this leads to a crash?
-
The stack frame is created *only* for function call.
Local variables are created and destroyed inside the *current* frame.
EBP register points always to function call frame creation point, this is consistent with stack unroll of exception handler (slightly modifyied for 64bits programs).
No frame is created and never will be to avoid any mess with stack handling.
If any stack extension is required for local variables, scoped or not, is handled through use of alloca() function.
-
Tater if you defined large structures local to scopes inside the same function they are created on the stack.
Well, the structs themeselves are in the range of 20 to 32k ... big yes, but the stack should be able to deal with it if the space is either re-used or released at the end of each scope. The problem might be that the overall program is deeply parenthetical, with function nesting getting quite deep in some points... so maybe there's not enough stack to begin with.
Your problem can be a stack corruption if you write beyond the structure (bad pointer), a stack overflow if data structures are too large or optimization problem.
Knowing you I am sure that you already recompiled everything with no optimization ;D
Actually, its a bit weird... turning off optimizations crashed the function sooner....
Remains the other two options.
If you are sure that there is no pointers error, try to increase stack allocation using the switch /STACK on the linker.
I'm pretty sure there's no overruns or pointers to non-existent stuff... and I will try the /STACK switch and maybe bump it up to 2mb instead of the default 1...
-
The stack frame is created *only* for function call.
Local variables are created and destroyed inside the *current* frame.
EBP register points always to function call frame creation point, this is consistent with stack unroll of exception handler (slightly modifyied for 64bits programs).
No frame is created and never will be to avoid any mess with stack handling.
If any stack extension is required for local variables, scoped or not, is handled through use of alloca() function.
Actually, from what I was reading earlier this morning there's a second reason why a stack frame cannot be created for each scope... apparently the function parameters and internal variables are often referenced as offsets from EBP... [ebp + 8] and such. To create a second stack frame would mess up the location of ebp and cause it to temporarily lose track of the function's (more or less) global data.
I wish I'd known all this 8 years ago... It would have changed the way I write C programs rather significantly... That'll teach me to trust text books! (In Pascal this was never an issue, far as I know)
I'm still trying to get my head around how the stack is handled with respect to scopes...
It appears that with optimizations off, every variable gets it's own chunk of stack which was what seems to have caused my overflow problem.
With any optimization turned on, it appears to re-use a "big as the biggest" chunk of stack to manipulate variables... which, if I'm understanding correctly, means that when a variable goes out of scope it's space is re-used as a means of reducing overall stack usage.
As you know, there have been times when the optimization bugs have caused me some grief, so I typically compose and test software with optimizations off then I will turn on the most appropriate optimization (usually "maximize speed") and test again...
In this case enabling optimizations actually corrected the problem... It remains for more extensive testing but it seems that I need the stack optimizations on to make this work. Maybe the answer is in this and making a larger stack through the linker...
I wonder if Pelle would consider making the Stack Optimizations available on a separate switch... so they can be tested independently of the other strategies.
-
The stack frame is created *only* for function call.
...
No frame is created...
You mean "no frame is created for code inside brackets", I suppose?
...programs with a large structs, where I've used nested scopes in hope of avoiding stack overflows, believing the variables in the inner scopes were released from the stack at the closing brace.
Thing is the books I learned from both said that variables are released from memory at the end of each scope... now, years later, I'm discovering this isn't always true...
What is "released"? It can be
- GlobalFree
- leave (i.e. destroy the stack frame)
- leave everything "as is" but declare the memory as part of another bit of code.
Again, with Olly that could be clarified in seconds.
-
Thing is the books I learned from both said that variables are released from memory at the end of each scope... now, years later, I'm discovering this isn't always true...
What is "released"? It can be
- GlobalFree
- leave (i.e. destroy the stack frame)
- leave everything "as is" but declare the memory as part of another bit of code.
In this case "released" pretty much means the destruction of the stack frame at the end of the function, which is really just resetting EBP and the stack pointer back to where they were when the function was called. (i.e. To point at the previous stack frame)
Again, with Olly that could be clarified in seconds.
Ok... so write some C code and experiment with it in Ollydbg ... which is exactly what I've been doing throught this whole fiasco...
-
Ok... so write some C code and experiment with it in Ollydbg ... which is exactly what I've been doing throught this whole fiasco...
What I meant is you compile your snippet with your optimised and not-so-optimised compiler settings and post the two executables here. And no, I am not asking that because I am a mean old man or unable to compile a C snippet but rather because your executable fails, and it has to do with your compiler (settings)
;-)
-
Yes Tater every dynamic allocation is relative addressed with reference to base stack pointer which is EBP.
If you take a look to assembler generated you will see a lot of move in some register the value from location pointed by EBP plus an offset like "mov eax, [EBP+0x12]".
That's why you have a prolog where you save the actual stack pointer in EBP, that will be base frame, than some value will be added to stack pointer to create space on the stack. Because for IAPX architecture the stack grows toward lower addresses, we must *subtract* from stack pointer the space we need. The subtracted value must leave the stack aligned on natural size (on DWORD boundary for 32bits and QWORD boundary for 64). This way anything that will be pushed expressily (like a push instruction), or implicitly (like the return address of a subroutine call) will lay beyond that point.
The value added to the stack pointer is the space required to store local or better dynamic variables. The parameter passed to the function are above (for IAPX) the base pointer EBP, so tipically you access local variables with positive offsets referred to base pointer and negative offsets for parameters.
The function alloca() can squeeze or enlarge the local variables area (between EBP and actual stack pointer ESP) in some cases when the requested space could not be determined statically during compilation (the compiler calculates the amount required for local variables and hardcodes the value to add to stack in prologue) is dinamically chnged by alloca().
The use of this function largely discouraged for the problems that can cause considering it dangerous (this is common opinion around), but the real reason is that the abuse of such a practice generally lead to stack overflow problems.
From what you said the stack overflow could be the culprite, anyway I suggest to enlarge stack at least 4M. Consider that the stack can grow by itself, the OS should dinamically increase it on request, but because it have to cause a memory exception to trigger the virtual memory manager to allocate more data, maybe that on very large structures your access is much far away leading to memory violation before memory reallocation. This means that the standard 4k (1 page) of reallocation could be insufficient (is the second parameter of /STACK switch), so maybe you want allocate 4 or even 8 pages at time.
This depend on your code, consider that enlarging the stack will erode memory resources that will remain allocated in your program, enlarging the allocation size could fit the request determining almost automatically the memory you need.
Last consider that the stack memory on program entry is allocated, but not committed. This means that the memory manager take a note of how much memory you require, but the real allocation, committment, is done when you *access* memory (or try to do it).
JJ yes, the creation of stack frame is *only* for functions. The scoping determine the visibility of a variable, but the memory allocation is the same along the whole function. Using optimizations the compiler could reuse memory, but this is not limiting it can do whatever it considers usefull to reduce resources (CPU, memory, time).
EDIT: I correct some incongruences, and attach an image to better explain the mechanism.
For 64bits the thing is a little bit more complex due to the calling convention (__fastcall) used, but the base working of stack handling remains the same.
-
Here is Olly's output for Pelles C with standard Win32 console settings:
main Ú$ 55 push ebp
00401001 ³. 89E5 mov ebp, esp ; create the stack frame
00401003 ³. 81EC 08010000 sub esp, 108 ; reserve 108h bytes, i.e. SIZEOF the bigger structure
00401009 ³. 57 push edi
0040100A ³. 68 08704000 push offset 00407008 ; ÚArg1 = ASCII "Hello"
0040100F ³. E8 5C000000 call puts ; ÀSimpleConsole.puts
00401014 ³. 59 pop ecx
00401015 ³. 8DBD F8FEFFFF lea edi, [local.66] ; get the address of first (bigger) struct
0040101B ³. 31C0 xor eax, eax
0040101D ³. B9 42000000 mov ecx, 42
00401022 ³. F3:AB rep stosd ; and clear it
00401024 ³. 8D85 F8FEFFFF lea eax, [local.66]
0040102A ³. 50 push eax ; ÚArg2
0040102B ³. 68 04704000 push offset 00407004 ; ³Arg1 = ASCII "%p
"
00401030 ³. E8 CB000000 call printf ; ÀSimpleConsole.printf
00401035 ³. 83C4 08 add esp, 8
00401038 ³. 8DBD FCFEFFFF lea edi, [local.65] ; get the address of the second struct
0040103E ³. 31C0 xor eax, eax
00401040 ³. B9 41000000 mov ecx, 41
00401045 ³. F3:AB rep stosd ; and clear it
00401047 ³. 8D85 FCFEFFFF lea eax, [local.65]
0040104D ³. 50 push eax ; ÚArg2
0040104E ³. 68 04704000 push offset 00407004 ; ³Arg1 = ASCII "%p
"
00401053 ³. E8 A8000000 call printf ; ÀSimpleConsole.printf
00401058 ³. 83C4 08 add esp, 8
0040105B ³. 68 00704000 push offset 00407000 ; ÚArg1 = ASCII "bye"
00401060 ³. E8 0B000000 call puts ; ÀSimpleConsole.puts
00401065 ³. 59 pop ecx
00401066 ³. 31C0 xor eax, eax
00401068 ³. 5F pop edi
00401069 ³. 89EC mov esp, ebp ; release the frame
0040106B ³. 5D pop ebp
0040106C À. C3 retn
It's all very straightforward, nothing mysterious. And it shouldn't fail, so your problems are elsewhere.
-
Yes Tater every dynamic allocation is relative addressed with reference to base stack pointer which is EBP.
If you take a look to assembler generated you will see a lot of move in some register the value from location pointed by EBP plus an offset like "mov eax, [EBP+0x12]".
Yep... read about that, then once alerted I saw it in the dissassembly...
The value added to the stack pointer is the space required to store local or better dynamic variables. The parameter passed to the function are below the base pointer EBP, so tipically you access local variables with positive offsets referred to base pointer and negative offsets for parameters.
Ahhh... ok, good to know...
The function alloca() can squeeze or enlarge the local variables area (between EBP and actual stack pointer ESP) in some cases when the requested space could not be determined statically during compilation (the compiler calculates the amount required for local variables and hardcodes the value to add to stack in prologue) is dinamically chnged by alloca().
The use of this function largely discouraged for the problems that can cause considering it dangerous (this is common opinion around), but the real reason is that the abuse of such a practice generally lead to stack overflow problems.
Which is almost certainly what's going on here... There is one condition in the program I was working on when this happened that can send it off into a recursive folder search to find the file it needs to open. This can really suck up the stack space and it was after that it would crash... I suspect because the huge number of pointer variables caused an overflow condition that didn't manifest itself until the structs were instantiated later.
From what you said the stack overflow could be the culprite, anyway I suggest to enlarge stack at least 4M. Consider that the stack can grow by itself, the OS should dinamically increase it on request, but because it have to cause a memory exception to trigger the virtual memory manager to allocate more data, maybe that on very large structures your access is much far away leading to memory violation before memory reallocation.
I increased it to 2097152 : 8192 over the course of several tries and it seems OK now. I won't know for certain until I do some further testing but this is promising...
With this new stuff in my head... doing a code review suggested that the folder search would best be relocated to it's own function, using heap allocation rather than stack space. (I rarely make functions that are only called from one place.) Fortunately it's the only part of that monster function that only needs two parameters... the start path and the filename.
I will be reorganizing it sometime this evening or tomorrow I'll let you know how it goes...
-
It's all very straightforward, nothing mysterious. And it shouldn't fail, so your problems are elsewhere.
Yet, strangely enough... using Frankie's advice --increasing the stack-- and a bit of background research seems to have me on the way to fixing the problem.
Yes, it was a stack overflow. (Exception code : 0x000003E9)
-
There is one condition in the program I was working on when this happened that can send it off into a recursive folder search to find the file it needs to open. This can really suck up the stack space and it was after that it would crash...
We discussed that elsewhere (http://www.masmforum.com/board/index.php?topic=12460.msg128535#msg128535) some time ago, and the conclusion was that a recursive folder search can go max 125 levels deep - with 1MB of stack, you need around 8k to achieve a stack overflow. MAX_PATH is 260 bytes...
-
We discussed that elsewhere (http://www.masmforum.com/board/index.php?topic=12460.msg128535#msg128535) some time ago, and the conclusion was that a recursive folder search can go max 125 levels deep - with 1MB of stack, you need around 8k to achieve a stack overflow. MAX_PATH is 260 bytes...
If I agree with you, will you stop trying to help me?
-
If I agree with you, will you stop trying to help me?
Tater, you are a funny guy ;-)
I just wanted to return the generosity you showed in the other forum (http://masm32.com/board/index.php?topic=1079.msg10111#msg10111). But your reaction shows that you are very tired of chasing your bug, and that right now is not a good moment for offering help. Good luck! And Merry Christmas!!
-
But your reaction shows that you are very tired of chasing your bug, and that right now is not a good moment for offering help.
I have the program working... so no more bug.
-
Ok... moved the folder search to it's own subroutine, used true recursion and used heap allocations for it's structs, made the stack bigger (2mb) and redid a couple of small sections of code...
Ran my monster sub into the deepest nether reagions of my system and everything worked on a dozen tries.
So... in future I'm going to have to change my decision making about a couple of things.
I'm going to have to do a code review on a bunch of other programs I've written...
but I'm pretty sure I'll still be alive at the end of it... :D
Thanks a bunch guys .... your help is deeply appreciated
-
Tater I'm happy you solved the problem :)
Anyway I should correct my previous description that refers to an architecture with stack that grows toward higher memory addresses. In reality in IAPX it grows toward lower addresses, so more space you reserve more lowers the address.
I wrote it in a rush, but anyway the concept is absolutely correct ;D
Your parameters, with return address and stack pointer original value are on one side of frame pointer EBP, local variables on the other. Later I will correct it, now I'm very busy.
-
Tater I'm happy you solved the problem :)
Anyway I should correct my previous description that refers to an architecture with stack that grows toward higher memory addresses. In reality in IAPX it grows toward lower addresses, so more space you reserve more lowers the address.
I wrote it in a rush, but anyway the concept is absolutely correct ;D
Your parameters, with return address and stack pointer original value are on one side of frame pointer EBP, local variables on the other. Later I will correct it, now I'm very busy.
No worries... I understood the stack's behaviour and realized you'd made a minor factual error.
It was the behaviour when closing a scope that was of the most concern...
What I now understand is that Pelles C has two sets of behaviours depending if the optimizations are on or off... with something as crucial as stack organization, this was totally unexpected.
Optimizations off ... every variable gets stack space at function entry... which is not cleared until the function exits. This essentially transforms functions with private scopes into standard functions, in terms of stack utilization.
Optimizations on ... scoped stack space is reused ... "big enough for biggest" stack space is released on function exit. This would seem to indicate that private scoping offers no advantage in stack utilization unless you are creating 2 or more private scopes (but still has other advantages).
Like I said... totally unexpected.
Fortunately it is very rare that I get into these "monster functions" (this is perhaps the 4th time in all the years) although I do use private scoping a fair bit. Far as I know this is the first time I've run into this issue and (silly me) trusting the text books really didn't help this time...
Thanks again for helping me sort it out.