optimiser emits ud2 (invalid-opcode trap) for a fully defined C loop at -Ot and

[KEL26]

Hello Pelle,

First, thank you for Pelles C — it has just carried a full port of CFITSIO
4.7.0 (NASA's FITS library, 59 sources), zlib, libpng and FFTW 3.3.11 to a
clean finish, so the report below comes from a codebase your compiler
otherwise handles admirably.

SUMMARY
-------
Pelles C 14.50.0 for Windows x64, compiling the attached self-contained
program (about one screen of code), translates the loop

    for (i = 0; i < 200; i += 1) {
        data = (unsigned char)(i % 256);
    }

(over a local  unsigned char data[200])  into a loop body whose FIRST
instruction is  ud2 , so the program dies with exception 0xC000001D on the
first iteration. The store that follows the trap reads  al , which no
instruction in the function ever writes — the value computation has been
deleted entirely. The statement is fully defined ISO C: i is in [0,200),
so i % 256 is well defined and non-negative, the conversion to unsigned
char is well defined, and the store is in bounds.

ENVIRONMENT
-----------
    Pelles C 14.50.0, Windows 10 x64 (build 19045)
    pocc -Tx64-coff -Ot -W1 -std:C17 -Ze pelle_repro.c
    polink -machine:x64 -subsystem:console pelle_repro.obj kernel32.lib

OBSERVED (same source, three settings)
--------------------------------------
    -Ot          RESULT : EXCEPTION 0xC000001D
    -Os          RESULT : EXCEPTION 0xC000001D
    (no -O...)   RESULT : SURVIVED   checksum 19900 (expect 19900)

DISASSEMBLY (podump /disasm of the -Ot object, the loop in full)
----------------------------------------------------------------
    [0050] 0F0B            ud2
    [0052] 4863D3          movsxd  rdx,ebx
    [0055] 888414EC000000  mov     byte ptr [rsp+rdx+0ECh],al
    [005C] 83C301          add     ebx,1
    [005F] 81FBC8000000    cmp     ebx,0C8h
    [0065] 7CE9            jl      0050

NOTES THAT MAY HELP LOCATE IT
-----------------------------
1. The mistranslation is sensitive to the surrounding frame: the identical
   loop in a different function (static buffer, checksum in a separate
   loop) compiles correctly at -Ot on the same machine. The attached
   reproducer therefore keeps the exact frame of the function in which the
   fault was first observed — the same locals in the same order, and calls
   through a volatile function pointer where the original code called into
   a library. Simplifying the frame may make the fault vanish.
2. The pattern (unsigned char)(expr % 256) appears in a dozen other files
   of the same project that compile correctly, consistent with note 1.
3. Discovered because a unit test of the CFITSIO port died with
   "CRT: unhandled exception" before its first library call; an
   __try/__except wrapper reported 0xC000001D, and podump named the ud2.

A SECOND, UNRELATED ISSUE, MENTIONED FOR COMPLETENESS
-----------------------------------------------------
The same project also hit what the evidence indicates is a stack-slot
assignment fault at -Ot in one large function of CFITSIO's imcompress.c:
two live locals (a 4-byte char array, address escaping through a pointer
array, and a 12-byte char array) appear to have shared storage, so a
12-byte strcpy destroyed the 4-byte string; rearranging the locals into
one union cured it. An isolated replica of the declarations does NOT
reproduce it — the full function seems to be needed — so I am not
attaching that one until it has a reproducer worth your time. I can supply
the full analysis on request.

With thanks and best regards,
Kelly

Attachment: pelle_repro.c

[TimoVJL]

A smaller test:

Code Select Expand

//#include <stdio.h>
int __cdecl printf(const char * restrict format, ...);
int __cdecl main(void)
{
    int i;
    unsigned char data[200];
    for (i = 0; i < 200; i += 1) {     /* the suspect loop, verbatim    */
        data[i] = (unsigned char)(i % 256);
    }
    for (i = 0; i < 200; i++)
        printf("%i ", data[i]);
    return 0;
}


Code Select Expand

test_loop.c(8): warning #2803: Attempt to divide by zero.

[Michele]

The instruction UD2 should have been designed specifically for testing.
Perhaps this is a typo from the compiler testing phase.

[KEL26]

Thank you TimoVJL for minimising my test case. Your version was much better than
mine, and it changed what I think the bug is.

My original file preserved the whole stack frame of the function where I met
the fault. That was based on a real
observation but the inference was wrong. You removed the frame and the fault
survived, so please treat my frame-sensitivity note as withdrawn.

What your version did that mine did not was move the fault to compile time,
where the compiler names its own reason:

    test_loop.c  ( 8 )   :   warning #2803: Attempt to divide by zero

on   data = (unsigned char)(i % 256);

I have since measured it: thirty-two variants, four optimisation levels, both
extension switch sets, all first validated under gcc -O2 -Wall -Wextra where
all thirty-two pass without a diagnostic.

THE PART THAT MATTERS MOST
--------------------------
The trap is the lucky case. Change 256 to 300 and there is no warning, no
trap, and no ud2 for anyone to find:

    int CDECL main(void)
    {
        unsigned char data[200];
        int i; unsigned long sum = 0;

        for (i = 0; i < 200; i += 1)
            data = (unsigned char)(i % 300);

        for (i = 0; i < 200; i += 1)
            sum += data;

        printf("checksum %lu (correct 19900)\n", sum);
        return 0;
    }

    -Ot            checksum 4060
    no optimising  checksum 19900
    gcc anything   checksum 19900

i is bounded to [0,200) and 300 is larger than 199, so i % 300 is i and the
answer must be 19900. 4060 is the sum of i % 44, and 300 modulo 256 is 44.

THE RULE, AS FAR AS I CAN MEASURE IT
------------------------------------
    IF   the operator is %  (not /)
    AND  the divisor is a signed integer literal K
    AND  the compiler can prove the numerator stays below 256
    AND  the result is converted to an unsigned type narrower than int
    AND  optimisation is on
    THEN K is replaced by K mod 256.

         K mod 256 == 0  ->  #2803 and ud2
         K mod 256 != 0  ->  nothing said, wrong number computed

The eight is fixed. It does not come from the destination type:

    (unsigned char)((i*300) % 256)   numerator to 59700, dest 8 bits   CORRECT
    (unsigned short)(i % 65536)      numerator to 199,   dest 16 bits  TRAPS

and it is not the exact width of the numerator's range either:

    (unsigned char)((i*3) % 1024)    numerator to 597, exactly 10 bits  CORRECT
    (unsigned char)((i*300) % 65600) numerator to 59700, 16 bits        CORRECT

WHAT IT IS NOT
--------------
Division is unaffected. 256u is correct but 256L is not, so the signedness of
the literal matters and its width does not. A non-literal divisor is correct,
even a plain int k = 256 that propagation would fold. A signed destination is
correct. A numerator of rank above int is correct. Unoptimised code is always
correct. And -Ze makes no difference at all: 256 combinations run twice,
identical results.

WHY IT MAY HAVE GONE UNNOTICED
------------------------------
In every failing case the numerator can never reach the divisor, so the
remainder was mathematically redundant to begin with. Ordinary code, where
the divisor actually bites, is untouched. The shape that breaks is the
defensive one -- a remainder written as a safety net over a value already
known to be a byte.

The cure, for anyone who needs one today:

    int t = i % 256;
    data = (unsigned char)t;      /* correct at every level */

My full set: the twelve-line case above, all thirty-two
tests, the harness, the generators, etc and the raw logs - no idea where to post them, if they are needed.


Kelly