Pelles C > Bug reports
fread() - slow on 4GB file
tony74:
OK, maybe I'm not configued right, but here's the thing:
In VS 2022 build tools CLI this code fread()s a 3.9GB file into memory in 1.5 secs, so that's 2581.5 MB/sec for almost 4GB.
In Pelles-C using IDE, the same code fread()s the same 3.9GB file into memory in 12.8 secs, so that's 307.1 MB/sec.
I'm gonna drop my code here so you guys can play with it, but that's a pretty big discrepancy...I was tearing my hair out (what little is left) while working with this in Pelles, then decided to try it in VS 2022 to see if it was something I was coding wrong or my machine was just slow, turns out it was the compiler after all...by several orders of magnitude (1.5 vs 12.8 secs).
OS: Win10 build 19045
Machine: Xeon, 64GB, 2TB SSD system drive
Test File: Avatar.The.Way.Of.Water.mp4
File Size: 3941952295
Compiler settings:
-Pelles C 12.00.2-
Debug: None
Warnings: level 1
Diagnostics: Classic
Microsoft Extensions Yes
Multithreaded Yes
Maximise speed Yes
inlining: Default
Architecture: SSE2
Floating Point: Fast
Calling Conv: __cdecl
Linker: kernel32.lib advapi32.lib delayimp64.lib user32.lib
Compiler settings:
-VS 2022 Build Tools-
All default, no libs added, no compiler-flags
command line: cl dde.c
Try it with a file of around the same size (I limit the size to 4GB in the code).
See what you come up with, I'd like to know why it's doing that, as I'm sure you would too.
Drop me a line if you think of any other info I can provide.
--- Code: ---#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>
#include <time.h>
#define MEG 1000000.0
#define GIG 1000000000.0
#define MAXLEN 4294967295
#define MEMERR -100
#define LENERR -200
#define FILERR -300
#define FRDERR -400
#define NOERR 0
#define uchar unsigned char
struct {
uchar *fbuf;
uchar *ip;
uchar *op;
uint32_t addr;
uint32_t sz;
double et;
}M;
//parse errors
///////////////////////////////////////////////////////////////
char *geterror(uint32_t e)
{
static char ebuf[32];
switch(e){
case MEMERR:
strcpy(ebuf,"\nMEMERR\n");
break;
case LENERR:
strcpy(ebuf,"\nLENERR\n");
break;
case FILERR:
strcpy(ebuf,"\nFILERR\n");
break;
case FRDERR:
strcpy(ebuf,"\nFRDERR\n");
break;
default:
strcpy(ebuf,"\nNOERR\n");
break;
}
return ebuf;
}
//get filesize up to MAXLEN
///////////////////////////////////////////////////////////////
uint32_t getfsize(char *fname)
{
FILE *fp;
__int64 offsetzed=0;
__int64 offsetend=0;
fp=fopen(fname,"rb");
_fseeki64(fp, offsetzed, SEEK_END);
offsetend = _ftelli64(fp);
fclose(fp);
if(offsetend>MAXLEN) return LENERR;
M.sz=(uint32_t)offsetend;
return M.sz;
}
//load input file into memory
///////////////////////////////////////////////////////////////
uint32_t loadinput(char *fname)
{
FILE *fp;
uint32_t fsize =0;
uint32_t r =0;
clock_t t;
fsize=getfsize(fname);
if(fsize==LENERR) return LENERR;
M.fbuf =(uchar *)malloc((fsize+10) * sizeof(char));
if(M.fbuf==NULL) return MEMERR;
memset(M.fbuf,0,(fsize+10));
fp=fopen(fname,"rb");
if(fp==NULL) return FILERR;
t = clock();
r=(uint32_t)fread(M.fbuf,1,fsize,fp); //// slow fread! ////
t = clock() - t;
M.et = ((double)t)/CLOCKS_PER_SEC; //// elapsed time ////
fclose(fp);
if(r!=fsize){
printf ("fread error: r:%u, fsize:%u\n",r,fsize);
}
return r;
}
// MAIN
///////////////////////////////////////////////////////////////
int main(int argc, char **argv)
{
uint32_t r=0;
if(argc<2){
printf("Syntax: dde filename\n");
return 0;
}
r=loadinput(argv[1]);
if(r!=M.sz){
printf(" %s\n",geterror(r));
free(M.fbuf);
return 0;
}
printf(" fread() took %.1f seconds to read a %.1fGB file.\n", M.et,(float) M.sz/GIG);
printf(" That's %.1fMB per sec. \n",(M.sz/(float)M.et)/MEG);
free(M.fbuf);
return 0;
}
--- End code ---
John Z:
Hi tony74,
Is is possible for you to post the entire Pelles project version using the zip feature under Project?
I tried to compile your code but there are a lot of warnings (15), even though level 1, so I'm guessing
something was not included in your post maybe?
I copied your post directly and the build was for console 64 bit and tried console 32 bit.
Just a few of them:
--- Code: ---C:\Program Files\PellesC\Files\big_open\big_open.c(59): warning #2073: Conversion from 'unsigned int' to 'int' changed value from 4294966996 to -300.
C:\Program Files\PellesC\Files\big_open\big_open.c(84): warning #2018: Undeclared function '_fseeki64' (did you mean: _fseek64?); assuming 'extern' returning 'int'.
C:\Program Files\PellesC\Files\big_open\big_open.c(85): warning #2018: Undeclared function '_ftelli64' (did you mean: _ftell64?); assuming 'extern' returning 'int'.
C:\Program Files\PellesC\Files\big_open\big_open.c(89): warning #2073: Conversion from 'int' to 'unsigned int' changed value from -200 to 4294967096.
--- End code ---
John Z
Update: OK fixed all the warnings and undeclared functions. Largest file I could find was about 2G
Yes - results are disappointing at :
--- Code: ---fread() took 18.8 seconds to read a 2.0GB file.
That's 108.7MB per sec.
--- End code ---
So even worse than your test, but I don't know how fast your VS 2022 code would run on my system to compare.
Perhaps you'll post that executable here.
Not to nit pick :) but your results are only one order of magnitude (approximately) different not "several orders" which would mean 100X or more ;)
John Z:
Hi tony74,
So if you can switch to using CreateFile, ReadFile, CloseHandle [will require windows.h]
or
try switching to _open,_read,_close [will require io.h and fcntl.h for just constant _O_RDONLY or just use 0]
it is much faster, over an order of magnitude faster :) on my system...
--- Code: ---C:\Program Files\PellesC\Files\big_open>big_open EIHH8305.MP4
LoadFile ReadFile took 1.1 seconds
file length 2039467900 read length 2039467900
_read() took 1.0 seconds to read a 2.0GB file.
That's 1951.6MB per sec.
--- End code ---
The same file using fread() on my system was about 18 seconds.
John Z
My guess if VS is doing something sneaky with the fread which is supposed to use fgetc ... but I don't know for sure as I don't have VS 2022, last version I have is VC 5.
I'd still like to try your VS 2022 executable on my system for reference....
tony74:
Ok John, thanks for having a look.
I don't have a clue why you got those errors.
My Pelles compile doesn't show anything...wait while I try Level 2 (Level 1 and 2 are all I show under Compiler Options for Warnings..)...just recompiled with Level 2, got nothing, no errors.
I'm using Pelles version 12.00.2, make sure you're on the same version.
The Pelles project files and exe...in dde.zip.
The VS executable...in VCdde.zip
CLI Sceenshot...fread.jpg
Try an A:B test with both executables, notice the VC exe is about twice the size of the Pelles exe.
Read times will differ on every system, but there's still a big difference with the compilers.
I'd try a compile on gcc just to see what it does, but I don't have it installed on this machine (curious to find out, though).
Tony
Vortex:
Hi Tony,
Can you try the code below based on the simple Win32 file handling functions? ReadFileToMem is a function to load a binary file into memory.
--- Code: ---#include <windows.h>
#include <stdio.h>
// pFileName : pointer fo file name
// pMem : pointer to a pointer indicating the reserved memory to read the file
// pNumOfBytesRead : pointer to a variable that receives the number of the bytes read
BOOL ReadFileToMem(LPSTR pFileName,LPBYTE* pMem,int* pNumOfBytesRead)
{
HANDLE hFile;
DWORD FileSize;
LPVOID hMem;
DWORD nBytesRead;
BOOL t;
hFile=CreateFile(pFileName,GENERIC_READ,0,0,OPEN_EXISTING,0,0);
if(hFile==INVALID_HANDLE_VALUE ){
return 0;
}
FileSize=GetFileSize(hFile,0);
if(!(FileSize)){
return 0;
}
hMem=VirtualAlloc(0,FileSize,MEM_COMMIT,PAGE_READWRITE);
if(!(hMem)){
return 0;
}
*pMem=hMem;
t=ReadFile(hFile,hMem,FileSize,&nBytesRead,0);
if(!(t)){
VirtualFree(hMem,0,MEM_RELEASE);
return 0;
}
*pNumOfBytesRead=nBytesRead;
CloseHandle(hFile);
if(!(t)){
return 0;
}
return 1;
}
int main(void)
{
LPBYTE pMemory;
int BytesRead;
ReadFileToMem("test.txt",&pMemory,&BytesRead);
printf("%s",pMemory);
VirtualFree(pMemory,0,MEM_RELEASE);
return 0;
}
--- End code ---
Navigation
[0] Message Index
[#] Next page
Go to full version