Pelles C > Bug reports
fread() - slow on 4GB file
John Z:
Hi tony74,
--- Quote from: tony74 on November 10, 2023, 09:26:03 PM ---I'm using Pelles version 12.00.2, make sure you're on the same version.
--- End quote ---
Yes same Pelles version BUT I'm on WIN 11 I don't have my WIN 10 system anymore.
The code Vortex sent is the same as one of my tests id'd as Readfile in my last previous post.
Nice of him to send it :) saving you the coding....
Thanks for the DDE source, and the VS 2022 exe it will be interesting, I'll try shortly.
John Z
Update: OK - I missed checking Micro$oft Extensions <- with = no warnings
The benefit of sending the zip :) guaranteed to be the same...
----------------------------------
Test of VS 2022 executable:
--- Code: ---C:\Program Files\PellesC\Files\big_open>vcdde EIHH8305.MP4
File Size: 2.0GB
Read Time: 1.1sec
Read Rate: 1898.9 MB/sec
--- End code ---
This 1.1 sec matches the test I did using ReadFile method (Like Vortex sent)
--- Code: ---C:\Program Files\PellesC\Files\big_open>big_open EIHH8305.MP4
LoadFile ReadFile took 1.1 seconds
file length 2039467900 read length 2039467900
_read() took 1.0 seconds to read a 2.0GB file.
That's 1951.6MB per sec.
--- End code ---
and
similar to the 3rd method using _open,_read,_close
Both of my tests used your source but substituted just the file handling method.
So I definitely think VS 2022 is optimizing the code and not really using fread() - still just a guess
but somewhat supported by the output results.
-------------------------
Executing the DDE Pelles C version as sent results in:
--- Code: ---C:\Program Files\PellesC\Files\big_open>dde.exe EIHH8305.MP4
File Size: 2.0GB
Read Time: 19.4sec
Read Rate: 105.0 MB/sec
--- End code ---
-------------------------
To prove it one might try using fgetc until eof and see how long it takes instead of using fread....
John Z
tony74:
Thanks John and Vortex,
One might think MS is somehow pulling a fast one here, but I went ahead and installed gcc, just to see if a straight ahead compile would differ significantly from VC, results are in the screengrab.
I attached the resulting exe, as well, in case you want to A:B it.
If you guys have more compilers (I only have the three, now) you could see if the trend continues.
The point here is not to have to substitute methods to gain equal performance, but to find out why at least two other compilers are performing up to seven times faster on this issue with plain old fread()...at least with the test file used on my machine.
I haven't done this yet, but gcc is open source, their version of fread() might shine a light on this.
But at this point, it'd be up to the maintainer/s (Orinius et al?) since Pelles-C isn't open source and we can't compare functions.
And John, I'm inclined to agree that the venerable fread() function has been upgraded, I might root through the gcc code to see if it might now look a lot more like yours and Vortex's, might be time to send Orinius a pull request.
Thanks again John for having a look, and to Vortex for your code!
Tony
John Z:
Hi tony74,
I think the gcc test is significant.
I managed to get VC 6 running and compiled your original DDE as 32 bit
--- Code: ---C:\Program Files (x86)\Microsoft Visual Studio\Files\fread\Release>fread superbig.exe
fread() took 0.8 seconds to read a 1.6GB file.
That's 2058.3MB per sec.
--- End code ---
Of course it is MS but nevertheless it is fast too.
I changed your original to use fgetc in Pelles without loading the memory, so almost reflects raw fgetc speed.
--- Code: --- C:\Program Files\PellesC\Files\big_open>dde EIHH8305.MP4
File Size: 2.0GB
Read Time: 92.9sec
Read Rate: 21.9 MB/sec
--- End code ---
Shows a hypothesis of fgetc is invalid as far as it goes.
Also tried Pelles C version 10 - it performs similarly to version 12
As this is not open source Pelle does not work by pull requests. He will review the forum bug reports and if he can reproduce the reported issue he usually looks further into it. I think you've provided enough that is reproducible. He's a fairly busy guy so patience is warranted.
Can't think of anything more within my capability ....
John Z
MrBcx:
Tony,
I compiled your test using Pelles, MSVC, Mingw/Clang, and Embarcadero.
I used a 2GB mp4 file to test.
Pelle's took 6 seconds and the others took 1 second.
Dell i7, 16gb, ssd, Win10 Pro
tony74:
Thanks John, Vortex and MrBcx.
After all the various compilers we've tested, I think it's safe to say there is nothing 'fishy' or 'upgraded' going on with the fread implementation on any of them, they all behave similarly.
That suggests the Pelles-C implementation may differ and should be looked at.
Until then, we have a workaround, thanks to John and Vortex.
Tony
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version