I'm capturing the output of a CLI to an edit control, but this particular CLI app uses '\r' to show it's progress (I think... I didn't build it).
So on the console, the numbers stay in one place while the progress increments. But, the edit control doesn't handle '\r' like the console, so the progress numbers just run all over the edit window.
In all other respects, everything works fine, it's just when it gets to the last line, where it shows the progress percentage.
Is there any way to handle this that won't impact normal lines in an edit control? Like emulate the way the console handles '\r'?
My test rig:
#include <windows.h>
#include <stdio.h>
int main(int argc, char **argv)
{
for(int i=0; i<=50; i++){
printf("%03d%%\r ",i);
Sleep(200);
}
return 0;
}
My Run/Capture function:
#include <windows.h>
#include <windowsx.h>
#include <stdio.h>
#include <stdlib.h>
#include <direct.h>
#define RBYTES 4096
#define PIPERR -1
#define CPRERR -2
void appendtext(char *text);
//args: HWND of edit control, char* Path of CLI program to run
//////////////////////////////////////////////////////////////////////
int runapp(HWND hread, char *cliapp)
{
BOOL ok = TRUE;
HANDLE hStdInPipeRead = NULL;
HANDLE hStdInPipeWrite = NULL;
HANDLE hStdOutPipeRead = NULL;
HANDLE hStdOutPipeWrite = NULL;
DWORD dwRead = 0;
DWORD dwExitCode = 0;
char lpCmdLine[1024] = {0};
LPSTR lpAppName = NULL;
char buf[RBYTES] = {0};
memset(buf,0,sizeof(buf));
strcpy(lpCmdLine, cliapp);
// Create two pipes.
SECURITY_ATTRIBUTES sa = { sizeof(SECURITY_ATTRIBUTES), NULL, TRUE };
ok = CreatePipe(&hStdInPipeRead, &hStdInPipeWrite, &sa, 0);
if (ok == FALSE) return PIPERR;
ok = CreatePipe(&hStdOutPipeRead, &hStdOutPipeWrite, &sa, 0);
if (ok == FALSE) return PIPERR;
// Create the CLI process.
STARTUPINFO si = {0};
si.cb = sizeof(STARTUPINFO);
si.dwFlags = STARTF_USESTDHANDLES;
si.hStdError = hStdOutPipeWrite;
si.hStdOutput = hStdOutPipeWrite;
si.hStdInput = hStdInPipeRead;
PROCESS_INFORMATION pi = {0};
LPSECURITY_ATTRIBUTES lpProcessAttributes = NULL;
LPSECURITY_ATTRIBUTES lpThreadAttribute = NULL;
BOOL bInheritHandles = TRUE;
DWORD dwCreationFlags = CREATE_NO_WINDOW;
LPVOID lpEnvironment = NULL;
LPSTR lpCurrentDirectory = NULL;
ok = CreateProcess (
lpAppName,
lpCmdLine,
lpProcessAttributes,
lpThreadAttribute,
bInheritHandles,
dwCreationFlags,
lpEnvironment,
lpCurrentDirectory,
&si,
&pi );
if (ok == FALSE) return CPRERR;
// Close pipes we don't need.
CloseHandle(hStdOutPipeWrite);
CloseHandle(hStdInPipeRead);
// main loop for reading output from the cli app.
ok = ReadFile(hStdOutPipeRead, buf, RBYTES, &dwRead, NULL); //initial read
while (ok == TRUE) //read while TRUE
{
buf[dwRead] = '\0';
appendtext(buf);
ok = ReadFile(hStdOutPipeRead, buf, RBYTES, &dwRead, NULL); //next read
}
// Clean up and exit.
CloseHandle(hStdOutPipeRead);
CloseHandle(hStdInPipeWrite);
GetExitCodeProcess(pi.hProcess, &dwExitCode);
return dwExitCode;
}
// append text to edit control
//////////////////////////////////////////////////////////////////////
void appendtext(char *text)
{
int len = GetWindowTextLength(hedit);
SendMessage(hedit, EM_SETSEL , (WPARAM)len, (LPARAM)len);
SendMessage(hedit, EM_REPLACESEL, TRUE , (LPARAM)text);
}
// clear edit control window
//////////////////////////////////////////////////////////////////////
void editclear(HWND hedit)
{
SetWindowText(hedit,0);
}
Hello,
I see 3 solutions:
1-To subclass the EDIT control
2-To use RICHEDIT
3-To use SetDlgItemText
Philippe RIO
Hi Tony74,
As an initial thing you might write the capture to a file so tha you can examine the data and be sure of the format.
It would be easy to write the buffer each time. The use a hex editor to view the binary. You will know for sure the line ending.
So you get the line and if it has a \r set that to 0 to remove it, you know or can get the line length
so you know the character location to check easy if one line at a time. if more than one line (which could be likely given you allocate a 4K buffer, then you'll parse the line into multiple line and process each before sending to the edit control.
When the line has a \r you will you send the text without the \r because it now is end-of-string \0
then after the line is sent to the edit box immediately move the cursor back to the start of the line with
SendMessage (HANDLE of Edit control,EM_SETSEL, 0, -1); // selects all text
then the next entry will overwrite the old value emulating a line return without carriage feed. If there is text above the progress text you'lll replace the 0 with the character count before the progress text.
This will not impact other uses of the edit box because those won't be using the \r strip.
John Z
Other option is to select and delete the prior text before each subsequent text output to the control
perhaps a replace function is solution for that.
first get position in edit control and then in loop replace added text to new.
buffer parsing is needed for processing needed text.
@John Z I think I'll give your suggestion a try. Maybe something like this to look at line endings:
// if EOL is '\r', return 1
int isCREOL(char *s)
{
char *p;
for( p=s; *p; p++ );
--p;
return(*p==0x0d?1:0);
}
And then follow-up with handling the carrot.
Luckily it's just the last line in this case, but it shouldn't matter if subsequent lines follow.
My only concern is reading the pipes fast enough, don't want to miss chars or, worse yet, deadlock because of more code slowing the read down.
Should be ok, thought. Thanks everybody for having a look, I'll post the project when everything comes together (or be back for more expert help if it doesn't...).
( This is a ytdlp front-end, I've used it for about a year but have just put up with letting ytdlp run in it's console, figured it was about time to incorporate it's output into the GUI ).
Thanks.
Well, the good news is it didn't bog-down the pipes. But unless I got it all wrong, it didn't have any effect on the number updatating.
ok = ReadFile(hStdOutPipeRead, buf, RBYTES, &dwRead, NULL); //initial read
while (ok == TRUE) //read while TRUE
{
buf[dwRead] = '\0'; //terminate current read
if(isCREOL(buf)) //if '\r' EOL
{
buf[dwRead-1]='\0'; //dump '\r'
appendtext ( buf ); //send txt to edit
SendMessage( hedit, EM_SETSEL, cr.bytes, -1 ); //cursor to SOL
//Don't update position
plog(buf, cr.bytes, dwRead, fp); //log to ConsoleLog.txt
}
else
{
appendtext(buf); //normal EOL
cr.bytes += dwRead; //update text position
plog(buf, cr.bytes, dwRead, fp); //log to ConsoleLog.txt
}
ok = ReadFile(hStdOutPipeRead, buf, RBYTES, &dwRead, NULL); //next read
}
The logs looked a little 'funny', but the edit control put everything in the right places, here's an excerpt:
buf:L cr.bytes:1 dwRead:1
buf:ine one of test normal text
cr.bytes:30 dwRead:29
buf:L cr.bytes:31 dwRead:1
buf:ine two of test normal text
cr.bytes:60 dwRead:29
buf:L cr.bytes:61 dwRead:1
buf:ine three of test normal text
cr.bytes:92 dwRead:31
buf:L cr.bytes:93 dwRead:1
buf:ine four of test normal text
cr.bytes:123 dwRead:30
buf:L cr.bytes:124 dwRead:1
buf:ine five of test normal text
cr.bytes:154 dwRead:30
buf:0 cr.bytes:155 dwRead:1
buf:00% cr.bytes:155 dwRead:5
buf:0 cr.bytes:156 dwRead:1
buf:01% cr.bytes:156 dwRead:5
buf:0 cr.bytes:157 dwRead:1
buf:02% cr.bytes:157 dwRead:5
buf:0 cr.bytes:158 dwRead:1
buf:03% cr.bytes:158 dwRead:5
buf:0 cr.bytes:159 dwRead:1
buf:04% cr.bytes:159 dwRead:5
buf:0 cr.bytes:160 dwRead:1
buf:05% cr.bytes:160 dwRead:5
buf:0 cr.bytes:161 dwRead:1
buf:06% cr.bytes:161 dwRead:5
buf:0 cr.bytes:162 dwRead:1
buf:07% cr.bytes:162 dwRead:5
buf:0 cr.bytes:163 dwRead:1
buf:08% cr.bytes:163 dwRead:5
buf:0 cr.bytes:164 dwRead:1
buf:09% cr.bytes:164 dwRead:5
buf:0 cr.bytes:165 dwRead:1
buf:10% cr.bytes:165 dwRead:5
buf:0 cr.bytes:166 dwRead:1
buf:11% cr.bytes:166 dwRead:5
buf:0 cr.bytes:167 dwRead:1
buf:12% cr.bytes:167 dwRead:5
You notice it did separate the 'normal' text from the lines ending in '\r', so there's that.
I will say it 'looked good on paper', seemed like it should have worked, but at least in my attempts, it didn't. Could be I'm just not doing it right.
This was close to work with concount.exe output
DWORD nPos = 0;
while (ReadFile(hStdOutPipeRead, buf, RBYTES, &dwRead, NULL))
{
buf[dwRead] = '\0';
//OutputDebugString(buf);
//appendtext(hread, buf, nPos);
SendMessage(hread, EM_SETSEL , (WPARAM)nPos, (LPARAM)-1);
SendMessage(hread, EM_REPLACESEL, TRUE , (LPARAM)buf);
//if (dwRead > 1 && buf[dwRead-1] != '\n') nPos--;
if (dwRead > 1 && buf[dwRead-1] == '\r') nPos--;
else nPos += dwRead;
}
void appendtext(HWND hedit, char *text, DWORD pos)
{
SendMessage(hedit, EM_SETSEL , (WPARAM)pos, (LPARAM)-1);
SendMessage(hedit, EM_REPLACESEL, TRUE , (LPARAM)text);
}
DbgView output[41228] L
[41228] ine one of test normal text
[41228] L
[41228] ine two of test normal text
[41228] L
[41228] ine three of test normal text
[41228] L
[41228] ine four of test normal text
[41228] L
[41228] ine five of test normal text
[41228] 0
[41228] 00%
[41228] 0
[41228] 01%
[41228] 0
[41228] 02%
[41228] 0
[41228] 03%
[41228] 0
[41228] 04%
[41228] 0
[41228] 05%
[41228] 0
[41228] 06%
[41228] 0
[41228] 07%
[41228] 0
[41228] 08%
[41228] 0
[41228] 09%
[41228] 0
[41228] 10%
[41228] 0
[41228] 11%
[41228] 0
[41228] 12%
Quote from: tony74 on December 04, 2024, 02:12:50 AM
Well, the good news is it didn't bog-down the pipes. But unless I got it all wrong, it didn't have any effect on the number updatating.
You notice it did separate the 'normal' text from the lines ending in '\r', so there's that.
I will say it 'looked good on paper', seemed like it should have worked, but at least in my attempts, it didn't. Could be I'm just not doing it right.
So from the picture that was attached it shows that the percentage line is not being returned to the beginning of the line. So it seems the 'cr.bytes' variable is being incremented when \r lines are found. In the line
SendMessage( hedit, EM_SETSEL, cr.bytes, -1 );
so in this line try SendMessage( hedit, EM_SETSEL, cr.bytes-k, -1 );
k=0 until the first \r line is found thereafter k=4 this should backup the the next output to the start of that line.
John
Got your code and I'll look too.
Hi Tony74
Here is code to try, and resulting screen output. Not optimized - but works -
For example I used a static variable so the second run after using 'clear' won't be right but
you can easily fix that.
John Z
@Timo: Thanks for all your work. I couldn't figure out why the buffer was doing that, it looked like an extra one-byte 'read'. It was all there, just split up.
That split-read is kinda critical.
@John Z: I'll run your code when I log off, looks pretty promising from the screen-grab. I guess you saw the 'error of my ways'?
While you guys were working on that, I reworked the concount.exe a little to more accurately mirror the actual ytdlp output. The reason I'm not using ytdlp itself is that it has to go online, grab a file, then output it's stats as it downloads. That's not conducive to repetitive testing. So I 'fake it' instead.
I threw together a function to grab the updating-substrings (ytdlp updates more than just percentage-complete) and send them to a control, which worked out ok. Just wanted to see if it would work.
The 'fly in the ointment' with any of this is not so much the edit control as it is the split buffer reads, maybe you've fixed that, but I'm still dealing with it (pipe buffer, not the edit control).
Anyway, John, I'll grab your code and get back to it.
Thanks again Timo and John, I'll post back with updates.
@Timo: I saw what you did there, good catch. Getting back to the ReadFile split-read issue, are the one-byte reads just an artifact of ReadFile or of the way the pipes are working? Would it be ok to combine the two reads in another buffer? Do you think this is consistent behavior or would it change with different input?
If this is consistent behavior, I could combine the reads before sending them to the edit control, or updating-substrings to a static control, etc.
Thanks again
@John Z: Dude, that was a master stroke, you nailed it!
I added variable-length line support, because the real-world apps that use this need it (including ytdlp) - Line 106 readapp.c.
I also added updating static views in the GUI - Line 108 readapp.c.
Only thing now is I'm still losing the first byte. I know you fixed this, but when I padded out K more than 1 (via cr.len), it all fell apart. keeping it at srlen(buf)+1 works, so far. Maybe you can have a look, you know what you're doing better than I do for edit controls.
But yeah, it's coming right along.
Hi Tony74,
Here is a revision.
Changes:
Static variable is removed, cr structure is global with added parameters, CLEAR now works, Last line looks ok.
Cheers,
John Z
Just one way to do it, not claiming the best :)
In Windows 7, strange problems comes occasionally :(
This was from console3
[youtube] Extracting URL: http://www.youtube.com/watch?v=FNGAvwzgKf4
[youtube] FNGAvwzgKf4: Downloading webpage
[youtube] FNGAvwzgKf4: Downloading ios player api JSON
[youtube] FNGAvwzgKf4: Downloading web creator player API JSON
[youtube] FNGAvwzgKf4: Downloading m3u8 information
[info] FNGAvwzgKf4: Downloading 1 format(s): 623+140
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total Fragments: 2388
[download] Destination: C:\yt-dlp\Video\New fornthite chapte 6 Bosses and Mythics!.f623.mp4
download] 006% of ~ 1.2GiB at 923dowdownload] 030% of ~ 6.0GiB at 923.57KiB/s ETA 13:30 (frag 31/2388)
Hi TimoVJL
Quote from: TimoVJL on December 05, 2024, 03:14:57 PM
In Windows 7, strange problems comes occasionally :(
This was from console3
I ran on test with console3_b on WIN 7 Pro.
Not good. Weird it ran ok once, but the next time badly.
Attached log file, you can see what is happening - - -
EVERY read gets just 1 byte - it could be a problem with
the pipes.
I then ran on WIN 7 Home as well. Same results.
In both cases the first run is good subsequent runs are impacted as the log shows. This basically continues until the
system is rebooted then it will work again for the first time. After that nope.
I think this points to an issue with the pipes where the reboot clears out something - so that's probably where to investigate. Too bad I don't have WIN 10 anymore.....
Tony74 - you might want to attach the code for the concount.exe portion - perhaps the issue originates there.
John Z
@John Z: OK, attached concount.zip. I'm running Win10 pro: build 19045, haven't had any issues. It does sound like something needs to be cleared. I don't have Win7 anymore, so can't help with tracking it down. I'm pretty sure I closed all the handles, the external process closes normally and I can re-run it without restarting the GUI. Without Win7, I'm limited to just guesswork. If it requires a reboot to clear on 7 and runs ok on 10 and 11, there's a system anomaly of some kind. Something 7 isn't letting go of between runs. I'll search for similar behavior online.
Hi Tony74,
Yes it is simple, thought it might be more. Didn't see anything wrong of course.
Thanks,
John
P.S. suggest reducing some of the console sleeps for testing operation at higher speeds too, once happy with the basics.
concount linked with msvcrt.dll leads other problems, as it use larger buffering.
Z console3 crash with it.
with unbuffered version works better.
EDIT:
concount3c_tl.zip is msvcrt.dll version with normal buffer.
idea is to test printf from different CRTs.
function setbuf(stdout, NULL); set minimal buffering.
test_msvcrt.zip have 32-bit msvcrt.lib and minimal runtime code for it.
@John Z: That looks great, have to try it out online with live data and see if it holds up. On my end, this means a rewrite of the front-end, so may take a minute.
I'm attaching the redone concount files. You can change the Sleep duration in main.h of console3c from 0 to whatever (50 seem pretty good, defaults to 200).
@Timo: So concount is having trouble with msvcrt? Too big a buffer? What crt did you replace it with?
concount is about as 'bare-bones' as it comes, a super simple test exe. Surprised the crt buffer size induces a crash.
Still trying to figure out why console3 isn't running right on 7, I can't see anything in the code that seems amiss, at least not yet.
Will post back when I have something.
as yt-dlp needs Window 8+ and use msvcrt.dll and setbuf, Windows 7 can't run it.
perhaps testing using OutputDebugString() and DbgView helps to understand, how it works.
i haven't tested _popen() family for pipes yet.
Getting closer check this log file -
Almost there, hopefully.
John Z
Then I'll try WIN 7 again....
next thing to try with real yt-dlp ...
So I had to, and did, solve the line splitting issue. Weirdly fixing that broke the move back overwrite code so now I have to revise the move back to overwrite the last progress line. It is almost there but I have to replace a fixed 510 char count with a measured char, so it is close. Fixing the splitting took some doing so we will also need to assess speed. I'm sure it can be optimized once it works with my gross code...
John Z
If you want I can test under w11, just tell me what I have to do. I did not followed this post. 8)
Hi Tony74,
Here is the completed display project.
Fixed split lines, a major issue not just a byte two.
Fixed status line after split lines fixed.
Attached project zip, screen shot, and last log file.
I won't say optimized, and there is the ReadLine routine for split lines replicated in used in two places, it could go in a proc and be called, but is not a speed issue and can stay as is.
Have not yet tested at speed. Have not tested in Windows 7 (although I think it may fly)
I suspect several memset 500's will need to be changed to the full buffer size when data is streaming. But ok to test for now.
Cheers,
John
Works in Windows 7 x64 with concount3c stream :)
@John Z:
Dude! You've been busting your a$$ on this thing! You can just pm me with the name you want on the copyright...
Anyway, I haven't done a re-write yet, this is still the original so you can see why I needed the live edit-control stuff.
So, two downloads: the Release Package and the Project files.
The Release folder exceeds the site's limits, so have to get it here: https://file.io/Y2b7eTjHLJVF (https://file.io/Y2b7eTjHLJVF)
kiss.exe has to run in it's Release folder to access yt-dlp and the files yt-dlp uses.
I just overwrite the Release kiss.exe with the latest compile from the Project folder.
I run it from a desktop shortcut, thereafter.
It's not pretty, it's not 'slick', but it works good enough for the kids (who don't want to use command-lines or complicated full-featured front-ends).
It needs a complete refactoring at this point, we'll see.
Oh yeah, 'Kiss' stands for 'Keep It Simple Stupid'.
When you look at the code you'll see the irony in that.
Here's a proj version with the inclusion of runapp, setup with an edit control, everything ready to go.
I Figured since I already knew my way around the code, it'd be faster if I did it... BUT... I haven't run it yet.
I think John should get first crack after all the work he's put in to get it this far.
It's the same thing we've been doing all along, except now it's going for a live stream, not a local test app.
In case anyone blows it up, there's a 'Save' folder included that has the complete original proj. You can refer to that if things 'go south'.
@John Z: Check to make sure I've got the most recent version of runapp in there, there's been a lot of revs.
( and make sure I'm not calling getnsub anywhere, I don't use it in the current build ).
Hi Tony74,
I've got the project you posted, It does look like the latest version of readapp.c. My clean-up was incomplete and there is still some fluff in there.
I have tested the readapp down to 1 millisecond i.e. Sleep(1) and it worked fine on my somewhat under powered laptop.
It fails at sleep 0 most likely because the receive buffer may have more than one line or one line plus a segment , etc. This can be fixed if needed. If this version fails you can 1st try turning off the logging as that consumes time.
As far as testing the entire project I had no clue what yt-dlp was. I've learned at least what it is supposed to do. I don't use youtube and have no idea how to get a video..... maybe once in a while someone sent me a link - maybe.
Anyway, better for others more experienced in this area to test under real conditions to see if it all works as hoped.
Post back if more improvement is needed.
Enjoy,
John Z
@John Z: Thanks for all the effort you've put in, I'll run it and see how it goes.
I don't use youtube either, but I do use Freetube. However, you don't need to bother with it.
Note: youtube defaults to MP4, you can select MP3 if you just want audio track.
Paste any of these links into the Kiss app and click [Download]:
https://www.youtube.com/watch?v=pPStdjuYzSI (CUDA intro - 3:20 min)
https://youtu.be/F2HH7J-Sx80 (Warren Zevon: Lawyers, Guns and Money - 3:30 min) MP3 POP
https://youtu.be/VU0nDHtV-ww (Cal Tjader: Tjader album - 41:29 min) MP3 JAZZ
https://youtu.be/Zup7rmDj6fE (Mambisimo big band: Sofrito Mongo Santamaria - 6:21 min) MP3 LATIN
https://youtu.be/_ovdm2yX4MA Avici, Levels - 3:19 min) MP3 EDM
https://youtu.be/R0gWefJXBuU (Smooth Jazz Guitar - 11:37:04) MP3 CHILL
https://youtu.be/Sq7Qd9PSmR0 (Mussorgsky, Pictures at an Exhibition, Leipzig - 37:47) MP3 CLASS
https://youtu.be/H4YRPdRXKFs (Dennis Ritchy: Memorial, 'Write in C' tune - 3:37)
https://youtu.be/EY6q5dv_B-o (Ken Thompson, Brian Kernigan, Interview 2019 - 1:03:51)
https://youtu.be/rXuswL_3Qto (Mike Abrash, Interview 2022 - 48:40)
https://youtu.be/I845O57ZSy4 (John Carmac, Interview 2022 - 5:15:50)
The music will be in the ..\Kiss\Audio folder.
The videos will be in the ..\Kiss\video folder.
Hi Tony74,
Is it all working?
Hopefully so.
Attached here is a rewrite of readapp.c changing the original loop method.
This version does not need a pre ok = ReadFile before the loop process
This version will handle the double \n\n correctly (your test file has it not sure if real data does)
This version will handle more than one line in the ReadFile buffer at a time i.e. abd\ndef\n correctly
This version eliminates calls to the isCREOL proc
This version writes the log file faster
Top of the file shows 12/10/2024 Version 2 - so clear it is new
Should be, might be, faster overall - but still won't work with sleep 0, although closer.
John Z
Now, I think it is done :)
Thanks John.
Everything works well with static text, but still no-go with dynamic (updating) text with live data.
Might have to shift gears, consoles work, edit controls don't.
In all the examples I've seen, none have handled the kind of updating we tried for, just straight-ahead non-updating text lines, which we can already do.
I've considered two things: a custom, stripped-down edit control that handles '\r' internally and an actual console 'control'.
Don't know how feasible the second would be, but I may look into it. I'm dealing with stable diffusion extraction and video restoration at the moment, a reliable text-output console would be a nice adjunct to several projects.
I'll check back in if anything looks promising.
Hi Tony74,
Disappointing certainly.
It is unlikely that is is just \r processing. There are a number of 'slowing' items. Starting with the send buffer being set to NULL, setbuf(stdout, NULL); set minimal buffering (TimoVJL pointed out), as well as the double pipes read/write buffers set at only 4K. I ran RBYTES set at 32K once and performance was a bit better, but as you pointed out the tester is a one shot stream of static text....
So as to not just be pointing out problems - here is another approach to compensate for the differential between incoming speed and outgoing capability.
stdout buffer needs to be higher, stdout is piped to a shared file using the console command line and ">" . Your program then reads the file and processes the lines. Of course there will be a lag, but stdout continues to stream to the file while the readapp continues to process the lines from top to bottom. I'm guess the input does not go on forever so when there are no more lines for readapp to process the temp file can be deleted assuming there is an end point for one video viewing thing.
So no pipes other than std console pipe ">" which is quite fast. No rush for lost data or buffer overrunning, an expected small lag from the file to edit box display which should not to bad.
Well just a thought -
John Z
Perhaps time to look up this again:
EasyPipe console redirection lib (https://forum.pellesc.de/index.php?topic=7781.msg29026#msg29026)
Long time ago:
Error line piping/modifier for example msvc (https://forum.pellesc.de/index.php?topic=3124.msg11807#msg11807)
@Timo: I think we're already doing what RTconsole did, but without needing the 'stub' program.
We're doing it with 'lines' instead of individual 'chars' and it works as expected with live-data.
What's interesting is that the program does differentiate between line endings, even with live streaming data, but fails to respond to the '\r' line ending, as opposed to the '\n' line ending. Curious, that.
It's possible that since the edit control doesn't have any intrinsic ability to deal with CR and we have to implement that externally, we're just not fast enough to 'catch' it at live-data speeds.
But everything works as expected at local-data speeds using any CLI test program.
Which, speed alone, sort of doesn't make sense in a way. We should be 'hammering' the pipes with local-data faster or as fast as the tcp buffers would.
Of course 'should' doesn't account for 'is', so it remains questionable as to what's going on here.
@John: I've sort of considered something similar to that, earlier. In this case, we're just monitoring the yt-dlp (or any) application's activity.
We don't need it to be 'real-time' in any strict sense, it just has to inform the user than something is actually 'going on', per se.
But as I told Timo, it's still somewhat curious that we can respond to '\n' and yet seem to miss '\r', when monitoring live-data.
The buffers/pipes seem to hold up ok for recognized line endings, but fail for non-intrinsics (as per the edit control).
That's why I'm kind of leaning in that direction at the moment. Processing '\r' externally seems to make the difference, but we could just write everything to a buffer as fast as the pipes could deliver it and process the buffer at our own rate later. We don't have <vectors> in C, but we could cobble up a variable length array to act as one.
That might simplify things considerably and make the finished component reusable regardless of SDK changes in the future.
As you say, just a thought.
Hi Tony74,
As you mention the test case works as expected. I hope you tested the new version too as it should be faster and catch \r 'better'. Having said that it would help to pipe one session to a file so that the 'real data' can be used for testing, and evaluated for any nuance in the stream.
John Z
Haven't tested the newest version yet, will download it now.
Yes, I'll download the live-data stream to a file, so we can be sure were not chasing 'rainbows'.
@John Z: I re-wrote the test GUI to accommodate both local and live inputs and put a SLEEP spinner on the UI. The inputs are pre-loaded so that it can be used without having to access youtube to get urls.
You can see the test results in the images. The tests with readapp.c 'D' version (your latest) looked great locally but didn't fair as well live, it looks like it missed crlf as well as cr.
I went back and tested all the versions of readapp.c back to version 'A', the first one where we had some success. Of them all, version 'B' did the best live, but it still only got 23% of 555MB before it stopped.
All the others, including your latest, stopped well before that.
Should anyone want to run this, you do have to have ffmpeg, ffprobe and yt-dlp on your machine. yt-dlp needs to be in the same directory as the GUI. I don't think the ffmpeg stuff needs to be there, as long as it's in the PATH.
This is as bare-bones as it can be and still run. I'll get around to the live-buffer reads tomorrow when I can carve out some time.
https://github.com/yt-dlp/yt-dlp/releases
https://ffmpeg.org/download.html
Hi Tony74,
Well that is totally disappointing! The only problem expected in the latest rewrite was if a CRLF entry was split across two ReadFile executions putting CR at the end of one line and LF at the start of another. There is one other thing that may have ended up an issue. Once the statistics started streaming, those with just \r, the old versions would not go back to non \r state until the next run. In the last version I thought that it would be beneficial to do that.
So if the raw data shows that once the textual lines are done all following lines should be \r statistic lines that could simplify the code.
Being able to revert to textual lines probably should be removed though.
if you want to try it in appendtext under (!flag) remove this line
cr->first = 0; // future allows switch back to separate lines
When you say 23% before it stopped does that mean a crash or just the edit box became out of synch or something else?
Meanwhile I guess "b" deserves another look to see if some optimizations can be done.
There are some assembly language GURUs in the forum if it can interest them (I have not done assembly for a long time)
John Z :(
Also if after the first lone \r there will never be another CRLF then that would improve the speed as well.
I revised runapp.c to send the raw buffer to a file.
It completed downloading the the ~200MB video portion, then downloaded the ~5MB audio portion, and combined them into a finished file that plays as expected.
ok=ReadFile(hStdOutPipeRead, buf, RBYTES, &dwRead, NULL);
while (ok == TRUE)
{
buf[dwRead] = '\0';
plog(buf,fp);
ok=ReadFile(hStdOutPipeRead, buf, RBYTES, &dwRead, NULL);
}
The buffer-log is unremarkable, all line-endings are as expected when viewed in Hxd.
I'm attaching the buffer-log and the log-version of runapp.c
There's a possibility processing of the read-buffer might have to be done in a worker thread, and in the thread's own buffer.
That would reduce the loop-processing to just a write to the thread-buffer and offload edit-control processing to the thread.
That might mitigate any buffer-hangs and makeup for the speed disparity between input and edit-control process timing.
But at this point, it's just a guess.
Hi Tony74,
This was a good idea!
The runapp.c could gradually be built up to see where the major slow down is.
For example next step could be just add looking for \r, take no action just look for it..
It would be more work too.. anyway I've got the files you attached for a look see.
Will ponder it a while.
Worker thread makes sense too.
Here is a new idea:
Keep the current Multiline edit control
Add a single line edit control
After 9 lines sent to Multiline edit control (all CRLF lines)
all other lines to single line edit control (all CR lines)
This should minimize a lot of process checking.
I'll work up test code. Unless a huge flaw is already obvious to you....
John Z
Well adding a single line control just for the stats is faster but still not fast enough ;(
Looks like it is going to need to be threaded, and use a buffer method either to a file or to a memory buffer.
For buffer method -
Main writes all data as fast as it can to the buffer file or buffer memory, while the program thread reads from the buffer file or buffer memory as fast as it can and updates the display.
Memory method carries some limitations in how much is tied up, File method has some read/write limitations disk space should not be an issue - mostly.
John Z
With memory method (I'd never considered file IO), both threads have to access the buffer at about the same time.
Normally, a mutex or semiphore signals the write-thread to 'wait' while read occurs and the opposite when a write occurs.
In both cases, we're looking at some hold-up in the loop (more on the read-end, if it uses that same buffer to process).
Instead, maybe a linked-list, similar to a 'stack' (but on the heap) might be preferable to either a monolithic buffer or even a string array.
One could use _strdup, which would isolate the Readfile buffer and provide a stand-alone string which only requires a pointer to link to the list. The edit-processor reads -it's- string, processes it and loops back to get the next one.
By that time, ReadFile has 'pushed' several more strings onto our 'stack'.
Edit-process doesn't care about speed, it takes what it takes. If it doesn't find another string, it waits until either a string shows up or it gets an EOF string, after which it exits and a counter frees all the _strdup ptrs.
And except for the 'pushes', all that takes place in the edit-process thread, without any concurrent buffer read-writes.
But, the devil's in the details.
Hi Tony74,
Well no matter which way it is clear that it is not going to be simple considering the incoming data rate.
In my experience there is no conflict with two let's say threads accessing the same block of global memory, no need for a semaphore - one threads pointer reads the buffer starting at the top, while the ReadFile thread is appending data at the bottom. The issue normally addressed is if the read thread out performs the append thread. Second issue is how big can the block of memory be? In the 'old' days one might get away with a 16K block and make a circular buffer, but the process rate(read thread) was faster than the append rate in that case.
Linked-list issue could be the same as faced now, if the linked list is one of logical lines, then line end processing must still occur which is where the current code seems to be failing at speed.
It is looking like a more practical solution might be a custom control that works as desired and handles \r and \n in the stream inherently.
John Z
John Z,
As you say, normally we wouldn't be concerned with accessing the same memory at the same time, but there are two things at play here (and you mentioned both) that require consideration:
1-The pipe-buffers are faster than the edit-control processing.
2-We can't predict incoming data size.
The thread/psuedo-stack idea treats both those issues (in theory).
Then there is the '\r' question and I share your opinion, as previous posts indicate.
So maybe that's the next thing I'll try, a custom, stripped-down edit and test it out.
The timing disparity may or may not still be an issue, but one thing at a time.
Tony