Handle CLI progress '\r' in an edit control

[John Z]

Hi Tony74,

Is it all working?

Hopefully so.

Attached here is a rewrite of readapp.c changing the original loop method.
This version does not need a pre ok = ReadFile before the loop process
This version will handle the double \n\n correctly (your test file has it not sure if real data does)
This version will handle more than one line in the ReadFile buffer at a time i.e.   abd\ndef\n correctly
This version eliminates calls to the isCREOL proc
This version writes the log file faster
Top of the file shows 12/10/2024 Version 2 - so clear it is new

Should be, might be, faster overall - but still won't work with sleep 0, although closer.

John Z

Now, I think it is done 

[tony74]

Thanks John.

Everything works well with static text, but still no-go with dynamic (updating) text with live data.
Might have to shift gears, consoles work, edit controls don't.

In all the examples I've seen, none have handled the kind of updating we tried for, just straight-ahead non-updating text lines, which we can already do.
I've considered two things: a custom, stripped-down edit control that handles '\r' internally and an actual console 'control'.

Don't know how feasible the second would be, but I may look into it. I'm dealing with stable diffusion extraction and video restoration at the moment, a reliable text-output console would be a nice adjunct to several projects.

I'll check back in if anything looks promising.

[John Z]

Hi Tony74,

Disappointing certainly.

It is unlikely that is is just  \r processing.  There are a number of 'slowing' items.  Starting with the send buffer being set to NULL,  setbuf(stdout, NULL); set minimal buffering (TimoVJL pointed out), as well as the double pipes read/write buffers set at only 4K.  I ran RBYTES set at 32K once and performance was a bit better, but as you pointed out the tester is a one shot stream of static text....

So as to not just be pointing out problems - here is another approach to compensate for the differential between incoming speed and outgoing capability.

stdout buffer needs to be higher, stdout is piped to a shared file using the console command line and ">" .  Your program then reads the file and processes the lines.  Of course there will be a lag, but stdout continues to stream to the file while the readapp continues to process the lines from top to bottom.  I'm guess the input does not go on forever so when there are no more lines for readapp to process the temp file can be deleted assuming there is an end point for one video viewing thing.

So no pipes other than std console pipe ">" which is quite fast.  No rush for lost data or buffer overrunning, an expected small lag from the file to edit box display which should not to bad.

Well just a thought -

John Z

[TimoVJL]

Perhaps time to look up this again:
EasyPipe console redirection lib


Long time ago:
Error line piping/modifier for example msvc

[tony74]

@Timo: I think we're already doing what RTconsole did, but without needing the 'stub' program.
We're doing it with 'lines' instead of individual 'chars' and it works as expected with live-data.
 
What's interesting is that the program does differentiate between line endings, even with live streaming data, but fails to respond to the '\r' line ending, as opposed to the '\n' line ending. Curious, that.

It's possible that since the edit control doesn't have any intrinsic ability to deal with CR and we have to implement that externally, we're just not fast enough to 'catch' it at live-data speeds.

But everything works as expected at local-data speeds using any CLI test program.

Which, speed alone, sort of doesn't make sense in a way. We should be 'hammering' the pipes with local-data faster or as fast as the tcp buffers would.
Of course 'should' doesn't account for 'is', so it remains questionable as to what's going on here.

[tony74]

@John: I've sort of considered something similar to that, earlier. In this case, we're just monitoring the yt-dlp (or any) application's activity.
We don't need it to be 'real-time' in any strict sense, it just has to inform the user than something is actually 'going on', per se.

But as I told Timo, it's still somewhat curious that we can respond to '\n' and yet seem to miss '\r', when monitoring live-data.
The buffers/pipes seem to hold up ok for recognized line endings, but fail for non-intrinsics (as per the edit control).

That's why I'm kind of leaning in that direction at the moment. Processing '\r' externally seems to make the difference, but we could just write everything to a buffer as fast as the pipes could deliver it and process the buffer at our own rate later. We don't have <vectors> in C, but we could cobble up a variable length array to act as one.

That might simplify things considerably and make the finished component reusable regardless of SDK changes in the future.

As you say, just a thought.   

[John Z]

Hi Tony74,

As you mention the test case works as expected. I hope you tested the new version too as it should be faster and catch \r ‘better’.  Having said that it would help to pipe one session to a file so that the ‘real data’ can be used for testing, and evaluated for any nuance in the stream.

John Z

[tony74]

Haven't tested the newest version yet, will download it now.
Yes, I'll download the live-data stream to a file, so we can be sure were not chasing 'rainbows'.

[tony74]

@John Z: I re-wrote the test GUI to accommodate both local and live inputs and put a SLEEP spinner on the UI. The inputs are pre-loaded so that it can be used without having to access youtube to get urls.
You can see the test results in the images. The tests with readapp.c 'D' version (your latest) looked great locally but didn't fair as well live, it looks like it missed crlf as well as cr.

I went back and tested all the versions of readapp.c back to version 'A', the first one where we had some success. Of them all, version 'B' did the best live, but it still only got 23% of 555MB before it stopped.
All the others, including your latest, stopped well before that.

Should anyone want to run this, you do have to have ffmpeg, ffprobe and yt-dlp on your machine. yt-dlp needs to be in the same directory as the GUI. I don't think the ffmpeg stuff needs to be there, as long as it's in the PATH.

This is as bare-bones as it can be and still run. I'll get around to the live-buffer reads tomorrow when I can carve out some time.

https://github.com/yt-dlp/yt-dlp/releases
https://ffmpeg.org/download.html

[John Z]

Hi Tony74,

Well that is totally disappointing!  The only problem expected in the latest rewrite was if a CRLF entry was split across two ReadFile executions putting CR at the end of one line and LF at the start of another.  There is one other thing that may have ended up an issue.  Once the statistics started streaming, those with just \r, the old versions would not go back to non \r state until the next run.   In the last version I thought that it would be beneficial to do that.
So if the raw data shows that once the textual lines are done all following lines should be \r statistic lines that could simplify the code.
 
Being able to revert to textual lines probably should be removed though.
if you want to try it in appendtext under (!flag) remove this line

Code: [Select]

cr->first = 0;       // future allows switch back to separate lines
When you say 23% before it stopped does that mean a crash or just the edit box became out of synch or something else?

Meanwhile I guess "b" deserves another look to see if some optimizations can be done.

There are some assembly language GURUs in the forum if it can interest them (I have not done assembly for a long time)

John Z   

Also if after the first lone \r there will never be another CRLF then that would improve the speed as well.

[tony74]

I revised runapp.c to send the raw buffer to a file.
It completed downloading the the ~200MB video portion, then downloaded the ~5MB audio portion, and combined them into a finished file that plays as expected.

Code: [Select]

    ok=ReadFile(hStdOutPipeRead, buf, RBYTES, &dwRead, NULL);   
    while (ok == TRUE)                                                                         
    {
        buf[dwRead] = '\0';                                                                     
        plog(buf,fp);
        ok=ReadFile(hStdOutPipeRead, buf, RBYTES, &dwRead, NULL); 
    }


The buffer-log is unremarkable, all line-endings are as expected when viewed in Hxd.
I'm attaching the buffer-log and the log-version of runapp.c

There's a possibility processing of the read-buffer might have to be done in a worker thread, and in the thread's own buffer.
That would reduce the loop-processing to just a write to the thread-buffer and offload edit-control processing to the thread.

That might mitigate any buffer-hangs and makeup for the speed disparity between input and edit-control process timing.

But at this point, it's just a guess.
 

[John Z]

Hi Tony74,

This was a good idea! 
The runapp.c could gradually be built up to see where the major slow down is.
For example next step could be just add looking for \r, take no action just look for it..
It would be more work too.. anyway I've got the files you attached for a look see.

Will ponder it a while.

Worker thread makes sense too.

Here is a new idea:
Keep the current Multiline edit control
Add a single line edit control
After 9 lines sent to Multiline edit control (all CRLF lines)
all other lines to single line edit control (all CR lines)

This should minimize a lot of process checking.

I'll work up test code.  Unless a huge flaw is already obvious to you....


John Z

[John Z]

Well adding a single line control just for the stats is faster but still not fast enough ;(

Looks like it is going to need to be threaded, and use a buffer method either to a file or to a memory buffer.

For buffer method -
Main writes all data as fast as it can to the buffer file or buffer memory, while the program thread reads from the buffer file or buffer memory as fast as it can and updates the display. 

Memory method carries some limitations in how much is tied up, File method  has some read/write limitations disk space should not be an issue - mostly.

John Z

[tony74]

With memory method (I'd never considered file IO), both threads have to access the buffer at about the same time.
Normally, a mutex or semiphore signals the write-thread to 'wait' while read occurs and the opposite when a write occurs.

In both cases, we're looking at some hold-up in the loop (more on the read-end, if it uses that same buffer to process).

Instead, maybe a linked-list, similar to a 'stack' (but on the heap) might be preferable to either a monolithic buffer or even a string array.

One could use _strdup, which would isolate the Readfile buffer and provide a stand-alone string which only requires a pointer to link to the list. The edit-processor reads -it's- string, processes it and loops back to get the next one.

By that time, ReadFile has 'pushed' several more strings onto our 'stack'.

Edit-process doesn't care about speed, it takes what it takes. If it doesn't find another string, it waits until either a string shows up or it gets an EOF string, after which it exits and a counter frees all the _strdup ptrs.

And except for the 'pushes', all that takes place in the edit-process thread, without any concurrent buffer read-writes.

But, the devil's in the details.

[John Z]

Hi Tony74,

Well no matter which way it is clear that it is not going to be simple considering the incoming data rate.

 In my experience there is no conflict with two let's say threads accessing the same block of global memory, no need for a semaphore  - one threads pointer reads the buffer starting at the top, while the ReadFile thread is appending data at the bottom. The issue normally addressed is if the read thread out performs the append thread.  Second issue is how big can the block of memory be?  In the 'old' days one might get away with a 16K block and make a circular buffer, but the process rate(read thread) was faster than the append rate in that case.

Linked-list issue could be the same as faced now, if the linked list is one of logical lines, then line end processing must still occur which is where the current code seems to be failing at speed.
 
It is looking like a more practical solution might be a custom control that works as desired and handles \r and \n in the stream inherently.

John Z