I have an interactive shell-like program, that outputs a lot of irrelevant error messages, that confuse our customers. Each of these error messages are full-line messages starting with "%".

So the easy way to get rid of them would be something like this:

command | grep -v "^%[^\n]*$"

With command being the shell-like program.

But the problem with that is, that it only outputs anything to the user, after it received a newline. So when the user types anything into the program, there is no feedback of what they typed, until they press enter.

Is there a way to do something like this non-line based? I thought of programming a little something in Python, but on that machine (very slow embedded device) starting Python takes multiple seconds, which is not acceptable.

To add more clarity, I'll explain a bit more:

The program that this is about is not under my control, so I can't change what it does (otherwise I'd just get the program to stop outputting these error messages). The program is used to view or change settings on the device in a Cisco-Shell style. A typical user interaction might look like this:

linux-shell# command
% Error: some meaningless error
fake-cisco-shell# show status
Status report: status ok
% Error: some meaningless error
fake-cisco-shell# exit

Every time the user presses a key, the letter that they typed appears on the screen as they type it, so that they can see what they typed, similar as on a regular bash shell.

Now if I do something like this:

command | grep -v "%[^\n]*\n"

Then the letters typed (and also the input prompt) will only show up after the user pressed enter and thus ended the line, because grep caches lines. So something like this ("<" = inputs from the user, ">" outputs from the program):

> fake-cisco-shell# 
< s
> s
< h
> h
< o
> o
< w
> w
< \n
> \n
> Result of the command\n
> fake-cisco-shell#

Now looks like this:

>(no output)
< s
< h
< o
< w
< \n
> fake-cisco-shell# show\n
> Result of the command\n

So as you see, it does not output each character on it's own when the program sends it, but instead waits for the next \n and only then grep forwards the content.

I am not sure grep can actually do what I want, so I am also open to suggestions to other programs.

  • 1
    Can you print the prompts for the user to stderr instead of stdout? – Eric Renouf Jun 15 '16 at 11:44
  • If I'm reading your question right, it sounds like the program has disabled terminal echo. This would be a rather odd action for a program to take, but you can try to fix it by doing something like cat | command | grep -v .... If you mean something else, can you please clarify what you mean by "it only outputs anything to the user, after it received a newline" – Patrick Jun 15 '16 at 12:26
  • I added some more explanation to what I am trying to do. – Dakkaron Jun 15 '16 at 12:49
  • If you can compile C programs, one way to get what you want is to modify the script program. Script allocates a pty, then runs a command that uses that pty. Everything the user types is sent to the command; everything the command outputs is sent to the user's terminal and to a file. It's straightforward to modify the code to remove the part that writes to the file and to modify the output that is sent to the user's terminal. – Mark Plotnick Jun 15 '16 at 14:53

grep is defined to match, and retain or discard, lines, so it has to read the whole line before doing the match; that can't accomplish what you want.

First you need to verify if command does this echoing of input chars one by one to a pipe. Standard C programs (and sometimes other programs using C stdio) by default use line-buffering when stdout is an 'interactive device' as defined by the implementation, but full-buffering otherwise, and pipe is usually not defined as interactive. Your command is already behaving unusually on a tty, but check it still does so on a pipe with command | dd bs=1.

If that doesn't work, you'll need a program that sets up a pseudo-tty (pty) for command, such as the stripped-down script suggested by Mark Plotnick.

If echoing to pipe does work on the command side, all you need is something on the right side of the pipe that doesn't wait for a full line. A simple C program can do it, something like:

#include <stdio.h>
int main (void) {
  setvbuf (stdout, NULL, _IONBF, 0);
  int c;
    int sel = c != '%';
    for( ; c != '\n' && c != EOF; c = getchar() ) 
      if( sel ) putchar (c);
    if( sel ) putchar ('\n');

Or if you have bash -- which IMLE embedded devices tend NOT to -- try a script containing

while read -rn1 c; do y=true; [[ "$c" == "%" ]] && y=false;
  while ! [[ "$c" == "" ]]; do $y && printf "%c" "$c"; read -rn1 c || exit; done
  $y && printf "\n"; done
  • 3
    Wow! post number 290000!!! Do I get any prize? – dave_thompson_085 Jun 15 '16 at 20:10
  • This almost works! Thanks for the help! The only problem now is that read does not distinguish between spaces and newlines. So it strips only the first word and inserts newlines instead of spaces. Any idea on what I can do there? And congratulations for the nice post number! – Dakkaron Jun 16 '16 at 8:35
  • I found the missing part. IFS needs to be set to IFS=$'\n'. Thanks a lot for the help! – Dakkaron Jun 16 '16 at 8:39

Start of line is done with ^ and end of line with $. The characters \n is not recognized. And end of line is never matched (by default).

So You should use something like grep -v "^%.*$".

  • This is not enough: you need to stop the terminal from doing line buffering. – Charles Stewart Jun 15 '16 at 14:16
  • Ok, this is not really helping. The regex I threw in in the example is not the exact one that I was using, and in reality I didn't use \n. This is just a small mistake in the regex and does not really have anything to do with my problem. – Dakkaron Jun 15 '16 at 15:25

sed has an unbuffered mode, so the following should work as well:

… | sed -u -r '/%[^\n]*\n/d'

Your Answer

