title: Best way to aggregate multiple log files from several servers
body: <p>I need a simple way to monitor multiple text log files distributed over a number of HP-UX servers. They are a mix of text and XML log files from several distributed legacy systems. Currently we just ssh to the servers and use <em>tail -f</em> and <em>grep</em>, but that doesn't scale when you have many logs to keep track of.</p>
  
  <p>Since the logs are in different formats and just files in folders (automatically rotated when they reach a certain size) I need to both collect them remotely and parse each one differently.</p>
  
  <p>My initial thought was to make a simple daemon process that I can run on each server using a custom file reader for each file type to parse it into a common format that can be exported over the network via a socket. Another viewer program running locally will connect to these sockets and show the parsed logs in some simple tabbed GUI or aggregated to a console.</p>
  
  <p>What log format should I try to convert to if I am to implement it this way?</p>
  
  <p>Is there some other easier way? Should I attempt to translate the log files to the log4j format to use with <a href="http://logging.apache.org/chainsaw/" rel="noreferrer">Chainsaw</a> or are there better log viewers that can connect to remote sockets? Could I use <a href="http://www.baremetalsoft.com/baretail/" rel="noreferrer">BareTail</a> as suggested in <a href="https://stackoverflow.com/questions/18632/how-to-monitor-a-text-file-in-realtime#18690">another log question</a>? This is not a <a href="https://stackoverflow.com/questions/35292/log-files-in-massively-distributed-systems">massivly distributed system</a> and changing the current logging implementations for all applications to use UDP broadcast or put messages on a JMS queue is not an option.</p>