Logstash

Logstash is an open-source data collection system which supports horizontally-scaling data processing pipelines constructed from a wide collection of community-built input, filter, and output plugins. Originally designed as a log collection system, Logstash has grown into a more general centralized data processing engine that can process various event data ingested from custom data sources. Its data processing engine can handle a variety of tasks such as aggregation, anonymization, checksuming, pruning, throttling, translation, etc. The processing pipeline can then be integrated with a variety of third-party analystics (e.g., Elasticsearch), monitoring (Nagios, Zabbix, Ganglia, Graphite) and storage (e.g., HDFS, AWS S3, Google Cloud Storage) engines via Logstash output plugins.