Logstash pipeline tester

Code is community submitted, community supported, and recognized as ‘Use At Your Own Risk’.

Short Description

A tool that makes developing logstash pipelines much much easier.

 

Problem solved by this Code Snippet

Oh. The problem... Have you ever tried to write a logstash pipeline? Did you suffer hair loss and splitting migraines? So did I. 

Presenting, logstash pipeline tester which gives you a web interface where you can paste raw logs, send them to the included logstash instance and see the result directly in the interface. The included logstash instance is also configured to automatically reload once it detects a config change.

How to use this Code Snippet

TLDR; 

  1. Don't do this, read the manual or checkout the video below
  2. Still here? Ok then! 🙂
  3. Install docker
  4. Clone the repo
  5. Run these commands in the repo root folder:sudo docker-compose build # Skip sudo if running Windows sudo docker compose up​ # Skip sudo if running WindowsGo to http://localhost:8080 on your PC/Mac
  6. Pick a pipeline and send data
  7. Edit the pipeline
  8. Send data
  9. Rince, repeat

Version info

v1.0.0

Video on how to get started:
https://youtu.be/Q3IQeXWoqLQ

Please note that I accidentally started the interface on port 3000 in the video while the shipped version uses port 8080. It took me roughly 5 hours and more retakes than I can count to make this video so that mistake will be preserved for the internet to laugh at. 🙂

The manual:
https://loadbalancing.se/2020/03/11/logstash-testing-tool/

Code Snippet Meta Information

  1. Version: Check GitHub
  2. Coding Language: NodeJS, Typescript + React

Full Code Snippet

https://github.com/epacke/logstash-pipeline-tester

Updated Feb 06, 2023
Version 3.0
  • A whole bunch of dependencies updated:

    https://github.com/epacke/logstash-pipeline-tester/releases/tag/v1.0.14

  • v1.23 Better logging, error handling and more.

    • Adding Pino as logging library, adding env BACKEND_ENDPOINT to make development easier
    • Adding instructions on how to start a development environment
    • Adding environment variable BACKEND_ENDPOINT in the logstash container to allow configuring logstash output to use the express web app instead of the docker name resolution
    • Defining backend endpoint with environment variable and updating README.md
    • Adding more detailed error message when backend is unable to contact logstash
    • Upgrading React to 18.3.1
    • build(deps-dev): bump rollup from 2.79.1 to 2.79.2 in /pipeline-ui/frontend by @dependabot in #253
    • [Snyk] Security upgrade express from 4.21.0 to 4.21.1 by @epacke in #254
    • [Snyk] Upgrade nodemon from 3.1.4 to 3.1.7 by @epacke in #255
    • build(deps): bump cookie and express in /pipeline-ui/frontend by @dependabot in #256
    • build(deps): bump cross-spawn from 7.0.3 to 7.0.6 in /pipeline-ui/frontend by @dependabot in #258
    • Fixing postcss@^7.0.35 depnendency vulnerability by @epacke in #260
  • v1.0.25 Adding TCP retries when sending data to logstash. This makes the unit tests more robust.