Skip to content

A First-class Native Windows to Redis/Elasticsearch Logstash Agent utilizing Microsoft's LogParser

License

Notifications You must be signed in to change notification settings

TerribleDev/TimberWinR

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

TimberWinR

A Native Windows to Redis/Elasticsearch Logstash Agent which runs as a service.

Development and Roadmap

TimberWinR's development has been and in the short term will continue to be on a hiatus. The project's ownership has been recently transferred, and thus the project will be slow to pick up steam again. Currently a roadmap is being established, and a vision for the project is currently being formulated.

Getting Involved

If you wish to be involved with TimberWinR in an on-going basis file an issue.

Why have TimberWinR?

TimberWinR is a native .NET implementation utilizing Microsoft's LogParser. This means no JVM/JRuby is required, and LogParser does all the heavy lifting. TimberWinR collects the data from LogParser and ships it to Logstash via Redis (or can ship direcly to Elasticsearch)

Release Notes

View Version History

Basics

TimberWinR uses a configuration file to control how the logs are collected, filtered and shipped off.
These are broken down into:

  1. Inputs (Collect data from different sources)
  2. Filters (Are applied to all Inputs)
  3. Outputs (e.g. Redis, Elasticsearch, Stdout, StatsD)

Support

Please use the TimberWinR Google Group for discussion and support:

https://groups.google.com/forum/#!forum/timberwinr

Latest Build:

alt tag

Inputs

The current supported Input format sources are:

  1. Logs (Files, a.k.a Tailing a file)
  2. Tcp (listens on TCP port for JSON messages)
  3. IISW3C(Internet Information Services W3C Format)
  4. WindowsEvents (Windows Event Viewer)
  5. Stdin (Standard Input for Debugging)
  6. W3C(Internet Information Services W3C Advanced/Custom Format)
  7. Udp (listens for UDP on port for JSON messages)
  8. TailFiles (Tails log files efficiently)
  9. Generator (Generate logs for testing New)

Codecs

The current list of supported codecs are:

  1. Multiline

Filters

The current list of supported filters are:

  1. Grok
  2. Mutate
  3. Date
  4. Json
  5. GeoIP

Note that there are now two syntaxes for filters, array and single, if you want more than one filter of the same type then you must use the array tag instead of the singular tag. i.e:

A single Json filter using the single tag (this is only provided as a convienience, the array syntax is preferred)

"Filters": [          
    {
        "json": {
            "type": "Win32-FileLog",
            "condition": "\"[logSource]\" == \"dev\"",
            "source": "Text",
            "add_field": [
                "_index",
                "dev-%{yyyy.MM.dd}"
            ]
        }
    }
]

Multiple Json filters must use the jsonFilters and array syntax, also mutateFilters, grokFilters, dateFilters, geoipFilters.

"Filters": [
    {
        "jsonFilters": [
            {
                "type": "Win32-FileLog",
                "condition": "\"[logSource]\" == \"dev\"",
                "source": "Text",
                "add_field": [
                    "_index",
                    "dev-%{yyyy.MM.dd}"
                ]
            },
            {
                "type": "Win32-FileLog",
                "condition": "\"[logSource]\" == \"sta\"",
                "source": "Text",
                "add_field": [
                    "_index",
                    "sta-%{yyyy.MM.dd}"
                ]
            }
        ]
    }
]

JSON

Since TimberWinR only ships to Redis and Elasticsearch, the format generated by TimberWinR is JSON. All fields referenced by TimberWinR can be represented as a JSON Property or Array.

Outputs

  1. Redis
  2. Elasticsearch
  3. Stdout
  4. File
  5. StatsD

Sample Configuration

TimberWinR reads a JSON configuration file, an example file is shown here:

{
"TimberWinR": {
    "Inputs": {
        "WindowsEvents": [
            {
                "source": "System,Application",
                "binaryFormat": "PRINT",
                "resolveSIDS": true
            }
        ]
    },
    "Filters": [          
        {
            "grok": {
                "condition": "\"[type]\" == \"Win32-Eventlog\"",
                "match": [
                    "Message",
                    ""
                ],                   
                "remove_field": [
                    "ComputerName"                   
                ]
            }
        }
    ],
    "Outputs": {
        "Redis": [
            { 
                "_comment": "Shuffle these hosts",
                "host": [
                   "server1.host.com", 
                   "server2.host.com"
                ]
            }
        ]
    }
}

This configuration:

  1. Inputs: Events from the Windows Event Logs (System, Application)
  2. Filters: Removes the ComputerName field
  3. Sends the event to Redis services (server1.host.com, server2.host.com) in a shuffling manner (balanced).

Installation from Source

If installing from source, you must first install LogParser, then install TimberWinR. Install LogParser from here:

Install LogParser from Microsoft.

After installing, follow the remaining directions here. If you install from Chocolatey then LogParser will automatically be installed first.

Running Interactively

You can run TimberWinR interactively when you are developing your JSON config file, to do so use the following options:

TimberWinR.ServiceHost.exe -configFile:myconfig.json -logLevel:Debug

Automatic Installation via Chocolatey (embedded)

TimbeWinR Chocolatey

alt tag

Manual Installation as a Windows Service

TimberWinR uses TopShelf to install as a service, so all the documentation for installing and configuring the service is show here TopShelf Doc

Specifically the command line options are listed here in Topshelf Command-Line Reference guide.

Install and set to Automatically Start the service:

; Install Service (will autostart on reboot)
TimberWinR.ServiceHost.exe install --autostart
; Start the Service
TimberWinR.ServiceHost.exe start

To Start/Stop the Service from the Command Line

TimberWinR.ServiceHost.exe start
TimberWinR.ServiceHost.exe stop

Alternatively you can use the Services Control Panel.

Usage

TimberWinR.ServiceHost.exe [options]

Options:
-logDir:           Specifies the directory where TimberWinR will write its log file TimberWinR.txt
                   Default is -logDir:"C:\logs"
-logLevel:         Specifies the logging level for TimberWinR
                   Legal Values: Trace|Debug|Info|Warn|Error|Fatal|Off
                   Default is -logDir:Info
-configFile:       Specifies the path to the JSON config file, or directory which contains .json file(s).
                   Default is -configFile:default.json
-diagnosticPort:   Specifies the diagnostic port which can be used to get a health check of the service.
                   Default Port is 5141, A value of 0 will disable it.  Open a browser
                      http://localhost:5141

-configFile

This may be a single .json file or a directory containing .json file(s). If it is a directory, all files will be read and processed, the order in which the files will be processed will match the alphabetical order on disk.

Quickstart Guide

If you really just want to try it out, grab the binary distribution, extract the .zip file into a directory, e.g. C:\TimberWinR

Grab the JSON example file and place it into C:\TimberWinR\default.json. Edit the default.json file and change the Redis instance to match yours, replace 'tstlexiceapp006.mycompany.svc' with the IP or DNS name of the machine running redis. Fire up the collector, enable the verbose debugging to see some Windows Events.

TimberWinR.ServiceHost.Exe -configFile:default.json -logLevel:Debug

You should see

To run it as a service

TimberWinR.ServiceHost.exe install --autostart
TimberWinR.ServiceHost.exe start

Builds

TimberWinR is distributed as an installable package via Chocolatey, and it is dependent on

TimbeWinR Chocolatey

alt tag

Notes

After you install it via Chocolatey, you will have a folder:

C:\Program Files (x86)\TimberWinR

TimberWinR will be configured to read the file default.json located in the folder above. You can edit this file to customize your installation.

Use these commands to Stop/Start the service.

sc stop TimberWinR ; stop the service
sc start TimberWinR; start the service

About

A First-class Native Windows to Redis/Elasticsearch Logstash Agent utilizing Microsoft's LogParser

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages