If you’re a data enthusiast or work with data on a regular basis, you’re probably familiar with the Kusto Query Language (KQL) and its power for querying and analyzing data. However, until now, users have had to upload their data to storage before being able to query it with KQL. This process often added unnecessary steps and delays. But fear not, Microsoft has come to the rescue with their new Kql Tools, which allow you to process event streams with KQL queries in real-time!
The traditional method of processing data with KQL involves uploading the data to storage and then querying it. This workflow can be time-consuming and inefficient, especially when working with real-time data. But with Microsoft’s Kql Tools, you can say goodbye to these extra steps. The Kql Tools process event streams as events arrive, eliminating the need for data uploads and providing a seamless real-time processing experience.
In addition to the convenience of real-time processing, the Kql Tools also offer a range of features and capabilities. You can choose from multiple tools including a command-line tool, a Python module, and a PowerShell module. Each tool comes with its own set of documentation and is available for download from the respective links provided in the README.
Supported event sources include not only CSV files but also various Windows and Linux event logs. Windows users can take advantage of WinLog, which encompasses logs seen in EventVwr or log files on disk. Linux users, on the other hand, can leverage Syslog as their event source. For high-volume tracing, the Kql Tools support Etw (Event Tracing for Windows), and there are plans to add support for EBPF (dynamic interception of kernel and user mode functions) in the future.
When it comes to event destinations, the Kql Tools provide multiple options to suit your needs. Real-time output options include JSON format and table format. You can choose to have the results printed to standard output in either format. If you prefer a file output, the Kql Tools allow you to save the results as a JSON file. And if you want to upload the results to Kusto (Azure Data Explorer) or BlobStorage as JSON objects, the Kql Tools have got you covered.
To foster collaboration and innovation, Microsoft welcomes contributions and suggestions to the Kql Tools project. If you’re interested in contributing, you’ll need to agree to a Contributor License Agreement (CLA) that grants Microsoft the rights to use your contribution. The README provides further details on how to contribute and links to the necessary documentation.
In summary, Microsoft’s Kql Tools offer a powerful and efficient solution for real-time processing with KQL. By eliminating the need for data uploads and providing seamless event stream processing, the Kql Tools bring significant time savings and improved data processing capabilities. Whether you’re a data analyst, software engineer, or solution architect, the Kql Tools are definitely worth exploring.
Have you tried out Microsoft’s Kql Tools yet? We would love to hear your thoughts and experiences. Don’t hesitate to share your questions or comments below!
References:
- Kql Tools Documentation: https://github.com/microsoft/kqltools/blob/master/README.md#list-of-tools
- Microsoft Open Source Code of Conduct: https://opensource.microsoft.com/codeofconduct/
- Contributor License Agreement (CLA): https://cla.opensource.microsoft.com
Leave a Reply