We wants to scale out the usage of logging by normalising raw logging to a normalised schema. The normalisation is targeted to be done in fully automated fashion to ensure changes in the format can be detected over time and normalisation adjusted to accommodate for the changes.

The normalisation needs to be placed within the Azure Monitor data pipeline and requires identification of fields and values, next to transformation to be done using KQL and Regex concepts. These need to be generated based-on a small sample of logging and applied directly within the data pipeline .

What you will achieve together with the team:

• A design is made or adapted to match the use case described enabling the normalisation of logging in a continuous way at scale.
• At least 30% of the schema's identified are normalised by EOY
• Metrics and steering elements to be generated by EOY to ensure steering on the following aspects:
• Identification ratio in % for a stream of logging including not identified events
• Quality % per stream of logging

What we expect from you:

• Statistics and mathematics backgrounds
• Data savviness. You know your way with data and getting all the insights out. Keen on finding the 'real' problem that needs to be solved. Relentless but friendly, following processes;
• Communication and Data Visualization: Knowing Your Audience;
• Data engineering. Implementing data analytics from source to consumption. Knowledge of Azure related pipelines is preferred.
• Understanding of logging in diverse formats and schema's
• Machine Learning algorithms (such as Scikit-learn, Tensorflow, Keras, PyTorch),
• Programming: Python and SQL
• Statistics, experiment design, mathematics.
• Data Wrangling, ETL, Big Data systems
• Data engineering/ML engineering. Implement advanced analytics robustly in various target (cloud) systems. Experience with databases (Azure Monitor/Splunk), containerizing (Docker, Kubernetes)
• Communication and Data Visualization: Knowing Your Audience
• And it would be great if you have some typical domain knowledge too like:
• Internet technology: networks, web applications, HTTP, json, XML;
• Information security: identity and access, cybercrime, cyber security;
• Banking or financial sector;
• Agile-scrum way of working.