Artikelserie: Kodgranskning del 1 av 3
Insufficient Monitoring
Description
Great focus is placed on implementing software correctly, but knowing that a system performs as intended is equally important. Often this is done by measuring technical aspects such as CPU load, memory pressure, or garbage collector pause times. But to fully understand how a system performs requires a combination of technical measurements and softer observations – such as response times, active sessions, and other significant business events – which requires a design that facilitates efficient monitoring.
Security issue
Exploitation of insufficient monitoring is the bedrock of nearly every major incident.[1] A commonality among vulnerable systems is that the system design doesn't facilitate monitoring – for example logs aren't centralized, alarms are lacking or misplaced, and aggregation of significant business events are missing. As a result, attackers operate without firing alarms and achieve their end goal without being detected.
Remedies
To achieve efficient monitoring, you need to recognize monitoring as a first class citizen of your system design. Sometimes this requires invasive changes, sometimes it only takes minor tweaks to your design, but regardless of how your system looks, applying the advice below is a great start to achieve the end goal of efficient monitoring.
Identifying significant events
Having adequate data is essential when trying to understand how a system performs. This means that all system components must supply information about their state and activity. Often this is confused with debugging, but monitoring isn't about debugging at all – it's really about establishing a complete picture of the system state from a technical and business perspective. On application level, this means you need to log all significant business events (for example account registration, order cancellation, etc) along with information regarding overall performance (for example execution times, number of active sessions, etc). On infrastructure level, you need to log information that shows how the system consumes technical resources – for example, memory pressure, CPU load, and number of active processes. This way, it's possible to correlate significant application events with technical measurements but to do this efficiently, you need to ensure that log data is structured and easy to visualize.
Structuring log data to facilitate visualization
To simplify aggregation and correlation of events, it's important to centralize log data and make sure it's written on a structured format (for example as JSON). Some may argue that structuring data at the source is unnecessary because it could be done by the log management system. This certainly works, but logging unstructured data tends to favor output of "unknown" strings – for example, a serialized version of an exception or user object. This in turn opens up potential security weaknesses – for example sensitive data leakage and 2nd order injection attacks – that you easily avoid by simply placing care on structuring data. Another benefit is the ease of separating sensitive audit data from other log data – an important aspect when viewing logs from a GDPR perspective.
But regardless of which strategy you choose, having structured log data is the key to visualization and efficient monitoring. There are several open source tools that allows you to render structured data. By visualizing technical measurements and other events in the system, you quickly see odd behaviors but what happens if you don’t watch the dashboard?
Using alarms to detect odd behaviors
Having big dashboards with graphs representing everything from CPU load to number of active sessions to average response times is certainly a powerful view of the system. But attackers often take advantage of the fact that nobody is watching at odd hours – for example after office hours or at 3 am in the morning. Therefore, to achieve efficient monitoring, you need to set alarms on individual and aggregated data and make sure the alarms trigger on odd behavior. This way, your system will be monitored all around the clock and make it significantly harder for an attacker to operate without being detected.
References
[1] OWASP top 10 2017, Insufficient Logging & Monitoring