Elastic - Overview
When considering data storage in Elasticsearch, it's common to think of relational databases. However, Elasticsearch data storage (e.g., indices) is not exactly the same as storage in a relational database. Nevertheless, analogies can be drawn between Elasticsearch indices and relational databases to enhance your understanding of Elasticsearch.
Think of an Elasticsearch cluster as a database that can contain many indices, similar to how a relational database contains tables. Within each index, you have many documents, which are akin to rows and columns in a relational database table.
RDBMS: Databases => Tables => Columns/Rows
Elasticsearch: Clusters => Indices => Shards => Documents with key-value pairs
Elasticsearch - Shards
????
Elasticsearch - Data Streams
10 key concepts in elastic
Example elastic ECS
"To achieve multi-tenancy, there are a few more steps to take to segregate the logs across multiple indices for the different customers:
If the data is directly shipped to Elasticsearch, you can simply use a special pattern in naming the sensors to help you distinguish which tenant the data belongs to (e.g., *-<customer-Id>).
The name of the sensor will be ingested under the observer.name field of ECS, and a custom ingest pipeline can be then used to read this value and set the appropriate index name for the event.
If the data is first sent to a different Kafka topic for each customer, then you can also use the name of this topic from the event metadata to place the data on the appropriate indices as well."
Kibana - Spaces
"Kibana spaces can be thought of as separate instances of Kibana, where each space contains all the dashboards, detections and alerts, and machine learning jobs for one customer. Those spaces are accessible via role-based access control to achieve the required level of segregation. Granular levels of access control are provided by the document and field-level security features, which can help enforce robust access for compliance with strict security and privacy standards."
data Viewers???
Elastic Security Overview - Log Sources
The diagram below from Elastic displays the Elastic security product (e.g. Elastic Security)
The Elastic Security product 1st starts on the foundational base of Elastic of getting data into Elastic. This is shown by the various types of log sources and that data being sent into Elastic.
It is important to understand these sources because this serves as the instrumentation and telemetry for data into the SIEM as illustrated by the SOC Triad.
Elastic Security - Alerting
The diagram below illustrates the SOC Triad, with typical cybersecurity instrumentation such as NDR and EDR at the base. This instrumentation feeds telemetry in the form of logs to the top of the SOC Triad, which is a specialized logging system called a Security Information & Event Management (SIEM) system. Note that with cloud and other types of data, there are now additional logging sources besides NDR and EDR.
Logging systems, like Elastic Security, then query the ingested log data for various conditions that could indicate a security issue. Elastic calls these alert rules. If the conditions of the alert rule are met, an alert is generated in the logging system. This alert event is a separate log from the initial log or logs ingested that triggered the alert event. Note that there are some SIEM systems that hold data in memory and look for rule alert conditions on ingest rather than by query.
Traditionally, SIEMs referred to the logs coming into the logging system from log sources as 'Base Events', and logs generated from alert rules as 'Correlated Events'. In Elastic, the base event logs are called 'Ancestors', and the alert logs are called 'Parents'.
The diagram below also introduces the concept of Security Orchestration and Response (SOAR) as it relates to SIEM. A SOAR use case for 'SIEM Triage' can help standardize and automate many activities related to dealing with SIEM alerts.
When SOAR interacts with the SIEM, it retrieves the alert events either via API call or receives them via push from the SIEM (i.e., webhook). The example below demonstrates an API call pulling the current SIEM alerts. Once the SIEM alerts are pulled, actions such as Investigate, Contain, Notify, and Document can be taken by the SOAR on the alerts
However, to take these additional actions, the base event or events that triggered the alert are often required. In such cases, an additional process step is necessary. The initial alert log event needs to be parsed to find the base event (in Elastic, the ancestor event) that triggered that alert event. Then, an additional API query needs to be made for the base event details.
Signal ancestor concept.
Log Source - NGFW
The Next generation Firewall (NGFW) includes Firewall logs, Intrusion Prevent System (IPS), and Web Prroxy Logs.
Fortigate divides logs into major types:
Traffic
Event
UTM
UTM stands for Unified Threat Managmeent and has several major subtypes.
IPS
Example Fortgate UTM logs.
GEO IP with beats (mainly an ECS thing)
???Integrate throuugh the agent SRE????
Log Source - Servers
Windows and Linux servers running Elastic agent. These are controlled by policy, but currently its different than the elastic security endpoint agent which is similar to an EDR (Elastic purchased a company called Endgame).
Log Source - Network Monitoring
This includes zeek data from Corelight and AC Hunnter alerting
Log Source - Other Sources
Tenable indetity exposure
File beat CEF module
Type of Data
Log Source - Endpoint Security Agent
Eastic acquired an endpoint security product called Endgame security. Elastic acquired Endgame in 2019 https://en.wikipedia.org/wiki/Endgame,_Inc. . Subsequently Elastic has the Endgame cybersecurity detection engine into its Elastic Agent product and refers to it as Elastic security Agent. The same Elastic Agent that colelcts logs from the host also performs the EDR like security function, and would only need to be enabled by policy.
???example policy picture???
When enabling the Edgame Functunality addiitonal telemtry from the host it is running on comes to the SIEM and there are sveeral different rules that alert on this telemetry. In many cases you will still see the term 'Endgame' referenced in the rules.
Elastic SIEM Overview - Detection Engine
The Elastic Security app in Kibana is employed for overseeing the Detection engine, Cases, and Timeline, in addition to administering hosts running Endpoint Security.
The Detection engine automatically searches for suspicious host and network activity via the following:
Detection rules involve conducting periodic searches on data (Elasticsearch indices) sent from your log sources (e.g., intrumentation telemtry) to identify any suspicious events. Upon discovering a suspicious event, a detection alert is then generated.
Machine learning jobs automatically detect anomalies in host and network events, providing anomaly scores per host that can be utilized in conjunction with detection rules.
KQL Queries and Aggregations which allow free search of the logs.
Timeline: Workspace for investigating alerts and events. Timelines use queries and filters to drill down into events related to a specific incident. Timeline templates are attached to rules and use predefined queries when alerts are investigated. Timelines can be saved and shared with others, as well as attached to Cases.
Detection Rules
Elastic coems with severak built in Detection rules. Create detection rules:
Tunin rules
Elastic built rules
Building custom rules
Rule exceptions
Exceptions: Reduce noise and the number of false positives. Exceptions are associated with rules and prevent alerts when an exception’s conditions are met. Value lists contain source event values that can be used as part of an exception’s conditions. When Elastic Endpoint Security is installed on your hosts, you can add malware exceptions directly to the endpoint from the Security app.
The impact of lists on rules
Machine Learning
Machine learning Note
"Machine learning jobs look back and analyse two weeks of historical data prior to the time they are enabled. After jobs are enabled, they continuously analyse incoming data. When jobs are stopped and restarted within the two week timeframe, previously analysed data is not processed again."
Kibana Info
Aggrehation
KQL
Dashboards and Lens
Cases: An internal system for opening, tracking, and sharing security issues directly in the Security app. Cases can be integrated with external ticketing systems.
Administration: View and manage hosts running Endpoint Security."
"Timeline: Workspace for investigating alerts and events. Timelines use queries and filters to drill down into events related to a specific incident. Timeline templates are attached to rules and use predefined queries when alerts are investigated. Timelines can be saved and shared with others, as well as attached to Cases.
Cases: An internal system for opening, tracking, and sharing security issues directly in the Security app. Cases can be integrated with external ticketing systems.
Administration: View and manage hosts running Endpoint Security."
Creating Cases
Other Notes
standard analyzer
elastic audit logging notes
There are two places where you can enable the collection of Audit logs in your Elastic environment: https://www.elastic.co/guide/en/elasticsearch/reference/8.9/enable-audit-logging.html https://www.elastic.co/guide/en/kibana/current/xpack-security-audit-logging.html
Audit logs are disabled by default in kibana.yml. Use the Kibana audit logs in conjunction with Elasticsearch audit logging to get a holistic view of all security related events. Kibana defers to the Elasticsearch security model for authentication, data index authorization, and features that are driven by cluster-wide privileges. For more information on enabling audit logging in Elasticsearch, refer to Auditing security events.
Kibana audit log functionality now also captures some event coming from the Fleet server activity:
zeek ecs rules
Comments