We’re excited to announce the release of Elastic 7.11. This new release brings a number of new features to Elastic enterprise search, observability, and security solutions built on Elastic Stack, including Elasticsearch and Kibana. With this release, you can optimize for cost, performance, insight, and flexibility using both the searchable snapshot feature official and the read-time pattern public beta.
An open beta version of Elastic’s new web crawler for enterprise Search makes it easy to search for content from publicly accessible websites. Elastic Observability has added service health and host detail views to provide more powerful root cause analysis, troubleshooting, and application observability. Elastic security has improved detection and repair processes with the addition of pre-built detection rules, Machine Learning jobs and customizable alarm notifications, and a unified analyst workspace greatly simplifies SecOps processes.
At the same time, we announced a number of key improvements to Elastic Cloud, which is an excellent Elastic hosting service and the only one that includes our solution. Elastic Cloud offers rich support for searchable snapshots, automatic scaling of data, and Machine Learning nodes, as well as improved availability and enhanced search performance with enhanced cross-cluster replication (CCR) and cross-cluster search (CCS). Elastic 7.11 is now available on Elastic Cloud and is the only managed Elasticsearch product that includes all the new features of the latest release. You can also download Elastic Stack and our Cloud Choreography products (Elastic Cloud Enterprise and Elastic Cloud for Kubernetes) for self-managed deployment.
Elastic Stack 7 11
As previously announced, Elastic will be changing license options for Elasticsearch and Kibana in version 7.11. We will be changing the code licensed under the Apache 2 license to a dual license with Elastic + SSPL. In addition, we have made a significant update to the Elastic license to simplify content and make licensing conditions more lenient. Source code for our distribution and all free and paid features is available under Elastic V2, and source code for free core features is available under SSPL V1. This change will have no impact on our customers or the vast majority of our community.
Read on to explore the key highlights of this release. For a full description of the features, please read the various blog posts about the solution and products.
Known issues
The installation of Elasticsearch 7.11.0 cannot begin with an Active Directory or LDAP domain configured. We will release a fix in 7.11.1. Please refer to 7.11.0 Release Notes for more details.
Elastic Stack
More data can be retained and searched by storing searchable snapshots in low-cost object storage services and in a new cold data layer.
With the searchable snapshot function, you can search for snapshot data stored in low-cost object Storage services such as AWS S3, Microsoft Azure Storage, and Google Cloud Storage to make tradeoffs in a better way: Storage cost, search performance, and the depth of insight gained from the data in the Elasticsearch cluster. Searchable snapshots can significantly reduce storage costs; They support a new cold layer feature (now available and available in Elastic Cloud) that can reduce infrastructure costs by up to 50% while minimizing performance impact.
Leveraging read-time patterns in run-time fields gives you the choice of flexibility and cost-effectiveness of read-time patterns, or high-speed performance using write-time patterns, all in one stack.
Runtime fields enable you to define schemas for indexes at query time. This new feature (available in public beta in 7.11) allows you to discover new data and new workflows by dynamically creating patterns, giving you unprecedented flexibility in discovering new insights while choosing a trade-off between cost and performance.
Elasticsearch is known for its extremely fast distributed search and analysis engine because the data is stored in a structured index that is created when the data is written to disk, also known as write time mode. This ordered structure requires understanding and planning how to represent data in Elasticsearch, but the biggest payoff is speed, scale, and relevance. Sometimes, however, you need to explore and examine data in a new way without having to plan the data schema in advance. Create fluid data structures when searching based on read-time patterns using run-time fields. This flexibility reduces the time to first gain insight, but there are trade-offs in overall performance. The Elastic method gives you the flexibility to choose write – and read-time modes depending on your search scenario.
This release initially supported runtime fields in Elasticsearch, and we plan to extend support to Kibana as a whole. Read time mode has been released as a public beta in version 7.11. For more information, please read the special run-time field blog post.
Use the newly released alarm framework to create, manage, and monitor alarms and notifications in the Elastic Stack and external systems.
No matter what you do, knowing when something important is happening in a digital ecosystem is a critical task. From threat detection to application performance warnings to physical asset tracking, receiving timely alerts when significant changes occur is critical to workflows that take action based on data insights. Eight months ago, we launched a public beta version of our new alarm framework to do just that in Elastic Stack, which is now available with the release of version 7.11.
During the open beta, we felt great enthusiasm from the community and noticed a very high adoption rate, which reinforced our commitment to building this framework. The framework is deeply integrated into every solution within Elastic Stack, easy to centrally manage, focused not only on enabling detection, but also on driving action, and integrating Elastic directly into your workflow. The alarm interface is directly integrated into Elastic security and Elastic Observability solutions, and we’ve extended the framework to include third-party alarms that integrate with platforms like PagerDuty, ServiceNow, and Microsoft Teams. By using role-based access control, you can easily manage alarms.
Elastic alerts
The alarm framework helps drive workflow and collaboration
Read Kibana 7.11 and Elasticsearch 7.11 for more details on these features and more.
Elastic Enterprise Search
Make it easy to Search publicly accessible web content with a new web crawler for Elastic App Search.
Allow content search to take many forms. Elastic App Search already allows users to collect content by uploading or pasting JSON and through API terminals. With Elastic Enterprise Search 7.11, users can now gather content through a powerful web crawler that retrits information from publicly accessible websites, making it easy to Search for content in your App Search engine. Like any collection method on App Search, this pattern is inferred at collection time and updated in near real time with a single click. With one click (without writing code), users can customize web crawler rules to specify entry points while exclusion rules instruct web crawlers to avoid certain pages, content, and terms.
Elastic scrawler 1
New web crawler in Elastic App Search
Use Elastic Workplace Search to Search for content in Box, the leading cloud content management system.
One of the early pioneers in cloud-based storage, Box has evolved into a leading cloud content management system with millions of users worldwide. Elastic Enterprise Search now supports Box as a source of content in Workplace Search. Pre-built connectors contain document-level permissions, so users with the right permissions see what they should see and nothing else. The addition of Box expands Workplace Search’s already powerful portfolio of content sources, including Google Drive and Dropbox.
Extend fine-grained access control with Atlassian Jira Cloud and Confluence Cloud document-level permissions in Elastic Workplace Search.
Not all content is created or shared equally. Sensitive and personal content needs to be shared with clearly defined individuals or groups, and when these files can be easily searched, it becomes important to have document-level access to these files. Elastic Workplace Search now includes document level permissions for Atlassian Jira Cloud and Confluence Cloud, so, The same permissions set in these source applications can be inherited by Elastic Workplace Search.
For an update on all the new features of Elastic Enterprise Search, read the 7.11 Elastic Enterprise Search blog post.
Elastic observability
Speed root cause analysis and troubleshooting with new service health views in Elastic APM.
Modern cloud-native applications are typically made up of hundreds of microservices, so the ability to quickly determine the performance and health of individual services is critical to an incident investigation workflow. The new service Overview page centralizes all information about service health, making it easier for developers and SRE to troubleshoot performance problems.
Time series diagrams of service latency, traffic, and error rates provide a summary view of service KPIs over time. Superimposed annotations, such as deployment markers and exception alarms, provide rich context for critical events that can cause behavior changes. The service Overview page uses waveforms to provide a compact view of how each sub-component is trending over time, making it easy to spot unusual changes in behavior and drive investigations. In addition, the service Overview page shows the health of the service broken down by the underlying infrastructure instances (such as containers) where the service is deployed, so that you can relate related issues to those of the underlying infrastructure.
Future releases will introduce more context and views to further simplify and speed up troubleshooting and root cause analysis workflows.
Elastic APM
New service health view in Elastic APM
Troubleshoot infrastructure problems faster with the new host detail view in Elastic Metrics.
Resource heat maps in the Elastic Metrics application can help you identify problems in your infrastructure and narrow the scope for further investigation. With the new view in the Metrics UI, you can easily check the current health of a single host from the profile view. When you click a tile in the heat map, a pop-up window displays key information, including time charts of key host indicators, logs generated by the host, processes running on the host, and host metadata.
Add more context by automatically linking logging and tracing to a new logging library in Elastic Common Schema (ECS).
Associating application logs and traces and navigating between them without losing context is critical to the application troubleshooting workflow. Using the logging library in Elastic Common Schema (ECS), application developers can easily inject trace contexts captured by APM agents into their application logs automatically, allowing the logs to track dependencies needed to simplify analysis.
The ECS Logging library is a plug-in to your favorite logging framework, such as Log4j, allowing developers to write application logs using ECS-compliant JSON formats without changing their native workflows. The ECS logger automatically logs the relevant trace context captured by the APM agent, helping developers create observable applications without additional work.
For a closer look at all the new features, read the Elastic Observability 7.11 blog post.
Elastic safety
Massive access to secure data through the cold layer of searchable snapshots.
Elastic 7.11 brings searchable snapshots into the official release and introduces a cold layer that leverages object storage such as Amazon S3. Security teams now have direct access to vast amounts of data accumulated over many years and can achieve cost savings of up to 50% in support of use cases such as hunting, investigation, compliance, threat analysis, forensic analysis, adversary simulation and much more. The extended retention of secure data ensures that practitioners have access to the data they need even in the face of very long dwell times. Cloud platforms and applications, IDS/IPS, DNS, wired data, host activity, observability data, MDM, IoT, OT, and many other data sources that are often too costly to be incorporated into daily operations can maintain operational readiness on a larger scale. The security team can even automatically detect data that would otherwise be archived or discarded.
Attacks against cloud applications and hosts are detected through preset Machine Learning jobs and detection rules supporting MITER sub-technologies.
Elastic Security 7.11 helps secure modern enterprise stacks with updated Machine Learning jobs and new detection rules. These detection rules, developed by Elastic, support the MITRE ATT&CK® sub-technology, improve understanding of how attacks can be launched within an organization and improve consistency with the ATT&CK® framework.
Detection rules pre-built for cloud applications automatically discover technologies and behaviors associated with attacks against SaaS technologies such as Google Workspace, Microsoft 365, and Okta, complemicating existing Elastic protection for IaaS technologies. Prebuilt security analysis content for Windows and Linux environments focuses on detecting a wide range of attacker activity, focusing on persistence, promoted permissions, and lateral movement.
Elastic security researchers recently detailed an advanced method for discovering domain generation algorithms (DGA) to detect SUNBURST and other attacks. For more information, please read the Machine Learning blog for DGA detection.
Streamline SOC workflows and shorten response times with improved alarm management, a broader set of rules actions, updated timeline workspaces, and navigation that is easy for everyone to use.
Streamlined alarm management allows analysts to quickly respond to threats. Customizable alarm notifications can provide critical context to third-party workflow tools, such as Slack and ServiceNow, reducing hectic analysis time and speeding alarm triage. Analysts can now attach alarms directly to cases to coordinate responders and centralize the management of closely related information. An extended set of rules operations enhances integration with Jira, ServiceNow, and IBM to improve SOC efficiency.
The new timeline workspace facilitates effective threat hunting, alarm triage, and investigation. You can view key information on dedicated tabs, view events in a full-screen view, and pay close attention to surrounding events while viewing event details. In addition, you can smoothly switch between multiple timelines and quickly update them by dragging and dropping fields.
Elastic Security
A refreshed timeline workspace in version 7.11
Elastic Security 7.11 offers frictionless navigation through enhanced keyboard navigation and screen reader support for users who want frictionless access in their everyday tools. We want these users and other power users (hello, hotkeys! Will like it too.
For full details, please read Elastic Security 7.11.
Elastic Cloud
Double storage density or save infrastructure costs with a new cold layer powered by searchable snapshots and object storage such as S3.
Elastic Cloud takes advantage of new searchable snapshots and cold layers to provide an easy-to-use cold layer slider in the Elastic Cloud console. This control provides an easy way to implement cost-effective data retention strategies and hold more data for longer at the same cost.
With a thermo-cooling architecture, you can control how and where sequential data is stored over time, maximizing the cost effectiveness of data storage. You can select a hot layer or a warm layer for recent, highly relevant, and frequently accessed data. With index lifecycle management, you can automatically move inactive read-only data to the cold tier, taking advantage of cost-effective and persistent object Storage (including Amazon S3, Azure Blob Storage, or Google Cloud Storage).
Deployment templates help you get your deployment up and running quickly, and templates in both existing and new deployments can use warm and cold sliders to manage your data policies. Elastic Cloud is a straightforward way to take advantage of searchable snapshots and cold layers.
Expand and streamline cloud operations with automatic scaling of data and Machine Learning nodes.
Free yourself from constantly monitoring and managing resources and let your cluster grow automatically to meet your capacity needs. The flexibility of automatically scaling data and Machine Learning nodes will soon be available in Elastic Cloud. Automatic scaling of data nodes ensures that you always have the capacity you need, even as the amount of data collected and indexed grows. Automatic ML-based scaling allows you to seamlessly add Machine Learning functionality without hitting memory limits for a seamless experience. We plan to continue extending auto scaling in future releases to support more metrics and use cases. When extending Elastic use cases, be aware that automatic scaling allows you to seamlessly scale the infrastructure.
With enhanced Cross-cluster Replication (CCR) and Cross-cluster Search (CCS), data is replicated and searched across regions and cloud service providers for higher availability and better search performance.
Enhanced cross-cluster replication and cross-cluster search capabilities between different regions and cloud service providers are now available. Customers are free to search and replicate their data across clusters in different regions and among cloud service providers.
With CCS, you can search in any number of clusters of your choice, allowing you to visualize all your data in a clear, easy-to-understand view. This breaks down data silos and connects all data as if it were in a cluster, making it easier for customers to come up with new insights. CCR allows you to replicate and store copies of data between clusters, process search requests even if you encounter data center outages, and create centralized data from a location in two or more clusters for easy analysis and aggregation locally, and bring the data closer to end users to reduce latency.
Making these capabilities available across regions and cloud service providers requires significant innovation to address issues such as security, trust, and network topology, but we have done the hard work of making it easy for our customers to leverage these capabilities in their deployments.
Elastic cloud
The enhancements in 7.11 search and replicate across clusters
For full details on Elastic Cloud news, read the New features in Elastic Cloud 7.11.
I don’t have space to list them all…
More functions released, please continue to pay attention. For more details on all the new features we’ve added in version 7.11, please check out the following blog posts about the solution and products:
Elastic Stack
Elasticsearch 7.11.0 hit
Kibana 7.11.0 hit the market
Elastic Solutions
Elastic enterprise Search 7.11.0 blockbuster release
Elastic observability 7.11.0
Elastic Security 7.11.0 blockbuster release
Elastic Cloud
New features in Elastic Cloud 7.11
Start a free trial on Elastic Cloud