https://londoncalling.hu/ Recent content on LONDON CALLING Hugo en Wed, 12 Mar 2025 00:00:00 +0000 https://londoncalling.hu/posts/elasticsearch_backup/ Wed, 12 Mar 2025 00:00:00 +0000 https://londoncalling.hu/posts/elasticsearch_backup/ You cannot back up an Elasticsearch cluster by copying its nodes’ data directories. Similarly, there are no officially supported methods for restoring data from a filesystem-level backup. Attempting to recover a cluster using this method may result in operational failures, including reports of data corruption, missing files, or other inconsistencies. Sometimes, the restoration process might appear successful while silently omitting portions of the data. A direct copy of the cluster’s nodes fails to deliver a consistent snapshot of the data at a specific moment. https://londoncalling.hu/posts/elasticsearch_data_streams_aliases_and_logstash/ Mon, 25 Mar 2024 00:00:00 +0000 https://londoncalling.hu/posts/elasticsearch_data_streams_aliases_and_logstash/ TLDRData streams store time series data that is not modified after indexing. Also, data streams streamline index creation, rollover, and life-cycle management. Logstash benefits greatly from these features. Indices in ElasticsearchIn Elasticsearch, indices are the basic unit for storing data. Each index is referred to by its unique name and is backed by one or more shards that split the data behind the scenes. This terminology demonstrates the overloaded meanings of the term “index” in computer science. https://londoncalling.hu/posts/customelasticbeats/ Sun, 26 Dec 2021 00:00:00 +0000 https://londoncalling.hu/posts/customelasticbeats/ Disclaimer: Elastic have deprecated the custom Beat creation as of 7.16. So you won’t be able to create custom Beats in the future using the generators. However, you can use existing Beats as a template. IntroElasticsearch with its Beats ecosystem provides a robust monitoring framework for multitude of use cases. Additionally the libbeats library and complementary tooling allows creating custom Beats for purposes not covered by the official Beats1. This post aims to document my adventures in creating such a custom Beat for my use case and the bits of information I needed to piece together to do it. https://londoncalling.hu/posts/datapipelines/ Sun, 10 May 2020 00:00:00 +0000 https://londoncalling.hu/posts/datapipelines/ How to start?A data pipeline is a chain of services for ingesting, processing, and storing data. Extra components on top allow querying data and presenting the results. For example, Hadoop and its ecosystem provide such services and components. Hadoop went a long way from its original set of services: storage (HDFS) and compute (MapReduce). Its growing number of services and the along growing complexity is hard to keep up with. On top of that, these days more and more deployments are utilizing the cloud.