==================================================
Changes from 2019-07-31 00:00:00 +0000 to present.
==================================================
----------------------------------------
version 7.9 at 2021-07-30 18:19:19 +0000
----------------------------------------
Change: f386e7003a442a0419fc62aa30d576312afa9efa
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-30 11:19:09 +0000
Release 7.9
* Fix timeout handling * Allow for connection preferences into the `meta`
settings section * Allow `_id` to be shown by `es-search.pl` * Look in
`~/.config` for `es-utils.yaml` * Started some work on library for handling
aggregation queries
Change: ba56db091ef15fd365100e034b17a8de530333e6
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-14 15:07:21 +0000
WIP: Substantial progress on the multi level aggregation script
Change: 5c2517a1264b4a381b11387125e63dada6fcef2a
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-08 14:50:45 +0000
Adding es-aggregate.pl script
`es-search.pl` has a lot of cruft. Creating `es-aggregate.pl` as a
proof-of-concept starting from a clean slate to see if I can make the code
more readable and more reusable. The idea being to push this code back into
the library to make working with aggregations easier and cleaner.
Change: cfd98ad2e84ee1ea8791debe7f6e1809bd6a9af6
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-08 07:34:21 +0000
Move the aggregation parsing out to it's own sub
This is a long time coming, the handling of aggregations needs a major
cleanup. First step is to break things up into more parseable blocks.
Change: ea453e9190173af63adce15b684dcad13e3b13c5
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-08 07:15:14 +0000
Pass timeout to the object creation
When converting to `App::ElasticSearch::Utilities::Connection`, I
mistakenly removed the pass through of the `--timeout` parameter to the
object creation. This restores the timeout capability.
Change: c8f664cfb09b928831945f82607eab5830faa066
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-08 07:14:50 +0000
Allow HTTP username/password in meta
The previous commit allowed overriding the destination for the connection.
This allows users to override the HTTP basic auth username/password.
```yaml host: es-main.int.example.com meta:
access:
timestamp: t
security:
timestamp: @timestamp
host: es-security.int.example.com
proto: https
http-username: bob
password-exec: ~/bin/get-es-security-password.sh ```
Change: 7c56813974cc069a6214daef8667551a8d31de65
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-08 06:55:14 +0000
Added connection details to the meta section
This update allows the metadata in the `~/.es-utils.yaml` file to include
the `host`, `port`, and `proto`.
Example:
```yaml host: es-main.int.example.com port: 80 meta:
security:
host: es-security.int.example.com
port: 443
proto: https ```
Change: 797199f7af436597ffa67c486d5ea275b825e471
Author: Matthew Feinberg <mattf@intex.com>
Date : 2021-06-24 14:03:07 +0000
For es-search.pl allow _id to be shown as a column when using --show and
display _id when showing full result records.
Change: 6547b9893ad05e7cef565f4a2caf51b321091de0
Author: Andrei Grechkin <andrei.grechkin@booking.com>
Date : 2021-06-11 10:11:34 +0000
Add support for the XDG Base Directory Specification
Support configuration file placed in
`$XDG_CONFIG_HOME/es-utils/config.yaml`
Change: 0f799636f22bf8bff51c25a890f12341f6f7b9d7
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-06-05 14:57:35 +0000
Fix warning in es_request()
When the index is undefined, the code throws a warning.
Change: fa084386a44bbda1a75ee961f8ea6228ab62fe0c
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-03-20 17:17:13 +0000
The GitHub::Meta plugin stopped working and needed configuration
Change: c118b27b05d5e5e75b444b9537362d78e099176f
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2020-10-08 13:42:10 +0000
Fix warnings when we encounter an unallocated shard
----------------------------------------
version 7.8 at 2020-09-16 18:12:23 +0000
----------------------------------------
Change: 7dbb50ff27914898417120f617def09279f8adc0
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2020-09-16 11:12:23 +0000
Better handling of percentages
For `--top` queries, be consistent about handling of the number of decimal
places. Default to showing 3 digits to the right of the decimal, but allow
a user to override it with `--precision`.
Show the percentages in the summary when using `--days` greater than 1.
Update the documentation.
Change: e55ca5e32f64bc10c1def43c4766b93fe1430679
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2020-09-04 13:00:37 +0000
Add percentages to es-search.pl aggs
Whenever an aggregation is used, output the percentage from 0-1 with the
doc count. Additionally, in sub aggs, calculate the percentage of the
parent bucket.
Change: 27a31a79da70aef08ec04586b9496160f2127007
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2020-06-08 09:52:42 +0000
Remove the HOME paths from the lookup
In the previous release, I added a check for the HOME environment variable,
but I neglected to remove the originals from the search path.
----------------------------------------
version 7.7 at 2020-06-08 15:34:40 +0000
----------------------------------------
Change: 5cbe34afdc85b490d2938aa9e37a39f9b34ae6d6
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2020-06-08 08:34:40 +0000
Bugfix release
* Sending "0" value query parameters did not work previously * Only add the
~/.es-utils.* to the config search path if HOME is set * Fix typo in the
documentation
----------------------------------------
version 7.6 at 2020-03-27 22:43:58 +0000
----------------------------------------
Change: 6cb7406e8cf4b112efe4fffd0a0138089d841597
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2020-03-27 15:43:58 +0000
Release Version 7.6
* In `::Query`, the `fields` element was deprecated a long time ago, so
overload it as a source filter. * `es-search.pl`
* Ensure `--missing` and `--exists` can be specified more than once
* Enable source filtering when the user specifies `--show`
* Honor `--max-batch-size` when `--all` is specified
Change: 8ef098984362da46348bec5bbb66de15f57c07a7
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2020-03-09 17:20:26 +0000
Added es-index-blocks.pl to manage index blocks
Change: 7804940e5562e5edc9e4472c7e9dcb3c9eeeb97e
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-12-31 11:05:36 +0000
Ensure error message isn't blank
Change: 6743298fac9c15c55ea719e61eb63d5397e2f60e
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-12-31 11:00:24 +0000
Change aggregation ID and make debug output more useful.
The debug output is now in a better format to paste to a Kibana instance
for debugging.
Change: cc0f7b846e85d6a8397dea24ff4102083990cd3e
Author: Vitaly Shupak <vitaly.shupak@deshaw.com>
Date : 2019-12-09 18:03:10 +0000
use LWP::UserAgent request instead of simple_request
simple_request does not handle authentication responses, which breaks in
environments where ElasticSearch requires Kerberos authentication.
----------------------------------------
version 7.5 at 2019-11-21 21:39:18 +0000
----------------------------------------
Change: eef552859755c3b446755232ef5cdab4fe39713f
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-11-21 13:39:18 +0000
Version 7.5 Release
Change: fdccadaf74838e2876b5d370978ba780a8bbb819
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-11-21 13:33:31 +0000
Include count in the --by ordering
The default sort order for --by does not take into account the count of the
sub buckets. This adds a secondary sort descending on the count of objects
in each bucket.
Change: a27a8b23f9bfd25259539dbae83a54ffc5a40cc7
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-11-21 13:10:17 +0000
Make ::Query objects able to execute
This was an oversight on my part. The `::Query` object now supports
`->execute( $list_of_indexes )` directly. This makes working with the
library easier as folks don't need to read docs on the `es_request()`
function to figure out how to construct a search.
Additionally, `scroll_id`'s are automatically extracted from the result
set. Calls to `->scroll_results()` will allow scrolling without having to
understand the mechanics of the scroll.
----------------------------------------
version 7.4 at 2019-10-20 13:24:50 +0000
----------------------------------------
Change: be1145db74f2394ae9c4f2747332cc03d3f7ca07
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-10-20 09:24:50 +0000
Ready v7.4 Release
* Docs updated * Update tests to handle the new IP range query
Change: feca6053d575b523277743a937a51cb226781d66
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-10-20 09:03:07 +0000
Add Support for --or and IPv6
* Added support to change the default token joiner to OR to es-search.pl *
Added support for IPv6 to the ::QueryString::IP expander * Converted
::QueryString::IP to use a range query
Change: 17e80653b2a90e665e59c545eda7776b2fae7c90
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-10-19 10:46:48 +0000
Update the examples/es-utils.yml file for new options
This file had gotten outdated.
Change: 31c3caf10224615723e8036eae28ac7789a1824d
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-10-09 20:23:57 +0000
Use a better name for fields metadata
Change: f6369b88f311751908dec2addad33d999a452c75
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-10-09 20:19:38 +0000
Some code cleanup for es-search.pl
Change: 29bbf1415b2be39cf79d0281608832ade50565a9
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-10-09 20:18:57 +0000
Allow field metadata sharing with ::Query
This will enable automatic detection of and working with nested fields in
the near term. Possibly other features in the long term.
----------------------------------------
version 7.3 at 2019-09-08 18:32:53 +0000
----------------------------------------
Change: 56b99467e968c2a4dcb6eeb5e49f62ff55213ca2
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-09-08 11:32:53 +0000
Release 7.3 ready to go
Change: cb5e39b38c73c19f02def6851daecb257b228759
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-09-08 11:27:57 +0000
Fix search API for ES < 6.x
The `track_total_hits` was added in ES 6.0.0. Remove that parameter for
versions before 6.0.0.
Change: 5a681c12f47a607293148b44be469a54dd490597
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-09-05 15:42:16 +0000
Add documentation for the new parameters and allow scroll to be undef
When working with aggregations, we need to disable the scroll, so allow
scroll to be undef.
Change: 37fb74c14732e998163a2b46c8960717e69514b7
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-09-04 18:25:02 +0000
Fixes for ::Query to handle parameters better
Drop all the `Maybe` from `::Query`. Make handling non-array parameters in
the `bool` section of the query work. Added the following parameters:
* `search_type` - Set to an `Enum` consistent with viable ES options *
`track_scores` - Can be 'true' or 'false' * `track_total_hits` - Defaults
'true' * `rest_total_hits_as_int` - Defaults 'true' *
`minimum_should_match` - Allow users to control the minimum should
match parameter.
`es-search.pl` remove redundant parameters. Set `--tail` batch size to
`max-batch-size` and check if `size` is less than `max-batch-size` and just
return `size` docs instead of `max-batch-size` docs.
Change: 350c919e7b7128c306bc4b5ef8237a9d2caf7910
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-09-04 17:24:28 +0000
Make ::Query->add_bool() more flexible
Now `add_bool()` supports adding multiple conditions at once. Allow users
to set `search_type` and `track_scores`.
----------------------------------------
version 7.2 at 2019-08-28 22:38:01 +0000
----------------------------------------
Change: 1c04c77be9fd7c0366ce28e2b8abe694babe468f
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-08-28 15:38:01 +0000
Documentation update for 7.2 release
Change: 7469f0aab1a9f93a2ebf4fbe8848a25689ff5cf5
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-08-28 15:35:07 +0000
Enhance es-apply-settings.pl
* New `es_flatten_hash()` function to perform compatible list flattening
to that found in Elastic's documentation. * Replace all instances of
`Hash::Flatten::flatten` with a call to
`es_flatten_hash()` for consistency * Use the `es_flatten_hash()` function
with `Test::Deep`'s
`cmp_details()` and `subhashof()` to check if an index requires a
settings change. * Configure `es-apply-settings.pl` to ignore indices with
protected
aliases. * Allow protected indices to have settings applied with
`--no-skip` * Skip checking the existing index settings with `--no-diff`
Change: bc5ce3686a5d7542fea53d70352f618bd1676a15
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-08-23 15:04:28 +0000
Tweaks for working with ElasticSearch 6.x and 7.x
* Field metadata extraction works on both versions * Filter out invalid
parameters to `_search` when using an older ES
version * Set `es-search.pl` to use the new parameters to track the total
hits
Change: 84142a2fa49ad8e819be1cc189b17a246783c762
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-08-05 11:40:16 +0000
Make skipped aliases in maintenance more visible.
Moving the skipped message from `verbose` to `output` to ensure proper
reporting of these indexes.
Change: 75e684c51aa125e6d8590bf95c3a7c0e3c43933f
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-08-01 17:10:46 +0000
Adapt es-search.pl --bases to use _all
If the user sets a default base in their `~/.es-utils.yaml`, then they'd
only see that base in via `--bases`. This ensures users can see all of the
indices available.
----------------------------------------
version 7.1 at 2019-08-01 19:25:21 +0000
----------------------------------------
Change: eaf5f3ca17bfa2abd653715cc69de098283a2ffc
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-08-01 12:25:21 +0000
Fixes to allow index aliases to inform the maintenance scripts
Change: 3c04257f11032df5e0b10694b77da17d4bd02ab0
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2019-07-31 20:27:52 +0000
Fixes to allow tagging indexes with special aliases
These aliases can be used to inform other processes, or simply as ways to
indicate they have special value.
* `es-alias-manager.pl` now ignores any alias applied to an index that
starts with an '_' or '.' and allows a user to specify an ignored
alias using a YAML null element: `ignored_alias: ~` *
`es-daily-index-maintenance.pl` now allows `--skip-alias pickle` to
skip any index aliased `pickle` from it's activities. Additionally,
any indexes tagged with `_hold` or `_do_not_erase` (in honor of every
whiteboard in the world) will be spared from maintenance.
=================================================
Plus 77 releases after 2019-07-31 00:00:00 +0000.
=================================================