Files
logseq/pages/hls__icse2023-paper9_1666614163725_0.md
T
2025-06-02 17:15:13 +02:00

553 lines
21 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
file:: [icse2023-paper9_1666614163725_0.pdf](../assets/icse2023-paper9_1666614163725_0.pdf)
file-path:: ../assets/icse2023-paper9_1666614163725_0.pdf
- metric selection, remains manual to a large extent
ls-type:: annotation
hl-page:: 1
hl-color:: purple
id:: 635edb08-6126-4cd8-9e7b-abe26aac629b
- we develop a metric recommendation service for online systems, which can automate the metrics selection practice and greatly ease the burden in managing an online system.
ls-type:: annotation
hl-page:: 1
hl-color:: purple
id:: 635edb16-59e2-4c01-aa40-66b5557d59b7
- anomaly detectio
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edb1e-7930-4318-8d8e-a5c88b6c4b79
- fault diagnosis
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edb25-704c-422b-bfd3-0e69092375ef
- metric recommendation mechanisms
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edb2c-e1cd-4179-b42e-e32e5ce8551a
- selecting metrics for anomaly detection
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edb3f-85f5-4588-8429-e485f6ac4eab
- retrieving metrics for faults diagnosis,
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edb46-f12d-499b-a37a-ae0eb2a288ce
- becomes increasingly important yet difficult.
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edb81-f6e9-4cf5-abb4-c0dceaa1b377
- Metrics are time series data that record the real-time state of a system.
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edb93-96aa-4dd4-93de-1b9fa00c0b4f
- metrics selection usually needs to be performed on demand to accelerate and enhance the management process
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edb9f-4a7f-4be1-9abe-d0b8c7038528
- when a software engineer wants to confirm whether the system is working properly, he/she shall first select some metrics and then perform anomaly detection on them
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edbb6-6d3e-4322-96b3-70e27e3dcd93
- Another example is, when a software engineer is diagnosing a fault, he/she usually needs to pick out relevant metrics to better understand what is going on.
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edca7-76e8-44c5-82fd-07b54743b8dd
- their effectiveness is of critical importance
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edcad-de36-4ae4-8f8c-b0d47bbf3085
- Typically, software engineers create and customize KPI dashboards to aid their daily operations
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edd26-35d9-49c5-b037-97c60e6f1902
-
- PI dashboard is a group of selected KPIs that are organized together in a Web-hosted site.
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edd55-67ea-466e-b350-47680c4e64df
- over 500TB a day.
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635edd8b-eff0-441f-b12a-0c6424cc558f
- equiring software engineers to manually inspect all these data to find useful metrics is impractical.
ls-type:: annotation
hl-page:: 1
hl-color:: purple
id:: 635edd99-4dfc-4770-8832-fb87fed32dcb
- the usefulness of some metrics might change as the system evolves.
ls-type:: annotation
hl-page:: 1
hl-color:: purple
id:: 635eddd1-bd13-4122-b20a-dbde6483f90d
- software engineers to frequently update their KPI dashboards, which can be especially exhausting.
ls-type:: annotation
hl-page:: 1
hl-color:: purple
id:: 635eddd9-ad48-444a-8109-895c62ce1b6f
- Understanding which metrics are more useful can be challenging.
ls-type:: annotation
hl-page:: 1
hl-color:: purple
id:: 635eddff-6604-4a55-8a64-312cf2bc70ac
- it is unrealistic to assume that every engineer is experienced enough to select metrics manually
ls-type:: annotation
hl-page:: 1
hl-color:: purple
id:: 635ede0e-11b0-4746-8b0b-06e6a02a2448
- rigger the service and set their KPI dashboards according to the recommendation.
ls-type:: annotation
hl-page:: 1
hl-color:: purple
id:: 635ede19-15e1-4881-8f71-d3d98c0c5128
- In this paper, we focus on metric recommendation for failure management, including anomaly detection and fault diagnosis.
ls-type:: annotation
hl-page:: 1
hl-color:: purple
id:: 635ede2b-19f7-4dc5-af4e-64e20f521c4c
- [30] I. Baradari, M. Shoar, N. Nezafati, and M. Motadel, “A new approach for kpi ranking and selection in itil processes: Using simultaneous evaluation of criteria and alternatives (seca),” Journal of Industrial Engineering and Management Studies, vol. 8, no. 1, pp. 152179, 2021.
ls-type:: annotation
hl-page:: 11
hl-color:: green
id:: 635ede56-7260-4486-b3b4-a38f84135b2f
- 30] I. Baradari, M. Shoar, N. Nezafati, and M. Motadel, “A new approach for kpi ranking and selection in itil processes: Using simultaneous evaluation of criteria and alternatives (seca),” Journal of Industrial Engineering and Management Studies, vol. 8, no. 1, pp. 152179, 2021
ls-type:: annotation
hl-page:: 11
hl-color:: green
id:: 635ede5b-0f4a-446c-a664-869b4f516f10
- . Baradari, M. Shoar, N. Nezafati, and M. Motadel, “A new approach for kpi ranking and selection in itil processes: Using simultaneous evaluation of criteria and alternatives (seca),” Journal of Industrial Engineering and Management Studies, vol. 8, no. 1, pp. 152179, 2021
ls-type:: annotation
hl-page:: 11
hl-color:: green
id:: 635ede60-f316-4105-aa0d-cd67e1f7fef3
- atency, traffic pressure, error count and saturation as KPIs.
ls-type:: annotation
hl-page:: 1
hl-color:: green
id:: 635ede84-bf11-45d1-8ea7-3a3138d8274b
- recommendations are static
ls-type:: annotation
hl-page:: 2
hl-color:: purple
id:: 635ede9e-ae82-4419-8f38-9434dd4c2084
- require continuous labelling of the anomaly time to supervisedly learn the correlation between metrics and real faults, which is still very labor-intensive and difficult to apply in practice.
ls-type:: annotation
hl-page:: 2
hl-color:: green
id:: 635edef5-50be-4055-b226-367d68808acf
- this paper proposes to automatically recommend metrics against actual conditions with minimal human effort.
ls-type:: annotation
hl-page:: 2
hl-color:: green
id:: 635edf0a-9d82-4d81-9e7a-710eba058112
- anomaly detection and fault diagnosis
ls-type:: annotation
hl-page:: 2
hl-color:: purple
id:: 635edf9e-8117-45de-9587-6ab7d166bd66
hl-stamp:: 1667162016382
- : i) How to accommodate the need of different failure management tasks when performing metric recommendation?
ls-type:: annotation
hl-page:: 2
hl-color:: purple
id:: 635edfb5-19b7-4927-add1-fb11d172a3f7
hl-stamp:: 1667162040311
- ii) How to strike a balance between the effectiveness and the reliance on human effort during the recommendation process?
ls-type:: annotation
hl-page:: 2
hl-color:: purple
id:: 635edfc2-b344-465a-acd8-36b94519d4d2
- metric selection for anomaly detectio
ls-type:: annotation
hl-page:: 2
hl-color:: green
id:: 635edfe8-5c3d-4efc-b606-a78f3363fe55
- unmonitored anomaly-related metric retrieval for fault diagnosis
ls-type:: annotation
hl-page:: 2
hl-color:: green
id:: 635edff3-291a-4560-9a98-1cfbc5a903a8
- for online systems based on graph learning
ls-type:: annotation
hl-page:: 2
hl-color:: green
id:: 635ee00c-51fb-4f7c-a49d-bd7fa7351963
- automate metric selection for anomaly detection and metric retrieval for fault diagnosis.
ls-type:: annotation
hl-page:: 2
hl-color:: purple
id:: 635ee01d-2fac-4a23-8b2c-4d80a5a54a6d
- Software engineers usually use it to inspect the state of a system, or configure the inputs for some intelligent anomaly detection and fault diagnosis algorithms
ls-type:: annotation
hl-page:: 2
hl-color:: green
id:: 635ee049-5c23-4749-bab9-623370c6eea4
- KPIs in the same panel are metrics of the same type but with different attributes. F
ls-type:: annotation
hl-page:: 2
hl-color:: green
id:: 635ee058-8adf-4618-9340-bc1a1fcdfb47
- However, which metric types and which attributes should be selected as KPIs for display and anomaly detection still remain a question.
ls-type:: annotation
hl-page:: 2
hl-color:: green
id:: 635ee074-3978-4c03-8bc6-9e266e4c2dfe
- in this paper, we provide a mechanism to automatically recommend KPIs for anomaly detection on the basis of the historical data of different metrics.
ls-type:: annotation
hl-page:: 2
hl-color:: green
id:: 635ee081-7ef5-4ff6-8c16-a09807835a5f
- unmonitored metrics
ls-type:: annotation
hl-page:: 2
hl-color:: purple
id:: 635ee0d2-9b8a-414a-8a30-3a7cf9db4c6a
- Although they are neglected in a KPI dashboard, they may still be useful for the diagnosis of some specific faults.
ls-type:: annotation
hl-page:: 2
hl-color:: purple
id:: 635ee0db-6275-4964-bf4e-d2b898f07dc7
- esource-related metrics
ls-type:: annotation
hl-page:: 2
hl-color:: purple
id:: 635ee0f7-1d19-4c59-8a67-25d21b796bc7
- component-specific metrics
ls-type:: annotation
hl-page:: 2
hl-color:: purple
id:: 635ee0fb-13b7-44d9-a61d-542e6c1381b7
- business-level metrics.
ls-type:: annotation
hl-page:: 2
hl-color:: purple
id:: 635ee0fe-1984-4d7e-9a75-bbb46f26c14c
- develop
ls-type:: annotation
hl-page:: 2
hl-color:: red
id:: 635ee154-b095-4b2b-bb67-d14741b8ef19
- equiring software engineers to manually select the most appropriate metrics to monitor every component is labor-intensive and error-prone.
ls-type:: annotation
hl-page:: 2
hl-color:: purple
id:: 635ee1b1-3ee3-48cc-9d2f-3a8e781ef8d5
- the configuration of some attributes has a great impact on the effectiveness of the selected metrics.
ls-type:: annotation
hl-page:: 3
hl-color:: purple
id:: 635ee208-16b4-491e-b6d6-98dd521576c2
- C. Using Metrics for Failure Management
ls-type:: annotation
hl-page:: 3
hl-color:: purple
id:: 635ee219-9550-425d-b0f9-89ff957d511d
- Anomaly Detection
ls-type:: annotation
hl-page:: 3
hl-color:: purple
id:: 635ee30d-4573-4246-8535-090fb007b673
- selecting useful metrics to assess the symptom of a fault needs to be conducted first
ls-type:: annotation
hl-page:: 3
hl-color:: green
id:: 635ee328-943c-406c-bbbd-de0d4da90474
- intelligent anomaly detection
ls-type:: annotation
hl-page:: 3
hl-color:: green
id:: 635ee456-82bb-4a80-8e10-99a250bc2e18
- fault diagnosis
ls-type:: annotation
hl-page:: 3
hl-color:: green
id:: 635ee459-2d0a-4d72-86e2-8bcd5316a14d
- It can be observed that more than three quarters of KPI dashboards have less than 25 metrics.
ls-type:: annotation
hl-page:: 3
hl-color:: green
id:: 635ee48d-203a-40e8-9723-ede05a6a4159
- many metrics are noisy or contain redundant information.
ls-type:: annotation
hl-page:: 3
hl-color:: green
id:: 635ee4c0-ddcc-48a2-aa82-b00ab11e27ce
- uch a practice is not automatic, and there is still no a unified standard for selecting KPIs, rendering it difficult to release the full potential of these anomaly detection methods in different systems.
ls-type:: annotation
hl-page:: 3
hl-color:: purple
id:: 635ee4e1-92fa-470d-8879-db295d39eae7
hl-stamp:: 1667251293778
- ngineers tend to use more metrics for analysis.
ls-type:: annotation
hl-page:: 4
hl-color:: green
id:: 635ee522-7f76-4ee3-9a0e-00f6639dcb88
- Since more available metrics can provide more clues, it is better to take more metrics into consideration when developing an intelligent fault diagnosis method.
ls-type:: annotation
hl-page:: 4
hl-color:: green
id:: 635ee5be-2b59-4c34-958d-4875381701ef
- This paper proposes to recommend metrics against actual conditions, and targets at making full use of metrics that can be collected, not limited to KPIs.
ls-type:: annotation
hl-page:: 4
hl-color:: yellow
id:: 635ee5f7-93af-41f1-8d05-9bca0d75025f
- Metric Selection for Anomaly Detection.
ls-type:: annotation
hl-page:: 4
hl-color:: green
id:: 635ee61c-6414-4ac3-83f2-0df2fcbdfe75
- For each entity, which metrics can better reveal the entity state and should be utilized for anomaly detection?”
ls-type:: annotation
hl-page:: 4
hl-color:: green
id:: 635ee62c-4cc7-4fc7-9eb2-ab3f558c2a60
- ain a recommended subset of M, and the recommended subset should comply with the fault revealing goal, noise removal goal and pattern diversity goal,
ls-type:: annotation
hl-page:: 4
hl-color:: purple
id:: 635ee649-b9d1-46df-aa03-ba55252077f3
- Unmonitored Anomaly-Related Metric Retrieval for Fault Diagnosis.
ls-type:: annotation
hl-page:: 4
hl-color:: green
id:: 635ee64e-bdf1-40b4-944d-ad97029a5235
- “Given an anomalous KPI, which unmonitored metrics may expose some clues about the anomaly and need to be retrieved for analysis?”.
ls-type:: annotation
hl-page:: 4
hl-color:: purple
id:: 635ee673-e622-4890-acb1-6710d1b10e74
- Some metrics are noisy and using them for alerting might give rise to a flood of false alerts.
ls-type:: annotation
hl-page:: 4
hl-color:: green
id:: 635ee6d0-b0c7-4bdf-9dd6-7048acf28fc5
- some metrics might correlate with each other.
ls-type:: annotation
hl-page:: 4
hl-color:: green
id:: 635ee6ea-3df1-40c2-bd5e-1a6cf0f65ab6
- redundancy
ls-type:: annotation
hl-page:: 4
hl-color:: purple
id:: 635ee706-faef-4fff-bc32-af9f2b4fe1d5
- pattern diversity
ls-type:: annotation
hl-page:: 4
hl-color:: purple
id:: 635ee70d-8ab9-42db-831c-497f3bcc102d
- we propose a metric recommendation service including offline metric selection for anomaly detection and online unmonitored anomaly-related metric retrieval for fault diagnosis.
ls-type:: annotation
hl-page:: 5
hl-color:: purple
id:: 635ee87e-a503-4548-8663-ea43f8e18ab5
- dopt anomaly detection techniques to identify anomalies in these metrics
ls-type:: annotation
hl-page:: 5
hl-color:: green
id:: 635ee8aa-f201-4ecc-8247-6a34e5145510
- When there exist anomalies, the software engineers should diagnose and repair the system faults based on the anomalous metrics. However, it is difficult to ensure that metrics can be selected completely and accurately. The engineers also need a service to assist them in automatically selecting metrics that can characterize the availability of their systems. Hence, we first design a metric selection method for monitoring and detecting anomalies. In addition, the unmonitored metrics can also assist in revealing and diagnosing anomalies. There are far more unmonitored metrics than monitored metrics in an industrial environment. Adopting only monitored metrics may make it difficult to diagnose a fault. Therefore, we design an anomaly-related unmonitored metric retrieval method for fault diagnosis.
ls-type:: annotation
hl-page:: 5
hl-color:: red
id:: 635ee8c8-9b94-4a0f-912b-519ee19122dd
- we adopt the metric selection service to recommend KPIs which should be monitored.
ls-type:: annotation
hl-page:: 5
hl-color:: red
id:: 635ee94e-82f2-49b4-ba57-f5de2b167576
- we use the metric retrieval service to identify the unmonitored anomaly-related metrics.
ls-type:: annotation
hl-page:: 5
hl-color:: green
id:: 635ee991-28fb-4310-9e08-4d88344cbde4
- unmonitored anomaly-related metric
ls-type:: annotation
hl-page:: 6
hl-color:: yellow
id:: 635ee99e-49d2-45cf-95a2-0f77f791b335
- orrelation between metric anomalies and system faults to automatic select metric in a supervised way
ls-type:: annotation
hl-page:: 6
hl-color:: green
id:: 635eea9b-fff2-4965-8947-0c66c0d48ec2
- here are two issues requiring us to design a new unsupervised solution in practice.
ls-type:: annotation
hl-page:: 6
hl-color:: purple
id:: 635eeaa5-4451-4aa4-89c0-cd075eb3c982
- Eventually, we can construct the anomaly graph of metrics based on the metric anomaly relationships and select the KPIs via graph learning
ls-type:: annotation
hl-page:: 6
hl-color:: purple
id:: 635eead5-a422-4861-ac27-f6d8ab79453f
- we adopt the detection results to construct the graph which can represent the topological relationships between metrics
ls-type:: annotation
hl-page:: 6
hl-color:: purple
id:: 635eeb08-6cba-4796-b180-567563da78f5
- we use graph representation learning techniques to learn the metric importance and select the representative KPIs.
ls-type:: annotation
hl-page:: 6
hl-color:: purple
id:: 635eeb24-f253-4e5a-92a7-b5378c0149bf
- ime series metric anomaly detection
ls-type:: annotation
hl-page:: 6
hl-color:: yellow
id:: 635eeb34-230f-4c6e-88b2-44e9c2f9f759
- SLAVAE
ls-type:: annotation
hl-page:: 6
hl-color:: green
id:: 635eeb80-2537-4fa7-9a36-464bbfad911b
- we prune the anomaly graph based on the similarity between metrics and remove edges between metrics with similar shapes.
ls-type:: annotation
hl-page:: 6
hl-color:: green
id:: 635eebbd-2cd0-4687-a14e-993d0b1b4e27
- e adopt dynamic time warping (DTW) [38] algorithm to compute the similarity. If the DTW distance of two metrics is lower than a relevant threshold, we will remove the edge between them. Eventually, we will obtain a pruned anomaly graph which can characterize the anomaly relations between metrics.
ls-type:: annotation
hl-page:: 6
hl-color:: purple
id:: 635eebc9-d73d-498d-86d2-e8c3eb5b76d7
- 3) Random walk to select metrics:
ls-type:: annotation
hl-page:: 6
hl-color:: purple
id:: 635eebdb-e77d-456a-bea3-98efcd8b74a3
- We group metrics with similar implications into one category, and then adopt relevant classification approach to classify the metrics.
ls-type:: annotation
hl-page:: 7
hl-color:: green
id:: 635eec25-14cf-45c1-b304-2e40f302a2bc
- MSG
ls-type:: annotation
hl-page:: 7
hl-color:: purple
id:: 635eec66-8037-4c71-95a8-ba843d570dea
- MRG
ls-type:: annotation
hl-page:: 7
hl-color:: purple
id:: 635eec6f-f65f-4a9d-a7fd-74d258f5bdae
- there is no unified evaluation standard for selecting metrics.
ls-type:: annotation
hl-page:: 8
hl-color:: green
id:: 635eecb3-fe18-4027-ba1f-c54bd8c5812e
- the exporter has collected 56 metrics from 9 perspectives which includes disk usage, i/o usage, memory usage, CPU usage, and so on.
ls-type:: annotation
hl-page:: 8
hl-color:: green
id:: 635eecdc-e156-4703-807c-78a9651c70b7
- effectiveness of metric selection method
ls-type:: annotation
hl-page:: 8
hl-color:: green
id:: 635eed00-350e-4a2d-bdf0-6c9d964c9b54
- e counted
ls-type:: annotation
hl-page:: 8
hl-color:: red
id:: 635eed17-81e4-47e2-b7e5-03cccb34d9e7
- we use the selected metrics to train model and detect anomalies respectively.
ls-type:: annotation
hl-page:: 8
hl-color:: purple
id:: 635eed3a-e7c3-4b85-bd08-fb3a41810567
- MSG
ls-type:: annotation
hl-page:: 8
hl-color:: purple
id:: 635eed44-7179-4d1d-895b-98db9462f4d6
- using more metrics to detect anomalies will result in better performance when randomly selecting metrics.
ls-type:: annotation
hl-page:: 8
hl-color:: red
id:: 635eed8e-9f85-40ca-bb78-08c8727ad481
- . In addition, we also find that MSG outperforms baselines
ls-type:: annotation
hl-page:: 8
hl-color:: red
id:: 635eedca-2184-439b-b5bb-abe4fc30253f
- unmonitored metrics will be classified into the same category as those detected as anomalie
ls-type:: annotation
hl-page:: 8
hl-color:: green
id:: 635eede7-f7b1-4726-99c3-b6329312b60b
- ormal patterns
ls-type:: annotation
hl-page:: 9
hl-color:: red
id:: 635eee45-f0bf-4bda-977a-a277196c0aca
- nomalous patterns
ls-type:: annotation
hl-page:: 9
hl-color:: red
id:: 635eee49-310f-4682-b4c2-5a7c52f9ae0b
- If metrics have the same normal and anomalous patterns, we consider that they may belong to the same category
ls-type:: annotation
hl-page:: 9
hl-color:: red
id:: 635eee52-943b-497e-a198-15575f256972
- feature extraction and classification model
ls-type:: annotation
hl-page:: 9
hl-color:: purple
id:: 635eee71-263d-4b96-9d86-e84786902f45
- This paper proposes a metric recommendation service for online systems on the basis of graph learning.
ls-type:: annotation
hl-page:: 10
hl-color:: green
id:: 635ef68f-73a0-4e93-96dd-5a79ce6013f2
- wo essential usage scenarios of failure management in online systems, namely metric selection for anomaly detection and metric retrieval for fault diagnosis
ls-type:: annotation
hl-page:: 10
hl-color:: green
id:: 635ef6a2-2238-4ffa-ad6c-d2c0317e15d5
- Fault Diagnosis.
ls-type:: annotation
hl-page:: 3
hl-color:: purple
id:: 63603b06-e72b-4986-87c4-8a117e1a9e30
- However, no matter how this task is performed
ls-type:: annotation
hl-page:: 3
hl-color:: yellow
id:: 63603b24-4cd8-4ae4-a371-fccc69aeda56
- which is representative enough to reveal the practice of engineers in selecting metrics on KPI dashboards for monitoring.
ls-type:: annotation
hl-page:: 3
hl-color:: yellow
id:: 63603c11-1a15-43da-a024-39cceebc546f
- intelligent anomaly det
ls-type:: annotation
hl-page:: 3
hl-color:: green
id:: 63603c2e-9725-4f25-b6bd-9f5a19636baa
hl-stamp:: 1667251250644
- ection and fault diagnosis for online system
ls-type:: annotation
hl-page:: 3
hl-color:: green
id:: 63603c3a-b07b-4f13-b1d4-6b972ad9046d
- or baseline group, we consider the expert experience and random selection. Besides, we uniformly adopt SLA-VAE[3] as anomaly detection model in this paper
ls-type:: annotation
hl-page:: 8
hl-color:: purple
id:: 63604237-cd0c-42d7-b5dc-96ac01306521