logseq/pages/hls__icse2023-paper9_1666614163725_0.md

file:: icse2023-paper9_1666614163725_0.pdf file-path:: ../assets/icse2023-paper9_1666614163725_0.pdf

• metric selection, remains manual to a large extent ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 635edb08-6126-4cd8-9e7b-abe26aac629b
• we develop a metric recommendation service for online systems, which can automate the metrics selection practice and greatly ease the burden in managing an online system. ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 635edb16-59e2-4c01-aa40-66b5557d59b7
• anomaly detectio ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edb1e-7930-4318-8d8e-a5c88b6c4b79
• fault diagnosis ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edb25-704c-422b-bfd3-0e69092375ef
• metric recommendation mechanisms ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edb2c-e1cd-4179-b42e-e32e5ce8551a
• selecting metrics for anomaly detection ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edb3f-85f5-4588-8429-e485f6ac4eab
• retrieving metrics for faults diagnosis, ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edb46-f12d-499b-a37a-ae0eb2a288ce
• becomes increasingly important yet difficult. ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edb81-f6e9-4cf5-abb4-c0dceaa1b377
• Metrics are time series data that record the real-time state of a system. ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edb93-96aa-4dd4-93de-1b9fa00c0b4f
• metrics selection usually needs to be performed on demand to accelerate and enhance the management process ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edb9f-4a7f-4be1-9abe-d0b8c7038528
• when a software engineer wants to confirm whether the system is working properly, he/she shall first select some metrics and then perform anomaly detection on them ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edbb6-6d3e-4322-96b3-70e27e3dcd93
• Another example is, when a software engineer is diagnosing a fault, he/she usually needs to pick out relevant metrics to better understand what is going on. ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edca7-76e8-44c5-82fd-07b54743b8dd
• their effectiveness is of critical importance ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edcad-de36-4ae4-8f8c-b0d47bbf3085
• Typically, software engineers create and customize KPI dashboards to aid their daily operations ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edd26-35d9-49c5-b037-97c60e6f1902
• PI dashboard is a group of selected KPIs that are organized together in a Web-hosted site. ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edd55-67ea-466e-b350-47680c4e64df
• over 500TB a day. ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635edd8b-eff0-441f-b12a-0c6424cc558f
• equiring software engineers to manually inspect all these data to find useful metrics is impractical. ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 635edd99-4dfc-4770-8832-fb87fed32dcb
• the usefulness of some metrics might change as the system evolves. ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 635eddd1-bd13-4122-b20a-dbde6483f90d
• software engineers to frequently update their KPI dashboards, which can be especially exhausting. ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 635eddd9-ad48-444a-8109-895c62ce1b6f
• Understanding which metrics are more useful can be challenging. ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 635eddff-6604-4a55-8a64-312cf2bc70ac
• it is unrealistic to assume that every engineer is experienced enough to select metrics manually ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 635ede0e-11b0-4746-8b0b-06e6a02a2448
• rigger the service and set their KPI dashboards according to the recommendation. ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 635ede19-15e1-4881-8f71-d3d98c0c5128
• In this paper, we focus on metric recommendation for failure management, including anomaly detection and fault diagnosis. ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 635ede2b-19f7-4dc5-af4e-64e20f521c4c
• [30] I. Baradari, M. Shoar, N. Nezafati, and M. Motadel, “A new approach for kpi ranking and selection in itil processes: Using simultaneous evaluation of criteria and alternatives (seca),” Journal of Industrial Engineering and Management Studies, vol. 8, no. 1, pp. 152–179, 2021. ls-type:: annotation hl-page:: 11 hl-color:: green id:: 635ede56-7260-4486-b3b4-a38f84135b2f
• 30] I. Baradari, M. Shoar, N. Nezafati, and M. Motadel, “A new approach for kpi ranking and selection in itil processes: Using simultaneous evaluation of criteria and alternatives (seca),” Journal of Industrial Engineering and Management Studies, vol. 8, no. 1, pp. 152–179, 2021 ls-type:: annotation hl-page:: 11 hl-color:: green id:: 635ede5b-0f4a-446c-a664-869b4f516f10
• . Baradari, M. Shoar, N. Nezafati, and M. Motadel, “A new approach for kpi ranking and selection in itil processes: Using simultaneous evaluation of criteria and alternatives (seca),” Journal of Industrial Engineering and Management Studies, vol. 8, no. 1, pp. 152–179, 2021 ls-type:: annotation hl-page:: 11 hl-color:: green id:: 635ede60-f316-4105-aa0d-cd67e1f7fef3
• atency, traffic pressure, error count and saturation as KPIs. ls-type:: annotation hl-page:: 1 hl-color:: green id:: 635ede84-bf11-45d1-8ea7-3a3138d8274b
• recommendations are static ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 635ede9e-ae82-4419-8f38-9434dd4c2084
• require continuous labelling of the anomaly time to supervisedly learn the correlation between metrics and real faults, which is still very labor-intensive and difficult to apply in practice. ls-type:: annotation hl-page:: 2 hl-color:: green id:: 635edef5-50be-4055-b226-367d68808acf
• this paper proposes to automatically recommend metrics against actual conditions with minimal human effort. ls-type:: annotation hl-page:: 2 hl-color:: green id:: 635edf0a-9d82-4d81-9e7a-710eba058112
• anomaly detection and fault diagnosis ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 635edf9e-8117-45de-9587-6ab7d166bd66 hl-stamp:: 1667162016382
• : i) How to accommodate the need of different failure management tasks when performing metric recommendation? ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 635edfb5-19b7-4927-add1-fb11d172a3f7 hl-stamp:: 1667162040311
• ii) How to strike a balance between the effectiveness and the reliance on human effort during the recommendation process? ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 635edfc2-b344-465a-acd8-36b94519d4d2
• metric selection for anomaly detectio ls-type:: annotation hl-page:: 2 hl-color:: green id:: 635edfe8-5c3d-4efc-b606-a78f3363fe55
• unmonitored anomaly-related metric retrieval for fault diagnosis ls-type:: annotation hl-page:: 2 hl-color:: green id:: 635edff3-291a-4560-9a98-1cfbc5a903a8
• for online systems based on graph learning ls-type:: annotation hl-page:: 2 hl-color:: green id:: 635ee00c-51fb-4f7c-a49d-bd7fa7351963
• automate metric selection for anomaly detection and metric retrieval for fault diagnosis. ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 635ee01d-2fac-4a23-8b2c-4d80a5a54a6d
• Software engineers usually use it to inspect the state of a system, or configure the inputs for some intelligent anomaly detection and fault diagnosis algorithms ls-type:: annotation hl-page:: 2 hl-color:: green id:: 635ee049-5c23-4749-bab9-623370c6eea4
• KPIs in the same panel are metrics of the same type but with different attributes. F ls-type:: annotation hl-page:: 2 hl-color:: green id:: 635ee058-8adf-4618-9340-bc1a1fcdfb47
• However, which metric types and which attributes should be selected as KPIs for display and anomaly detection still remain a question. ls-type:: annotation hl-page:: 2 hl-color:: green id:: 635ee074-3978-4c03-8bc6-9e266e4c2dfe
• in this paper, we provide a mechanism to automatically recommend KPIs for anomaly detection on the basis of the historical data of different metrics. ls-type:: annotation hl-page:: 2 hl-color:: green id:: 635ee081-7ef5-4ff6-8c16-a09807835a5f
• unmonitored metrics ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 635ee0d2-9b8a-414a-8a30-3a7cf9db4c6a
• Although they are neglected in a KPI dashboard, they may still be useful for the diagnosis of some specific faults. ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 635ee0db-6275-4964-bf4e-d2b898f07dc7
• esource-related metrics ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 635ee0f7-1d19-4c59-8a67-25d21b796bc7
• component-specific metrics ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 635ee0fb-13b7-44d9-a61d-542e6c1381b7
• business-level metrics. ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 635ee0fe-1984-4d7e-9a75-bbb46f26c14c
• develop ls-type:: annotation hl-page:: 2 hl-color:: red id:: 635ee154-b095-4b2b-bb67-d14741b8ef19
• equiring software engineers to manually select the most appropriate metrics to monitor every component is labor-intensive and error-prone. ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 635ee1b1-3ee3-48cc-9d2f-3a8e781ef8d5
• the configuration of some attributes has a great impact on the effectiveness of the selected metrics. ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 635ee208-16b4-491e-b6d6-98dd521576c2
• C. Using Metrics for Failure Management ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 635ee219-9550-425d-b0f9-89ff957d511d
• Anomaly Detection ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 635ee30d-4573-4246-8535-090fb007b673
• selecting useful metrics to assess the symptom of a fault needs to be conducted first ls-type:: annotation hl-page:: 3 hl-color:: green id:: 635ee328-943c-406c-bbbd-de0d4da90474
• intelligent anomaly detection ls-type:: annotation hl-page:: 3 hl-color:: green id:: 635ee456-82bb-4a80-8e10-99a250bc2e18
• fault diagnosis ls-type:: annotation hl-page:: 3 hl-color:: green id:: 635ee459-2d0a-4d72-86e2-8bcd5316a14d
• It can be observed that more than three quarters of KPI dashboards have less than 25 metrics. ls-type:: annotation hl-page:: 3 hl-color:: green id:: 635ee48d-203a-40e8-9723-ede05a6a4159
• many metrics are noisy or contain redundant information. ls-type:: annotation hl-page:: 3 hl-color:: green id:: 635ee4c0-ddcc-48a2-aa82-b00ab11e27ce
• uch a practice is not automatic, and there is still no a unified standard for selecting KPIs, rendering it difficult to release the full potential of these anomaly detection methods in different systems. ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 635ee4e1-92fa-470d-8879-db295d39eae7 hl-stamp:: 1667251293778
• ngineers tend to use more metrics for analysis. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 635ee522-7f76-4ee3-9a0e-00f6639dcb88
• Since more available metrics can provide more clues, it is better to take more metrics into consideration when developing an intelligent fault diagnosis method. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 635ee5be-2b59-4c34-958d-4875381701ef
• This paper proposes to recommend metrics against actual conditions, and targets at making full use of metrics that can be collected, not limited to KPIs. ls-type:: annotation hl-page:: 4 hl-color:: yellow id:: 635ee5f7-93af-41f1-8d05-9bca0d75025f
• Metric Selection for Anomaly Detection. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 635ee61c-6414-4ac3-83f2-0df2fcbdfe75
• For each entity, which metrics can better reveal the entity state and should be utilized for anomaly detection?” ls-type:: annotation hl-page:: 4 hl-color:: green id:: 635ee62c-4cc7-4fc7-9eb2-ab3f558c2a60
• ain a recommended subset of M, and the recommended subset should comply with the fault revealing goal, noise removal goal and pattern diversity goal, ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 635ee649-b9d1-46df-aa03-ba55252077f3
• Unmonitored Anomaly-Related Metric Retrieval for Fault Diagnosis. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 635ee64e-bdf1-40b4-944d-ad97029a5235
• “Given an anomalous KPI, which unmonitored metrics may expose some clues about the anomaly and need to be retrieved for analysis?”. ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 635ee673-e622-4890-acb1-6710d1b10e74
• Some metrics are noisy and using them for alerting might give rise to a flood of false alerts. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 635ee6d0-b0c7-4bdf-9dd6-7048acf28fc5
• some metrics might correlate with each other. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 635ee6ea-3df1-40c2-bd5e-1a6cf0f65ab6
• redundancy ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 635ee706-faef-4fff-bc32-af9f2b4fe1d5
• pattern diversity ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 635ee70d-8ab9-42db-831c-497f3bcc102d
• we propose a metric recommendation service including offline metric selection for anomaly detection and online unmonitored anomaly-related metric retrieval for fault diagnosis. ls-type:: annotation hl-page:: 5 hl-color:: purple id:: 635ee87e-a503-4548-8663-ea43f8e18ab5
• dopt anomaly detection techniques to identify anomalies in these metrics ls-type:: annotation hl-page:: 5 hl-color:: green id:: 635ee8aa-f201-4ecc-8247-6a34e5145510
• When there exist anomalies, the software engineers should diagnose and repair the system faults based on the anomalous metrics. However, it is difficult to ensure that metrics can be selected completely and accurately. The engineers also need a service to assist them in automatically selecting metrics that can characterize the availability of their systems. Hence, we first design a metric selection method for monitoring and detecting anomalies. In addition, the unmonitored metrics can also assist in revealing and diagnosing anomalies. There are far more unmonitored metrics than monitored metrics in an industrial environment. Adopting only monitored metrics may make it difficult to diagnose a fault. Therefore, we design an anomaly-related unmonitored metric retrieval method for fault diagnosis. ls-type:: annotation hl-page:: 5 hl-color:: red id:: 635ee8c8-9b94-4a0f-912b-519ee19122dd
• we adopt the metric selection service to recommend KPIs which should be monitored. ls-type:: annotation hl-page:: 5 hl-color:: red id:: 635ee94e-82f2-49b4-ba57-f5de2b167576
• we use the metric retrieval service to identify the unmonitored anomaly-related metrics. ls-type:: annotation hl-page:: 5 hl-color:: green id:: 635ee991-28fb-4310-9e08-4d88344cbde4
• unmonitored anomaly-related metric ls-type:: annotation hl-page:: 6 hl-color:: yellow id:: 635ee99e-49d2-45cf-95a2-0f77f791b335
• orrelation between metric anomalies and system faults to automatic select metric in a supervised way ls-type:: annotation hl-page:: 6 hl-color:: green id:: 635eea9b-fff2-4965-8947-0c66c0d48ec2
• here are two issues requiring us to design a new unsupervised solution in practice. ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 635eeaa5-4451-4aa4-89c0-cd075eb3c982
• Eventually, we can construct the anomaly graph of metrics based on the metric anomaly relationships and select the KPIs via graph learning ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 635eead5-a422-4861-ac27-f6d8ab79453f
• we adopt the detection results to construct the graph which can represent the topological relationships between metrics ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 635eeb08-6cba-4796-b180-567563da78f5
• we use graph representation learning techniques to learn the metric importance and select the representative KPIs. ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 635eeb24-f253-4e5a-92a7-b5378c0149bf
• ime series metric anomaly detection ls-type:: annotation hl-page:: 6 hl-color:: yellow id:: 635eeb34-230f-4c6e-88b2-44e9c2f9f759
• SLAVAE ls-type:: annotation hl-page:: 6 hl-color:: green id:: 635eeb80-2537-4fa7-9a36-464bbfad911b
• we prune the anomaly graph based on the similarity between metrics and remove edges between metrics with similar shapes. ls-type:: annotation hl-page:: 6 hl-color:: green id:: 635eebbd-2cd0-4687-a14e-993d0b1b4e27
• e adopt dynamic time warping (DTW) [38] algorithm to compute the similarity. If the DTW distance of two metrics is lower than a relevant threshold, we will remove the edge between them. Eventually, we will obtain a pruned anomaly graph which can characterize the anomaly relations between metrics. ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 635eebc9-d73d-498d-86d2-e8c3eb5b76d7
• 3. Random walk to select metrics: ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 635eebdb-e77d-456a-bea3-98efcd8b74a3
• We group metrics with similar implications into one category, and then adopt relevant classification approach to classify the metrics. ls-type:: annotation hl-page:: 7 hl-color:: green id:: 635eec25-14cf-45c1-b304-2e40f302a2bc
• MSG ls-type:: annotation hl-page:: 7 hl-color:: purple id:: 635eec66-8037-4c71-95a8-ba843d570dea
• MRG ls-type:: annotation hl-page:: 7 hl-color:: purple id:: 635eec6f-f65f-4a9d-a7fd-74d258f5bdae
• there is no unified evaluation standard for selecting metrics. ls-type:: annotation hl-page:: 8 hl-color:: green id:: 635eecb3-fe18-4027-ba1f-c54bd8c5812e
• the exporter has collected 56 metrics from 9 perspectives which includes disk usage, i/o usage, memory usage, CPU usage, and so on. ls-type:: annotation hl-page:: 8 hl-color:: green id:: 635eecdc-e156-4703-807c-78a9651c70b7
• effectiveness of metric selection method ls-type:: annotation hl-page:: 8 hl-color:: green id:: 635eed00-350e-4a2d-bdf0-6c9d964c9b54
• e counted ls-type:: annotation hl-page:: 8 hl-color:: red id:: 635eed17-81e4-47e2-b7e5-03cccb34d9e7
• we use the selected metrics to train model and detect anomalies respectively. ls-type:: annotation hl-page:: 8 hl-color:: purple id:: 635eed3a-e7c3-4b85-bd08-fb3a41810567
• MSG ls-type:: annotation hl-page:: 8 hl-color:: purple id:: 635eed44-7179-4d1d-895b-98db9462f4d6
• using more metrics to detect anomalies will result in better performance when randomly selecting metrics. ls-type:: annotation hl-page:: 8 hl-color:: red id:: 635eed8e-9f85-40ca-bb78-08c8727ad481
• . In addition, we also find that MSG outperforms baselines ls-type:: annotation hl-page:: 8 hl-color:: red id:: 635eedca-2184-439b-b5bb-abe4fc30253f
• unmonitored metrics will be classified into the same category as those detected as anomalie ls-type:: annotation hl-page:: 8 hl-color:: green id:: 635eede7-f7b1-4726-99c3-b6329312b60b
• ormal patterns ls-type:: annotation hl-page:: 9 hl-color:: red id:: 635eee45-f0bf-4bda-977a-a277196c0aca
• nomalous patterns ls-type:: annotation hl-page:: 9 hl-color:: red id:: 635eee49-310f-4682-b4c2-5a7c52f9ae0b
• If metrics have the same normal and anomalous patterns, we consider that they may belong to the same category ls-type:: annotation hl-page:: 9 hl-color:: red id:: 635eee52-943b-497e-a198-15575f256972
• feature extraction and classification model ls-type:: annotation hl-page:: 9 hl-color:: purple id:: 635eee71-263d-4b96-9d86-e84786902f45
• This paper proposes a metric recommendation service for online systems on the basis of graph learning. ls-type:: annotation hl-page:: 10 hl-color:: green id:: 635ef68f-73a0-4e93-96dd-5a79ce6013f2
• wo essential usage scenarios of failure management in online systems, namely metric selection for anomaly detection and metric retrieval for fault diagnosis ls-type:: annotation hl-page:: 10 hl-color:: green id:: 635ef6a2-2238-4ffa-ad6c-d2c0317e15d5
• Fault Diagnosis. ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 63603b06-e72b-4986-87c4-8a117e1a9e30
• However, no matter how this task is performed ls-type:: annotation hl-page:: 3 hl-color:: yellow id:: 63603b24-4cd8-4ae4-a371-fccc69aeda56
• which is representative enough to reveal the practice of engineers in selecting metrics on KPI dashboards for monitoring. ls-type:: annotation hl-page:: 3 hl-color:: yellow id:: 63603c11-1a15-43da-a024-39cceebc546f
• intelligent anomaly det ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63603c2e-9725-4f25-b6bd-9f5a19636baa hl-stamp:: 1667251250644
• ection and fault diagnosis for online system ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63603c3a-b07b-4f13-b1d4-6b972ad9046d
• or baseline group, we consider the expert experience and random selection. Besides, we uniformly adopt SLA-VAE[3] as anomaly detection model in this paper ls-type:: annotation hl-page:: 8 hl-color:: purple id:: 63604237-cd0c-42d7-b5dc-96ac01306521