Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reproducibility Issue #4

Open
lizeyan opened this issue Sep 3, 2022 · 1 comment
Open

Reproducibility Issue #4

lizeyan opened this issue Sep 3, 2022 · 1 comment

Comments

@lizeyan
Copy link
Member

lizeyan commented Sep 3, 2022 •
edited
Loading

Here I list my environment, command and the outputs to run an experiment.

Environment

Machine

硬件概览：

  型号名称：	MacBook Pro
  型号标识符：	MacBookPro14,3
  处理器名称：	四核Intel Core i7
  处理器速度：	2.9 GHz
  处理器数目：	1
  核总数：	4
  L2缓存（每个核）：	256 KB
  L3缓存：	8 MB
  超线程技术：	已启用
  内存：	16 GB
  系统固件版本：	451.140.1.0.0
  操作系统加载程序版本：	540.120.3~19
  SMC版本（系统）：	2.45f5

Docker Info

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2)
  compose: Docker Compose (Docker Inc., v2.6.1)
  extension: Manages Docker extensions (Docker Inc., v0.2.7)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 2
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc version: v1.1.2-0-ga916309
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.774GiB
 Name: docker-desktop
 ID: WKZ5:6KJZ:3K6S:I7WY:LTLR:3TNP:D23G:N3C7:6QPG:KG44:WEEK:CVMR
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Username: lizytalk
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

Docker Image Info

REPOSITORY        TAG       IMAGE ID       CREATED        SIZE
lizytalk/dejavu   latest    32d6db301926   2 months ago   17.3GB

Command

docker run -it --rm -v $(realpath .):/workspace lizytalk/dejavu bash -c 'source .envrc && python exp/run_GAT_node_classification.py -H=4 -L=8 -fe=GRU -bal=True --data_dir=./data/A1 --max_epoch=20'
Note that --max_epoch=20 is used to validate the program fastly.

Output


=============
== PyTorch ==
=============

NVIDIA Release 21.11 (build 29224839)
PyTorch Version 1.11.0a0+b6df043

Container image Copyright (c) 2021, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copyright (c) 2014-2021 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

NVIDIA Deep Learning Profiler (dlprof) Copyright (c) 2021, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

NOTE: The SHMEM allocation limit is set to the default of 64MB.  This may be
   insufficient for PyTorch.  NVIDIA recommends the use of the following flags:
   docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 ...

Using backend: pytorch
2022-09-03 03:02:35.245 | INFO     | failure_dependency_graph.FDG_config:process_args:62 - torch.cuda.is_available()=False
2022-09-03 03:02:35.332 | INFO     | DejaVu.workflow:_train_exp_CFL:34 -
================================================Config=============================================
{'FI_feature_dim': 3,
 'GAT_layers': 8,
 'GAT_num_heads': 4,
 'GAT_residual': True,
 'GAT_shared_feature_mapper': False,
 'augmentation': False,
 'balance_train_set': True,
 'batch_size': 16,
 'cache_dir': PosixPath('/tmp/SSF/.cache'),
 'checkpoint_metric': 'val_loss',
 'cuda': False,
 'data_dir': PosixPath('data/A1'),
 'dataset_split_ratio': (0.4, 0.2, 0.4),
 'display_epoch_freq': 10,
 'display_second_freq': 5,
 'drop_FDG_edges_fraction': 0.0,
 'dropout': False,
 'early_stopping_epoch_patience': 500,
 'es': True,
 'faults_path': None,
 'feature_projector_type': 'GRU',
 'flush_dataset_cache': True,
 'gradient_clip_val': 1.0,
 'graph_config_path': None,
 'init_lr': 0.01,
 'max_epoch': 20,
 'metrics_path': None,
 'output_base_path': PosixPath('/SSF/output'),
 'output_dir': PosixPath('/SSF/output/run_GAT_node_classification.py.2022-09-03T03:02:35.245139'),
 'p': 0.25,
 'q': 0.25,
 'random_walk_length': 8,
 'rec_loss_weight': 1.0,
 'test_batch_size': 128,
 'test_epoch_freq': 100,
 'test_second_freq': 30.0,
 'train_set_repeat': 1,
 'train_set_sampling': 1.0,
 'ts_feature_mode': 'full',
 'use_anomaly_direction_constraint': False,
 'valid_epoch_freq': 10,
 'weight_decay': 0.01,
 'window_size': (10, 10)}
===================================================================================================

2022-09-03 03:02:36.842 | INFO     | DejaVu.workflow:_train_exp_CFL:39 - reproducibility info: {'command_line': 'python exp/run_GAT_node_classification.py -H=4 -L=8 -fe=GRU -bal=True --data_dir=./data/A1 --max_epoch=20', 'time': 'Sat Sep  3 03:02:35 2022', 'git_root': '/workspace', 'git_url': 'https://github.com/NetManAIOps/DejaVu/tree/00d36dd07eed266840840769ecbc4abf0322319a', 'git_has_uncommitted_changes': False}
2022-09-03 03:02:37.800 | INFO     | failure_dependency_graph.failure_dependency_graph:_load_FDG:206 - Loading FDG from data/A1/FDG.pkl
2022-09-03 03:02:40.971 | INFO     | failure_dependency_graph.model_interface:__init__:47 - dataset_cache_dir=/tmp/SSF/.cache/faults=data_A1_faults.csv.graph=data_A1_graph.yml.metrics=data_A1_metrics.norm.pkl.use_anomaly_direction_constraint=False
2022-09-03 03:02:41.019 | INFO     | failure_dependency_graph.model_interface:split_failures_by_type:209 - fault ids with multiple root causes: []
2022-09-03 03:02:41.019 | INFO     | failure_dependency_graph.model_interface:split_failures_by_type:234 - fault_type=('Docker CPU',)
train_length=7     train_ids=[5, 23, 12, 37, 0, 58, 30]
validation_length=4     validation_ids=[75, 29, 3, 15]
test_length=8     test_ids=[65, 59, 18, 50, 44, 41, 8, 52]
(7   recurring faults)
2022-09-03 03:02:41.020 | INFO     | failure_dependency_graph.model_interface:split_failures_by_type:234 - fault_type=('Docker',)
train_length=12    train_ids=[35, 25, 56, 71, 53, 17, 13, 9, 6, 1, 7, 26]
validation_length=6     validation_ids=[2, 21, 63, 68, 70, 54]
test_length=12    test_ids=[24, 31, 28, 77, 48, 49, 16, 67, 76, 62, 60, 69]
(12  recurring faults)
2022-09-03 03:02:41.021 | INFO     | failure_dependency_graph.model_interface:split_failures_by_type:234 - fault_type=('DB Session',)
train_length=2     train_ids=[36, 47]
validation_length=2     validation_ids=[74, 45]
test_length=3     test_ids=[19, 4, 57]
(1   recurring faults)
2022-09-03 03:02:41.021 | INFO     | failure_dependency_graph.model_interface:split_failures_by_type:234 - fault_type=('DB State',)
train_length=2     train_ids=[46, 11]
validation_length=1     validation_ids=[27]
test_length=2     test_ids=[34, 10]
(2   recurring faults)
2022-09-03 03:02:41.022 | INFO     | failure_dependency_graph.model_interface:split_failures_by_type:234 - fault_type=('OS Network',)
train_length=6     train_ids=[22, 39, 64, 61, 20, 38]
validation_length=4     validation_ids=[51, 42, 55, 32]
test_length=7     test_ids=[40, 14, 33, 66, 43, 73, 72]
(4   recurring faults)
2022-09-03 03:02:41.023 | INFO     | failure_dependency_graph.model_interface:split_failures_by_type:247 - repeat [5, 23, 12, 37, 0, 58, 30] for 1 times
2022-09-03 03:02:41.024 | INFO     | failure_dependency_graph.model_interface:split_failures_by_type:247 - repeat [35, 25, 56, 71, 53, 17, 13, 9, 6, 1, 7, 26] for 1 times
2022-09-03 03:02:41.024 | INFO     | failure_dependency_graph.model_interface:split_failures_by_type:247 - repeat [36, 47] for 6 times
2022-09-03 03:02:41.024 | INFO     | failure_dependency_graph.model_interface:split_failures_by_type:247 - repeat [46, 11] for 6 times
2022-09-03 03:02:41.025 | INFO     | failure_dependency_graph.model_interface:split_failures_by_type:247 - repeat [22, 39, 64, 61, 20, 38] for 2 times
2022-09-03 03:02:41.025 | INFO     | failure_dependency_graph.model_interface:split_failures_by_type:265 - len(train_list)=55 len(set(train_list))=29 len(validation_list)=17 len(test_list)=32
2022-09-03 03:02:41.081 | INFO     | DejaVu.workflow:_train_exp_CFL:51 -
==========================Model Summary================================================================================================================
Layer (type:depth-idx)                   Param #
=================================================================
GAT                                      --
├─FIFeatureExtractor: 1-1                --
│    └─ModuleList: 2-1                   --
│    │    └─GRUFeatureModule: 3-1        --
│    │    │    └─GRU: 4-1                81
│    │    │    └─Sequential: 4-2         --
│    │    │    │    └─Reshape: 5-1       --
│    │    │    │    └─Conv1d: 5-2        100
│    │    │    │    └─GELU: 5-3          --
│    │    │    │    └─Flatten: 5-4       --
│    │    │    │    └─Linear: 5-5        543
│    │    │    │    └─Reshape: 5-6       --
│    │    └─GRUFeatureModule: 3-2        --
│    │    │    └─GRU: 4-3                288
│    │    │    └─Sequential: 4-4         --
│    │    │    │    └─Reshape: 5-7       --
│    │    │    │    └─Conv1d: 5-8        100
│    │    │    │    └─GELU: 5-9          --
│    │    │    │    └─Flatten: 5-10      --
│    │    │    │    └─Linear: 5-11       543
│    │    │    │    └─Reshape: 5-12      --
│    │    └─GRUFeatureModule: 3-3        --
│    │    │    └─GRU: 4-5                81
│    │    │    └─Sequential: 4-6         --
│    │    │    │    └─Reshape: 5-13      --
│    │    │    │    └─Conv1d: 5-14       100
│    │    │    │    └─GELU: 5-15         --
│    │    │    │    └─Flatten: 5-16      --
│    │    │    │    └─Linear: 5-17       543
│    │    │    │    └─Reshape: 5-18      --
│    │    └─GRUFeatureModule: 3-4        --
│    │    │    └─GRU: 4-7                117
│    │    │    └─Sequential: 4-8         --
│    │    │    │    └─Reshape: 5-19      --
│    │    │    │    └─Conv1d: 5-20       100
│    │    │    │    └─GELU: 5-21         --
│    │    │    │    └─Flatten: 5-22      --
│    │    │    │    └─Linear: 5-23       543
│    │    │    │    └─Reshape: 5-24      --
│    │    └─GRUFeatureModule: 3-5        --
│    │    │    └─GRU: 4-9                90
│    │    │    └─Sequential: 4-10        --
│    │    │    │    └─Reshape: 5-25      --
│    │    │    │    └─Conv1d: 5-26       100
│    │    │    │    └─GELU: 5-27         --
│    │    │    │    └─Flatten: 5-28      --
│    │    │    │    └─Linear: 5-29       543
│    │    │    │    └─Reshape: 5-30      --
│    │    └─GRUFeatureModule: 3-6        --
│    │    │    └─GRU: 4-11               153
│    │    │    └─Sequential: 4-12        --
│    │    │    │    └─Reshape: 5-31      --
│    │    │    │    └─Conv1d: 5-32       100
│    │    │    │    └─GELU: 5-33         --
│    │    │    │    └─Flatten: 5-34      --
│    │    │    │    └─Linear: 5-35       543
│    │    │    │    └─Reshape: 5-36      --
│    │    └─GRUFeatureModule: 3-7        --
│    │    │    └─GRU: 4-13               81
│    │    │    └─Sequential: 4-14        --
│    │    │    │    └─Reshape: 5-37      --
│    │    │    │    └─Conv1d: 5-38       100
│    │    │    │    └─GELU: 5-39         --
│    │    │    │    └─Flatten: 5-40      --
│    │    │    │    └─Linear: 5-41       543
│    │    │    │    └─Reshape: 5-42      --
│    │    └─GRUFeatureModule: 3-8        --
│    │    │    └─GRU: 4-15               54
│    │    │    └─Sequential: 4-16        --
│    │    │    │    └─Reshape: 5-43      --
│    │    │    │    └─Conv1d: 5-44       100
│    │    │    │    └─GELU: 5-45         --
│    │    │    │    └─Flatten: 5-46      --
│    │    │    │    └─Linear: 5-47       543
│    │    │    │    └─Reshape: 5-48      --
│    │    └─GRUFeatureModule: 3-9        --
│    │    │    └─GRU: 4-17               63
│    │    │    └─Sequential: 4-18        --
│    │    │    │    └─Reshape: 5-49      --
│    │    │    │    └─Conv1d: 5-50       100
│    │    │    │    └─GELU: 5-51         --
│    │    │    │    └─Flatten: 5-52      --
│    │    │    │    └─Linear: 5-53       543
│    │    │    │    └─Reshape: 5-54      --
│    │    └─GRUFeatureModule: 3-10       --
│    │    │    └─GRU: 4-19               54
│    │    │    └─Sequential: 4-20        --
│    │    │    │    └─Reshape: 5-55      --
│    │    │    │    └─Conv1d: 5-56       100
│    │    │    │    └─GELU: 5-57         --
│    │    │    │    └─Flatten: 5-58      --
│    │    │    │    └─Linear: 5-59       543
│    │    │    │    └─Reshape: 5-60      --
│    │    └─GRUFeatureModule: 3-11       --
│    │    │    └─GRU: 4-21               54
│    │    │    └─Sequential: 4-22        --
│    │    │    │    └─Reshape: 5-61      --
│    │    │    │    └─Conv1d: 5-62       100
│    │    │    │    └─GELU: 5-63         --
│    │    │    │    └─Flatten: 5-64      --
│    │    │    │    └─Linear: 5-65       543
│    │    │    │    └─Reshape: 5-66      --
│    │    └─GRUFeatureModule: 3-12       --
│    │    │    └─GRU: 4-23               81
│    │    │    └─Sequential: 4-24        --
│    │    │    │    └─Reshape: 5-67      --
│    │    │    │    └─Conv1d: 5-68       100
│    │    │    │    └─GELU: 5-69         --
│    │    │    │    └─Flatten: 5-70      --
│    │    │    │    └─Linear: 5-71       543
│    │    │    │    └─Reshape: 5-72      --
│    │    └─GRUFeatureModule: 3-13       --
│    │    │    └─GRU: 4-25               54
│    │    │    └─Sequential: 4-26        --
│    │    │    │    └─Reshape: 5-73      --
│    │    │    │    └─Conv1d: 5-74       100
│    │    │    │    └─GELU: 5-75         --
│    │    │    │    └─Flatten: 5-76      --
│    │    │    │    └─Linear: 5-77       543
│    │    │    │    └─Reshape: 5-78      --
│    │    └─GRUFeatureModule: 3-14       --
│    │    │    └─GRU: 4-27               63
│    │    │    └─Sequential: 4-28        --
│    │    │    │    └─Reshape: 5-79      --
│    │    │    │    └─Conv1d: 5-80       100
│    │    │    │    └─GELU: 5-81         --
│    │    │    │    └─Flatten: 5-82      --
│    │    │    │    └─Linear: 5-83       543
│    │    │    │    └─Reshape: 5-84      --
│    │    └─GRUFeatureModule: 3-15       --
│    │    │    └─GRU: 4-29               243
│    │    │    └─Sequential: 4-30        --
│    │    │    │    └─Reshape: 5-85      --
│    │    │    │    └─Conv1d: 5-86       100
│    │    │    │    └─GELU: 5-87         --
│    │    │    │    └─Flatten: 5-88      --
│    │    │    │    └─Linear: 5-89       543
│    │    │    │    └─Reshape: 5-90      --
│    │    └─GRUFeatureModule: 3-16       --
│    │    │    └─GRU: 4-31               243
│    │    │    └─Sequential: 4-32        --
│    │    │    │    └─Reshape: 5-91      --
│    │    │    │    └─Conv1d: 5-92       100
│    │    │    │    └─GELU: 5-93         --
│    │    │    │    └─Flatten: 5-94      --
│    │    │    │    └─Linear: 5-95       543
│    │    │    │    └─Reshape: 5-96      --
│    │    └─GRUFeatureModule: 3-17       --
│    │    │    └─GRU: 4-33               153
│    │    │    └─Sequential: 4-34        --
│    │    │    │    └─Reshape: 5-97      --
│    │    │    │    └─Conv1d: 5-98       100
│    │    │    │    └─GELU: 5-99         --
│    │    │    │    └─Flatten: 5-100     --
│    │    │    │    └─Linear: 5-101      543
│    │    │    │    └─Reshape: 5-102     --
│    │    └─GRUFeatureModule: 3-18       --
│    │    │    └─GRU: 4-35               144
│    │    │    └─Sequential: 4-36        --
│    │    │    │    └─Reshape: 5-103     --
│    │    │    │    └─Conv1d: 5-104      100
│    │    │    │    └─GELU: 5-105        --
│    │    │    │    └─Flatten: 5-106     --
│    │    │    │    └─Linear: 5-107      543
│    │    │    │    └─Reshape: 5-108     --
│    │    └─GRUFeatureModule: 3-19       --
│    │    │    └─GRU: 4-37               81
│    │    │    └─Sequential: 4-38        --
│    │    │    │    └─Reshape: 5-109     --
│    │    │    │    └─Conv1d: 5-110      100
│    │    │    │    └─GELU: 5-111        --
│    │    │    │    └─Flatten: 5-112     --
│    │    │    │    └─Linear: 5-113      543
│    │    │    │    └─Reshape: 5-114     --
├─Identity: 1-2                          --
├─ModuleList: 1-3                        --
│    └─GATConv: 2-2                      --
│    │    └─Linear: 3-20                 36
│    │    └─Dropout: 3-21                --
│    │    └─Dropout: 3-22                --
│    │    └─LeakyReLU: 3-23              --
│    │    └─Linear: 3-24                 36
│    └─GATConv: 2-3                      --
│    │    └─Linear: 3-25                 144
│    │    └─Dropout: 3-26                --
│    │    └─Dropout: 3-27                --
│    │    └─LeakyReLU: 3-28              --
│    │    └─Identity: 3-29               --
│    └─GATConv: 2-4                      --
│    │    └─Linear: 3-30                 144
│    │    └─Dropout: 3-31                --
│    │    └─Dropout: 3-32                --
│    │    └─LeakyReLU: 3-33              --
│    │    └─Identity: 3-34               --
│    └─GATConv: 2-5                      --
│    │    └─Linear: 3-35                 144
│    │    └─Dropout: 3-36                --
│    │    └─Dropout: 3-37                --
│    │    └─LeakyReLU: 3-38              --
│    │    └─Identity: 3-39               --
│    └─GATConv: 2-6                      --
│    │    └─Linear: 3-40                 144
│    │    └─Dropout: 3-41                --
│    │    └─Dropout: 3-42                --
│    │    └─LeakyReLU: 3-43              --
│    │    └─Identity: 3-44               --
│    └─GATConv: 2-7                      --
│    │    └─Linear: 3-45                 144
│    │    └─Dropout: 3-46                --
│    │    └─Dropout: 3-47                --
│    │    └─LeakyReLU: 3-48              --
│    │    └─Identity: 3-49               --
│    └─GATConv: 2-8                      --
│    │    └─Linear: 3-50                 144
│    │    └─Dropout: 3-51                --
│    │    └─Dropout: 3-52                --
│    │    └─LeakyReLU: 3-53              --
│    │    └─Identity: 3-54               --
│    └─GATConv: 2-9                      --
│    │    └─Linear: 3-55                 144
│    │    └─Dropout: 3-56                --
│    │    └─Dropout: 3-57                --
│    │    └─LeakyReLU: 3-58              --
│    │    └─Identity: 3-59               --
├─NodeWeightPredictor: 1-4               --
│    └─Sequential: 2-10                  --
│    │    └─Linear: 3-60                 1,664
│    │    └─GELU: 3-61                   --
│    │    └─Linear: 3-62                 128
=================================================================
Total params: 17,267
Trainable params: 17,267
Non-trainable params: 0
=======================================================================================================================================================

GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
preprocess metrics for each instance type: 100%|████████████████████████████████████████| 19/19 [00:06<00:00,  2.95it/s]
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Missing logger folder: /SSF/output/run_GAT_node_classification.py.2022-09-03T03:02:35.245139/lightning_logs

  | Name    | Type | Params
---------------------------------
0 | _module | GAT  | 17.6 K
---------------------------------
17.6 K    Trainable params
0         Non-trainable params
17.6 K    Total params
0.070     Total estimated model params size (MB)
2022-09-03 03:02:52.634 | INFO     | DejaVu.models.interface.callbacks:on_validation_epoch_end:52 - epoch=0     val_loss=1.0618     A@1=0.00 % A@2=0.00 % A@3=0.00 % A@5=0.00 % MAR=47.71
2022-09-03 03:02:54.752 | INFO     | DejaVu.models.interface.callbacks:on_train_epoch_end:41 - epoch=0     loss=1.2861    A@1=7.27 % A@2=14.55% A@3=16.36% A@5=16.36% MAR=36.62
2022-09-03 03:02:59.639 | INFO     | DejaVu.models.interface.callbacks:on_train_epoch_end:41 - epoch=4     loss=0.5538    A@1=16.36% A@2=21.82% A@3=47.27% A@5=49.09% MAR=8.98
2022-09-03 03:03:04.952 | INFO     | DejaVu.models.interface.callbacks:on_train_epoch_end:41 - epoch=8     loss=0.1951    A@1=78.18% A@2=80.00% A@3=83.64% A@5=96.36% MAR=2.18
2022-09-03 03:03:06.374 | INFO     | DejaVu.models.interface.callbacks:on_validation_epoch_end:52 - epoch=9     val_loss=0.4763     A@1=29.41% A@2=47.06% A@3=70.59% A@5=82.35% MAR=3.82
Metric val_loss improved. New best score: 0.476
2022-09-03 03:03:07.945 | INFO     | DejaVu.models.interface.callbacks:on_train_epoch_end:41 - epoch=10    loss=0.1300    A@1=76.36% A@2=83.64% A@3=90.91% A@5=96.36% MAR=1.56
2022-09-03 03:03:14.117 | INFO     | DejaVu.models.interface.callbacks:on_train_epoch_end:41 - epoch=15    loss=0.0551    A@1=89.09% A@2=100.00% A@3=100.00% A@5=100.00% MAR=1.11
2022-09-03 03:03:19.016 | INFO     | DejaVu.models.interface.callbacks:on_validation_epoch_end:52 - epoch=19    val_loss=0.2789     A@1=64.71% A@2=88.24% A@3=94.12% A@5=94.12% MAR=2.12
Metric val_loss improved by 0.197 >= min_delta = 0.0. New best score: 0.279
2022-09-03 03:03:19.274 | INFO     | utils.callbacks:on_fit_end:106 - Average epoch time: 1.33
2022-09-03 03:03:19.275 | INFO     | DejaVu.workflow:_train_exp_CFL:99 - trainer.checkpoint_callback.best_model_path='/SSF/output/run_GAT_node_classification.py.2022-09-03T03:02:35.245139/lightning_logs/version_0/checkpoints/epoch=19-A@1=0.647059-val_loss=0.278938-MAR=2.117647.ckpt'
2022-09-03 03:03:19.848 | INFO     | DejaVu.workflow:_train_exp_CFL:100 - {'command_line': 'python exp/run_GAT_node_classification.py -H=4 -L=8 -fe=GRU -bal=True --data_dir=./data/A1 --max_epoch=20', 'time': 'Sat Sep  3 03:03:19 2022', 'git_root': '/workspace', 'git_url': 'https://github.com/NetManAIOps/DejaVu/tree/00d36dd07eed266840840769ecbc4abf0322319a', 'git_has_uncommitted_changes': False}
2022-09-03 03:03:19.863 | WARNING  | utils.load_model:best_checkpoint:35 - ckpt_path=PosixPath('/SSF/output/run_GAT_node_classification.py.2022-09-03T03:02:35.245139/lightning_logs/version_0/checkpoints/last.ckpt') not match
Restoring states from the checkpoint path at /SSF/output/run_GAT_node_classification.py.2022-09-03T03:02:35.245139/lightning_logs/version_0/checkpoints/epoch=19-A@1=0.647059-val_loss=0.278938-MAR=2.117647.ckpt
Loaded model weights from checkpoint at /SSF/output/run_GAT_node_classification.py.2022-09-03T03:02:35.245139/lightning_logs/version_0/checkpoints/epoch=19-A@1=0.647059-val_loss=0.278938-MAR=2.117647.ckpt
2022-09-03 03:03:21.327 | INFO     | DejaVu.models.interface.callbacks:on_test_epoch_end:107 -
A@1=53.12% A@2=90.62% A@3=100.00% A@5=100.00% MAR=1.56
|id  |     |FR |AR |recurring|timestamp                |root cause                                                            |rank-1              |rank-2              |rank-3              |
|65  |✅    |  1|  1|True     |2020-05-30T04:13:00+08:00|docker_002 CPU                                                        |docker_002 CPU      |docker_002          |db_003 Session      |
|59  |✅    |  1|  1|True     |2020-05-29T03:41:00+08:00|docker_001 CPU                                                        |docker_001 CPU      |docker_008          |docker_007          |
|18  |✅    |  1|  1|False    |2020-05-23T00:05:00+08:00|docker_004 CPU                                                        |docker_004 CPU      |docker_004          |db_009              |
|50  |✅    |  1|  1|True     |2020-05-27T05:09:00+08:00|docker_001 CPU                                                        |docker_001 CPU      |os_020 Network      |docker_001          |
|44  |✅    |  1|  1|True     |2020-05-27T01:23:00+08:00|docker_006 CPU                                                        |docker_006 CPU      |docker_006          |docker_002 CPU      |
|41  |✅    |  1|  1|True     |2020-05-26T05:15:00+08:00|docker_002 CPU                                                        |docker_002 CPU      |docker_002          |os_021              |
|8   |✅    |  1|  1|True     |2020-04-11T04:40:00+08:00|docker_008 CPU                                                        |docker_008 CPU      |docker_008          |db_007 Session      |
|52  |✅    |  1|  1|True     |2020-05-28T00:47:00+08:00|docker_001 CPU                                                        |docker_001 CPU      |docker_001          |os_021              |
|24  |❌    |  2|  2|True     |2020-05-23T05:20:00+08:00|docker_005                                                            |docker_005 CPU      |docker_005          |docker_003          |
|31  |✅    |  1|  1|True     |2020-05-24T04:47:00+08:00|docker_004                                                            |docker_004          |os_021 Network      |docker_004 CPU      |
|28  |❌    |  3|  3|True     |2020-05-24T02:47:00+08:00|docker_002                                                            |db_007 Session      |docker_002 CPU      |docker_002          |
|77  |❌    |  2|  2|True     |2020-05-31T05:48:00+08:00|docker_003                                                            |docker_003 CPU      |docker_003          |db_007 Session      |
|48  |✅    |  1|  1|True     |2020-05-27T03:23:00+08:00|docker_001                                                            |docker_001          |os_022 Network      |os_021              |
|49  |❌    |  2|  2|True     |2020-05-27T04:39:00+08:00|docker_007                                                            |docker_007 CPU      |docker_007          |db_007 Session      |
|16  |❌    |  3|  3|True     |2020-05-22T05:18:00+08:00|docker_007                                                            |docker_003 CPU      |docker_007 CPU      |docker_007          |
|67  |❌    |  2|  2|True     |2020-05-30T05:43:00+08:00|docker_002                                                            |os_022 Network      |docker_002          |db_007 Session      |
|76  |❌    |  2|  2|True     |2020-05-31T04:47:00+08:00|docker_006                                                            |docker_006 CPU      |docker_006          |docker_004 CPU      |
|62  |❌    |  2|  2|True     |2020-05-30T00:43:00+08:00|docker_005                                                            |docker_005 CPU      |docker_005          |docker_004 CPU      |
|60  |❌    |  2|  2|True     |2020-05-29T05:11:00+08:00|docker_006                                                            |docker_006 CPU      |docker_006          |docker_004 CPU      |
|69  |❌    |  2|  2|True     |2020-05-31T00:47:00+08:00|docker_001                                                            |os_022 Network      |docker_001          |db_007 Session      |
|19  |❌    |  3|  3|False    |2020-05-23T00:40:00+08:00|db_003 Session                                                        |db_003 Load         |db_003              |db_003 Session      |
|4   |❌    |  2|  2|True     |2020-04-11T02:15:00+08:00|db_007 Session                                                        |db_007 Load         |db_007 Session      |db_007              |
|57  |❌    |  2|  2|False    |2020-05-29T02:11:00+08:00|db_003 Session                                                        |os_021 Network      |db_003 Session      |db_007 Session      |
|34  |✅    |  1|  1|True     |2020-05-25T04:47:00+08:00|db_003 State                                                          |db_003 State        |db_007 Session      |db_003 Session      |
|10  |✅    |  1|  1|True     |2020-04-11T05:45:00+08:00|db_003 State                                                          |db_003 State        |db_007 Session      |os_017              |
|40  |✅    |  1|  1|True     |2020-05-26T04:15:00+08:00|os_020 Network                                                        |os_020 Network      |os_021 Network      |docker_004          |
|14  |✅    |  1|  1|True     |2020-05-22T01:48:00+08:00|os_018 Network                                                        |os_018 Network      |docker_002          |os_018              |
|33  |❌    |  2|  2|False    |2020-05-25T03:47:00+08:00|os_017 Network                                                        |os_019 Network      |os_017 Network      |docker_008 CPU      |
|66  |❌    |  2|  2|True     |2020-05-30T05:13:00+08:00|os_018 Network                                                        |os_022 Network      |os_018 Network      |docker_002          |
|43  |✅    |  1|  1|False    |2020-05-27T00:53:00+08:00|os_017 Network                                                        |os_017 Network      |docker_001          |docker_002          |
|73  |✅    |  1|  1|False    |2020-05-31T03:17:00+08:00|os_017 Network                                                        |os_017 Network      |docker_005          |docker_001          |
|72  |✅    |  1|  1|True     |2020-05-31T02:47:00+08:00|os_021 Network                                                        |os_021 Network      |os_021              |os_022              |

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       Test metric             DataLoader 0
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
           A@1                    0.53125
           A@2                    0.90625
           A@3                      1.0
           A@5                      1.0
           MAR                    1.5625
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
2022-09-03 03:03:21.358 | INFO     | DejaVu.workflow:<lambda>:27 - Time Report:
|path                                                                                                   |%total     |%parent    |count   |total      |mean(±std)              |min-max              |
|/train_exp_CFL                                                                                         |    100.00%|    100.00%|       1|    46.105s|    46.105(±     0.000)s|    46.105~    46.105|
|/train_exp_CFL/DejaVuDataset.__getitem__                                                               |      8.96%|      8.96%|    1183|     4.132s|     0.003(±     0.011)s|     0.000~     0.134|
|/train_exp_CFL/DejaVuDataset.__getitem__/MetricPreprocessor.__call__                                   |      0.59%|      6.54%|      78|     0.270s|     0.003(±     0.003)s|     0.002~     0.021|
|/train_exp_CFL/DejaVuDataset.__getitem__/_get_global_id_getter                                         |      0.00%|      0.00%|       1|     0.000s|     0.000(±     0.000)s|     0.000~     0.000|
|/train_exp_CFL/DejaVuDataset.__init__                                                                  |      0.01%|      0.01%|       6|     0.003s|     0.001(±     0.001)s|     0.000~     0.003|
|/train_exp_CFL/DejaVuModelInterface.get_collate_fn.<locals>.collate_fn                                 |      0.37%|      0.37%|      84|     0.171s|     0.002(±     0.001)s|     0.001~     0.009|
|/train_exp_CFL/DejaVuModelInterface.test_step                                                          |      0.21%|      0.21%|       1|     0.098s|     0.098(±     0.000)s|     0.098~     0.098|
|/train_exp_CFL/DejaVuModelInterface.training_step                                                      |     18.07%|     18.07%|      80|     8.332s|     0.104(±     0.022)s|     0.061~     0.165|
|/train_exp_CFL/DejaVuModelInterface.validation_step                                                    |      0.63%|      0.63%|       3|     0.290s|     0.097(±     0.012)s|     0.080~     0.105|
|/train_exp_CFL/Epoch Time                                                                              |     57.58%|     57.58%|      20|    26.546s|     1.327(±     0.241)s|     0.978~     2.106|
|/train_exp_CFL/FDG.load                                                                                |      7.28%|      7.28%|       1|     3.357s|     3.357(±     0.000)s|     3.357~     3.357|
|/train_exp_CFL/GAT.__init__                                                                            |      0.08%|      0.08%|       1|     0.036s|     0.036(±     0.000)s|     0.036~     0.036|
|/train_exp_CFL/MetricPreprocessor.extract_features                                                     |     14.21%|     14.21%|       1|     6.553s|     6.553(±     0.000)s|     6.553~     6.553|
|/train_exp_CFL/MetricPreprocessor.extract_features/instance type iter                                  |     13.94%|     98.10%|      19|     6.429s|     0.338(±     0.213)s|     0.105~     0.888|
|/train_exp_CFL/MetricPreprocessor.extract_features/instance type iter/fill na                          |      2.28%|     16.38%|      19|     1.053s|     0.055(±     0.069)s|     0.005~     0.272|
|/train_exp_CFL/MetricPreprocessor.extract_features/instance type iter/fill na/metric iter              |      0.63%|     27.67%|     710|     0.291s|     0.000(±     0.000)s|     0.000~     0.005|
|/train_exp_CFL/MetricPreprocessor.extract_features/instance type iter/get idx dict                     |      0.00%|      0.01%|      19|     0.001s|     0.000(±     0.000)s|     0.000~     0.000|
|/train_exp_CFL/MetricPreprocessor.extract_features/instance type iter/get values from df               |     11.14%|     79.90%|      19|     5.136s|     0.270(±     0.134)s|     0.099~     0.564|
|/train_exp_CFL/MetricPreprocessor.extract_features/instance type iter/get values from df/fill into feat|      6.20%|     55.68%|      19|     2.860s|     0.151(±     0.121)s|     0.000~     0.377|
|/train_exp_CFL/MetricPreprocessor.extract_features/instance type iter/get values from df/index         |      4.81%|     43.17%|      19|     2.217s|     0.117(±     0.025)s|     0.090~     0.181|
|/train_exp_CFL/MetricPreprocessor.extract_features/ts select                                           |      0.15%|      1.04%|       1|     0.068s|     0.068(±     0.000)s|     0.068~     0.068|
|/train_exp_CFL/_get_global_id_resolver                                                                 |      0.00%|      0.00%|       1|     0.000s|     0.000(±     0.000)s|     0.000~     0.000|

2022-09-03 03:03:22.513 | INFO     | DejaVu.workflow:<lambda>:124 - command output one-line summary: 53.12,90.62,100.00,100.00,1.56,46.105078504000005,,,/SSF/output/run_GAT_node_classification.py.2022-09-03T03:02:35.245139,,,,python exp/run_GAT_node_classification.py -H=4 -L=8 -fe=GRU -bal=True --data_dir=./data/A1 --max_epoch=20,https://github.com/NetManAIOps/DejaVu/tree/00d36dd07eed266840840769ecbc4abf0322319a
train finished. saved to /SSF/output/run_GAT_node_classification.py.2022-09-03T03:02:35.245139
@lizeyan lizeyan pinned this issue Sep 3, 2022
@lizeyan
Copy link
Member Author

lizeyan commented Oct 24, 2022

File structure:
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@lizeyan