Refactor Directory Structure and Update CI/CD Pipelines

[TaekyungHeo]

Summary

This PR reorganizes the directory structure to enhance maintainability and readability. Additionally, it applies ruff format and ruff check fix to standardize the coding style. Import statements have been properly adjusted based on the updated paths. We have temporarily removed Pyre and GitHub Actions so that they can be properly set up in a subsequent PR.

• Remove blank line from cla.yml
• Remove build test and pyre from GitHub actions
• Rename *2chakra_converter to *_converter
• Adjust import statements
• Remove .flake8
• Fix ruff check errors in src/converter/pytorch2chakra_converter.py
• Fix ruff check errors in src/timeline_visualizer/timeline_visualizer.py
• Fix ruff check errors in src/trace_link/trace_link.py
• Apply ruff check --fix .
• Ignore EXE001 in ruff check
• Reorganize directories

Test Plan

1. chakra_trace_link

$ chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_et_0.json --kineto-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_kineto_0.json --output-file ~/megatron_0.json &
$ chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_et_1.json --kineto-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_kineto_1.json --output-file ~/megatron_1.json &
$ chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_et_2.json --kineto-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_kineto_2.json --output-file ~/megatron_2.json &
$ chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_et_3.json --kineto-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_kineto_3.json --output-file ~/megatron_3.json &
$ chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_et_4.json --kineto-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_kineto_4.json --output-file ~/megatron_4.json &
$ chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_et_5.json --kineto-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_kineto_5.json --output-file ~/megatron_5.json &
$ chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_et_6.json --kineto-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_kineto_6.json --output-file ~/megatron_6.json &
$ chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_et_7.json --kineto-file /Users/theo/Downloads/llama2_7b_bf16_4node/megatron_kineto_7.json --output-file ~/megatron_7.json &

2. chakra_converter

$ chakra_converter --input_filename ~/megatron_0.json --output_filename megatron_0.chakra --input_type PyTorch > /tmp/rank_0 &
$ chakra_converter --input_filename ~/megatron_1.json --output_filename megatron_1.chakra --input_type PyTorch > /tmp/rank_1 &
$ chakra_converter --input_filename ~/megatron_2.json --output_filename megatron_2.chakra --input_type PyTorch > /tmp/rank_2 &
$ chakra_converter --input_filename ~/megatron_3.json --output_filename megatron_3.chakra --input_type PyTorch > /tmp/rank_3 &
$ chakra_converter --input_filename ~/megatron_4.json --output_filename megatron_4.chakra --input_type PyTorch > /tmp/rank_4 &
$ chakra_converter --input_filename ~/megatron_5.json --output_filename megatron_5.chakra --input_type PyTorch > /tmp/rank_5 &
$ chakra_converter --input_filename ~/megatron_6.json --output_filename megatron_6.chakra --input_type PyTorch > /tmp/rank_6 &
$ chakra_converter --input_filename ~/megatron_7.json --output_filename megatron_7.chakra --input_type PyTorch > /tmp/rank_7 &

3. chakra_jsonizer

$ chakra_jsonizer --input_filename megatron_0.chakra --output_filename megatron_0.json

4. chakra_visualizer

$ chakra_visualizer --input_filename megatron_1.chakra --output_filename megatron_1.dot
$ chakra_visualizer --input_filename megatron_1.chakra --output_filename megatron_1.pdf
$ chakra_visualizer --input_filename megatron_1.chakra --output_filename megatron_1.graphml

5. chakra_timeline_visualizer

$ chakra_timeline_visualizer                                                          
usage: chakra_timeline_visualizer [-h] --input_filename INPUT_FILENAME --output_filename OUTPUT_FILENAME --num_npus NUM_NPUS --npu_frequency NPU_FREQUENCY [--log_filename LOG_FILENAME]
chakra_timeline_visualizer: error: the following arguments are required: --input_filename, --output_filename, --num_npus, --npu_frequency

6. Correlation
   6.1 Run trace_link

chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama_pytorch24.05/megatron_et_0.json --kineto-file /Users/theo/Downloads/llama_pytorch24.05/megatron_kineto_0.json --output-file ~/megatron_0.json &
chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama_pytorch24.05/megatron_et_1.json --kineto-file /Users/theo/Downloads/llama_pytorch24.05/megatron_kineto_1.json --output-file ~/megatron_1.json &
chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama_pytorch24.05/megatron_et_2.json --kineto-file /Users/theo/Downloads/llama_pytorch24.05/megatron_kineto_2.json --output-file ~/megatron_2.json &
chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama_pytorch24.05/megatron_et_3.json --kineto-file /Users/theo/Downloads/llama_pytorch24.05/megatron_kineto_3.json --output-file ~/megatron_3.json &
chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama_pytorch24.05/megatron_et_4.json --kineto-file /Users/theo/Downloads/llama_pytorch24.05/megatron_kineto_4.json --output-file ~/megatron_4.json &
chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama_pytorch24.05/megatron_et_5.json --kineto-file /Users/theo/Downloads/llama_pytorch24.05/megatron_kineto_5.json --output-file ~/megatron_5.json &
chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama_pytorch24.05/megatron_et_6.json --kineto-file /Users/theo/Downloads/llama_pytorch24.05/megatron_kineto_6.json --output-file ~/megatron_6.json &
chakra_trace_link --pytorch-et-file /Users/theo/Downloads/llama_pytorch24.05/megatron_et_7.json --kineto-file /Users/theo/Downloads/llama_pytorch24.05/megatron_kineto_7.json --output-file ~/megatron_7.json &

6.2 Run et_converter

chakra_converter --input_filename ~/megatron_0.json --output_filename megatron_0.chakra --input_type PyTorch > /tmp/rank_0 &
chakra_converter --input_filename ~/megatron_1.json --output_filename megatron_1.chakra --input_type PyTorch > /tmp/rank_1 &
chakra_converter --input_filename ~/megatron_2.json --output_filename megatron_2.chakra --input_type PyTorch > /tmp/rank_2 &
chakra_converter --input_filename ~/megatron_3.json --output_filename megatron_3.chakra --input_type PyTorch > /tmp/rank_3 &
chakra_converter --input_filename ~/megatron_4.json --output_filename megatron_4.chakra --input_type PyTorch > /tmp/rank_4 &
chakra_converter --input_filename ~/megatron_5.json --output_filename megatron_5.chakra --input_type PyTorch > /tmp/rank_5 &
chakra_converter --input_filename ~/megatron_6.json --output_filename megatron_6.chakra --input_type PyTorch > /tmp/rank_6 &
chakra_converter --input_filename ~/megatron_7.json --output_filename megatron_7.chakra --input_type PyTorch > /tmp/rank_7 &

6.3 Results

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅