Skip to content

Latest commit

 

History

History
executable file
·
121 lines (97 loc) · 5 KB

README_EXPS.md

File metadata and controls

executable file
·
121 lines (97 loc) · 5 KB
Raw

Distributed NMFk Experiments

Experiments for Test cases

For data1,data2,...,data10 example datasets in data/ directory use launchscript.sh

sh launchscript.sh

The required dataset to run can be specified in the script in matname variable.

  • This split data in 2d grid as shown below, if the processor grid needs to be changed, note that Pr should be divisible by m, Pc should be divisible by n.
  • A job on one node is submitted to perform distnmfk on the benchmark dataset. Once the job is completed, use the postprocessing.sh to extract the necessary files and plots.
sh postprocessing.sh "${PWD}/Results_${matname}/"

i.e. if the dataset you wish to run is data1

sh postprocessing.sh "${PWD}/Results_data1/"

The above script does the following

  • Extract the necessary files containing 1. 'Avg_sil.csv' 2. 'Min_sil.csv' 3. 'Relerr_Avg.csv' in Results_${matname}/ directory

  • Plots the results and saves it into png in Results_${matname}/ directory

Experiments for Swimmer Data

For Swimmer example dataset in data/ directory use launchscript_swimmer.sh

sh launchscript_swimmer.sh
  • A job on eight nodes is submitted to perform distnmfk on the benchmark dataset. Once the job is completed, use the postprocessing.sh to extract the necessary files and plots.
sh postprocessing.sh "${PWD}/Results_swim/"

The above script does the following

  • Extract the necessary files containing 1. 'Avg_sil.csv' 2. 'Min_sil.csv' 3. 'Relerr_Avg.csv' in Results_$swim/ directory

  • Plots the results and saves it into png in Results_$swim/ directory

MoreDetails (Inside the scripts)

Pre Processing (Input to Distnmfk)

This current version reads the input of the original data, split based on the processor grid. 1d and 2d procssor grid splitting can be done using the file split_data_2d.py

matname="data4"
pr=16
pc=16
python split_data_2d.py "data/${matname}.mat" $pr $pc "Results_${matname}/"

Running Distnmfk on the splitdata

Script involving the slurm scheduler job script and distnmfk execution is given in launchscript.sh and run_Datamats.sh and is submitted using the command

pr=16
pc=16
m=1024
n=256
sbatch run_Datamats.sh ${PWD}/Results_${matname}/ 1 $pr $pc $m $n

General running instructions and parameters

mpirun -np p "distnmfk" -a $i -k $k -i "random" -t $j -d "240 180" -p "$pr $pc" -o "nmfout_${out}" -r "0.001 0 0.001 0" -e 1 

p : Number of Processors/cores
a : algorithm
k : For k, rank of NMF
i : input type or data location
t : Number of iterations
d : Dimensions of the matrix to be decomposed
o : directory to produce output files
e : Flag to print relative error

Post Processing (After Distnmfk completion)

use _postprocessing.sh _ to collect silhouettes and relative error after distnmfk decomposition

python collectsil.py "${PWD}/Results_${matname}/"
python collectrel.py "${PWD}/Results_${matname}/"

Plotting Results

Use the function plot_results.py to plot the collected Silhouette widths and Relative error using

python plot_results.py "${PWD}/Results_${matname}/" "${matname}"

This is the example figure obtained for data1 in benchmark datasets

Output interpretation

For W matrix row major ordering. That is., W_0, W_1, .., W_p For H matrix column major ordering. That is., for 6 processes with pr=3, pc=2, interpret as H_0, H_2, H_4, H_1, H_3, H_5

Citation:

If you are using this implementation, kindly cite.

Chennupati, G., Vangara, R., Skau, E., Djidjev, H., & Alexandrov, B. (2020). Distributed non-negative matrix factorization with determination of the number of latent features. The Journal of Supercomputing, 76(9), 7458–7488. https://doi.org/10.1007/s11227-020-03181-6