Skip to content
This repository has been archived by the owner on May 19, 2023. It is now read-only.

cyBERT and WEP fixes after upstream cuDF updates #487

Merged
merged 5 commits into from
May 12, 2022
Merged

cyBERT and WEP fixes after upstream cuDF updates #487

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8b4643b
remove problem deps and update pytorch in gpu build
efajardo-nv May 11, 2022
ab2bba3
fix cybert after cudf upstream updates
efajardo-nv May 11, 2022
2c226ff
fix wep after upstream cudf updates
efajardo-nv May 11, 2022
22234c8
revert event_parser change
efajardo-nv May 11, 2022
bd12569
add rapids dependencies back to gpu build after ops fix
efajardo-nv May 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ci/gpu/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Copyright (c) 2018-2020, NVIDIA CORPORATION.
# Copyright (c) 2018-2022, NVIDIA CORPORATION.
##########################################
# CLX GPU build & testscript for CI #
##########################################
Expand Down Expand Up @@ -58,7 +58,7 @@ gpuci_mamba_retry install -y \
# gpuci_conda_retry remove --force rapids-build-env rapids-notebook-env
# gpuci_conda_retry install -y "your-pkg=1.0.0"

pip install -U torch==1.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
pip install -U torch==1.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
pip install "git+https://github.com/rapidsai/cudatashader.git"
pip install "git+https://github.com/slashnext/SlashNext-URL-Analysis-and-Enrichment.git#egg=slashnext-phishing-ir&subdirectory=Python SDK/src"
pip install mockito
Expand Down
2 changes: 1 addition & 1 deletion python/clx/analytics/cybert.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ def inference(self, raw_data_col, batch_size=160):
confidences, labels = torch.max(logits, 2)
confidences_list.extend(confidences.detach().cpu().numpy().tolist())
labels_list.extend(labels.detach().cpu().numpy().tolist())
infer_pdf = pd.DataFrame(meta_data).astype(int)
infer_pdf = pd.DataFrame(meta_data.cpu()).astype(int)
infer_pdf.columns = ["doc", "start", "stop"]
infer_pdf["confidences"] = confidences_list
infer_pdf["labels"] = labels_list
Expand Down
11 changes: 6 additions & 5 deletions python/clx/parsers/windows_event_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,11 +53,12 @@ def parse(self, dataframe, raw_column):
for eventcode in self.event_regex.keys():
pattern = "eventcode=%s" % (eventcode)
input_chunk = self.filter_by_pattern(dataframe, raw_column, pattern)
temp = self.parse_raw_event(
input_chunk, raw_column, self.event_regex[eventcode]
)
if not temp.empty:
output_chunks.append(temp)
if not input_chunk.empty:
temp = self.parse_raw_event(
input_chunk, raw_column, self.event_regex[eventcode]
)
if not temp.empty:
output_chunks.append(temp)
parsed_dataframe = cudf.concat(output_chunks)
# Replace null values with empty.
parsed_dataframe = parsed_dataframe.fillna("")
Expand Down