template.py AutoTemplate.load_from method read file with encoding utf8

[anexplore]

PR types

Bug fixes

PR changes

Others

Description

because Template.save write to file with encoding utf-8, however AutoTemplate.load_from() read file with platform default encoding, So when the default encoding is GBK，AutoTemplate.load_from() will read a utf8 encoded file with gbk, this will throw error.

Template.save()

    def save(self, save_path):
        if not os.path.exists(save_path):
            os.makedirs(save_path, exist_ok=True)
        template_config_file = os.path.join(save_path, TEMPLATE_CONFIG_FILE)
        template_class = self.__class__.__name__
        with open(template_config_file, "w", encoding="utf-8") as fp:
            fp.write(json.dumps(self._prompt, ensure_ascii=False) + "\n")
            fp.write(json.dumps({"class": template_class}, ensure_ascii=False) + "\n")
        template_param_file = os.path.join(save_path, TEMPLATE_PARAMETER_FILE)

AutoTemplate.load_from()

    def load_from(
        cls, data_path: os.PathLike, tokenizer: PretrainedTokenizer, max_length: int, model: PretrainedModel = None
    ):
        template_config_file = os.path.join(data_path, TEMPLATE_CONFIG_FILE)
        if not os.path.isfile(template_config_file):
            raise ValueError("{} not found under {}".format(TEMPLATE_CONFIG_FILE, data_path))
        with open(template_config_file, "r") as fp:
            config = [x.strip() for x in fp]
            prompt = json.loads(config[0])
            if len(config) > 1:
                template_class = json.loads(config[1])["class"]

[CLAassistant]

All committers have signed the CLA.

[codecov]

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (5500d09) 57.91% compared to head (5f6b9be) 57.91%.
Report is 1 commits behind head on develop.

Files	Patch %	Lines
...erimental/transformers/fused_transformer_layers.py	0.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #7558      +/-   ##
===========================================
- Coverage    57.91%   57.91%   -0.01%     
===========================================
  Files          579      579              
  Lines        86390    86392       +2     
===========================================
  Hits         50036    50036              
- Misses       36354    36356       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sijunhe]

lgtm. thank you for the contribution!