Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unicode issue #3

Open
veksha opened this issue Jun 7, 2023 · 3 comments
Open

unicode issue #3

veksha opened this issue Jun 7, 2023 · 3 comments
Assignees

Comments

@veksha
Copy link

veksha commented Jun 7, 2023

Hi, i have an issue when ident contains non-latin characters.
for example, one character Ы will be parsed as two characters: Р«

@glipari
Copy link
Owner

glipari commented Jun 7, 2023

Hello, I am sorry, for the moment there is no Unicode support, just plain ASCII. Introducing Unicode should not too difficult, however, but at the moment I have no time to work on this project.
I may consider Unicode support for the next version, though.
Can you please provide one or more test cases? Thanks in advance.

Best regards,

@veksha
Copy link
Author

veksha commented Jun 11, 2023

here is my test:

unicode_test.cpp
#include <sstream>
#include <tinyparser.hpp>
using namespace tipa;
using namespace std;

struct Record {
    string name, unlock, message;
    void clear() { name = ""; unlock = ""; message = ""; }
};

int main()
{
    const token tk_ident = create_lib_token("^[^\\d\\W][\\w\\s\\.,!?:\\\\]*");
    
    rule unlock  =   rule("unlock") >> rule(":") >> rule("\"") >> rule(tk_ident) >> rule("\"");
    rule message =  rule("message") >> rule(":") >> rule("\"") >> rule(tk_ident) >> rule("\"");
    rule property = unlock | message;
    rule button = rule("\"") >> rule(tk_ident) >> rule("\"") >> rule('{') >> *property >> rule('}');
    rule root = *button;

    vector<Record> records;
    Record temp;

     unlock.read_vars(temp.unlock);
    message.read_vars(temp.message);
    button.set_action([&temp, &records](parser_context &pc) {
            vector<string> v;
            pc.collect_tokens(1, back_inserter(v));
            temp.name = v[0];
            records.push_back(temp);
            temp.clear();
        });

    stringstream s(R"(
        "door ascii" {
            unlock: "silver key"
            message: "This place is unknown to you. Would you like to enter?"
        }
        "door ascii_2" {
            unlock: "test key"
            message: "test"
        }
        "door unicode" {
            unlock: "silver key"
            message: "Это место не изведано. Желаете войти?"
        }
    )");
    
    parser_context pc;
    pc.set_stream(s);

    bool f = false;
    try { f = parse_all(root, pc); }
    catch(parse_exc exc) { cout << "exception caught" << endl; cout << exc.what() << endl; f = false; }

    cout << "Parser status : " << boolalpha << f << endl;
    if (!f) { cout << pc.get_formatted_err_msg() << endl; }
    
    for (auto x: records)
        cout << "> " << x.name << ", unlock: " << x.unlock << ", message: " << x.message << endl;     
}

                                                    /*
g++ unicode_test.cpp tinyparser.a -Itipa/src && a.exe
                                                    */

@glipari
Copy link
Owner

glipari commented Jun 14, 2023

Thank you :) I should have some free time next week, let's see what I can do.

@glipari glipari self-assigned this Jun 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants