Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a from_pro function to the Process class in TM1py #383

Closed
wimgielis opened this issue Oct 2, 2020 · 11 comments
Closed

Add a from_pro function to the Process class in TM1py #383

wimgielis opened this issue Oct 2, 2020 · 11 comments
Labels
question

Comments

@wimgielis
Copy link
Contributor

Describe what did you try to do with TM1py
It would be nice to have hot promotion for TI processes, where the source is a, existing PRO file. Say, a process was created in development, and we want to promote it to production without bouncing the TM1 server.

Describe what's not working the way you expect
This would be an enhancement.

Refer to: cubewise-code/tm1py-samples#77
@scrambldchannel
Copy link
Contributor

scrambldchannel commented Oct 2, 2020 •
edited
Loading

This is an interesting use case @wimgielis. As Marius already alluded to, the tricky part is parsing a .pro file to pull out the necessary fields to create an instance of the Process object. Once you have that, it can be created on the server.

I'm not sure I'm brave enough to try right now, it seems a bit hairy :) I wonder though if there's anything out there documenting the structure of the .pro files? Maybe someone involved in the Bedrock project has published something?

@wimgielis
Copy link
Contributor Author

wimgielis commented Oct 2, 2020 via email

@scrambldchannel
Copy link
Contributor

Thanks, that's interesting... I am looking at the IBM docs with trepidation ;)

I think this weekend I'd like to get outside a bit more! Let's see, those codes are really useful but it's still a bit fiddly

Cheers
Alex

@adscheevel
Copy link
Collaborator

Does the source have to be a PRO file? I use TM1Py to hot promote from dev environment to PROD but I do that by connecting to both environments, getting the desired process from the source environment, and then create in the target environment with the process object I just got from Dev. Below is example script of what I've used in the past where I can change the name of the process when promoting.

tm1Dev = TM1Service(....)
tm1Prod = TM1Service(...)

SrcProcName = 'zTEMP - Hot Promote ~20201002'
TarProcName = 'zTEMP - Hot PromotED ~20201002'

oProcess = tm1Dev.processes.get(SrcProcName)
oProcess.name = TarProcName

tm1Prod.processes.create(oProcess)

@wimgielis
Copy link
Contributor Author

wimgielis commented Oct 2, 2020 •
edited
Loading

Hello @adscheevel

I think there are 2 things to mention here:

  • we could use a PRO file, since that would make most sense. We are not using text files (other than PRO files) or databases to store coding, variables, etc. So PRO files make sense, more than any other source, I would argue. It could be a download of an earlier TM1 model, or a model from a different customer, anything.

  • I was thinking about a PRO file that is not an active process in a TM1 model. If we talk about promoting a process (as part of the }Processes dimension) from 1 model to a different model, we could use your approach. However, the idea was to make a PRO file available in a target TM1 model, where the source is a PRO text file (correct process syntax needed) and just sitting on the hard disk somewhere. For example, there could be a new Bedrock process, we download it, and promote it to a TM1 model without restarting TM1. Or I could have an own library of TI processes, I don't need all of the processes, but when I create or update a processs I would be happy to add it to certain TM1 models without having to restart the models.

Maybe...using TM1py to create a simple small TM1 model with just that 1 or a few TI processes (no cubes, dims, ...), then hot promoting like you said, would be a solution ? That's food for thought for @MariusWirtz probably :-)

@scrambldchannel
Copy link
Contributor

scrambldchannel commented Oct 4, 2020 •
edited
Loading

Hi @wimgielis

@adscheevel's method sounds a good solution and you could, as you say, potentially create a server that is just repository for all processes and promote them for there. Presumably, you're using one of the native clients or Arc to create the processes in the first place which is going to be easier than trying to edit the pro file directly.

It's nice to be able to save things as text though if you want to take advantage of version control. I toyed with exporting TI processes, dimensions and cubes to JSON using TM1py's built-in methods and then editing the JSON but it was a bit fiddly to try to edit objects like large dimensions.

The direction IBM seems to be going is to manage TM1 models in git like this repo from Hubert. Processes are defined in two files, one containing the metadata for the process and one containing the lines of TI code which makes it a bit nicer to edit them. I haven't tested this though.

Anyway, the weather was horrible here yesterday so I did manage to get a rough proof of concept going for your original request. It's a bit clunky but seems to work at least in some cases. It might give you something to work with, no promises though :)
I wrote a bit more about it here noting a few issues with handling embedded quotes and commas.

Create a dictionary of all codes and their values:

# codes to treat differently
multiline_codes = ['560', '561', '572', '573', '574', '575', '577', '578', '579', '580', '581', '582', '566']
multiline_codes_with_key = ['590','637']

# location of pro file to load
file = "}bedrock.cube.rule.processfeeders.pro"

with open(file, encoding='utf-8-sig') as f:

    process_dict = {}
    in_multiline = False
    in_multiline_with_key = False
    code = ''

    for line in f:
        if in_multiline:
            process_dict[code].append(line.replace('"', '').rstrip())
            lines = lines - 1
            if lines == 0:
                in_multiline = False
        elif in_multiline_with_key:
            fields = line.split(',')
            process_dict[code].append(fields[1].replace('"', '').rstrip())
            lines = lines - 1
            if lines == 0:
                in_multiline_with_key = False
        else:
            fields = line.split(',')
            code = fields[0]
            if code in multiline_codes:
                lines = int(fields[1])
                if lines > 0:
                    in_multiline = True
                process_dict[code] = []
            elif code in multiline_codes_with_key:
                lines = int(fields[1])
                if lines > 0:
                    in_multiline_with_key = True
                process_dict[code] = []
            else:
                # hacky way to deal with commas in the set values, eg where ',' is set as one of the delimiter character
                process_dict[code] = ''.join(fields[1:]).replace('"', '').rstrip() # hacky way to deal with commas in the set values

Try to build an instance of the Process class:

import TM1py

my_new_process = TM1py.Objects.Process(
    name=process_dict['602'],
    has_security_access=(True if process_dict['1217'] == 'True' else False),
    ui_data=process_dict['576'],
    prolog_procedure="\n".join(process_dict['572']),
    metadata_procedure="\n".join(process_dict['573']),
    data_procedure="\n".join(process_dict['574']),
    epilog_procedure="\n".join(process_dict['575']),
    datasource_type='None',
    datasource_ascii_decimal_separator=process_dict['588'],
    datasource_ascii_delimiter_char=process_dict['567'],
    datasource_ascii_delimiter_type='Character', # doesn't seem to have a corresponding code
    datasource_ascii_header_records=process_dict['569'],
    datasource_ascii_quote_character=process_dict['568'],
    datasource_ascii_thousand_separator=process_dict['589'],
    datasource_data_source_name_for_client=process_dict['585'],
    datasource_data_source_name_for_server=process_dict['586'],
    datasource_password=process_dict['565'],
    datasource_user_name=process_dict['564'],
    datasource_query=process_dict['566'],
    datasource_uses_unicode=process_dict['559'],
    datasource_view=process_dict['570'],
    datasource_subset=process_dict['571']
)

# now add parameters and variables

for index, item in enumerate(process_dict['560']):

    if process_dict['561'][index] == "2":
        parameter_type = "String"
        value = process_dict['590'][index]
    else:
        parameter_type = "Numeric"
        if process_dict['590'][index] == "":
            value = 0
        else:
            value = float(process_dict['590'][index])

    my_new_process.add_parameter(
        name=item,
        prompt=process_dict['637'][index],
        value=value,
        parameter_type=parameter_type
    )

for index, item in enumerate(process_dict['577']):

    variable_type = "String" if process_dict['578'][index] == "2" else "Numeric"        

    my_new_process.add_variable(
        name=item,
        variable_type=variable_type
    )

Connect to TM1 and add the process:

import configparser

# establish connection / how you do this is up to you
config = configparser.ConfigParser()
config.read('config.ini')

with TM1py.Services.TM1Service(**config['tm1srv01']) as tm1:

    if tm1.processes.exists(my_new_process.name):
        tm1.processes.delete(my_new_process.name)

    response = tm1.processes.create(my_new_process)

    # check status of response
    print(response.status_code)

@wimgielis
Copy link
Contributor Author

wimgielis commented Oct 4, 2020 via email

@wimgielis
Copy link
Contributor Author

wimgielis commented Oct 5, 2020 via email

@scrambldchannel
Copy link
Contributor

Hi Alexander, Is it true that you explicit set None data source for the process ?

Ah, well spotted, I must have hacked that in. I'll take a look once I've had some sleep :)

@cubewise-tryan
Copy link
Member

Hi all,

Just a warning on going down this path, it is easy enough for simple examples but it will take a lot of effort to make it work generically across all of the options that are available in TI. There are lots of little quirks in the pro file format, we know from our experience parsing it for Pulse.

It is much easier to just extract the process via the REST API and then updating it using that source. It isn't very difficult to start up a TM1 server from the command line.

@MariusWirtz
Copy link
Collaborator

I tend to agree with @cubewise-tryan on this one.

Happy to accept a PR though if someone can actually make it work :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@MariusWirtz @wimgielis @cubewise-tryan @adscheevel @scrambldchannel