Running simulations will pass sequentially but fail if using –num-threads. (ACTIVE HDL)

[Dr34d-P1r4t3-R0b3rts]

Using Vunit with ACTIVE HDL.
Using multiple threads (run.py –num-threads n) will result in some of the tests failing with the following error.

ASDB: Error: remove: "C:\Projects\Devlop\Vunit\vunit_out\activehdl\wave.asdbw\strings.data": The process cannot access the file because it is being used by another process.

I found that a directory called wave.asdbw is created during the simulations and is then deleted once the simulation is finished. If multiple simulations are being run, then multiple processes are trying to strings.data at the same time.

Is there a configuration within Vunit which can change the directory name to avoid this?

[LarsAsplund]

@Dr34d-P1r4t3-R0b3rts What versions of VUnit and Active-HDL are you using?

[Dr34d-P1r4t3-R0b3rts]

@LarsAsplund Thanks for getting back to me. I am using the following.
VUnit - 4.6.0
Active-HDL - v12 (64-bit)

I have done some digging around and it seems like a similar fix has been applied for other simulators, but I am new to python so replicating such a fix is a little beyond me at the moment.

[LarsAsplund]

How many Active-HDL licenses do you have? Note that every thread needs a license so if you run more threads than licenses you will fail. In the log you would see something like:

VSIM: Fatal Error: (FLEXlm error = -4) Maximum number of users reached.
VSIM: Fatal Error: License unavailable

Sounds like you have another problem but I just want to make sure.

If this is not the case could you show me the complete output? I only have one license so I can't really test multiple threads with Active-HDL.

[Dr34d-P1r4t3-R0b3rts]

I have enough licences to cover the number of threads I am using.
I have attached some output.txt files and the transcript from the cmd window. Let me know if there is anything else i can send over which will be useful.
output - TC001.txt
output - TC002.txt
output - TC003.txt
output - TC004.txt
output - TC005.txt
VUnit CMD output.txt

[Andy-Darlington]

As previously mentioned, the failures are caused by ActiveHDL attempting to write to the same simulation data file for multiple tests. By default ActiveHDL, will create a file called wave.asdb. We can change this using the argument -asdb with the asim command. Unfortunately this doesn't do much good when adding this to a Run.py because this will be applied to every test case, so we still have the same problem of the same simulation database file being written to by multiple tests running in parallel.

I had a poke around the VUnit source code and managed to implement a bodge that proves out the functionality.
Here's a snippet of the code I've added to the testcase class.

`        # Create a unique name based on the test case and configuration
        if (str(config.name) != "None"):
            uniqueTestCaseName = str(config._design_unit.name) + "_" + str(config.name) + "_" + str(test.name)
        else:
            uniqueTestCaseName = str(config._design_unit.name) + "_" + str(test.name)
        uniqueAsdbName = ("-asdb " + uniqueTestCaseName + ".asdb")
        # Get the existing vsim_flags for this test case
        vsim_args = []
        vsim_args = config.sim_options.get("activehdl.vsim_flags", vsim_args)

        # Append our vsim_flags with the unique name for a asdb file
        vsim_args.append(uniqueAsdbName)
        # Update the test case's simulation options
        config.set_sim_option("activehdl.vsim_flags", vsim_args)`

This will update the sim_options for each test case to include the -asdb argument with a unique name for each test. This does fix the errors encountered when using the num_threads argument however as mentioned its a bodge and has a number of shortcomings such as:

1. Using the -asdb argument in the Run.py will be overwritten by this code
2. The vunit_out/activehdl folder will become cluttered with a unique simulation database for every test case in the codebase regardless of whether the num_threads / -p option is used.
3. This is a fix targeting ActiveHDL but will impact usage with any simulator. I have done no testing with any other simulator.

@LarsAsplund I don't have adequate knowledge of the VUnit source code to implement this properly or access to other simulators to adequately test it. Is this a feature you'd be interested in implementing? Its something I'd be very keen to see supported as it has the potential to massively reduce our simulation times.

[LarsAsplund]

I will see if I can setup an environment will multiple licenses so that I can recreate this.

[LarsAsplund]

@Dr34d-P1r4t3-R0b3rts I haven't had the time to do any work on this but I have a follow-up question: Did multiple threads work with previous versions of Active-HDL? Do you know?

[Marek-ADT]

There is need to change the way how Active-HDL vsimsa is started and use the working folder and configuration files.
I have changed the activehdl.py to support more than 1 thread.
Try this out.
activehdl.zip

[LarsAsplund]

@Marek-ADT Your changes look correct but I'm thinking about this line:

batch_do += f'cd "{fix_path(output_path)!s}"\n'

vsim is called with the batch script from the output_path directory. Will Active-HDL change that such that the batch script must change directory again?

@Dr34d-P1r4t3-R0b3rts and @Andy-Darlington, can you try this fix?

@Andy-Darlington If I understand your code correctly you're creating unique names for the ASDB file. @Marek-ADT solution have the same name but put that file in the unique directory that we already have for each test. For that reason I prefer that approach.

[Andy-Darlington]

@Marek-ADT I've given your code a go and it's working well for me, thank you very much! I've tested on ActiveHDL 12.0 x64.

@LarsAsplund Yes, completely agree. Marek-ADT approach to keeping in the activehdl.py file is much more sensible. I'd consider mine a proof of concept to myself, there are still unresolved issues.

Btw, I tested multiple threads on a few versions of Active. It works in 10.4 32-bit but fails in 12.0 x64 and 13.0 x64 (this is without using either mine or Marek's code).

[Marek-ADT]

There is change to define wave.asdb file in the new place
asdb_file_path = str(Path(output_path) / "wave.asdb")
asdb_file_path_fix = "-asdb " + fix_path(asdb_file_path)
vsim_flags = [
asdb_file_path_fix,
...
but it is not needed, if the pwd (working folder) for each job will be different.
If the job is one, there is no need to change working folder, if there are many jobs each have to have own folder for the task.
There is vsimsa.cfg file and keeps inside the asdb path and name and it is updated by the dedicated set or vsim -asdb call.
If you for many jobs will have save vsimsa.cfg file and change only the name or path of asdb file, still there will conflict on vsimsa.cfg, each of parallel jobs will try to write to it. That is why there is need to define unique working folder before vsim call.
At that time each job has his own library.cfg, vsimsa.cfg and also asdb file in case if user want to create waveform file.

[LarsAsplund]

Correction. All Active-HDL threads starts up in the same directory but test outputs go to a dedicated directory. The -asdb option moves the ASDB file to the separate directory and we do the same in Riviera-PRO but then it is called -dataset. In Riviera-PRO there is no vimsa.cfg so nothing else needs to be done.

As you mentioned, @Andy-Darlington, multiple threads worked with previous Active-HDL versions so this must have changed along the way.

We can make Active-HDL switch simulator working directory as @Marek-ADT showed but the problem is that the simulator class isn't expected to do that. The working directory is provided by higher layers of VUnit.

If we change that directory locally things like the simulator_output_path argument to a pre_config function (other name for the working directory) will no longer be correct. A more fundamental change has to be made for this to work all the way.

Fortunately that work has been done for other reasons in another PR, #785 , but that work has been blocked for some time because we need someone to verify the changes when using Xcelium.

It looks like we have a solution but there are some complicating factors.

[Marek-ADT]

Do you read or modify the library.cfg during vsim or run process? From: Lars Asplund ***@***.*** Sent: Saturday, January 28, 2023 12:16 AM To: VUnit/vunit ***@***.***> Cc: Marek-ADT ***@***.***>; Mention ***@***.***> Subject: Re: [VUnit/vunit] Running simulations will pass sequentially but fail if using –num-threads. (ACTIVE HDL) (Issue #862) Correction. All Active-HDL threads starts up in the same directory but test outputs go to a dedicated directory. The -asdb option moves the ASDB file to the separate directory and we do the same in Riviera-PRO but then it is called -dataset. In Riviera-PRO there is no vimsa.cfg so nothing else needs to be done. As you mentioned, @Andy-Darlington <https://github.com/Andy-Darlington> , multiple threads worked with previous Active-HDL versions so this must have changed along the way. We can make Active-HDL switch simulator working directory as @Marek-ADT <https://github.com/Marek-ADT> showed but the problem is that the simulator class isn't expected to do that. The working directory is provided by higher layers of VUnit. If we change that directory locally things like the simulator_output_path argument to a pre_config function (other name for the working directory) will no longer be correct. A more fundamental change has to be made for this to work all the way. Fortunately that work has been done for other reasons in another PR, #785 <#785> , but that work has been blocked for some time because we need someone to verify the changes when using Xcelium. It looks like we have a solution but there are some complicating factors. — Reply to this email directly, view it on GitHub <#862 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARIHLP46RFWLSW5YSTXXCQLWURJJFANCNFSM6AAAAAARKCTXR4> . You are receiving this because you were mentioned. <https://github.com/notifications/beacon/ARIHLP4RZMVK2NYPVVLYPSDWURJJFA5CNFSM6AAAAAARKCTXR6WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTST37SDC.gif> Message ID: ***@***.*** ***@***.***> >