Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unittest for PkPD conversion model fails #53

Closed
tph-thuering opened this issue May 7, 2015 · 7 comments
Closed

unittest for PkPD conversion model fails #53

tph-thuering opened this issue May 7, 2015 · 7 comments
Assignees
@tph-thuering
Labels
1: high
Milestone
Schema 34

Comments

@tph-thuering
Copy link
Contributor

Metabolite concentration values are negative.

relevant unittests are called testAR and testAS, both use the conversion model with DHA-metabolites

see Unittest output

For a more detailed unittest with intermediate values for g, h, i and qtyM, qtyP output,
see the first revision

Hint - if you'd like to reproduce the verbose output: remove comment from this line and compile OM.
@tph-thuering tph-thuering self-assigned this May 7, 2015
@tph-thuering tph-thuering added this to the Resistance milestone May 7, 2015
@tph-thuering tph-thuering modified the milestones: Schema 34, Resistance, Build infrastructure May 7, 2015
@tph-thuering
Copy link
Contributor Author

Monolix PKPDlibrary reference: http://www.lixoft.eu/wp-content/resources/docs/PKPDlibrary.pdf
For three compartment model see page 44. (Not sure if conversion model is related or not)

@tph-thuering tph-thuering assigned dhardy and unassigned tph-thuering May 13, 2015
tph-thuering added a commit that referenced this issue May 13, 2015
@tph-thuering
Drug conversion model: Fix one equation for #53
4c61277 
This patch generates closer results for concentrations, but not yet precise enough.

Fixes one equation, which Ian Hastings pointed out in "Openmalaria PKPD equations.pdf" Page 6.
Altough he meant x, he wrote z. -> fix: x should be k.

Unittests:
testAR()
Concentration has the wrong sign, but the amount is very near to the expected value.

testAS()
This patch generates closer results for concentration, but not yet precise enough.

TODO: Check AR parameterization.
tph-thuering added a commit that referenced this issue Jun 2, 2015
@tph-thuering
Revert sign corrections and revert Fix (#53)
c41dec1 
Reverts
- 0f5fcda "Drug conversion model: fix some signs in equations"
- 0f41f49 "PKPD conversion model: Another sign correction"
- 4c61277 "Drug conversion model: Fix one equation for #53"
@tph-thuering tph-thuering assigned tph-thuering and unassigned dhardy Jun 2, 2015
@tph-thuering tph-thuering modified the milestones: Schema 34, Resistance Jul 9, 2015
@tph-thuering
Copy link
Contributor Author

After applying this patch https://gist.github.com/tph-thuering/288474a8ae514b0d9d55#ian.patch

And running the unittests, we get the following results:

Running 58 tests.................................................
In PkPdComplianceSuite::testAR:

Artemether conversion unittest (AR):

Type day difference found expected
DF 0 0.00000012266375776 0.00002453275155652 0.00000017000000002
CO 0 0.00000100949828987 -0.00020189965787820 0.00020131259995317
DF 1 0.00000000000300366 0.00000000060073299 0.00000000000002840
CO 1 0.00000100949195449 -0.00020189839099318 0.00020131390006428
DF 2 0.00000000000000007 0.00000000000001471 0.00000000000000000
CO 2 0.00000100949195445 -0.00020189839098524 0.00020131390006428
DF 3 0.00000000000000007 0.00000000000001471 0.00000000000000000
CO 3 0.00000000000635000 0.00000000126063351 0.00000000127000000
DF 4 0.00000000000000007 0.00000000000001471 0.00000000000000000

.
In PkPdComplianceSuite::testAS:

Arthesunate conversion unittest(AS):

Type day difference found expected
DF 0 0.00002387377793806 0.00477475558105623 0.00053229079937151
CO 0 0.00000000000057577 0.00000000011515423 0.00000000011400000
DF 1 0.00000011399145468 0.00002279829092163 0.00000028300000004
CO 1 0.00000000000057577 0.00000000011515423 0.00000000011400000
DF 2 0.00000000054428133 0.00000010885626694 0.00000000015100000
CO 2 0.00000000000057577 0.00000000011515423 0.00000000011400000
DF 3 0.00000000054428133 0.00000010885626694 0.00000000015100000
CO 3 0.00000000000000000 0.00000000000000000 0.00000000000000000
DF 4 0.00000000054428133 0.00000010885626694 0.00000000015100000

......
Failed 2 of 58 tests
Success rate: 96%

tph-thuering added a commit that referenced this issue Jul 14, 2015
@tph-thuering
Update PkPd Conversion Drug factors #53
c69a010 
enable conversion unittests again
disable Cloroquine unittest for now
tph-thuering added a commit that referenced this issue Jul 14, 2015
@tph-thuering
PkPd: Fix #53 lower drug factor unittest precision
59949f1 
, as concentrations seem ok now for drug conversion unittests
@tph-thuering
Copy link
Contributor Author

Running 58 tests................................................

Testing AR1:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 1 0 0 -nan
1 1.03393e-12 -4.03897e-27 1 0.015352 -3.65369e-11 1
2 1.06825e-24 -6.19157e-28 0.999421 0.0156447 1.06909e-11 1
3 1.1037e-36 4.06951e-40 1.00037 0.0156502 4.77728e-05 1.00306
4 1.7314e-42 -2.99925e-42 0.365996 0.000298342 4.56162e-10 1
5 1.73054e-42 -3.0001e-42 0.365815 5.68734e-06 -3.66015e-12 0.999999

.

Testing AR:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 1 0 0 -nan 0 0
1 0.00148275 0.00148258 8746.55 0.000182523 -1.39572e-11 1 0.000201313 -2.97398e-11
2 2.15304e-06 2.15304e-06 7.58607e+07 0.000182524 3.29943e-11 1 0.000201314 -6.28492e-11
3 3.12634e-09 3.12634e-09 6.59562e+11 0.000182524 3.30015e-11 1 0.000201314 -6.28413e-11
4 3.12634e-09 3.12634e-09 6.57971e+11 1.14696e-09 -3.04129e-12 0.997355 1.2669e-09 -3.10151e-12
5 3.12634e-09 3.12634e-09 6.57971e+11 7.18952e-15 -4.80014e-19 0.999933 7.94134e-15 1.34264e-18

.

Testing AS1:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 1 0 0 -nan
1 1.20467e-05 4.67463e-08 1.0039 8.98336e-08 3.36194e-11 1.00037
2 1.45124e-10 1.24082e-13 1.00086 8.98336e-08 3.3625e-11 1.00037
3 1.74827e-15 -1.72717e-18 0.999013 8.98336e-08 3.3625e-11 1.00037
4 1.74827e-15 -1.72717e-18 0.999013 5.54818e-15 -1.82022e-18 0.999672
5 1.74827e-15 -1.72717e-18 0.999013 0 -3.43e-22 0

.

Testing AS:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 1 0 0 -nan 0 0
1 0.00477476 0.00424246 8.9702 2.3013e-14 1.30484e-17 1.00057 -1.15154e-10 -2.29154e-10
2 2.27983e-05 2.2515e-05 80.4645 2.3013e-14 1.30484e-17 1.00057 -1.15154e-10 -2.29154e-10
3 1.08856e-07 1.08705e-07 721.782 2.3013e-14 1.30484e-17 1.00057 -1.15154e-10 -2.29154e-10
4 1.08856e-07 1.08705e-07 721.782 8.2455e-28 -4.50008e-31 0.999455 -1.06782e-21 -2.13782e-21
5 1.08856e-07 1.08856e-07 inf 2.95434e-41 4.33563e-44 1.00147 -9.94054e-33 -1.98805e-32

.

Testing DHA:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 1 0 0 -nan
1 0.000355234 -3.76304e-10 0.999999 6.75839e-09 -1.61418e-12 0.999761
2 1.26191e-07 1.90927e-10 1.00152 6.75839e-09 -1.61416e-12 0.999761
3 4.48273e-11 2.72601e-14 1.00061 6.75839e-09 -1.61416e-12 0.999761
4 4.48273e-11 2.72601e-14 1.00061 1.70142e-17 1.42277e-20 1.00084
5 4.48273e-11 2.72601e-14 1.00061 0 -4.28e-26 0

.

Testing LF:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 1 0 0 -nan
1 0.0317463 4.73527e-10 1 1.01443 5.87699e-11 1
2 0.00100781 -1.4843e-09 0.999999 1.87888 -1.46805e-10 1
3 3.19935e-05 -6.50612e-09 0.999797 2.61551 -2.37216e-10 1
4 1.01565e-06 -4.34607e-09 0.995739 2.22879 -3.57779e-10 1
5 3.22426e-08 4.25871e-11 1.00132 1.89925 1.77621e-10 1

.

Testing MQ:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 1 0 0 -nan
1 0.0317458 2.32157e-10 1 0.37844 -3.81273e-07 0.999999
2 0.00100779 2.76226e-10 1 0.737344 -1.10487e-06 0.999999
3 3.1993e-05 -7.02364e-09 0.999781 1.07772 -2.13423e-06 0.999998
4 1.01564e-06 -4.36257e-09 0.995723 1.02209 -3.05396e-06 0.999997
5 3.22421e-08 4.20583e-11 1.00131 0.969327 -3.87312e-06 0.999996

.

Testing PPQ:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 1 0 0 -nan
1 0.0524515 4.20915e-10 1 0.116453 2.5821e-11 1
2 0.00168186 -5.37307e-12 1 0.229465 -4.40691e-11 1
3 5.34433e-05 2.06221e-10 1 0.339137 -4.97115e-08 1
4 1.69855e-06 1.34833e-14 1 0.329114 -3.12838e-11 1
5 5.39961e-08 -3.93245e-12 0.999927 0.319387 9.61325e-12 1

.

Testing PPQ2:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 1 0 0 -nan
1 0.0342259 -2.82181e-08 0.999999 0.0724459 3.31059e-11 1
2 0.00108659 -4.07372e-10 1 0.121802 5.33078e-12 1
3 3.4495e-05 -5.04796e-09 0.999854 0.156117 -5.19698e-12 1
4 1.09514e-06 -4.8571e-09 0.995584 0.108163 -1.41887e-11 1
5 3.47865e-08 -1.35093e-11 0.999612 0.076857 4.55472e-11 1

.

Testing PPQ3:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 1 0 0 -nan
1 0.0342176 5.96403e-09 1 0.0768788 1.41506e-11 1
2 0.00108631 2.00802e-10 1 0.120169 2.83906e-11 1
3 3.44858e-05 4.04864e-10 1.00001 0.152677 -1.6656e-11 1
4 1.09489e-06 1.91858e-13 1 0.101699 4.14293e-11 1
5 3.47787e-08 -4.33748e-12 0.999875 0.0798269 1.51654e-12 1

@tph-thuering
Copy link
Contributor Author

Not fixed, will supply a fix now.

@tph-thuering tph-thuering reopened this Aug 11, 2015
tph-thuering added a commit that referenced this issue Aug 11, 2015
@tph-thuering
Fix #53 once and for all
5c89b03
@tph-thuering
Copy link
Contributor Author

commit 5c89b03 fixes AR unittest.
-> https://travis-ci.org/SwissTPH/openmalaria/jobs/75018663#L645

AS needs probably more precise expected drug factor values from @KatherineKay

Testing AS:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 1 0 0 -nan 0 0
1 0.000515278 -1.70125e-05 0.968039 2.3013e-14 1.30484e-17 1.00057 1.14249e-10 2.49099e-13
2 2.65512e-07 -1.78218e-08 0.9371 2.3013e-14 1.30484e-17 1.00057 1.14249e-10 2.49099e-13
3 1.36812e-10 -1.40036e-11 0.907148 2.3013e-14 1.30484e-17 1.00057 1.14249e-10 2.49099e-13
4 1.36812e-10 -1.40036e-11 0.907148 8.2455e-28 -4.50008e-31 0.999455 -1.06782e-21 -2.13782e-21
5 1.36812e-10 -1.40036e-11 0.907148 2.95434e-41 4.33563e-44 1.00147 9.94054e-33 5.40864e-37

@tph-thuering
Copy link
Contributor Author

After verification our expected values we use to compare seem to be correct.

Output of the AR and AS conversion unittests are now as follows:

Testing AR:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 0% 0 0 -nan% 0 0
1 1.69524e-07 4.51805e-14 0% 0.000182523 -1.39572e-11 0.01% 0.000201313 -4.94356e-12
2 2.83815e-14 7.47577e-21 0% 0.000182524 3.29943e-11 0% 0.000201314 -3.8053e-11
3 4.75159e-21 1.15717e-23 -0.24% 0.000182524 3.30015e-11 0% 0.000201314 -3.80451e-11
4 4.75148e-21 -4.23176e-29 0.01% 1.14696e-09 -3.04129e-12 0.27% 1.2669e-09 -3.10151e-12
5 4.75148e-21 -4.23176e-29 0.01% 7.18952e-15 -4.80014e-19 0.01% 7.94134e-15 1.34264e-18

.

Testing AS:

day factor f_error f_rel_error concentration c_error c_rel_error concentration2 c2_error
0 1 0 0% 0 0 -nan% 0 0
1 0.000515278 -1.70125e-05 3.2% 2.3013e-14 1.30484e-17 -0.05% 1.14249e-10 2.49099e-13
2 2.65512e-07 -1.78218e-08 6.3% 2.3013e-14 1.30484e-17 -0.05% 1.14249e-10 2.49099e-13
3 1.36812e-10 -1.40036e-11 9.29% 2.3013e-14 1.30484e-17 -0.05% 1.14249e-10 2.49099e-13
4 1.36812e-10 -1.40036e-11 9.29% 8.2455e-28 -4.50008e-31 0.06% 0 -1.07e-21
5 1.36812e-10 -1.40036e-11 9.29% 2.95434e-41 4.33563e-44 -0.14% 9.94054e-33 5.40864e-37

between day 3&4: ERROR! concentration is lower than zero, setting it to zero for now: -1.06782e-21 drug_index: 4

@tph-thuering
Copy link
Contributor Author

Testing AR - DHA_AR Conversion

AR

day factor f_abs_error f_rel_err % conc c_abs_error c_rel_err % type
0 1 +0 +0 0 +0 -nan P
0 1 +0 +0 0 +0 -nan M
1 1.69524e-07 +4.51805e-14 +0 0.000182523 -1.39572e-11 -0.0001 P
1 1.69524e-07 +4.51805e-14 +0 0.000201313 -4.83963e-12 -0.0001 M
2 2.83815e-14 +7.47577e-21 +0 0.000182524 +3.29943e-11 +0 P
2 2.83815e-14 +7.47577e-21 +0 0.000201314 -3.79488e-11 -0.0001 M
3 4.75159e-21 +1.15717e-23 +0.2441 0.000182524 +3.30015e-11 +0 P
3 4.75159e-21 +1.15717e-23 +0.2441 0.000201314 -3.79409e-11 -0.0001 M
4 4.75148e-21 -4.23176e-29 -0.0001 1.14696e-09 -3.04129e-12 -0.2645 P
4 4.75148e-21 -4.23176e-29 -0.0001 1.2669e-09 -3.10151e-12 -0.2443 M
5 4.75148e-21 -4.23176e-29 -0.0001 7.18952e-15 -4.80014e-19 -0.0067 P
5 4.75148e-21 -4.23176e-29 -0.0001 7.94134e-15 +1.34264e-18 +0.0169 M

.
...

Testing AS - DHA_AS Conversion

AS

day factor f_abs_error f_rel_err % conc c_abs_error c_rel_err % type
0 1 +0 +0 0 +0 -nan P
0 1 +0 +0 0 +0 -nan M
1 0.000532291 +2.41417e-10 +0 2.3013e-14 +1.30484e-17 +0.0567 P
1 0.000532291 +2.41417e-10 +0 1.14249e-10 +2.49099e-13 +0.2185 M
2 2.83334e-07 +2.52769e-13 +0 2.3013e-14 +1.30484e-17 +0.0567 P
2 2.83334e-07 +2.52769e-13 +0 1.14249e-10 +2.49099e-13 +0.2185 M
3 1.50816e-10 +1.83276e-17 +0 2.3013e-14 +1.30484e-17 +0.0567 P
3 1.50816e-10 +1.83276e-17 +0 1.14249e-10 +2.49099e-13 +0.2185 M
4 1.50816e-10 +1.83276e-17 +0 8.2455e-28 -4.50008e-31 -0.0546 P
4 1.50816e-10 +1.83276e-17 +0 1.05936e-21 -1.06425e-23 -0.9947 M
5 1.50816e-10 +1.83276e-17 +0 2.95434e-41 +4.33563e-44 +0.1469 P
5 1.50816e-10 +1.83276e-17 +0 9.8618e-33 -7.82036e-35 -0.7868 M

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tph-thuering tph-thuering

Labels
1: high
Projects
None yet
Milestone
Schema 34
Development

No branches or pull requests
2 participants
@dhardy @tph-thuering