-
-
Notifications
You must be signed in to change notification settings - Fork 134
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix TV range color scaling (again) #479
Conversation
debian/patches/0007-add-bt2390-eetf-and-code-refactor-to-opencl-tonemap.patch
Outdated
Show resolved
Hide resolved
debian/patches/0007-add-bt2390-eetf-and-code-refactor-to-opencl-tonemap.patch
Outdated
Show resolved
Hide resolved
Is it possible that dovi reshape is also affected by this texture normalization issue? It doesn't use yuv2rgb. |
Most human eye noticeable off comes from the inaccurate yuv2rgb formula and the GPU quantization off is much harder to notice. Have not tested if dovi is affected though, but it should be much harder to tell after the reshaping. Just tested, I cannot tell the difference with my eye. |
debian/patches/0007-add-bt2390-eetf-and-code-refactor-to-opencl-tonemap.patch
Outdated
Show resolved
Hide resolved
I have a suggestion to use constant instead of marco in opencl header so as to be consistent with other floats? The previous discussion also reminded me, should we print extra digits for other coeffs that are more affected by precision? Many of them are calculated in real time, and the number of zeros after the decimal point cannot be predicted. [Parsed_tonemap_opencl_2 @ 0000023d5a7e15c0] Generated OpenCL header:
__constant float ref_white = 203.0000f;
__constant float tone_param = 1.0000f;
__constant float desat_param = 0.0000f;
__constant float target_peak = 1.0000f;
__constant float scene_threshold = 0.2000f;
__constant float pq_max_lum_div_ref_white = 49.261086f;
__constant float ref_white_div_pq_max_lum = 0.020300f;
#define TONE_FUNC bt2390
#define TONE_FUNC_BT2390
#define TONE_MODE_ITP
#define ENABLE_DITHER
__constant float dither_size2 = 65536.0f;
__constant float dither_quantization = 255.0f;
#define FULL_RANGE_IN
#define FULL_RANGE_OUT
#define INPUT_QUANTIZATION_OFFSET 0.000977f
#define INPUT_Y_SCALE 1.167808f
#define INPUT_UV_SCALE 1.141741f
#define chroma_loc 1
__constant float rgb2rgb[9] = {
1.660491f, -0.587641f, -0.072850f,
-0.124550f, 1.132900f, -0.008349f,
-0.018151f, -0.100579f, 1.118730f,
};
#define DOVI_RESHAPE
__constant float3 ycc2rgb_offset = {-0.151367f, -0.009644f, 0.322144f};
__constant float rgb_matrix[9] = {
1.000000f, 0.097534f, 0.205200f,
1.000000f, -0.113892f, 0.133179f,
1.000000f, 0.032593f, -0.676880f,
};
__constant float lms2rgb_matrix[9] = {
3.208390f, -2.295148f, 0.086695f,
-0.749679f, 1.908645f, -0.159048f,
-0.033198f, -0.041158f, 1.074456f,
};
__constant float yuv_matrix[9] = {
0.212600f, 0.715200f, 0.072200f,
-0.114572f, -0.385428f, 0.500000f,
0.500000f, -0.454153f, -0.045847f,
};
__constant float3 luma_dst = {0.212600f, 0.715200f, 0.072200f};
#define linearize eotf_st2084
#define delinearize inverse_eotf_bt1886 |
From current header there is nothing like the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The previous discussion also reminded me, should we print extra digits for other coeffs that are more affected by precision? Many of them are calculated in real time, and the number of zeros after the decimal point cannot be predicted.
From current header there is nothing like the
INPUT_QUANTIZATION_OFFSET
that is rounded to only 3 significant digits, and the color matrix are actually quite predictable. I have no problem printing more digits though if we want some consistency.
It may help in lin_lut[1024]
in the fast path, which is used on low perf iGPUs.
__constant float lin_lut[1024] = {
0.0000000000000f, 0.0000001991254f, 0.0000006458805f, 0.0000012924631f, 0.0000021256837f, 0.0000031402285f, 0.0000043341256f, 0.0000057071161f, 0.0000072600519f, 0.0000089945097f, 0.0000109125949f, 0.0000130167546f, 0.0000153097080f, 0.0000177944803f, 0.0000204743301f, 0.0000233523315f, 0.0000264316404f, 0.0000297165443f, 0.0000332098971f, 0.0000369164372f, 0.0000408391097f, 0.0000449815925f, 0.0000493490443f, 0.0000539446810f, 0.0000587722243f, 0.0000638365236f, 0.0000691419918f, 0.0000746928999f, 0.0000804923402f, 0.0000865460242f, 0.0000928589361f, 0.0000994339280f, 0.0001062767551f, 0.0001133914775f, 0.0001207842797f, 0.0001284567988f, 0.0001364170457f, 0.0001446682872f, 0.0001532134193f, 0.0001620633848f, 0.0001712175872f, 0.0001806827058f, 0.0001904637465f, 0.0002005660499f, 0.0002109970956f, 0.0002217604779f, 0.0002328619012f, 0.0002443018893f, 0.0002560952271f, 0.0002682396444f, 0.0002807444835f, 0.0002936111123f, 0.0003068499791f, 0.0003204689710f, 0.0003344702418f, 0.0003488530347f, 0.0003636353358f, 0.0003788222675f, 0.0003944064665f, 0.0004104095569f, 0.0004268246121f, 0.0004436713934f, 0.0004609477182f, 0.0004786600184f, 0.0004968152498f, 0.0005154194077f, 0.0005344906240f, 0.0005540185957f, 0.0005740109482f, 0.0005944868317f, 0.0006154456642f, 0.0006368983886f, 0.0006588429678f, 0.0006813015789f, 0.0007042654906f, 0.0007277561817f, 0.0007517638733f, 0.0007763124886f, 0.0008014013874f, 0.0008270337712f, 0.0008532219217f, 0.0008799834759f, 0.0009073165129f, 0.0009352278430f, 0.0009637108888f, 0.0009928048821f, 0.0010225101141f, 0.0010528110433f, 0.0010837255977f, 0.0011152932420f, 0.0011474787025f, 0.0011803027010f, 0.0012138090096f, 0.0012479558354f, 0.0012827697210f, 0.0013182732509f, 0.0013544684043f, 0.0013913358562f, 0.0014289506944f, 0.0014672399266f, 0.0015062806197f, 0.0015460291179f, 0.0015865379246f, 0.0016278108815f, 0.0016698248219f, 0.0017126105959f, 0.0017562048743f, 0.0018005799502f, 0.0018457319820f, 0.0018917096313f, 0.0019385394407f, 0.0019861613400f, 0.0020346120000f, 0.0020839765202f, 0.0021341373213f, 0.0021851751953f, 0.0022371395025f, 0.0022899962496f, 0.0023437477648f, 0.0023984024301f, 0.0024539933074f, 0.0025105453096f, 0.0025680549443f, 0.0026264807675f, 0.0026858700439f, 0.0027462851722f, 0.0028076793533f, 0.0028701105621f, 0.0029335361905f, 0.0029980072286f, 0.0030635287985f, 0.0031301225536f, 0.0031977850012f, 0.0032665214967f, 0.0033363343682f, 0.0034072392154f, 0.0034792933147f, 0.0035524556879f, 0.0036267905962f, 0.0037023038603f, 0.0037790010683f, 0.0038568286691f, 0.0039358502254f, 0.0040161707439f, 0.0040976991877f, 0.0041804667562f, 0.0042644506320f, 0.0043497784063f, 0.0044362517074f, 0.0045241541229f, 0.0046133371070f, 0.0047038565390f, 0.0047956635244f, 0.0048889443278f, 0.0049835257232f, 0.0050794850104f, 0.0051769739948f, 0.0052757556550f, 0.0053760875016f, 0.0054778438061f, 0.0055809319019f, 0.0056856987067f, 0.0057918117382f, 0.0058995080180f, 0.0060088136233f, 0.0061196065508f, 0.0062320139259f, 0.0063459347002f, 0.0064615081064f, 0.0065787266940f, 0.0066976188682f, 0.0068180565722f, 0.0069401864894f, 0.0070641594939f, 0.0071897059679f, 0.0073171434924f, 0.0074461465701f, 0.0075769117102f, 0.0077096060850f, 0.0078440709040f, 0.0079803075641f, 0.0081185428426f, 0.0082585727796f, 0.0084002390504f, 0.0085441088304f, 0.0086900172755f, 0.0088375778869f, 0.0089872200042f, 0.0091387173161f, 0.0092925187200f, 0.0094480216503f, 0.0096058668569f, 0.0097658373415f, 0.0099277542904f, 0.0100916298106f, 0.0102578923106f, 0.0104261515662f, 0.0105966031551f, 0.0107695292681f, 0.0109444865957f, 0.0111217079684f, 0.0113012148067f, 0.0114830052480f, 0.0116673875600f, 0.0118538169190f, 0.0120426267385f, 0.0122340274975f, 0.0124275693670f, 0.0126237841323f, 0.0128224026412f, 0.0130237583071f, 0.0132272336632f, 0.0134335327893f, 0.0136422570795f, 0.0138538060710f, 0.0140679022297f, 0.0142845017835f, 0.0145040042698f, 0.0147260464728f, 0.0149510204792f, 0.0151786189526f, 0.0154091455042f, 0.0156423300505f, 0.0158785320818f, 0.0161174200475f, 0.0163593664765f, 0.0166044030339f, 0.0168521348387f, 0.0171030387282f, 0.0173571053892f, 0.0176139567047f, 0.0178740043193f, 0.0181372854859f, 0.0184038206935f, 0.0186731982976f, 0.0189459193498f, 0.0192219577730f, 0.0195013564080f, 0.0197836440057f, 0.0200693756342f, 0.0203590579331f, 0.0206516440958f, 0.0209477432072f, 0.0212478823960f, 0.0215510483831f, 0.0218577999622f, 0.0221681147814f, 0.0224820822477f, 0.0227996483445f, 0.0231209229678f, 0.0234464798123f, 0.0237751621753f, 0.0241082571447f, 0.0244445148855f, 0.0247845314443f, 0.0251290686429f, 0.0254774186760f, 0.0258296895772f, 0.0261858813465f, 0.0265466887504f, 0.0269108507782f, 0.0272796731442f, 0.0276525523514f, 0.0280294269323f, 0.0284111704677f, 0.0287970472127f, 0.0291870869696f, 0.0295820441097f, 0.0299804676324f, 0.0303838867694f, 0.0307915825397f, 0.0312044136226f, 0.0316223651171f, 0.0320438146591f, 0.0324706248939f, 0.0329017676413f, 0.0333382524550f, 0.0337783023715f, 0.0342247150838f, 0.0346756018698f, 0.0351311191916f, 0.0355921387672f, 0.0360578745604f, 0.0365291759372f, 0.0370052717626f, 0.0374870151281f, 0.0379735305905f, 0.0384658649564f, 0.0389630272985f, 0.0394661352038f, 0.0399751290679f, 0.0404891408980f, 0.0410092435777f, 0.0415343306959f, 0.0420643761754f, 0.0426006615162f, 0.0431432016194f, 0.0436920896173f, 0.0442461296916f, 0.0448065735400f, 0.0453722029924f, 0.0459443405271f, 0.0465230830014f, 0.0471084564924f, 0.0476991720498f, 0.0482967533171f, 0.0489010065794f, 0.0495106801391f, 0.0501273535192f, 0.0507508926094f, 0.0513800494373f, 0.0520162545145f, 0.0526595786214f, 0.0533100701869f, 0.0539678335190f, 0.0546312555671f, 0.0553021654487f, 0.0559803582728f, 0.0566644445062f, 0.0573577359319f, 0.0580568797886f, 0.0587636530399f, 0.0594781339169f, 0.0602004937828f, 0.0609288662672f, 0.0616667605937f, 0.0624109134078f, 0.0631631761789f, 0.0639233216643f, 0.0646917372942f, 0.0654663890600f, 0.0662512108684f, 0.0670443549752f, 0.0678439810872f, 0.0686539486051f, 0.0694704800844f, 0.0702955722809f, 0.0711294487119f, 0.0719717815518f, 0.0728230997920f, 0.0736831799150f, 0.0745523720980f, 0.0754302889109f, 0.0763174742460f, 0.0772137045860f, 0.0781168863177f, 0.0790317356586f, 0.0799556598067f, 0.0808868408203f, 0.0818323045969f, 0.0827823951840f, 0.0837446600199f, 0.0847140848637f, 0.0856959149241f, 0.0866875201464f, 0.0876890271902f, 0.0887006446719f, 0.0897223353386f, 0.0907540917397f, 0.0917962640524f, 0.0928486660123f, 0.0939113423228f, 0.0949846580625f, 0.0960714817047f, 0.0971657782793f, 0.0982740595937f, 0.0993898957968f, 0.1005197092891f, 0.1016606390476f, 0.1028124988079f, 0.1039723455906f, 0.1051497235894f, 0.1063354387879f, 0.1075324192643f, 0.1087439954281f, 0.1099639832973f, 0.1111990958452f, 0.1124457716942f, 0.1137042045593f, 0.1149783805013f, 0.1162572577596f, 0.1175556108356f, 0.1188657209277f, 0.1201886162162f, 0.1215236857533f, 0.1228710711002f, 0.1242352873087f, 0.1256078332663f, 0.1269974708557f, 0.1283997446299f, 0.1298150718212f, 0.1312433332205f, 0.1326888352633f, 0.1341433376074f, 0.1356198489666f, 0.1371055394411f, 0.1386044919491f, 0.1401214748621f, 0.1416480094194f, 0.1431973427534f, 0.1447558850050f, 0.1463332176208f, 0.1479248553514f, 0.1495308727026f, 0.1511509418488f, 0.1527904272079f, 0.1544394642115f, 0.1561133116484f, 0.1577969193459f, 0.1595004647970f, 0.1612188220024f, 0.1629528105259f, 0.1647069603205f, 0.1664713174105f, 0.1682618856430f, 0.1700624227524f, 0.1718787401915f, 0.1737164258957f, 0.1755757778883f, 0.1774456650019f, 0.1793375015259f, 0.1812453418970f, 0.1831755042076f, 0.1851224154234f, 0.1870860755444f, 0.1890726536512f, 0.1910697072744f, 0.1930967271328f, 0.1951342970133f, 0.1971956044436f, 0.1992743611336f, 0.2013707160950f, 0.2034914195538f, 0.2056366503239f, 0.2077928036451f, 0.2099743783474f, 0.2121736854315f, 0.2143982946873f, 0.2166416496038f, 0.2189035415649f, 0.2211915254593f, 0.2234984785318f, 0.2258244901896f, 0.2281771302223f, 0.2305567711592f, 0.2329477369785f, 0.2353666722775f, 0.2378132194281f, 0.2402712255716f, 0.2427659183741f, 0.2452720999718f, 0.2478074580431f, 0.2503710687160f, 0.2529559731483f, 0.2555611729622f, 0.2581872344017f, 0.2608435750008f, 0.2635294497013f, 0.2662279009819f, 0.2689663469791f, 0.2717169523239f, 0.2745084166527f, 0.2773119807243f, 0.2801478207111f, 0.2830148935318f, 0.2859052419662f, 0.2888176441193f, 0.2917632460594f, 0.2947320938110f, 0.2977231144905f, 0.3007483780384f, 0.3038074970245f, 0.3068902790546f, 0.3099968433380f, 0.3131381869316f, 0.3163144588470f, 0.3195042014122f, 0.3227293789387f, 0.3260024785995f, 0.3292885720730f, 0.3326118290424f, 0.3359479606152f, 0.3393334746361f, 0.3427446782589f, 0.3461936116219f, 0.3496686816216f, 0.3531697094440f, 0.3567091822624f, 0.3602887988091f, 0.3638943731785f, 0.3675273954868f, 0.3712137043476f, 0.3749141693115f, 0.3786552250385f, 0.3824238777161f, 0.3862337768078f, 0.3900865018368f, 0.3939675688744f, 0.3978760540485f, 0.4018282592297f, 0.4058236479759f, 0.4098491668701f, 0.4139028489590f, 0.4180167317390f, 0.4221452474594f, 0.4263186752796f, 0.4305375814438f, 0.4347868263721f, 0.4390833079815f, 0.4434107244015f, 0.4477848410606f, 0.4521902203560f, 0.4566434919834f, 0.4611292481422f, 0.4656634628773f, 0.4702471792698f, 0.4748629629612f, 0.4795110821724f, 0.4842107594013f, 0.4889610409737f, 0.4937444627285f, 0.4985795915127f, 0.5034495592117f, 0.5083719491959f, 0.5133283138275f, 0.5183380842209f, 0.5233821868896f, 0.5284816026688f, 0.5336360335350f, 0.5388256311417f, 0.5440722107887f, 0.5493547320366f, 0.5546936392784f, 0.5600897669792f, 0.5655239224434f, 0.5709947943687f, 0.5765457749367f, 0.5821121931076f, 0.5877616405487f, 0.5934489369392f, 0.5991743206978f, 0.6049605011940f, 0.6107866168022f, 0.6166981458664f, 0.6226506233215f, 0.6286407709122f, 0.6346960663795f, 0.6408177018166f, 0.6469774842262f, 0.6531792283058f, 0.6594719886780f, 0.6658077239990f, 0.6722100973129f, 0.6786545515060f, 0.6851407289505f, 0.6917238831520f, 0.6983493566513f, 0.7050192952156f, 0.7117577791214f, 0.7185989022255f, 0.7254542708397f, 0.7323833107948f, 0.7393565773964f, 0.7464035153389f, 0.7535260319710f, 0.7607232928276f, 0.7679682374001f, 0.7752570509911f, 0.7826566100121f, 0.7901012301445f, 0.7976258993149f, 0.8051975965500f, 0.8128504157066f, 0.8205515146255f, 0.8283327817917f, 0.8361975550652f, 0.8441112041473f, 0.8521071672440f, 0.8601886630058f, 0.8683204054832f, 0.8765390515327f, 0.8848438262939f, 0.8931987285614f, 0.9016435146332f, 0.9101397991180f, 0.9187240004539f, 0.9273998141289f, 0.9361687302589f, 0.9449502229691f, 0.9538635611534f, 0.9628719091415f, 0.9719346761703f, 0.9810518622398f, 0.9902629852295f, 0.9995745420456f, 1.0089806318283f, 1.0184447765350f, 1.0280076265335f, 1.0376716852188f, 1.0473924875259f, 1.0572167634964f, 1.0670984983444f, 1.0770829916000f, 1.0871727466583f, 1.0973666906357f, 1.1076226234436f, 1.1179851293564f, 1.1284073591232f, 1.1389399766922f, 1.1496310234070f, 1.1603368520737f, 1.1711525917053f, 1.1820818185806f, 1.1931240558624f, 1.2042301893234f, 1.2154525518417f, 1.2267941236496f, 1.2382004261017f, 1.2497235536575f, 1.2613145112991f, 1.2730239629745f, 1.2849115133286f, 1.2968115806580f, 1.3088923692703f, 1.3210411071777f, 1.3333175182343f, 1.3456628322601f, 1.3581336736679f, 1.3707362413406f, 1.3834112882614f, 1.3962759971619f, 1.4091519117355f, 1.4222229719162f, 1.4354326725006f, 1.4486482143402f, 1.4620695114136f, 1.4756269454956f, 1.4891946315765f, 1.5029690265656f, 1.5168849229813f, 1.5308792591095f, 1.5450170040131f, 1.5592339038849f, 1.5735963582993f, 1.5881778001785f, 1.6027680635452f, 1.6175812482834f, 1.6324738264084f, 1.6475226879120f, 1.6626508235931f, 1.6779363155365f, 1.6933777332306f, 1.7089798450470f, 1.7246652841568f, 1.7405126094818f, 1.7565225362778f, 1.7726988792419f, 1.7889592647552f, 1.8053894042969f, 1.8219050168991f, 1.8386750221252f, 1.8555361032486f, 1.8724796772003f, 1.8896886110306f, 1.9070723056793f, 1.9244599342346f, 1.9421144723892f, 1.9599527120590f, 1.9778827428818f, 1.9959043264389f, 2.0142080783844f, 2.0325996875763f, 2.0511817932129f, 2.0699582099915f, 2.0889277458191f, 2.1079897880554f, 2.1272544860840f, 2.1467132568359f, 2.1662693023682f, 2.1861281394958f, 2.2060925960541f, 2.2261552810669f, 2.2464206218719f, 2.2670001983643f, 2.2876877784729f, 2.3084776401520f, 2.3294813632965f, 2.3508112430573f, 2.3722474575043f, 2.3937883377075f, 2.4155516624451f, 2.4376518726349f, 2.4597487449646f, 2.4821865558624f, 2.5047347545624f, 2.5275154113770f, 2.5505313873291f, 2.5737802982330f, 2.5971403121948f, 2.6207411289215f, 2.6445863246918f, 2.6685435771942f, 2.6927490234375f, 2.7173323631287f, 2.7420339584351f, 2.7668519020081f, 2.7920649051666f, 2.8172581195831f, 2.8428514003754f, 2.8687021732330f, 2.8946740627289f, 2.9209134578705f, 2.9474234580994f, 2.9741981029510f, 3.0011065006256f, 3.0282919406891f, 3.0557479858398f, 3.0834813117981f, 3.1113572120667f, 3.1395032405853f, 3.1679491996765f, 3.1966779232025f, 3.2257034778595f, 3.2548620700836f, 3.2843198776245f, 3.3139138221741f, 3.3439710140228f, 3.3743388652802f, 3.4046807289124f, 3.4354989528656f, 3.4666268825531f, 3.4979038238525f, 3.5295000076294f, 3.5614128112793f, 3.5936553478241f, 3.6262273788452f, 3.6589479446411f, 3.6919932365417f, 3.7253882884979f, 3.7589287757874f, 3.7928056716919f, 3.8270297050476f, 3.8616034984589f, 3.8965339660645f, 3.9318130016327f, 3.9672496318817f, 4.0030570030212f, 4.0392208099365f, 4.0755434036255f, 4.1124467849731f, 4.1495175361633f, 4.1869592666626f, 4.2247967720032f, 4.2627854347229f, 4.3013920783997f, 4.3399367332458f, 4.3791036605835f, 4.4186639785767f, 4.4584050178528f, 4.4987807273865f, 4.5393300056458f, 4.5800509452820f, 4.6214332580566f, 4.6629924774170f, 4.7052164077759f, 4.7473769187927f, 4.7902159690857f, 4.8332324028015f, 4.8769507408142f, 4.9208550453186f, 4.9651947021484f, 5.0097327232361f, 5.0549869537354f, 5.1007041931152f, 5.1466150283813f, 5.1927103996277f, 5.2395620346069f, 5.2868843078613f, 5.3344120979309f, 5.3824148178101f, 5.4309082031250f, 5.4798994064331f, 5.5290904045105f, 5.5787835121155f, 5.6289844512939f, 5.6796879768372f, 5.7309107780457f, 5.7823510169983f, 5.8343043327332f, 5.8867921829224f, 5.9401302337646f, 5.9933662414551f, 6.0474700927734f, 6.1018028259277f, 6.1566767692566f, 6.2121114730835f, 6.2681169509888f, 6.3243432044983f, 6.3814859390259f, 6.4388680458069f, 6.4968247413635f, 6.5550236701965f, 6.6141710281372f, 6.6735501289368f, 6.7339024543762f, 6.7945041656494f, 6.8553423881531f, 6.9171748161316f, 6.9792666435242f, 7.0419721603394f, 7.1053214073181f, 7.1693205833435f, 7.2339582443237f, 7.2992773056030f, 7.3648295402527f, 7.4310688972473f, 7.4979705810547f, 7.5655584335327f, 7.6338405609131f, 7.7023816108704f, 7.7716221809387f, 7.8415579795837f, 7.9122252464294f, 7.9835982322693f, 8.0552415847778f, 8.1276359558105f, 8.2012090682983f, 8.2750787734985f, 8.3492202758789f, 8.4245920181274f, 8.5002613067627f, 8.5771894454956f, 8.6544017791748f, 8.7319135665894f, 8.8106927871704f, 8.8903055191040f, 8.9702081680298f, 9.0509109497070f, 9.1324586868286f, 9.2148256301880f, 9.2980394363403f, 9.3815546035767f, 9.4664821624756f, 9.5517215728760f, 9.6372632980347f, 9.7242593765259f, 9.8121232986450f, 9.9003047943115f, 9.9899845123291f, 10.0799913406372f, 10.1709203720093f, 10.2627830505371f, 10.3549757003784f, 10.4481086730957f, 10.5428066253662f, 10.6378393173218f, 10.7338514328003f, 10.8302259445190f, 10.9282197952271f, 11.0265550613403f, 11.1265516281128f, 11.2269268035889f, 11.3283081054688f, 11.4307556152344f, 11.5335340499878f, 11.6380758285522f, 11.7429800033569f, 11.8489637374878f, 11.9560403823853f, 12.0642185211182f, 12.1727685928345f, 12.2831535339355f, 12.3939514160156f, 12.5058555603027f, 12.6189193725586f, 12.7331438064575f, 12.8485488891602f, 12.9643421173096f, 13.0821104049683f, 13.2003107070923f, 13.3196935653687f, 13.4395055770874f, 13.5613584518433f, 13.6844778060913f, 13.8080034255981f, 13.9328088760376f, 14.0588960647583f, 14.1871442794800f, 14.3149595260620f, 14.4449634552002f, 14.5754308700562f, 14.7081403732300f, 14.8412847518921f, 14.9758100509644f, 15.1117238998413f, 15.2480840682983f, 15.3867864608765f, 15.5259752273560f, 15.6666011810303f, 15.8086490631104f, 15.9531803131104f, 16.0972003936768f, 16.2436771392822f, 16.3906860351562f, 16.5402107238770f, 16.6902503967285f, 16.8418140411377f, 16.9949779510498f, 17.1486358642578f, 17.3049526214600f, 17.4617691040039f, 17.6213417053223f, 17.7814331054688f, 17.9431877136230f, 18.1054363250732f, 18.2717056274414f, 18.4373512268066f, 18.6058845520020f, 18.7737674713135f, 18.9445743560791f, 19.1184158325195f, 19.2915344238281f, 19.4677352905273f, 19.6444759368896f, 19.8243484497070f, 20.0047473907471f, 20.1870670318604f, 20.3712253570557f, 20.5573616027832f, 20.7440090179443f, 20.9339904785156f, 21.1259574890137f, 21.3185157775879f, 21.5130729675293f, 21.7096214294434f, 21.9082641601562f, 22.1089305877686f, 22.3102493286133f, 22.5151290893555f, 22.7206268310547f, 22.9282722473145f, 23.1380786895752f, 23.3500823974609f, 23.5643043518066f, 23.7791671752930f, 23.9962692260742f, 24.2172145843506f, 24.4388904571533f, 24.6628303527832f, 24.8891124725342f, 25.1177577972412f, 25.3471374511719f, 25.5805492401123f, 25.8146915435791f, 26.0512962341309f, 26.2903804779053f, 26.5319099426270f, 26.7760334014893f, 27.0226726531982f, 27.2700805664062f, 27.5218467712402f, 27.7744865417480f, 28.0297031402588f, 28.2876148223877f, 28.5481643676758f, 28.8115234375000f, 29.0756340026855f, 29.3444938659668f, 29.6141929626465f, 29.8866615295410f, 30.1641120910645f, 30.4423637390137f, 30.7214679718018f, 31.0055294036865f, 31.2926750183105f, 31.5806274414062f, 31.8716125488281f, 32.1678657531738f, 32.4627494812012f, 32.7652816772461f, 33.0664291381836f, 33.3730354309082f, 33.6804924011230f, 33.9936180114746f, 34.3076171875000f, 34.6249313354492f, 34.9456100463867f, 35.2696838378906f, 35.5971221923828f, 35.9280166625977f, 36.2599105834961f, 36.5978813171387f, 36.9367790222168f, 37.2793426513672f, 37.6254539489746f, 37.9752616882324f, 38.3286895751953f, 38.6859626770020f, 39.0442085266113f, 39.4062690734863f, 39.7749633789062f, 40.1447563171387f, 40.5184020996094f, 40.8960227966309f, 41.2747421264648f, 41.6604347229004f, 42.0502586364746f, 42.4411239624023f, 42.8391990661621f, 43.2384605407715f, 43.6419906616211f, 44.0497131347656f, 44.4619140625000f, 44.8751716613770f, 45.2961044311523f, 45.7181472778320f, 46.1481399536133f, 46.5792884826660f, 47.0150222778320f, 47.4554519653320f, 47.9005889892578f, 48.3504943847656f, 48.8051414489746f, 49.2610855102539f,};
In #472, we improved range scaling, but it still wasn’t quite right. While it prevented the green tint, it came at the expense of some color accuracy, with colors drifting too much toward the white point. After further investigation, I found that the green tint was likely caused by quantization errors within the pipeline, which are tied to the bit depth of the input and how GPUs handle textures.
The original YUV-to-RGB and RGB-to-YUV conversions introduced by Intel developers in the upstream filter (see colorspace_common.cl) used 8-bit quantization scaling for all inputs, which isn’t correct for 10-bit or 12-bit inputs. This will cause the generated RGB values to be slightly off.
When FFmpeg generates frames in formats like P010, it stores the 10 bits of color data in the high 10 bits of a 16-bit integer. This approach makes sense because it allows different bit depths to be interpreted consistently as any bit depth can be normalized as a full 16-bit per-channel texture by the GPU. However, this creates a quantization error in our tonemap filter because the max value for 10-bit color (1023) doesn’t align with the max value of 16-bit color after the bit shift. Instead, 10-bit color values max out at 65472 when interpreted as 16-bit color.
Both errors are only slightly off for the inputs. But after the power/logarithm-like transfer functions and the TMOs the slight off could be a huge off in certain scenes and will generate an artifact-like tint.
To fix those this PR now set scaling and offsets depending on the input and output depth, instead of using a hardcoded value.
This also contains an extra fix for videotoolboxenc which always set the pixel format info for the
VTCompressionSession
. Without setting this the tv range to pc range conversion will not correctly set the encoder to encode in pc range. This fix should be upstream-able but the upstream does not have any filters that can change the video range like our tonemap filter, so probably not needed.For reference:
The original scaling, derived from upstream:
Our initial fix, which does scaling in RGB:
This PR, account for quantization errors:
Changes
Issues