Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add WebGL shim in CLI/Node context to allow GL-based modules to run in Node #216

Open
jywarren opened this issue Apr 19, 2018 · 146 comments
Open

Add WebGL shim in CLI/Node context to allow GL-based modules to run in Node #216

jywarren opened this issue Apr 19, 2018 · 146 comments
Labels
new-feature performance project

Comments

@jywarren
Copy link
Member

jywarren commented Apr 19, 2018 •
edited
Loading

https://www.npmjs.com/package/gl

gl lets you create a WebGL context in node.js without making a window or loading a full browser environment.

This would no longer be pure JavaScript, but for some modules this is interesting. For example, the FisheyeGL module #27 can currently only be run in a browser, and webgl-distort #64 also would be this way.

Long-term project!

Hmm, maybe also these resources:
This was referenced May 4, 2018
Open
Open
@gitmate gitmate bot mentioned this issue Jun 6, 2018
Open
@gitmate gitmate bot mentioned this issue Jun 27, 2018
Open
@gitmate gitmate bot mentioned this issue Sep 29, 2018
Closed
@jywarren jywarren changed the title Add WebGL in CLI/Node context (long-term) Add WebGL shim in CLI/Node context to allow GL-based modules to run in Node Feb 8, 2019
@jywarren jywarren pinned this issue Feb 8, 2019
@Divy123
Copy link
Member

Divy123 commented Feb 8, 2019 •
edited
Loading

Sorry for being late on the fisheyeGl issue but I was facing some issues which I am going through.
Will soon come up with a solution.

@jywarren jywarren mentioned this issue Feb 10, 2019
Open
@tech4GT
Copy link
Member

tech4GT commented Feb 12, 2019

Okay I brainstormed this a lot and I came up with 2-3 solutions

  1. We can run a headless browser like puppeteer and run this module in it's context, then get the result using the callback.
  2. We can try to use jsdom in conjunction with gl and make a global document object that can get a web-gl context. I have been trying this out but so far it looks like this would require some changes in the fisheyeGL library by @jywarren . Since the library uses new Image() and I was not able to find anything to emulate that in node, I tried using jsdom but it does not work with the constructor, maybe we can change this to
document.querySelector("#image-node") || new Image();
  1. We can enable web-gl support in our modules by automatically requiring gl in node context. This would however mean that code for browser and node wold have to be different, which is what I am trying to avoid here.
    cc @jywarren thoughts?

@tech4GT
Copy link
Member

tech4GT commented Feb 12, 2019 •
edited
Loading

Also there's an issue with node_version for gl It was compiled for node 8.15.0 latest, so If we go with this we would have to use that version
Also please do see the warning on their npm page
screen shot 2019-02-12 at 9 34 58 am

@jywarren jywarren mentioned this issue Feb 16, 2019
Closed
@Divy123
Copy link
Member

Divy123 commented Feb 17, 2019

@tech4GT @jywarren I have done some research and here are some points on this:
First of all the gl produces the canvas element that does not have texImage2D function method on it .
So that's a drawback I think.
Also @jywarren can you please explain how can https://github.com/Overv/JSGL be useful here.
While researching for fisheyeGl module I went through but there seemed no good resources to move with it.
Also I came across basicHTML but not of much use as there canvas.getContext() not implemented.
WebReflection/basicHTML#32.
I was trying to learn webGl so that I can be helpful with this . Can you people please suggest some good resource to go forward?
And also puppeteer can be quite good to go with.

@Divy123
Copy link
Member

Divy123 commented Feb 18, 2019

@jywarren request your help here.

@jywarren
Copy link
Member Author

Hmm, to @tech4GT I think it's ok to have different code in browser or node if that code split exists only inside the module context. Although we could eventually try to generalize parts to make it easier to write webgl modules.

@jywarren
Copy link
Member Author

jywarren commented Feb 18, 2019 via email

@jywarren
Copy link
Member Author

jywarren commented Feb 18, 2019 via email

@jywarren
Copy link
Member Author

jywarren commented Feb 18, 2019 via email

@jywarren
Copy link
Member Author

jywarren commented Feb 18, 2019 via email

@jywarren
Copy link
Member Author

jywarren commented Feb 18, 2019 via email

@Divy123
Copy link
Member

Divy123 commented Feb 18, 2019 •
edited
Loading

GoogleChrome/puppeteer#1260

In that case it can't be.

@jywarren
Copy link
Member Author

jywarren commented Feb 18, 2019 via email

@Divy123
Copy link
Member

Divy123 commented Feb 18, 2019

Use at your own risk though, since @aslushnikov says Emulation.setDefaultBackgroundColorOverride doesn't work well.

What say on this?

@jywarren
Copy link
Member Author

jywarren commented Feb 18, 2019 via email

@jywarren
Copy link
Member Author

jywarren commented Feb 18, 2019 via email

@Divy123
Copy link
Member

Divy123 commented Feb 18, 2019 •
edited
Loading

Sure @jywarren
Not mentioned specifically something about it.
I think that's the only way to work around right now.

@tech4GT
Copy link
Member

tech4GT commented Feb 19, 2019

@jywarren Are we open to modifying your fisheye-gl coed then? I can make the changes and put in a pr.

@tech4GT
Copy link
Member

tech4GT commented Mar 28, 2019

Oh, the approach I tried on this was nothing special as well, I was running the headless browser inside node and the module was running on the sequencer inside the headless browser!

@tech4GT
Copy link
Member

tech4GT commented Mar 28, 2019

The issue I posted earlier relates to the fact that by default gl is disabled in puppeteer. You need to explicitly enable it.

@tech4GT
Copy link
Member

tech4GT commented Mar 28, 2019

This is not an ideal solution though since we would want to avoid running the headless browser if possible(coz it'll take up a lot of resources)

@jywarren
Copy link
Member Author

Even if it takes a lot of resources (starting up the browser) I still think we should move ahead with this implementation -- it's the closest I've seen to actually working so far. We could refactor and attempt another approach afterwards, but I think we have to try! Thank you all!!!

@jywarren
Copy link
Member Author

Hi @Divy123 could you open a PR with your attempt so far? I'd like to try a few things.

For example, https://github.com/stackgl/headless-gl#example says we can access the pixels like:

//Create context
var width   = 64
var height  = 64
var gl = require('gl')(width, height, { preserveDrawingBuffer: true })

//Clear screen to red
gl.clearColor(1, 0, 0, 1)
gl.clear(gl.COLOR_BUFFER_BIT)

//Write output as a PPM formatted image
var pixels = new Uint8Array(width * height * 4)
gl.readPixels(0, 0, width, height, gl.RGBA, gl.UNSIGNED_BYTE, pixels)
process.stdout.write(['P3\n# gl.ppm\n', width, " ", height, '\n255\n'].join(''))
for(var i=0; i<pixels.length; i+=4) {
  for(var j=0; j<3; ++j) {
    process.stdout.write(pixels[i+j] + ' ')
  }
}

@tech4GT
Copy link
Member

tech4GT commented Mar 31, 2019

@jywarren I have an idea, maybe I can create a binding between gl and node-canvas and add the gl based functions on node-canvas using gl! I think this might be worth a shot.

@tech4GT
Copy link
Member

tech4GT commented Mar 31, 2019

If this works, it would allow for 100% code sharing between browser and node.

@tech4GT
Copy link
Member

tech4GT commented Mar 31, 2019

@jywarren @Divy123 I think I have figured this out, what I am doing is using 2 libraries:
node-canvas and gl, and I have overriden the default canvas.getContext to return an instance of 'node-gl'
Apart from that I needed some more changes in the fisheye-gl code and I think this might be ready.
Now do you guys think we should modify jywarren/fisheyegl or I should bake the node-specific code in the module itself?

@jywarren
Copy link
Member Author

jywarren commented Mar 31, 2019 via email

@tech4GT
Copy link
Member

tech4GT commented Mar 31, 2019

@jywarren I have figured out a lot of things, can you explain a little about how this code is important though.
https://github.com/jywarren/fisheyegl/blob/89d28f0dee943681192a708b59c076e3d5854652/src/fisheyegl.js#L204-L217
Since we don't have window available in node context, how should we change this code?

@tech4GT
Copy link
Member

tech4GT commented Mar 31, 2019

The current code is outputting a black image and I am unable to figure out why.

@jywarren
Copy link
Member Author

jywarren commented Apr 1, 2019 via email

@tech4GT
Copy link
Member

tech4GT commented Apr 2, 2019

Okay there is clearly some issue with the gl module, so for now I think we should proceed with the headless browser implementation. We can circle back to the other implementation once the issues are rectified.
cc @jywarren @Divy123

@Divy123
Copy link
Member

Divy123 commented Apr 2, 2019

I agree @tech4GT .
Exploring puppeteer nowadays to work on this.
Also I request you and @jywarren to please review my SoC proposal :
https://publiclab.org/notes/lit2017001/04-01-2019/soc-proposal-image-sequencer-v3-boosting-the-performance.
Thanks!!

@jywarren
Copy link
Member Author

jywarren commented Apr 2, 2019

Did you open an issue to ask more about it, and if so could you link there for reference? Thank you!

@jywarren
Copy link
Member Author

jywarren commented Apr 2, 2019

I will review ASAP, apologies this has been taking a lot of time this week 😄 thanks for your patience!

@Divy123
Copy link
Member

Divy123 commented Apr 3, 2019

Thanks a lot @jywarren .

@tech4GT
Copy link
Member

tech4GT commented Apr 4, 2019

@jywarren here stackgl/headless-gl#149

@jywarren
Copy link
Member Author

jywarren commented Apr 4, 2019 via email

@tech4GT
Copy link
Member

tech4GT commented Apr 11, 2019

@jywarren @Mridul97 So I was trying this out and I think the initial reload we do for the pwa results in puppeteer execution context being destroyed on sequencer.publiclab.org, I think we need to work around it somehow. That was a temporary solution anyway I guess.

@tech4GT
Copy link
Member

tech4GT commented Apr 11, 2019

@publiclab/is-reviewers Can someone point me to the link for dist files.
I am trying to add image-sequencer.min.js as a script on a page.

tech4GT added a commit to tech4GT/image-sequencer that referenced this issue Apr 11, 2019
@tech4GT
now running gl using localhost fixes publiclab#216
3a21c29 
Signed-off-by: tech4GT <varun.gupta1798@gmail.com>
@harshkhandeparkar
Copy link
Member

harshkhandeparkar commented Apr 22, 2019 •
edited
Loading

We are getting help from @robertleeplummerjr in using headless-gl
cc @tech4GT @jywarren @Divy123 @publiclab/is-reviewers

@robertleeplummerjr
Copy link

I don't have a ton of time I can offer, already on borrowed, but I can look into getting the fisheye project running on headless-gl, and then we can connect the dots backwards to this project, so it can be loaded in. Does that sound reasonable?

@jywarren
Copy link
Member Author

Wow that is very generous, thank you! We are very interested in this bc our puppeteer based approach (running a whole headless Chrome) is not very efficient. We would /love/ to work with whatever you can figure out and work to generalize a narrow case.

For what it's worth, this is another WebGL module we use that has simpler parameters (four corner coordinate pairs): https://github.com/jywarren/webgl-distort

You can see them both running in the browser here with a standard input and output:

Thank you!!!

@robertleeplummerjr
Copy link

robertleeplummerjr commented Apr 24, 2019 •
edited by harshkhandeparkar
Loading

If I use headlessgl like this:

const gl = require('gl');
const context = gl(1,1);
const getPixels = require('get-pixels');
const fisheyeGL = require('../../src/fisheyegl.js');
getPixels('example/images/grid.png', function(err, pixels) {
  if (err) {
    console.log(err);
    return;
  }

  fisheyeGL({
    image: pixels.data,
    getGLContext: () => context,
  });
});

And in fisheyegl, if I force the flow to use the pixels lib mentioned above, and I change line 157 from:

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, img);

to:

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, gl.UNSIGNED_BYTE, img);

I believe it works.

Here is the full diff I did:

diff --git a/src/fisheyegl.js b/src/fisheyegl.js
index db7369b..0117c63 100644
--- a/src/fisheyegl.js
+++ b/src/fisheyegl.js
@@ -41,7 +41,7 @@ var FisheyeGl = function FisheyeGl(options){
   var image = options.image || "images/barrel-distortion.png";
 
   var selector = options.selector || "#canvas";
-  var gl = getGLContext(selector);
+  var gl = options.getGLContext ? options.getGLContext() : getGLContext(selector);
 
   var shaders = require('./shaders');
 
@@ -172,7 +172,7 @@ var FisheyeGl = function FisheyeGl(options){
 
     gl.bindTexture(gl.TEXTURE_2D, texture);
 
-    gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, img);
+    gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, gl.UNSIGNED_BYTE, img);
 
     gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
     gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR); //gl.NEAREST is also allowed, instead of gl.LINEAR, as neither mipmap.
@@ -201,6 +201,13 @@ var FisheyeGl = function FisheyeGl(options){
     return texture;
   }
 
+  function loadImageFromPixels(gl, pixels, callback){
+    console.log('pixels', pixels);
+    var texture = gl.createTexture();
+    loadImage(gl, pixels, callback, texture);
+    return texture;
+  }
+
   function run(animate, callback){
     var f = window.requestAnimationFrame || window.mozRequestAnimationFrame ||
       window.webkitRequestAnimationFrame || window.msRequestAnimationFrame;
@@ -272,7 +279,11 @@ var FisheyeGl = function FisheyeGl(options){
     });
   }
 
-  setImage(image);
+  if (typeof image === 'string') {
+    setImage(image);
+  } else {
+    loadImageFromPixels(gl, image);
+  }
 
   // asynchronous!
   function getImage(format) {

@robertleeplummerjr
Copy link

You'll notice I only had an image size of 1x1 above, so not a practical solution, but proof you can load in the pixels.

@harshkhandeparkar
Copy link
Member

Wow! Thanks @robertleeplummerjr
@jywarren @tech4GT @VibhorCodecianGupta @Divy123

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
new-feature performance project
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@jywarren @robertleeplummerjr @tech4GT @vibhorgupta-gh @Divy123 @harshkhandeparkar