Rdf Flushing Microservice #29

dannylamb · 2017-04-19T18:55:36Z

GitHub Issue: Part of Islandora/documentation#597

What does this Pull Request do?

Adds a microservice that retrieves Drupal entites as JSONLD and persists them in a Fedora repository.

What's new?

An entire new Silex microservice.

How should this be tested?

Run it with the PHP builtin php -S localhost:8088 -t src
Create a FedoraResource entity in Drupal. Let's pretend it's at http://localhost:8000/fedora_resource/1.
curl -X POST -H "Authorization: Bearer SOME_CRAZY_TOKEN" "http://localhost:8088/fedora_resource/1" should return the uri of the created resource. I got `http://localhost:8080/fcrepo/rest/4/e2/18/ab/14e218ab-aec4-4262-a71a-36d0466353dd".
Link the two using Gemini (Milliner purposefully does not do this itself, as the asynchronous workflow should handle that. PR to come soon). curl -X POST -H "Authorization: Bearer SOME_CRAZY_TOKEN" -H "Content-Type: application/json" -d '{"drupal": "fedora_resource/1", "fedora": "14/e2/18/ab/14e218ab-aec4-4262-a71a-36d0466353dd"}' "http://localhost:8000/gemini/" You'll have to provide your own fedora path using the uri you got in the previous step.
Update the resource in Drupal (like change title or description)
curl -X PUT -H "Authorization: Bearer SOME_CRAZY_TOKEN" "http://localhost:8088/fedora_resource/1" should return 204 No Content.
GET it from Fedora to verify your changes took effect curl -H "Authorization: Bearer SOME_CRAZY_TOKEN" "http://localhost:8080/fcrepo/rest/14/e2/18/ab/14e218ab-aec4-4262-a71a-36d0466353dd"
curl -X DELETE -H "Authorization: Bearer SOME_CRAZY_TOKEN" "http://localhost:8088/fedora_resource/1" should return 204 No Content
Try to GET it from Fedora again, and you should get a tombstone. curl -H "Authorization: Bearer SOME_CRAZY_TOKEN" "http://localhost:8080/fcrepo/rest/14/e2/18/ab/14e218ab-aec4-4262-a71a-36d0466353dd"

Interested parties

@Islandora-CLAW/committers

codecov · 2017-04-19T18:59:49Z

Codecov Report

Merging #29 into master will decrease coverage by 29.15%.
The diff coverage is 45.85%.

@@              Coverage Diff              @@
##             master      #29       +/-   ##
=============================================
- Coverage     98.41%   69.25%   -29.16%     
- Complexity       35       56       +21     
=============================================
  Files             3        6        +3     
  Lines           126      283      +157     
=============================================
+ Hits            124      196       +72     
- Misses            2       87       +85

Impacted Files	Coverage Δ	Complexity Δ
Hypercube/src/Controller/HypercubeController.php	`100% <ø> (ø)`	`4 <0> (ø)`	⬇️
Milliner/src/Converter/DrupalEntityConverter.php	`0% <0%> (ø)`	`3 <3> (?)`
Milliner/src/Service/MillinerService.php	`0% <0%> (ø)`	`9 <9> (?)`
Milliner/src/Controller/MillinerController.php	`100% <100%> (ø)`	`9 <9> (?)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78e5d2f...9b5f4b3. Read the comment docs.

ruebot · 2017-04-19T19:05:07Z

Paging @manez; Avatar time! 🎉

manez · 2017-04-19T19:06:35Z

Aw, c'mon man. flushing microservice?

ruebot · 2017-04-19T19:12:19Z

wobster in a toilet?

manez · 2017-04-19T19:20:15Z

I can't even...

ajs6f · 2017-04-19T20:17:53Z

Milliner/README.md

+- Install `composer`.  [Install instructions here.][4]
+- `$ cd /path/to/Milliner` and run `$ composer install`
+- Then either
+  - For production, configure your web server appropriately (e.g. add a VirtualHost for Houdini in Apache) OR


Should these references to Houdini refer to Milliner?

ajs6f · 2017-04-19T20:17:59Z

Milliner/README.md

+- `$ cd /path/to/Milliner` and run `$ composer install`
+- Then either
+  - For production, configure your web server appropriately (e.g. add a VirtualHost for Houdini in Apache) OR
+  - For development, run the PHP built-in webserver `$ php -S localhost:8888 -t src` from Houdini root.


ajs6f · 2017-04-19T20:21:05Z

Milliner/README.md

+
+For example, suppose you create an entity at `http://localhost:8000/fedora_resource/`.  If running the PHP built-in server command described in the Installation section:
+```
+$ curl -X "POST" "localhost:8888/metadata/fedora_resource/1"


This may be a really dumb question, but why is there a 1 on the end of that URI? What does it mean?

1 is the first one. It'll increment from there basically like drupalsite.foo/node/1.

The first what? The first time you did the sync (i.e. the first "push")?

First "Fedora Resource" created in Drupal. I don't have a VM fired up right now, but iirc, localhost:8000, the add content, then select "Fedora Resource" (that's our content type), and select what kind you want, and then go from there.

Hm, that's weird. When I fired up the VM, it just offered the choices "Page" or "Article" to create. Maybe I did something wrong.

Anyways, I'm sorry to be so thick about this, but what is the difference between localhost:8888/metadata/fedora_resource/1 and localhost:8888/metadata/other_fedora_resource/1 and localhost:8888/metadata/fedora_resource/2? If I do five POSTs to localhost:8888/metadata/fedora_resource/ I'll be at localhost:8888/metadata/fedora_resource/5, right? Will I have created five separate resources in Fedora, all of them copied from the same Drupal resource http://localhost:8000/fedora_resource/?

Or is it that localhost:8888/metadata/fedora_resource/1 is the endpoint associated with a Drupal resource http://localhost:8000/fedora_resource/1? In that case, should it say above "suppose you create an entity at http://localhost:8000/fedora_resource/1" (note that I added the 1)

@ajs6f

As I read it the difference between http://localhost:8888/metadata/fedora_resource/1 and http://localhost:8888/metadata/fedora_resource/2 is that the first refers to the Drupal entity at http://localhost:8000/fedora_resource/1 and the second refers to the other Drupal entity at http://localhost:8000/fedora_resource/2.

Or POSTing to <Milliner hostname/port>/metadata/<path> pushes the entity at <Drupal hostname/port>/<path> to Fedora.

Okay, cool, then I think I finally get it, but as I wrote above

In that case, should it say above "suppose you create an entity at http://localhost:8000/fedora_resource/1" (note that I added the 1)"

(Sidebar: This is why in any doc-ish situation where I need to use a random ID for a resource, I never use 1. I always pick some obviously random int like 88 or 1234 or something. 1 confuses really simpleminded people like me who suppose it to have some actual semantic.)

@whikloj has it right

POSTing to <Milliner hostname/port>/metadata/ pushes the entity at <Drupal hostname/port>/ to Fedora.

dannylamb · 2017-04-20T13:45:01Z

I give you a microservice with the name of a hat maker. I receive a toilet avatar. THANKS INTERNET

ajs6f · 2017-04-20T13:57:23Z

Milliner/README.md

+
+This retrieves a jsonld representation of the specified Drupal entity and inserts it in Fedora.
+
+For example, suppose you create an entity at `http://localhost:8000/fedora_resource/`.  If running the PHP built-in server command described in the Installation section:


In light of the discussion below, shouldn't this be "For example, suppose you create an entity at http://localhost:8000/fedora_resource/1."?

yes it should

ruebot · 2017-04-20T14:41:52Z

@dannylamb can you update the main Crayfish README too? https://github.com/dannylamb/Crayfish/tree/milliner#services

dannylamb · 2017-04-20T14:57:52Z

@ruebot yeah np. and i'm not sure if i should update config now to use yaml as per #27 or if i should just wait until @acoburn's work lands.

manez · 2017-04-20T17:45:41Z

Hats?

acoburn · 2017-04-20T17:48:54Z

@dannylamb don't feel like you should wait for me. I think you should just proceed, and I'll integrate whatever you do into my eventual PR. I am still figuring out the best way to structure all of this (thanks for the suggestions @whikloj!) and I don't want to hold you up.

acoburn · 2017-04-20T17:51:45Z

Milliner/cfg/cfg.php

+
+return [
+    'fedora base url' => 'http://localhost:8080/fcrepo/rest',
+    'drupal base url' => 'http://localhost:8000',


would it be possible to avoid spaces in these array keys? (just thinking about the future YAML-based config...)

yeah no prawb

whikloj · 2017-04-21T13:29:59Z

@manez I'm not sure the toilet handle is required, but I do love the hats. Perhaps you could include the artists palette from http://daviddunkley.me/queens-plate/??

manez · 2017-04-21T14:07:01Z

Ok no toilet hardware. Just hats. @dannylamb you have options 😄

whikloj · 2017-04-21T14:27:04Z

Milliner/src/Service/MillinerService.php

+        if ($fedora_path !== null) {
+            throw new \RuntimeException(
+                "$drupal_path already exists in Fedora at $fedora_path",
+                200


This should probably not be a 200 response code. Perhaps 409 Conflict?

yeah, 409 for sure

whikloj

Just the 200 OK -> 409 Conflict change (or some other code). But otherwise this works as expected.

I feel like Miliner or MilinerService should be adding the Gemini record so you don't end up generating more Fedora resources than Drupal resources?

dannylamb · 2017-04-21T16:51:42Z

@whikloj Milliner very purposefully does not create the link using gemini. That will be a separate operation in the async workflow. Suppose Fedora is up, and after you create the resource with Milliner but before you link it, the db goes down. If you retried the operation, you would create a duplicate resource until the db came up. If linking is a separate operation, then you can just keep retrying that until you succeed without creating duplicates.

whikloj · 2017-04-21T17:01:07Z

@dannylamb I'm not sure I understand that work flow. Here is what I did that raised the issue with me.

I created a resource in Drupal (http://localhost:8000/fedora_resource/999)
I POSTed that to Miliner and it responded with the new Fedora URL.
I forgot to link them in Gemini (or perhaps the database was down)
I tried to POST again to Miliner and it created another copy in Fedora.

So it seems that if you don't get that link made in Gemini you could end up in a situation of a GET from Gemini returning a 404 and then you POST to Miliner (because as far as you know it hasn't been posted yet) again and again.

I'm not really sure how to mitigate this issue...

ajs6f · 2017-04-21T17:07:19Z

@whikloj Is the problem here that Milliner doesn't create the link, or that the action against Milliner isn't idempotent? For example, if you could use PUT to create as well as update, and you tried the same workflow, would you avoid the problem?

whikloj · 2017-04-21T17:07:22Z

Milliner/cfg/config.example.yaml

+  port: 3306
+  dbname: gemini
+  user: changeme
+  password: changeme


Does this section need to be here for Milliner?

Yeah, it's for the PathMapper that gets injected into the MillinerService.

Oh I guess it does, so Gemini and Milliner both need a PathMapper DB connection.

dannylamb · 2017-04-21T17:07:41Z

@whikloj If you combine the operations, then it fails to be idempotent which is going to be a problem. Separating them leaves us with two operations that can be retried independently of each other without introducing duplicates or making weird state.

The solution is that we use an alpaca pipeline that pokes milliner first, then pokes gemini.

dannylamb · 2017-04-21T17:10:47Z

@ajs6f The problem with PUT is that then I need to know where I want to put it. For now I'm just POSTing and letting the default Fedora path minter kick in. I check to see if it's already been created and abort the POST to Milliner with 409 to keep it from introducing duplicates.

ajs6f · 2017-04-21T17:16:24Z

@dannylamb Makes sense, but from a µservice POV, would it be better to factor the default id mapping into Milliner as a service?

whikloj · 2017-04-21T17:32:54Z

@ajs6f I like the idea of mapping our own paths, I'm always just unclear on what would be a good method to generate a consistent path.

whikloj · 2017-04-21T17:35:27Z

I'm good to merge this. Comments/concerns @Islandora-CLAW/committers? Speak now or...it'll be too late.

ajs6f · 2017-04-21T17:53:51Z

Just leaving a note here to connect to some relevant IRC discussion: http://irclogs.islandora.ca/2017-04-21.html

ajs6f reviewed Apr 19, 2017

View reviewed changes

ajs6f reviewed Apr 20, 2017

View reviewed changes

acoburn reviewed Apr 20, 2017

View reviewed changes

whikloj reviewed Apr 21, 2017

View reviewed changes

whikloj requested changes Apr 21, 2017

View reviewed changes

dannylamb added 7 commits April 21, 2017 11:32

Committing before branch hopping

c3a5dea

First pass. Needs tests and cleanup before PR.

43593c1

Using refactored PathMapper

eb0f775

Manually tested and working. Gave it tests.

be55d9f

README

dd676b7

Cleanup before PR

1d73a92

README fixes and alignment with said README

9c60812

dannylamb force-pushed the milliner branch from a4124b8 to 9c60812 Compare April 21, 2017 14:47

dannylamb added 2 commits April 21, 2017 13:40

using yaml config and changede create exception to status code 409

4df9e6f

README update

9b5f4b3

whikloj reviewed Apr 21, 2017

View reviewed changes

whikloj approved these changes Apr 21, 2017

View reviewed changes

whikloj merged commit 29dbba8 into Islandora:master Apr 21, 2017

whikloj mentioned this pull request Apr 21, 2017

Issue 598: Update Syn configuration root element Islandora/Syn#7

Merged


		This retrieves a jsonld representation of the specified Drupal entity and inserts it in Fedora.

		For example, suppose you create an entity at `http://localhost:8000/fedora_resource/`. If running the PHP built-in server command described in the Installation section:

Rdf Flushing Microservice #29

Rdf Flushing Microservice #29

Conversation

dannylamb commented Apr 19, 2017

What does this Pull Request do?

What's new?

How should this be tested?

Interested parties

codecov bot commented Apr 19, 2017 • edited Loading

Codecov Report

ruebot commented Apr 19, 2017

manez commented Apr 19, 2017

ruebot commented Apr 19, 2017

manez commented Apr 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dannylamb commented Apr 20, 2017

ajs6f Apr 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruebot commented Apr 20, 2017

dannylamb commented Apr 20, 2017

manez commented Apr 20, 2017

acoburn commented Apr 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whikloj commented Apr 21, 2017

manez commented Apr 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whikloj left a comment

Choose a reason for hiding this comment

dannylamb commented Apr 21, 2017

whikloj commented Apr 21, 2017

ajs6f commented Apr 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dannylamb commented Apr 21, 2017 • edited Loading

dannylamb commented Apr 21, 2017

ajs6f commented Apr 21, 2017

whikloj commented Apr 21, 2017 • edited Loading

whikloj commented Apr 21, 2017

ajs6f commented Apr 21, 2017 • edited Loading

codecov bot commented Apr 19, 2017 •

edited

Loading

ajs6f Apr 20, 2017 •

edited

Loading

dannylamb commented Apr 21, 2017 •

edited

Loading

whikloj commented Apr 21, 2017 •

edited

Loading

ajs6f commented Apr 21, 2017 •

edited

Loading