Update for one-jar Alpaca #208
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ruebot
reviewed
Nov 15, 2021
roles/internal/Islandora-Devops.alpaca/templates/alpaca.conf.j2
Outdated
Show resolved
Hide resolved
|
I built this in conjunction with #207, and removed all the triplestore call, except I was able to add a test collection, image, and audio objects successfully. The objects ended up fcrepo, and derivatives were created. I also kept an eye on |
|
Oh, I noticed I get these too in the log: Not sure if that's just something to ignore. |
|
Since we're doing away with Karaf, we should also:
|
ruebot
reviewed
Nov 16, 2021
|
Was the Milliner 404 error causing a problem? Like was anything not happening correctly? There are definitely some messages that seem to fail, not sure if its timing or when the message is generated. Like the second error, this normally happens when you select a media file. Drupal immediately starts to upload it and we send a message. But there is no actual media created so Alpaca fails which is expected but also can look like any other error. |
Those? I don't think they were causing any problems because all the derivatives were created, so they were sent to the correct microservices. I don't have derivatives in fcrepo, so that could be it? That one is from when I select a file to upload, but haven't uploaded anything yet. So, your explanation makes sense there, and it make sense that it's a |
ruebot
added a commit
to yorkulibraries/yudl-playbook
that referenced
this pull request
Nov 16, 2021
ruebot
added a commit
to yorkulibraries/yudl-playbook
that referenced
this pull request
Nov 16, 2021
- Localized updates from: - Islandora-Devops/islandora-playbook#208 - Islandora-Devops/islandora-playbook#207 - Update to fcrepo 6.1.0 - Update PDF.js to most recent version as of this commit
ruebot
requested changes
Nov 18, 2021
|
So re: deleting the |
|
Actually maybe the current |
|
Yeah, I figured at some point the playbook would get updated to allow for a released artifact to be used instead of compiling. We're just in that chicken-egg situation for that now 😄 |
|
I ran the playbook build from scratch with this branch.
Is there some other config that needs to be done? |
|
Encountered a few errors in the run... seems to have come up, eventually: I see that "Failure okay" thing on the second, and "ignore_errors" or whatever in the code for the first... seems... odd to expect errors to be emitted from things, but anyway. All that said... doesn't seem to lead to a system with Alpaca setup? As in, doing a In the build logs, I do see: ... still trying to do something with Karaf? I see that the renamed - hosts: karaf
become: yes... does this also need to be updated to Update: Tried changing to Update x2: Seemed to do the trick; however, looks like it runs into the issue as reported over on the Alpaca PR, with the type conversion: Islandora/Alpaca#82 (comment) ... so will continue on things over there... Something of an aside, but... if we did want to put compiled artifacts somewhere, could possibly roll a "release candidate" kind of release, with the built artifacts, kind of thing? |
adam-vessey
reviewed
Dec 2, 2021
Comment on lines
3
to
5
| if [ -f "/usr/libexec/tomcat9/tomcat-locate-java.sh" ]; then | ||
| . /usr/libexec/tomcat9/tomcat-locate-java.sh | ||
| fi |
There was a problem hiding this comment.
Seems like unnecessary coupling for this to be referencing Tomcat stuff... not the end of the world, but... shrug?
There was a problem hiding this comment.
Yeah, I didn't want to copy the code and $JAVA_HOME doesn't seem to be set by default. Because this is only for Ubuntu I'll just assume where java is.
There was a problem hiding this comment.
Made a guess based on where Ubuntu should put OpenJDK 11. Resolved in a741462
Comment on lines
10
to
24
| - name: Install Alpaca via Gradle | ||
| command: ./gradlew install | ||
| command: ./gradlew build shadowJar | ||
| args: | ||
| chdir: "{{ alpaca_clone_directory }}" | ||
| when: alpaca_git.changed |
There was a problem hiding this comment.
Should... this be run as a non-root user? Seems like it will create/leave the "gradle daemon" dealio intact running as root... seems less-than ideal?
There was a problem hiding this comment.
Changed to run as the alpaca_user, in a741462
Comment on lines
8
to
9
| alpaca_user: tomcat | ||
| alpaca_group: tomcat |
There was a problem hiding this comment.
Should this create its own user/group by default? Just yeah... that coupling thing again.
There was a problem hiding this comment.
Apologies if this is not an issue, but can we make sure you aren't hardcoding 'vagrant' in there since we aren't always deploying to vagrant.
{{ ansible_os_user }} is the most current way to reference the user that becomes 'vagrant' on a vagrant box.
The Drupal role gives a more robust expression:
"{{ ansible_ssh_user | default(ansible_env.SUDO_USER, true) | default(ansible_env.USER, true) | default(ansible_user_id) }}"There was a problem hiding this comment.
I'm still using a role variable, sorry I should have been clearer. I just changed the default from tomcat to vagrant, so you can still change this to whatever you want by overridding it in your inventory.
|
Re-ran with this morning's updates, was able to generate derivatives for image and PDF objects. |
|
Ran this playbook update on top of an existing Vagrant box. Ansible showed no errors, but did get the following error in alpaca.log when uploading an image in the add media form: INFO 14:55:06.486 [Camel (camel-1) thread #13 - JmsConsumer[islandora-indexing-triplestore-index]] (TriplestoreIndexer) Unable to find JsonLD Url, this is an error or happens when a file is pre-uploaded to Drupal
ERROR 14:55:06.514 [Camel (camel-1) thread #13 - JmsConsumer[islandora-indexing-triplestore-index]] (DefaultErrorHandler) Failed delivery for (MessageId: ID:islandora8-38751-1638977195832-3:15:-1:1:2 on ExchangeId: A5D8E28EFD775B7-0000000000000000). Exhausted after delivery attempt: 1 caught: ca.islandora.alpaca.support.exceptions.MissingJsonldUrlException: Cannot find JSONLD URL in event message.. Processed by failure processor: FatalFallbackErrorHandler[Channel[log28]]
Message History (complete message history is disabled)
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[IslandoraTriplesto] [IslandoraTriplesto] [from[broker://queue:islandora-indexing-triplestore-index] ] [ 806]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
ca.islandora.alpaca.support.exceptions.MissingJsonldUrlException: Cannot find JSONLD URL in event message.
at ca.islandora.alpaca.support.event.AS2Object.getJsonldUrl(AS2Object.java:119)
at ca.islandora.alpaca.indexing.triplestore.TriplestoreIndexer.lambda$configure$0(TriplestoreIndexer.java:128)
at org.apache.camel.support.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:66)
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.doRun(RedeliveryErrorHandler.java:714)
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.run(RedeliveryErrorHandler.java:623)
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:147)
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:312)
at org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:83)
at org.apache.camel.support.AsyncProcessorSupport.process(AsyncProcessorSupport.java:41)
at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:130)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:318)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1237)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1227)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1120)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
WARN 14:55:06.522 [Camel (camel-1) thread #13 - JmsConsumer[islandora-indexing-triplestore-index]] (EndpointMessageListener) Execution of JMS message listener failed. Caused by: [org.apache.camel.RuntimeCamelException - ca.islandora.alpaca.support.exceptions.MissingJsonldUrlException: Cannot find JSONLD URL in event message.]
org.apache.camel.RuntimeCamelException: ca.islandora.alpaca.support.exceptions.MissingJsonldUrlException: Cannot find JSONLD URL in event message.
at org.apache.camel.RuntimeCamelException.wrapRuntimeCamelException(RuntimeCamelException.java:51)
at org.apache.camel.component.jms.EndpointMessageListener$EndpointMessageListenerAsyncCallback.done(EndpointMessageListener.java:213)
at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:134)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:318)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1237)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1227)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1120)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: ca.islandora.alpaca.support.exceptions.MissingJsonldUrlException: Cannot find JSONLD URL in event message.
at ca.islandora.alpaca.support.event.AS2Object.getJsonldUrl(AS2Object.java:119)
at ca.islandora.alpaca.indexing.triplestore.TriplestoreIndexer.lambda$configure$0(TriplestoreIndexer.java:128)
at org.apache.camel.support.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:66)
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.doRun(RedeliveryErrorHandler.java:714)
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.run(RedeliveryErrorHandler.java:623)
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:147)
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:312)
at org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:83)
at org.apache.camel.support.AsyncProcessorSupport.process(AsyncProcessorSupport.java:41)
at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:130)
... 11 common frames omitted
The triple store does seem to have entries for the added item so unsure what this error is indicating. Note this probably belongs in the closed PR for alpaca :P |
alxp
requested changes
Dec 20, 2021
| ``` | ||
| alpaca_karaf_dir: /opt/karaf | ||
| alpaca_group: vagrant |
There was a problem hiding this comment.
This should be an Ansible default ssh user variable, since it may not be running on Vagrant.
| - islandora-connector-broadcast | ||
| - islandora-indexing-triplestore | ||
| - islandora-indexing-fcrepo | ||
| alpaca_user: vagrant |
There was a problem hiding this comment.
This should be the Ansible default ssh user variable, since it may not be running on Vagrant.
| @@ -2,16 +2,14 @@ | |||
|
|
|||
| # Comment in to build Alpaca from source | |||
| alpaca_from_source: yes | |||
| alpaca_version: dev | |||
| alpaca_version: one-app | |||
There was a problem hiding this comment.
Can this be flipped back to 'dev' now?
There was a problem hiding this comment.
I was trying to release Alpaca first so I could put a tag in here, but I can switch it back
There was a problem hiding this comment.
Actually it should be 2.x. Dev is not up-to-date anymore.
| - mvn:org.apache.camel.karaf/apache-camel/2.20.4/xml/features | ||
| - mvn:org.apache.activemq/activemq-karaf/5.15.0/xml/features | ||
| - mvn:ca.islandora.alpaca/islandora-karaf/1.0.1/xml/features | ||
| alpaca_user: vagrant |
There was a problem hiding this comment.
This should be the ansible default ssh user variable and not 'vagrant' as this may not be being run on Vagrant.
There was a problem hiding this comment.
So I changed this to
alpaca_user: "{{ ansible_user }}"
alpaca_group: "{{ ansible_user }}"
now I get this
TASK [Islandora-Devops.alpaca : Install Alpaca via Gradle] *********************
Monday 20 December 2021 11:39:17 -0600 (0:00:00.316) 0:58:00.653 *******
fatal: [default]: FAILED! => {"msg": "The field 'become_user' has an invalid value, which includes an undefined variable. The error was: {{ ansible_user }}: 'ansible_user' is undefined. {{ ansible_user }}: 'ansible_user' is undefined"}
🤷
p.s. I used ansible_user and not ansible_ssh_user because of the note here https://docs.ansible.com/ansible/2.3/intro_inventory.html#list-of-behavioral-inventory-parameters
There was a problem hiding this comment.
I also tried ansible_ssh_user and it also fails... 🤷
There was a problem hiding this comment.
@alxp I'm going to leave this as is because I can't figure out what is supposed to work. You can override it with your own ansible config and set it to whatever you need.
There was a problem hiding this comment.
Actually for some reason it is using ansible_user in become_user that is the problem. Otherwise it is valid, but there it causes a failure. I'll do a little more digging
|
Got the following error while running this branch of the playbook: TASK [Gathering Facts] *********************************************************
Monday 20 December 2021 10:21:06 -0400 (0:00:00.691) 0:27:46.045 *******
ok: [default]
TASK [Islandora-Devops.alpaca : Clone CLAW Alpaca] *****************************
Monday 20 December 2021 10:21:07 -0400 (0:00:01.195) 0:27:47.241 *******
fatal: [default]: FAILED! => {"changed": false, "cmd": "/usr/bin/git checkout --force one-app", "msg": "Failed to checkout one-app", "rc": 1, "stderr": "error: pathspec 'one-app' did not match any file(s) known to git\n", "stderr_lines": ["error: pathspec 'one-app' did not match any file(s) known to git"], "stdout": "", "stdout_lines": []}
PLAY RECAP Probably a branch got deleted? |
|
Getting this building a new playbook: TASK [Islandora-Devops.alpaca : Clone CLAW Alpaca] *****************************
Tuesday 22 March 2022 13:01:55 -0300 (0:00:01.104) 1:07:47.311 *********
fatal: [default]: FAILED! => {"changed": false, "cmd": "/usr/bin/git checkout --force one-app", "msg": "Failed to checkout one-app", "rc": 1, "stderr": "error: pathspec 'one-app' did not match any file(s) known to git\n", "stderr_lines": ["error: pathspec 'one-app' did not match any file(s) known to git"], "stdout": "", "stdout_lines": []}
|
whikloj
force-pushed
the
one-app-updates
branch
from
7eb2539 to
4588e2c
Compare
|
Running playbook now... |
|
In post-install.yml, there's a server variable that needs to be updated. - name: Force Alpaca host setting to override install profile.
command: "{{ drush_path }} --root {{ drupal_core_path }} -y -l localhost:{{ apache_listen_port }} cset islandora.settings broker_url tcp://{{ hostvars[groups['alpaca'][0]].ansible_host }}:61613"
|
alxp
approved these changes
Mar 24, 2022
alxp
approved these changes
Mar 24, 2022
ruebot
approved these changes
Mar 29, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
GitHub Issue: Islandora/documentation#731
What does this Pull Request do?
Updates islandora-playbook to install the changes in Islandora/Alpaca#82
I'm leaving this as draft until things are merged and this can be pointed back to islandora repos.
How should this be tested?
This will build a system with the runnable jar instead of the karaf installation.
Interested parties
Tag (@ mention) interested parties or, if unsure, @Islandora-Devops/committers