Compiling graylog from master fails with yarn error

I try to start development of a new Graylog plugin for OpenID Connect authentication/authorization but fail to get my development environment running properly.

Building Graylog from the current master (commit e8ee0fb1b08d3c26f637c021e27cc0206246e20c) fails:

[INFO] --------------------< org.graylog2:graylog2-server >--------------------
[INFO] Building Graylog 3.2.0-SNAPSHOT                                    [3/7]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- buildnumber-maven-plugin:1.4:create (default) @ graylog2-server ---
[INFO] Executing: /bin/sh -c cd '/home/jan/projects/graylog/graylog-project-repos/graylog2-server/graylog2-server' && 'git' 'rev-parse' '--verify' 'HEAD'
[INFO] Working directory: /home/jan/projects/graylog/graylog-project-repos/graylog2-server/graylog2-server
[INFO] Storing buildNumber: e8ee0fb1b08d3c26f637c021e27cc0206246e20c at timestamp: 1570352232575
[INFO] Storing buildScmBranch: master
[INFO] 
[INFO] --- antlr4-maven-plugin:4.7.1:antlr4 (antlr) @ graylog2-server ---
[INFO] No grammars to process
[INFO] ANTLR 4: Processing source directory /home/jan/projects/graylog/graylog-project-repos/graylog2-server/graylog2-server/src/main/antlr4
[INFO] 
[INFO] --- build-helper-maven-plugin:3.0.0:parse-version (parse-version) @ graylog2-server ---
[INFO] 
[INFO] --- frontend-maven-plugin:1.6:install-node-and-yarn (install node and yarn) @ graylog2-server ---
[INFO] Node v10.13.0 is already installed.
[INFO] Installing Yarn version v1.12.3
[INFO] Unpacking /home/jan/.m2/repository/com/github/eirslett/yarn/1.12.3/yarn-1.12.3.tar.gz into /home/jan/projects/graylog/graylog-project-repos/graylog2-server/graylog2-web-interface/node/yarn
[INFO] Installed Yarn locally.
[INFO] 
[INFO] --- frontend-maven-plugin:1.6:yarn (yarn install) @ graylog2-server ---
[INFO] Running 'yarn install' in /home/jan/projects/graylog/graylog-project-repos/graylog2-server/graylog2-web-interface
[INFO] yarn install v1.12.3
[INFO] [1/4] Resolving packages...
[ERROR] warning Resolution field "upath@1.0.5" is incompatible with requested version "upath@^1.1.1"
[INFO] success Already up-to-date.
[INFO] Done in 0.94s.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Graylog Parent POM 3.2.0-SNAPSHOT:
[INFO] 
[INFO] Graylog Parent POM ................................. SUCCESS [  0.003 s]
[INFO] Graylog Project Parent POM ......................... SUCCESS [  0.064 s]
[INFO] Graylog ............................................ FAILURE [  2.809 s]
[INFO] integration-tests .................................. SKIPPED
[INFO] Graylog Plugin Parent POM .......................... SKIPPED
[INFO] Graylog Plugin with Web support Parent POM ......... SKIPPED
[INFO] graylog-plugin-archetype ........................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.112 s
[INFO] Finished at: 2019-10-06T10:57:14+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.6:yarn (yarn install) on project graylog2-server: Failed to run task: 'yarn install' failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 134 (Exit value: 134) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :graylog2-server

I use graylog-project-cli exec "mvn compile -B" to compile the graylog sources. I’m on Debian GNU/Linux 10 if that matters.

Do I do something wrong or are the npm/yarn dependencies in a broken state currently?

Hello,

Here is a link that explains the steps to compile graylog:

Regards,
Merouane AGAR

@merouaneagar thanks for the link. I came to similar conclusions how to get a build environment before but tried to get it running from scratch, just to avoid own mistakes.

I followed the guide and got the same compilation errors mentioned above when running the Initial Build step of the README file. I continued and my freshly created plugin failed to build because of missing dependencies that have not been built from the graylog2-server tree in the initial build step:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.445 s
[INFO] Finished at: 2019-10-07T18:23:39+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project graylog-plugin-oidc-auth: Could not resolve dependencies for project info.dittberner:graylog-plugin-oidc-auth:jar:1.0.0-SNAPSHOT: Could not find artifact org.graylog2:graylog2-server:jar:tests:3.2.0-SNAPSHOT in sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

unfortunatelly the Travis CI builds for Graylog2-server (master branch) and the graylog-plugin-auth-sso are failing too, so I assume this is not just a problem for me.

Hey @jandd,

I would love to help you getting your dev setup up and running. First thing, I think it would be a better idea to develop against the 3.1.2 tag of the server repo instead of master. Things are changing constantly on master and might break your build.

This alone would probably solve the issue you are seeing already, as the 3.1.2 artefacts are on sonatype. If you really want to develop against master, you have to run maven in the graylog-project directory, so it is able to resolve the snapshot dependencies from its module tree instead of looking them up on sonatype.

I hope this helps!

P.S.: We know that we really need to update our docs when it comes to the scripts/bootstrap-plugin script that we removed. Sorry for that!

Hi dennis,

how do you recover version 3.1.2
because I do not arrive with the following command:

…/graylog-project.linux bootstrap https://github.com/Graylog2/graylog-project.git --manifest=“manifests/3.1.2.json”
git clone https://github.com/Graylog2/graylog-project.git graylog-project
Cloning into ‘graylog-project’…
git checkout master
Already on ‘master’
Your branch is up to date with ‘origin/master’.
Unable to read manifest: open /home/agar/Documents/projects/graylog/3.1.2/graylog-project/manifests/3.1.2.json: no such file or directory

Regards,
Merouane AGAR

Right now you would need to create a manifest on your own. For 3.1.2, it would look like this:

{
  "modules": [
    {
      "repository": "git@github.com:Graylog2/graylog2-server.git",
      "revision": "3.1.2",
      "server": true,
      "submodules": [
        {
          "path": "graylog2-server"
        },
        {
          "path": "graylog2-web-interface"
        }
      ]
    },
    {
      "repository": "git@github.com:Graylog2/graylog-plugin-collector.git",
      "revision": "3.1.2",
      "assemblies": ["graylog"]
    },
    {
      "repository": "git@github.com:Graylog2/graylog-plugin-aws.git",
      "revision": "3.1.2",
      "assemblies": ["graylog"]
    },
    {
      "repository": "git@github.com:Graylog2/graylog-plugin-threatintel.git",
      "revision": "3.1.2",
      "assemblies": ["graylog"]
    }
  ]
}

thanks, it works now

cat 3.1.2.json 
{
  "modules": [
{
  "repository": "git@github.com:Graylog2/graylog2-server.git",
  "revision": "3.1.2",
  "server": true,
  "submodules": [
    {
      "path": "graylog2-server"
    },
    {
      "path": "graylog2-web-interface"
    }
  ]
},
{
  "repository": "git@github.com:Graylog2/graylog-plugin-collector.git",
  "revision": "3.1.2",
  "assemblies": ["graylog"]
},
{
  "repository": "git@github.com:Graylog2/graylog-plugin-aws.git",
  "revision": "3.1.2",
  "assemblies": ["graylog"]
},
{
  "repository": "git@github.com:Graylog2/graylog-plugin-threatintel.git",
  "revision": "3.1.2",
  "assemblies": ["graylog"]
}
  ]
}

../graylog-project.linux bootstrap https://github.com/Graylog2/graylog-project.git --manifest="3.1.2.json" 
    git clone https://github.com/Graylog2/graylog-project.git graylog-project
      Cloning into 'graylog-project'...
    git checkout master
      Already on 'master'
      Your branch is up to date with 'origin/master'.
Unable to read manifest: open /home/agar/Documents/projects/graylog/3.1.2/graylog-project/3.1.2.json: no such file or directory
agar@P67RGRE ~/Documents/projects/graylog/3.1.2 $ rm -rf graylog-project/
agar@P67RGRE ~/Documents/projects/graylog/3.1.2 $ ../graylog-project.linux bootstrap https://github.com/Graylog2/graylog-project.git --manifest="../3.1.2.json" 
    git clone https://github.com/Graylog2/graylog-project.git graylog-project
      Cloning into 'graylog-project'...
    git checkout master
      Already on 'master'
      Your branch is up to date with 'origin/master'.
Repository: https://github.com/Graylog2/graylog2-server.git
Cloning https://github.com/Graylog2/graylog2-server.git into /home/agar/Documents/projects/graylog/3.1.2/graylog-project-repos/graylog2-server
    git clone https://github.com/Graylog2/graylog2-server.git /home/agar/Documents/projects/graylog/3.1.2/graylog-project-repos/graylog2-server
      Cloning into '/home/agar/Documents/projects/graylog/3.1.2/graylog-project-repos/graylog2-server'...
Checkout revision: 3.1.2
    git branch 3.1.2 origin/3.1.2
    git checkout 3.1.2
      Note: checking out '3.1.2'.
      You are in 'detached HEAD' state. You can look around, make experimental
      changes and commit them, and you can discard any commits you make in this
      state without impacting any branches by performing another checkout.
      If you want to create a new branch to retain commits you create, you may
      do so (now or later) by using -b with the checkout command again. Example:
        git checkout -b <new-branch-name>
      HEAD is now at 9e96b08ff [graylog2-server] prepare release 3.1.2
Repository: https://github.com/Graylog2/graylog-plugin-collector.git
Cloning https://github.com/Graylog2/graylog-plugin-collector.git into /home/agar/Documents/projects/graylog/3.1.2/graylog-project-repos/graylog-plugin-collector
    git clone https://github.com/Graylog2/graylog-plugin-collector.git /home/agar/Documents/projects/graylog/3.1.2/graylog-project-repos/graylog-plugin-collector
      Cloning into '/home/agar/Documents/projects/graylog/3.1.2/graylog-project-repos/graylog-plugin-collector'...
Checkout revision: 3.1.2
    git branch 3.1.2 origin/3.1.2
    git checkout 3.1.2
      Note: checking out '3.1.2'.
      You are in 'detached HEAD' state. You can look around, make experimental
      changes and commit them, and you can discard any commits you make in this
      state without impacting any branches by performing another checkout.
      If you want to create a new branch to retain commits you create, you may
      do so (now or later) by using -b with the checkout command again. Example:
        git checkout -b <new-branch-name>
      HEAD is now at b4ecf46 [graylog-plugin-collector] prepare release 3.1.2
Repository: https://github.com/Graylog2/graylog-plugin-aws.git
Cloning https://github.com/Graylog2/graylog-plugin-aws.git into /home/agar/Documents/projects/graylog/3.1.2/graylog-project-repos/graylog-plugin-aws
    git clone https://github.com/Graylog2/graylog-plugin-aws.git /home/agar/Documents/projects/graylog/3.1.2/graylog-project-repos/graylog-plugin-aws
      Cloning into '/home/agar/Documents/projects/graylog/3.1.2/graylog-project-repos/graylog-plugin-aws'...
Checkout revision: 3.1.2
    git branch 3.1.2 origin/3.1.2
    git checkout 3.1.2
      Note: checking out '3.1.2'.
      You are in 'detached HEAD' state. You can look around, make experimental
      changes and commit them, and you can discard any commits you make in this
      state without impacting any branches by performing another checkout.
      If you want to create a new branch to retain commits you create, you may
      do so (now or later) by using -b with the checkout command again. Example:
        git checkout -b <new-branch-name>
      HEAD is now at 25ceaf1 [graylog-plugin-aws] prepare release 3.1.2
Repository: https://github.com/Graylog2/graylog-plugin-threatintel.git
Cloning https://github.com/Graylog2/graylog-plugin-threatintel.git into /home/agar/Documents/projects/graylog/3.1.2/graylog-project-repos/graylog-plugin-threatintel
    git clone https://github.com/Graylog2/graylog-plugin-threatintel.git /home/agar/Documents/projects/graylog/3.1.2/graylog-project-repos/graylog-plugin-threatintel
      Cloning into '/home/agar/Documents/projects/graylog/3.1.2/graylog-project-repos/graylog-plugin-threatintel'...
Checkout revision: 3.1.2
    git branch 3.1.2 origin/3.1.2
    git checkout 3.1.2
      Note: checking out '3.1.2'.
      You are in 'detached HEAD' state. You can look around, make experimental
      changes and commit them, and you can discard any commits you make in this
      state without impacting any branches by performing another checkout.
      If you want to create a new branch to retain commits you create, you may
      do so (now or later) by using -b with the checkout command again. Example:
        git checkout -b <new-branch-name>
      HEAD is now at 2953ede [graylog-plugin-threatintel] prepare release 3.1.2
Generating pom.xml file from template pom.xml.tmpl
Generating runner/pom.xml file from template runner/pom.xml.tmpl
Generating src/main/assembly/server-tarball.xml file from template src/main/assembly/server-tarball.xml.tmpl
Writing manifest state to .graylog-project-manifest-state
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for graylog-project 3.1.2:
[INFO] 
[INFO] Graylog Parent POM ................................. SUCCESS [  0.003 s]
[INFO] Graylog Project Parent POM ......................... SUCCESS [  0.079 s]
[INFO] Graylog ............................................ SUCCESS [06:27 min]
[INFO] Graylog Plugin Parent POM .......................... SUCCESS [  0.000 s]
[INFO] Graylog Plugin with Web support Parent POM ......... SUCCESS [  1.514 s]
[INFO] graylog-plugin-collector ........................... SUCCESS [01:40 min]
[INFO] graylog-plugin-aws ................................. SUCCESS [01:25 min]
[INFO] graylog-plugin-threatintel ......................... SUCCESS [01:27 min]
[INFO] graylog-project .................................... SUCCESS [  0.000 s]
[INFO] runner ............................................. SUCCESS [  0.127 s]
[INFO] integration-tests .................................. SUCCESS [  0.019 s]
[INFO] graylog-plugin-archetype ........................... SUCCESS [  0.067 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11:02 min
[INFO] Finished at: 2019-10-08T16:53:17+02:00
[INFO] ------------------------------------------------------------------------

Hello Dennis,

I encounter an error message in the web interface, can you tell me that I missed ?

Thanks for your help.

Regards,
Merouane AGAR

I can only guess, but I assume it is a CORS error. Can you try connecting to http://localhost:8080 (vs. http://127.0.0.1:8080) in your browser? it is a fine difference that only matters to CORS checks in your browser.

Hi Dennis,

Thanks for your help

I have practically the same result

Regards,
Merouane

Do you see anything in your browser’s JS console?

Yes ,

Hi Dennis,

I solved my problem, I forgot to run the server from the IDE
Thanks for your help

Regards,
Merouane AGAR

1 Like

Hi @dennis,

I just found time to test your advice. Compiling from the 3.1.2 tag worked fine. I will continue my plugin development based on this graylog version.

1 Like