Creating Content Packs

Magneton · January 10, 2018, 1:14pm

Dear All,

Can someone point me towards some guidance on how to build content packs please?

Jake

jochen · January 10, 2018, 1:23pm

You can export content packs based on your current Graylog cluster on the System/Content Packs page by clicking on Create a content pack.

Magneton · January 10, 2018, 1:29pm

HI Jochen,

I have several custom dashboards connected to pipeline connected to streams that I want to export from one system and import to another.

I have tried exporting the dashboards via the API and then tried to re-import them but noticed that I need to add fields to the JSON.

So I was looking for a resource to explain how to build content packs containing streams and dashboards as well as whether pipeline could be included.

Cheers

Jake

jochen · January 10, 2018, 1:42pm

Do you have some examples?

Creating content packs via the System/Content Packs page is currently the only supported way of creating content packs.

Pipelines can currently not be part of content packs. This will likely change in Graylog 3.0.0:

github.com/Graylog2/graylog2-server

Content Packs v2 - Meta Issue

opened 09:15PM - 09 Oct 17 UTC

closed 11:13AM - 02 Feb 18 UTC

lennartkoopmann

feature triaged

This issue collects all issues related to the next version of content packs. …- [ ] "Support parameterized content packs" #4224 - [ ] "Support views in content packs" #4223 - [ ] "Support processing pipelines in content packs" #4222 - [ ] "Allow plugins to support content packs" #4221 - [ ] "Support version contraints for content packs" #4220 - [ ] "Better process to update content packs" #3517 - [ ] "Processing Pipelines cannot be included in Content Packs" #2507 #4222 - [ ] "Content Pack rollback does not rollback the Grok Patterns" #2189 - [ ] "Alerts configuration is not saved when creating a content pack for a stream" #1758 - [ ] "Web UI: Make contents of imported content packs inspectable" #1653 - [ ] "Include licenses in content packs v2?" https://github.com/Graylog2/graylog-plugin-license/issues/34

Magneton · January 10, 2018, 2:50pm

HI Jochen,

Created a test dashboard with one widget.

Get a list of dashboards id’s via API
Get content of single test dash board via API using id
Delete dashboard
Try to restore using API , it fails showing it is missing some required fields.

The json output from the API is

{
  "creator_user_id": "admin",
  "description": "Dashboard to test resotre via api",
  "created_at": "2018-01-10T14:11:25.568Z",
  "positions": {
    "138e4814-8557-4402-89d0-c6b09cf97f94": {
      "width": 1,
      "col": 1,
      "row": 1,
      "height": 3
    }
  },
  "id": "5a561f0d0061f404694f9aa5",
  "title": "Test",
  "widgets": [
    {
      "creator_user_id": "admin",
      "cache_time": 10,
      "description": "Sysmon Events 24h",
      "id": "138e4814-8557-4402-89d0-c6b09cf97f94",
      "type": "QUICKVALUES",
      "config": {
        "timerange": {
          "type": "relative",
          "range": 86400
        },
        "field": "sysmon_event_id",
        "stream_id": "5a5392430061f42d271a703d",
        "query": "",
        "show_data_table": true,
        "limit": 5,
        "show_pie_chart": false,
        "sort_order": "desc",
        "stacked_fields": "",
        "data_table_limit": 50
      }
    }
  ]
}

Next I tried looking at other content packs to see structure and modified the json to

{
  "id" : null,
  "name" : "Test Content Pack",
  "description" : "Dashboards - Single Test",
  "category" : "Test",
  "inputs" : [ ],
  "streams" : [ ],
  "outputs" : [ ],
  "dashboards" : [ {
    "title" : "Test Restore",
    "creator_user_id": "admin",
	"description": "Dashboard to test resotre via api",
	"created_at": "2018-01-10T14:11:25.568Z",
	"positions": {
		"138e4814-8557-4402-89d0-c6b09cf97f94": {
		"width": 1,
		"col": 1,
		"row": 1,
		"height": 3
		}
	},
	"id": "5a561f0d0061f404694f9aa5",
	"widgets": [
		{
		"creator_user_id": "admin",
		"cache_time": 10,
		"description": "Sysmon Events 24h",
		"id": "138e4814-8557-4402-89d0-c6b09cf97f94",
		"type": "QUICKVALUES",
		"config": {
		"timerange": {
          "type": "relative",
          "range": 86400
        },
        "field": "sysmon_event_id",
        "stream_id": "5a5392430061f42d271a703d",
        "query": "",
        "show_data_table": true,
        "limit": 5,
        "show_pie_chart": false,
        "sort_order": "desc",
        "stacked_fields": "",
        "data_table_limit": 50
		}
    }
  ]
}
]
}

The above json is valid according to JSON lint

but graylog looks to expect a title at line 49

{
  "type": "ApiError",
  "message": "Can not construct instance of org.graylog2.rest.models.dashboards.requests.CreateDashboardRequest, problem: Null title\n at [Source: org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream@64bc727e; line: 49, column: 1]"
}

Magneton · January 10, 2018, 2:55pm

graylog-api-2

It looks like I might need some information on the expected structure of the JSON elements to solve this as it should be possible to import and export dashboard and streams and then combine them in a JSON content pack

Cheers

Jake

jochen · January 10, 2018, 3:44pm

I think you’ve mixed up a few things.

The content packs (or part of the content packs) are not compatible with the payload you have to use with the Graylog REST API to create a dashboard.

You have to import the content pack and apply it, if you want to restore the streams and dashboards in another Graylog cluster.

Magneton · January 10, 2018, 4:05pm

Hi Jochen,

So are we saying that if you use the API on graylog system ‘1’ to export dashboard ‘A’ you cant use the API on system 2 with the Manage Dashboards POST method to add the dashboard?

Jake

jochen · January 10, 2018, 4:15pm

No, that’s not at all what I’m saying.

But you’ve copied parts of a content pack, which is not compatible with the resource in the Graylog REST API which can be used to create dashboards.

Magneton · January 10, 2018, 4:22pm

Hi Jochen,

My idea behind using the content pack parts was that if I can’t restore dashboards directly via API from one system to another, then I would try to use a content pack containing a single dashboard and restore that way as a basic proof of concept. If I could get it to work then I would add streams etc.

I was attempting to create a content pack skeleton ( ie all the necessary fields left blank) with just the dashboard in it. This would allow me to move dashboards from customer to customer without having to rebuild each one each time.

Ideally, I would like to build a test system to prove use cases / dashboards and then move dashboards, streams and pipelines to production systems.

What is the best way to achieve this?

Regards

Jake

jochen · January 10, 2018, 4:35pm

Use content packs and their appropriate API.

Magneton · January 15, 2018, 12:45pm

Hi Jochen,

How do I create a content pack from within graylog?

I have at least 3 custom dashboards and multiple streams with pipeline rules.

How do I create the content pack with the API?

Magneton

jochen · January 15, 2018, 1:56pm

Take a look at the /system/bundles resource in the Graylog REST API browser (Swagger).

github.com

Graylog2/graylog2-server/blob/2.4.0/graylog2-server/src/main/java/org/graylog2/rest/resources/system/bundles/BundleResource.java

/**
 * This file is part of Graylog.
 *
 * Graylog is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 *
 * Graylog is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with Graylog.  If not, see <http://www.gnu.org/licenses/>.
 */
package org.graylog2.rest.resources.system.bundles;

import com.codahale.metrics.annotation.Timed;
import com.google.common.collect.ImmutableSetMultimap;

This file has been truncated. show original

Magneton · January 15, 2018, 6:21pm

Hi Jochen,

So we use:

GET /system/bundles to get a list of bundle id’s and then use GET /system/bundles/{bundleId} to get a content pack for that id.

We can then import the content pack to Graylog using POST/system/bundles or Graylog UI?

What is the difference between the above and POST /system/bundles/export (list entities to export) ?

Is POST /system/bundles/export able to export a whole configuration (Streams, Dashboard,Input etc) where as GET /system/bundles/{bundleId} exports just a single item.

Should I use POST /system/bundles/export to export whole configuration to import into new Graylog?

Cheers

Jake

jochen · January 16, 2018, 9:02am

You should think of content packs as templates which describe entities (inputs, extractors, streams, dashboards, etc.).

You can “apply” a content pack multiple times, which would create the described entities multiple times. This was a design decision when we planned to introduce parameters for content packs at some point.

If you want to create a new content pack, you have to use POST /system/bundles/export and describe the entities it should contain in the request body (basically list their IDs, see ExportBundle).

HTTP method	Resource	Description
`POST`	`/system/bundles`	Upload new content pack.
`GET`	`/system/bundles`	List available content packs.
`GET`	`/system/bundles/{bundleId}`	Show content pack with ID `{bundleId}`.
`PUT`	`/system/bundles/{bundleId}`	Update content pack with ID `{bundleId}`.
`DELETE`	`/system/bundles/{bundleId}`	Delete content pack with ID `{bundleId}` (but don’t delete any entities which have been created from this content pack).
`POST`	`/system/bundles/{bundleId}`	Set up entities described by content pack with ID `{bundleId}`.
`POST`	`/system/bundles/export`	Export entities listed in request body as a new content pack.

system · January 30, 2018, 9:02am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Content packs export / import permissions Graylog Central (peer support)	1	444	November 24, 2020
Content packs with dashboard not working Graylog Central (peer support)	1	562	April 19, 2019
Export all stream and dashboards unsing curl with POST Development	1	1650	June 8, 2017
Unable to import content pack Graylog Central (peer support)	7	1417	May 31, 2021
Graylog 3.2 create streams and dashboards via API Graylog Central (peer support)	13	2605	June 8, 2020

Creating Content Packs

Related Topics