Hello.
Can anyone tell me why my graylog syslog UDP extractors stop without any reason?
He is working fine but without any changes it stop’s!
Please elaborate.
Which extractors have you been using?
What’s the result you’ve expected and what’s the actual result?
Please check this prints please.
The first one is post error the last one is pre error
Remember to check the timestamp because this to message occours in the moment of the error.
There was no changes made to the grok patterns neither syslog-ng. Graylog continues to recive them bus dont do the extraction as planned.
Just to repeat my questions, because you didn’t answer any of them:
- Which extractors have you been using?
- What’s the result you’ve expected and what’s the actual result?
Additionally please add the configuration of your syslog daemon.
jochen, i gess the explication was good enougth…
But in casse is was not goot enougth where go the configuration file with all extractors.
And My goal is clearly is put the graylog working again because HE IS NOT EXTRATING MY MESSAGES!
{
"category": "Help",
"inputs": [
{
"extractors": [
{
"source_field": "message",
"title": "1.Watchguard Syslog Header",
"order": 1,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": "%{SYSLOG_STANDARD:timestamp} %{IPORHOST:source} .*\\) %{DATA:message:string}$",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "2.Watchguard Header",
"order": 2,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": "%{DATA:facility:string}: msg_id=\"%{WORD:watchguard_mgs_id_1}\\-%{WORD:watchguard_msg_id_2}\" %{ACTION:watchguar_action} %{DATA:watchguard_interface_src} %{DATA:watchguard_interface_dst}( %{NUMBER:watchguard_ip_package_size:int})?( %{PROTOCOL:watchguard_protocol:string})?( %{NUMBER:watchguard_ip_header_size:int})?( %{NUMBER:watchguard_ttl:int})? %{IPORHOST:watchguard_ip_src} %{IPORHOST:watchguard_ip_dst} %{NUMBER:watchguard_port_src} %{NUMBER:watchguard_port_dst} %{DATA:message}$",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "3.Watchguard_tcp_offset",
"order": 3,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*offset %{NUMBER:watchguard_tcp_offse:int} %{WORD:watchguard_some_word:string} %{NUMBER:watchguard_tcp_package_size:int} win %{NUMBER:watchguard_tcp_window:int} %{DATA:message}$",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "10.Watchguard_msg",
"order": 4,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*msg=\"%{DATA:watchguard_msg:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "11.Watchguard_proxy_act",
"order": 5,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*proxy_act=\"%{DATA:watchguard_proxy_act:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "12.Watchguard_cats",
"order": 6,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*cats=\"%{DATA:watchguard_cats:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "13.Watchguard_op",
"order": 7,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*op=\"%{DATA:watchguard_op:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "14.Watchguard_dstname",
"order": 8,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*dstname=\"%{DATA:watchguard_dstname:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "15.Watchguard_arg",
"order": 9,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*arg=\"%{DATA:watchguard_arg:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "16.Watchguard_sent_bytes",
"order": 10,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*sent_bytes=\"%{NUMBER:watchguard_sent_bytes:int}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "17.Watchguard_rcvd_bytes",
"order": 11,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*rcvd_bytes=\"%{NUMBER:watchguard_rcvd_bytes:int}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "18.Watchguard_elapsed_time",
"order": 12,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*elapsed_time=\"%{DATA:watchguard_elapsed_time}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "19.Watchguard_app_id",
"order": 13,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*app_id=\"%{NUMBER:watchguard_app_id:int}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "20.Watchguard_app_cat_id",
"order": 14,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*app_cat_id=\"%{NUMBER:watchguard_app_cat_id:int}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "21.Watchguard_app_name",
"order": 15,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*app_name=\"%{DATA:watchguard_app_name:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "22.Watchguard_app_cat_name",
"order": 16,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*app_cat_name=\"%{DATA:watchguard_app_cat_name:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "23.Watchguard_reputation",
"order": 17,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*reputation=\"%{NUMBER:watchguard_reputationreputation:int}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "24.Watchguard_service",
"order": 18,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*service=\"%{DATA:watchguard_service:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "25.Watchguard_src_user",
"order": 19,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*src_user=\"%{DATA:watchguard_src_user:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "26.Watchguard_app_beh_name",
"order": 20,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*app_beh_name=\"%{DATA:watchguard_app_beh_name:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "27.Watchguard_app_ctl_disp",
"order": 21,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*app_ctl_disp=\"%{NUMBER:watchguard_app_ctl_disp:int}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "28.Watchguard_dst_user",
"order": 22,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*dst_user=\"%{DATA:watchguard_dst_user:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "29.Watchguard_sni",
"order": 23,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*sni=\"%{DATA:watchguard_sni:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "30.Watchguard_cn",
"order": 24,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*cn=\"%{DATA:watchguard_cn:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "31.Watchguard_cert_issuer",
"order": 25,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*cert_issuer=\"%{DATA:watchguard_cert_issuer:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "32.Watchguard_cert_subject",
"order": 26,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*cert_subject=\"%{DATA:watchguard_cert_subject:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "33.Watchguard_action",
"order": 27,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*action=\"%{DATA:watchguard_action_action:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "34.Watchguard_rc",
"order": 28,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*rc=\"%{NUMBER:watchguard_return_code:int}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "35.Watchguard_src_ip_nat",
"order": 29,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*src_ip_nat=\"%{IPORHOST:watchguard_src_ip_nat}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "36.Watchguard_tcp_info",
"order": 30,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*tcp_info=\"%{DATA:watchguard_tcp_info:string}\"(.*)?",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "37.Watchguard_acl",
"order": 31,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*\\(%{DATA:watchguard_acl}\\)",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "38.Watchguard_message",
"order": 32,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {
"grok_pattern": ".*\\(%{DATA:message}\\)",
"named_captures_only": true
},
"type": "GROK",
"converters": [],
"target_field": ""
},
{
"source_field": "message",
"title": "0.Raw Message",
"order": 0,
"cursor_strategy": "COPY",
"condition_type": "NONE",
"condition_value": "",
"configuration": {},
"type": "COPY_INPUT",
"converters": [],
"target_field": "Watchguard_raw_input"
}
],
"static_fields": {},
"title": "Firewall",
"global": false,
"configuration": {
"expand_structured_data": false,
"port": 5900,
"store_full_message": false,
"bind_address": "127.0.0.1",
"recv_buffer_size": 262144,
"allow_override_date": true,
"force_rdns": false,
"override_source": null
},
"type": "org.graylog2.inputs.syslog.udp.SyslogUDPInput",
"id": "58e3ac4e300f7203d5eaf703"
}
],
"grok_patterns": [],
"name": "For Joshi",
"outputs": [],
"dashboards": [],
"streams": [],
"description": "...."
}I repeat my question again: What’s the result you’ve expected and what’s the actual result?
Without precise information about what you think is wrong, there’s no way anyone can help you.
Also, please post the configuration of your syslog daemon.
jochen, are you understanding why i put the first 2 prints in the first replay?
I repete, i have a error with graylog, graylog isn’t appling the extrator on my SYSLOG UDP, so the output of the data from that specific source is not indexed. i thing my goal can’t be more cleary than that. RESOLVE THE CURRENT PROBLEM OF NOT INDEXING MENSSAGES !!!
About the syslog-ng config’s sorry but i cant post that because have confidential information.
No. These are just 2 screenshots and you didn’t elaborate on what’s “wrong” with the messages in them.
You could redact the confidential information or at least check that your configuration is valid according to graylog-guide-syslog-linux/README.md at master · Graylog2/graylog-guide-syslog-linux · GitHub.
Allrigth jochen, if you can help it’s ok.
I gest with the title and all information and print i post anyone can detect what was the problem. I dont have time to debating this. This is the 4 time the same problem happen and in previous case i deploy a new instalation of graylog and accept to loose all the data. In this case is the last time my companny will losse to resolve this problem and chosse another solution if the comunity dont support this kind of problems, Dont i my companny that will loose, graylog inc will lose a potencial client.
If you don’t provide the required information, there’s no help. It’s that easy.
I’m keeping out of this topic now. Good luck!
i suggest you use GROK to extract infos in any type of messages, it’s not that hard once you understand the concept . use this site it’s basicly a GROK debuger, : https://grokdebug.herokuapp.com/.
dont start using Ready-extractors, it’s a bad habbit and you dont get all informations sometimes, create your own extractors.
cheers
Anas
Thank you Anas for your feedback. Yes, i use to make the groks extratores bye myself, is safer that way.
About my problem, I’ve allready resolve my problem. It was some miss configuration on plugins configuration. I accidentally turned off one plugin and graylog stoped to use the groks extrators and bypass them.
Bests
Patrick