How to set graylog to read mai lservers log.
i read mail log on graylog via filebeat.
logs appear as normal log not a mail messages. what should i do/missing?
I have log from mail server and when i using filebeat to read this, it’s a lot of info not showing because of log format (i think) . so my question is how or which type of file input using for log for mail.
Sorry, I don’t see any logs that you posted. You could use \a couple different inputs. This depends on how your mail logs are formatted. The only Input I know of at this point that you could use is Raw/Plaintext. I’m quit sure that it would be useful.
how do i combined this with filebeat?I mounted log file in /data/maillog/
then i need to use filebeat orwhat?
Hello,
If your using FileBeat as a log shipper to send MAIL logs to Graylog Server then you would use BEAT Input and configure it accordantly.
Hope that helps
I For got to add an example of a FileBeat configuration.
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
filebeat.inputs:
- input_type: log
paths:
- /var/log/*.log
- /data/maillog/*.log <-- Example of what you might want.
- /var/log/maillog <-- This configuration is on my server.
type: log
output.logstash:
hosts: ["192.168.1.1:5044"]
path:
data: /var/lib/graylog-sidecar/collectors/filebeat/data
logs: /var/lib/graylog-sidecar/collectors/filebeat/log
Hope that is a better example for you.
I use the filebeat. i know the configuration of filebeat. it’s not the issue.
My issue is when i add it as filebeat, the format is not what i can use it.
there is nothing on messages.
problem is i can’t see messages in graylog. but if i search som text (i know it in messages fx. regards …) then i have output. i mean it’s look like reading log file is not correct.
Hello,
The screen shot above looks like a letter to someone and also the File your sending to Graylog is not properly formatted for elasticsearch.
This might be the issue but I’m just guessing at this point.
If you send a letter text file over to graylog server you may want to use Raw/Plaintext INPUT to see if it works, Not sure how to configure FileBeat to do that .
Next I see the file path.
/var/lib/elasticsearch/log/nic.log
Not sure why you have a mail server in your elasticsearch library directory.
I see the name of the file is nic.log??? It dos not resemble anything for Mail server log. perhaps NIC meaning it a interface log file?? Just a guess.
I mount (read-only) mail log to graylog server and used filebeat (i move one to
filebeat space) . if i use raw/plaintext, how to tell graylog where my log file are?
Hello @Sharzad
Is it possible to show what you did?
I believe using FileBeat you need to use a BEAT input. Unfortunately, I’m not sure how you setup your environment.
please forget about the filebeat and beat input.
how to configure raw/plaintext in graylog, to read a maillog?
What are you using for a log shipper? This would depend on what input you can use.
I nfs mount maillog files.
is it matter? i have a email logfiles and i want to read in to graylog.
If you are using filebeat going to Graylog beats input, you can modify the sidecar configuration to accept multi line messages such as the one you have shown above. There are some Elasticsearch documentation on doing that here.
Reading through your post it is very difficult to answer the question. You haven’t posted much about your environment (text formatted with </> forum tool is much better than a screen shot wherever possible.) the more information you give about your environment, what you have tried etc. the easier it is to help you to a solution. Here are tips on asking questions on the forum that also has some of the general diagnostic commands
1 Like
ok no problem here is my env:
Graylog 4.2.6
MongoDB v4.0.27
Elasticsearch 7.10.2
and elasticsearch, mogodb and graylog installation
dpkg -l | grep -E ".*(elasticsearch|graylog|mongo).*"
ii elasticsearch-oss 7.10.2 amd64 Distributed RESTful search engine built for the cloud
ii graylog-4.2-repository 1-4 all Package to install Graylog 4.2 GPG key and repository
ii graylog-server 4.2.6-1 all Graylog server
ii graylog-sidecar 1.1.0-1 amd64 Graylog collector sidecar
ii mongodb-org 4.0.28 amd64 MongoDB open source document-oriented database system (metapackage)
ii mongodb-org-mongos 4.0.28 amd64 MongoDB sharded cluster query router
ii mongodb-org-server 4.0.28 amd64 MongoDB database server
ii mongodb-org-shell 4.0.28 amd64 MongoDB shell client
ii mongodb-org-tools 4.0.28 amd64 MongoDB tools
my conf file in graylog is:
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = my-pass
root_password_sha2 = mypass
root_timezone = UTC
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = ip:9000
http_bind_address = ipv6:9000
http_publish_uri = http://localhost.org
rotation_strategy = count
elasticsearch_max_docs_per_index = 200000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 1000
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
gc_warning_threshold = 30s
proxied_requests_thread_pool_size = 32
cat /etc/elasticsearch/elasticsearch.yml | egrep -v "^\s*(#|$)"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
cluster.name: graylog
cat /etc/graylog/sidecar/sidecar.yml | egrep -v "^\s*(#|$)"
server_url: "http://myhostname/api/"
server_api_token: "<secret>"
tls_skip_verify: true
curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
"cluster_name" : "graylog",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 68,
"active_shards" : 68,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 16,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 80.95238095238095
curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty
{
"index" : "graylog_3",
"shard" : 1,
"primary" : false,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "CLUSTER_RECOVERED",
"at" : "2022-04-01T11:25:52.012Z",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "pPiimofqSdSMFsgOuQjKiA",
"node_name" : "hostname.domain",
"transport_address" : "127.0.0.1:9300",
"node_decision" : "no",
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "a copy of this shard is already allocated to this node [[graylog_3][1], node[pPiimofqSdMMFagOuQjKiA], [P], s[STARTED], a[id=5qpoh5zUR8aMyzov78J_Bw]]"
}
]
}
]
}
curl -XGET http://localhost:9200/_cat/indices?pretty
green open gl-events_1 vce0mjFtTiC8a1iFkUs6TA 4 0 0 0 832b 832b
green open gl-events_0 -M7yVuuQRw2UL_YvOqHKbQ 4 0 0 0 832b 832b
green open gl-events_5 HKHnyWpfRC2r_d-z0bFu6Q 4 0 66546000 0 11.5gb 11.5gb
green open gl-events_4 BC_0S_SXRwuF9IIGfatUjA 4 0 315045628 0 48gb 48gb
green open gl-events_3 vr4jp5Q3SCWpCHv2FwTJCA 4 0 0 0 832b 832b
green open gl-events_2 Lm4DCmEeSy6SDMAjLND5Mw 4 0 0 0 832b 832b
green open gl-system-events_2 rVN8CxrPRt2Zm0-FSxcXbQ 4 0 0 0 832b 832b
yellow open graylog_1 x2w9kfLWRaCGZXO859L2HQ 4 1 20549492 0 6gb 6gb
green open gl-system-events_3 cHAwlO0gQPG5D1XojgmAnw 4 0 0 0 832b 832b
green open graylog_0 mYBO3Vu5Q-icyJ98ijX2RA 4 0 20707250 0 6.2gb 6.2gb
green open gl-system-events_0 bsivTyMySPCrTQA0dlV2VQ 4 0 0 0 832b 832b
green open gl-system-events_1 Yn0Uo6XfQLSW8mF5H5_YZA 4 0 0 0 832b 832b
yellow open graylog_4 EH_eODqGRAWD4s0e3sQ5tQ 4 1 4941486 0 1.5gb 1.5gb
yellow open graylog_3 K8mu_z9KT1eb2mVblCe6xQ 4 1 20597728 0 5.9gb 5.9gb
green open gl-system-events_4 0w46xyqfRsShPOA0RF9V5Q 4 0 0 0 832b 832b
yellow open graylog_2 2ilDajsjSNyBn7caMjJDVw 4 1 20688957 0 5.3gb 5.3gb
green open gl-system-events_5 w1OpxMXzQt6SGwInLCfrqQ 4 0 0 0 832b 832b
OK - That’s the general data. Although I appreciate it, you didn’t need to follow the entire post… 
What I needed you to glean from that is how to ask you initial question in the future… what is usually needed is targeted information… I should have been more specific. Also - PLEASE use the formatting tool </> when you are posting code by highlighting the code then clicking on the </> tool
as example, I will reformat your last post with it… it makes things MUCH easier to decipher.
It appears you have a filebeat input and you are receiving data from the filebeat sidecar but it is not in the format that you want? If the format you want is multi-line messages like the e-mail example , the elasticsearch link I posted previously is a good start to research how to pull in multi-line messages. You just have to put together a filebeat configuration in Graylog using the elastic file multiline parameters.
Give it a try putting in the parameters from the elasticsearch link, make sure to check the sidecare and filebeat logs on the client, they usually tell you exactly what the issue is (if any) if you run into further issues, post your sidecar configuration (using </> !!!) and comment on things you have tried and describe where your issue is as specifically as possible and with as much relevant information you can.
1 Like
On a side note, you have some issues in your Elasticsearch that you should address:
yellow open graylog_1 x2w9kfLWRaCGZXO859L2HQ 4 1 20549492 0 6gb 6gb
...
yellow open graylog_4 EH_eODqGRAWD4s0e3sQ5tQ 4 1 4941486 0 1.5gb 1.5gb
yellow open graylog_3 K8mu_z9KT1eb2mVblCe6xQ 4 1 20597728 0 5.9gb 5.9gb
...
yellow open graylog_2 2ilDajsjSNyBn7caMjJDVw 4 1 20688957 0 5.3gb 5.3gb
This is showing on viewing allocation:
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
If you google elasticsearch “cannot allocate because allocation is not permitted to any of the nodes” you will find the first link shows that you have replicas that are unallocated… (the light blue area that shows “4 1”… 4 shards, 1 replica) likely because you are running one server for Graylog. There is a command you can run to set replicas to zero -if- you don’t want/need them that I have handy - it looks something like this:
curl -XPUT -H 'Content-Type: application/json' http://<GraylogServer>:9200/<indice_name>/_settings?pretty -d '{"number_of_replicas":0}'
1 Like
yes i have filebeat from one server, format and everything is ok from this input.
what i want is i have maillog from other server when i mount nfs to graylog server.
i don’t want to use filebeat for this input, because format is not match. i want to use
raw/plaintext. how to create it i mean how to say where logs place are?