We are using graylog version 2.3.0 & Elasticsearch 5.5.2 in our environment and I was trying to index csv data to elasticsearch by configuring filebeat but the entire data was not indexed to elasticsearch properly some data were missed.
Below is the sample data,
Title,Shortdescription,knowledgecategory,Description DNS Issue Resolution,"Open Start ype command prompt into Start Click Command Prompt. It's at the top of the Start window. This will open Command Prompt Type in ipconfig /flushdns and press Enter Restart your web browser.",Network,
Below is my generated filebeat configuration,
filebeat: prospectors: - document_type: log encoding: plain exclude_lines: - ^Title fields: data: knowledgedata gl2_source_collector: f4749ffd-1f9b-4ef1-b065-a8fc32388fa1 ignore_older: 0 input_type: log paths: - /var/log/knowledgedata/*.csv scan_frequency: 10s tail_files: false output: logstash: hosts: - graylogdemo.cloudapp.azure.com:5044 loadbalance: false path: data: /var/cache/graylog/collector-sidecar/filebeat/data logs: /var/log/graylog/collector-sidecar tags: - linux - apache - knowledgedata
Since I am indexing csv files to elasticsearch so I used the csv output plugin in graylog and configured pipeline rule for my knowledge data below is my pipeline rule for csv,
rule "knowledgedata" when has_field("message") then let csv_fields = "Title,Shortdescription,knowledgecategory,description"; let csv_parsed = csv(csv_fields: csv_fields, csv_text: to_string($message.message), csv_separator: ","); set_fields(csv_parsed); end
After the file done harvested through filebeat I can see in Elasticsearch that knowledge data index has below data only,
“Title”: “DNS Issue Resolution”
"Shortdescription": "\"Open Start"
I couldn’t find any other data indexed to elasticsearch.
Please kindly share any thoughts and let me know what changes I need to make in order to index data to Elasticsearch properly