Graylog can't find documents created by Python app

[lmokarzel (Aesalon)]

Hi all,

I’m currently trying to create new documents in an already existing Graylog index from my own Python application. My application writes directly to ES. The index was created in Graylog. The documents are in the index but they can’t be displayed in searches even if I’m including a timestamp in a format which I believe is supported by Graylog.

Below I’m inserting an example of one of the documents.

{’_index’: ‘virustotal_5’, ‘_type’: ‘message’, ‘_id’: ‘88684822-2250-11ea-a988-021d5f3b286c’, ‘_score’: 1.0, ‘_source’: {’@timestamp’: ‘2019-12-19T11:13:05.347’, ‘sha1’: ‘3f702fd78ca7edf9ecc8aa79b3cf14288837a85b’, ‘sha256’: ‘bf8b4726229b05360bc727411c6bc90fdb20c86bc7029e19b7c3b7edffdaffe2’, ‘md5’: ‘449fcd34b72db953611c3b4de4c93828’, ‘vt_link’: ‘https://www.virustotal.com/file/bf8b4726229b05360bc727411c6bc90fdb20c86bc7029e19b7c3b7edffdaffe2/analysis/1502330110/’, ‘vt_positives’: 16, ‘vt_scan_date’: ‘2017-08-10 01:55:10’, ‘av_products’: {‘AegisLab’: ‘Troj.W32.EraseMBR.tnc9’, ‘Baidu’: ‘Win32.Trojan.WisdomEyes.16070401.9500.9996’, ‘Symantec’: ‘Trojan.Gen.2’, ‘TrendMicro-HouseCall’: ‘Suspicious_GEN.F47V0803’, ‘ClamAV’: ‘Win.Trojan.Agent-36952’, ‘Avast’: ‘Win32:Malware-gen’, ‘DrWeb’: ‘Trojan.KillFiles.10563’, ‘Invincea’: ‘heuristic’, ‘McAfee-GW-Edition’: ‘BehavesLike.Win32.Upatre.xz’, ‘Endgame’: ‘malicious (high confidence)’, ‘McAfee’: ‘Artemis!449FCD34B72D’, ‘WhiteArmor’: ‘Malware.HighConfidence’, ‘Rising’: ‘Malware.Generic.2!tfe (thunder:CkJgeqvmscN)’, ‘SentinelOne’: ‘static engine - malicious’, ‘AVG’: ‘Win32:Malware-gen’, ‘CrowdStrike’: ‘malicious_confidence_100% (W)’}}}

Could you please help me to understand what I should do to make the searches work for these documents?

Thanks

[Ponet (Jesse Hills)]

Why not just have your python app send the logs into an input on Graylog?
That way you can actually utilise Graylog’s features (Streams, Pipelines, etc) to process the messages.

Writing messages directly to ES for reading in Graylog isn’t really a supported method of doing things.

[jan (Jan Doberstein)]

what you try will not work.

when you bypass Graylog the needed meta information are not added to the data. Graylog will only display data that has the needed META information given.