How to load old squid logs to graylog/opensearch index?

Hi all!

Environment: Graylog 4.3.8 + OpenSearch 1.3.6

I already have my cluster Graylog/Opensearch receiving logs, configured streams, dashboard, etc…

I’m sending logs from my Squid Proxy server using this format (configured in squid.conf file):

logformat graylog {"version":"1.1","host":"%{Host}>h","short_message":"%rm %ru HTTP/%rv","timestamp":"%tl","_client_ip":"%>a","_dst_ip":"%<a","_response_time":%tr,"_request_size":%>st,"_reply_size":%<st,"_http_url":"%ru","_http_status":"%>Hs","_http_method":"%rm","_http_referer":"%{Referer}>h","_user_auth":"%[un","_user_agent":"%{User-Agent}>h","_squid_request_status":"%Ss","_squid_hierarchy_status":"%Sh","_from_squid":"true"}
access_log udp://<graylog_server_ip>:12201 graylog

But now I wish to import the old stored squid logs to my “squid_proxy_logs” index using this same format that I’m using to send the real time logs. Is this possible? How can I do this?

If squid is streaming it in real time, you probably can’t get it to read the stored logs. You can use an agent like filebeat or nxlog to grab them, but you would likely need to write a parser to put them in the same fields as the streamed logs.

Is there a reason you need the stored logs to be put into Graylog? Or just curiosity?


Hi, Chris.

I figured the solution would pass through some parser. I just wanted to make sure that I wouldn’t be putting effort into this unnecessarily if there was an easier solution (some API feature maybe).

I just want to be able to import the old logs to have historical data for analysis and to compose the statistics on the dashboard.

Anyway, thanks for the reply!

It’s not wasted effort if you need the historical data, but most people just start from Day 1 and go forward. The effort required to write a parser, though educational if you need to write more of them, might be more work than it is worth in the end. YMMV.

1 Like

One way to do it would be to save old logs in CSV format and then push the CSV to Graylog using powershell as described here :


Actually it’s not a “necessity”, it would just be interesting to import this data if it wouldn’t take too much work time. But it’s okay to “start from day 1” too. Thanks!

Hmm, interesting! It may fit for my case. I’ll try to adapt the solution and come back here to give feedback if it works. :grinning:

Thanks for the tip!

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.