Filebeat prevents files from being accessed by other applications

Graylog Central (peer support)
,
1

1. Describe your incident:
After letting filebeat read log files, programs that actually write the logs are not able to access these files.

2. Describe your environment:

  • OS Information:
    Sidecar agent (on Windows) is downloaded and installed from graylog (1.2.0-1), just vanilla installation no custom configs.
  • Package Version:
    Graylog 4.3.3+86369d3 on Ubuntu 20.04
  • Service logs, configurations, and environment variables:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["10.0.0.89:12526"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\filebeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - weblogfiles
filebeat:
  inputs:
    - type: filestream
      id: api_client
      enabled: true
      paths:
        - C:\temp\api-logs\*.txt
        - C:\temp\client-logs\*.txt
        - C:\temp\pdks-logs\*.txt

3. What steps have you already taken to try and solve the problem?
I thought maybe filebeat service is taking exclısive rights to the log files so other applıcations are not able to write to, but that does not hold much water as surely the developers must have thought of this, so I am actually at complete loss, all info I can get from the log writing program is:

2022-10-14 15:03:59.5822 Warn FileTarget(Name=fileLog): Failed to archive file ‘c:\temp\client-logs\2022-10-14.txt’. Exception: System.IO.IOException: The process cannot access the file because it is being used by another process.

as far as I can tell, no processes are accessing the file other than filebeat.

4. How can the community help?
What I would like to know,
is this a common thing?
are there any obvious solutions?
can I control filebeat file access rights?
since it was packaged by Graylog, is this the right place to ask or is it an Elasticsearch matter?

Thank you.

2

these are actually correct in the config file, there is a backslash before the asterisk, the text editor is removing them here.

3

When you post code/logs, use the </> tool for preformatted test to make sure your test is readable and it doesn’t remove pieces. you can edit the post, highlight the config and use the `</> tool in the tool bar.

I noticed with your error it says “Failed to archive file” which says to me that something is trying to move it. What is the application trying to move it? My first guess woudl be the issue is with that application rather than filebeat since there are a lot of people who use filebeat


2 Likes
4

It’s a remote server that I have limited troubleshooting capabilities to, so you’re saying this is not a common occurrence, and filebeat is not to blame, but is it possible to select file access mode say to be read only and also limit access to certain time periods?

5

I have just received word from the server admin saying he found other reasons why file access is impaired, so yeah filebeat is not to blame, however it would be nice if I can have greater control over file access rights and access frequency, I found some config controls but I’m not sure how to use them with Sidecar’s own filebeat implementation.

6

The filebeat commands you want would be put into the Graylog Sidecar Configuration you are applying to that machine - If you posted your configuration and the command you want, I can probably help with where to place it 
 if the online docs aren’t making sense. For file access rights, that’s between you and the server admin.

1 Like
7

Well thank you very much, I am most interested in “Scan Frequency”

my config is very simple, please find it in the question.

8

I took care of this for you
 :expressionless: 
so I could read the post and make sure there weren’t missing information or spacing
 yml files are very finnicky about stuff like that.

My guess is it would look like this
 You will have to test it to be sure.

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["10.0.0.89:12526"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\filebeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - weblogfiles
filebeat:
  inputs:
    - type: filestream
      id: api_client
      enabled: true
      ###################
      # Default is 10 seconds as below.  less than 1s is not recommended.
      scan_frequency: 10s
      ###################
      paths:
        - C:\temp\api-logs\*.txt
        - C:\temp\client-logs\*.txt
        - C:\temp\pdks-logs\*.txt
1 Like
9

Sorry I didn’t think to fix it like you did, you’re awesome man, now I can apply any setting like that, I didn’t know It goes here, if only the documentation were more clear about it, thank you very much.

2 Likes
10

Great!!! Glad it works now! :smiley:

2 Likes
11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.