Rest access log reformatting


#1

Hi Community,

I have added the rest-access-log step to graylog using the following part of the documentation: http://docs.graylog.org/en/2.4/pages/securing.html#configuring-the-access-log

What I am actually interested in is getting the format of the rest-access-log to be like the following: https://gist.github.com/joschi/f1ae18b3a37891374ef9294f88e8c80a

I tried exchanging the log4j2.xml configuration with the one given in this repo: https://gist.github.com/joschi/9c702417f81a8bf7037367d36bb20d4d

which corrupted both my server.log and restaccess.log and no logs are written to both of them anymore.

It would be much appreciated If any one can assist me on how to do the reformatting in such a way that I would see this format
46.139.54.184 [Tue Feb 22 23:25:11 UTC 2017] GET /_js/master.js HTTP/1.1 200 6943 http://www.google.com/url?sa=t&rct=j&q=log%20management&source=web&cd=4 Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1467.0 Safari/537.36

instead of that one:
2018-02-19 14:51:07,558 DEBUG: org.graylog2.rest.accesslog - 127.0.0.1 admin [-] "POST api/cluster/metrics/multiple" Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 200 -1

Cheers,


(Jochen) #2

So, which version of Graylog are you running and what did you actually add to the Log4j 2 configuration of your Graylog nodes?


#3

I am running graylog 2.3.2

Here’s my log file configuration:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.graylog2.log4j" shutdownHook="disable">
    <Appenders>
        <RollingFile name="rolling-file" fileName="/var/log/graylog-server/server.log" filePattern="/var/log/graylog-server/server.log.%i.gz">
            <PatternLayout pattern="%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5p [%c{1}] %m%n"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="50MB"/>
            </Policies>
            <DefaultRolloverStrategy max="10" fileIndex="min"/>
        </RollingFile>
        <Console name="STDOUT" target="SYSTEM_OUT">
            <PatternLayout pattern="%d %-5p: %c - %m%n"/>
        </Console>
        <!-- Simple appender that writes access log to specified file -->
        <File name="RestAccessLog" fileName="/var/log/graylog/server/restaccess.log" append="true">
            <PatternLayout pattern="%d %-5p: %c - %m%n"/>
        </File>
        <!-- Internal Graylog log appender. Please do not disable. This makes internal log messages available via REST calls. -->
        <Memory name="graylog-internal-logs" bufferSize="500"/>
    </Appenders>
    <Loggers>
        <!-- Application Loggers -->
        <Logger name="org.graylog2" level="info"/>
        <Logger name="com.github.joschi.jadconfig" level="warn"/>
        <!-- This emits a harmless warning for ActiveDirectory every time which we can't work around :( -->
        <Logger name="org.apache.directory.api.ldap.model.message.BindRequestImpl" level="error"/>
        <!-- Prevent DEBUG message about Lucene Expressions not found. -->
        <Logger name="org.elasticsearch.script" level="warn"/>
        <!-- Disable messages from the version check -->
        <Logger name="org.graylog2.periodical.VersionCheckThread" level="off"/>
        <!-- Suppress crazy byte array dump of Drools -->
        <Logger name="org.drools.compiler.kie.builder.impl.KieRepositoryImpl" level="warn"/>
        <!-- Silence chatty natty -->
        <Logger name="com.joestelmach.natty.Parser" level="warn"/>
        <!-- Silence Kafka log chatter -->
        <Logger name="kafka.log.Log" level="warn"/>
        <Logger name="kafka.log.OffsetIndex" level="warn"/>
        <!-- Silence useless session validation messages -->
        <Logger name="org.apache.shiro.session.mgt.AbstractValidatingSessionManager" level="warn"/>
        <!-- RestAccessLogFilter -->
        <Logger name="org.graylog2.rest.accesslog" level="debug" additivity="false">
                <AppenderRef ref="RestAccessLog"/>
        </Logger>
        <Root level="warn">
            <AppenderRef ref="rolling-file"/>
            <AppenderRef ref="STDOUT"/>
            <AppenderRef ref="graylog-internal-logs"/>
        </Root>
    </Loggers>
</Configuration>

(Jochen) #4

Does the system user running Graylog have sufficient permissions to write into the directory /var/log/graylog/server/ and into the file /var/log/graylog/server/restaccess.log?
You can check this with the following commands:

# namei -l /var/log/graylog/server/
# namei -l /var/log/graylog/server/restaccess.log

#5

I guess so. Anyway here’s the outcome of the two commands:

f: /var/log/graylog/server/
drwxr-xr-x root    root    /
drwxr-xr-x root    root    var
drwxr-xr-x root    syslog  log
drwxrwxr-x graylog graylog graylog
drwxr-xr-x graylog graylog server

f: /var/log/graylog/server/restaccess.log
drwxr-xr-x root    root    /
drwxr-xr-x root    root    var
drwxr-xr-x root    syslog  log
drwxrwxr-x graylog graylog graylog
drwxr-xr-x graylog graylog server
-rw-r--r-- graylog graylog restaccess.log

(Jochen) #6

And what’s the output of the following commands?

# namei -l /var/log/graylog-server
# namei -l /var/log/graylog-server/server.log

#7
f: /var/log/graylog-server
drwxr-xr-x root    root    /
drwxr-xr-x root    root    var
drwxr-xr-x root    syslog  log
drwxr-xr-x graylog graylog graylog-server
f: /var/log/graylog-server/server.log
drwxr-xr-x root    root    /
drwxr-xr-x root    root    var
drwxr-xr-x root    syslog  log
drwxr-xr-x graylog graylog graylog-server
-rw-r--r-- graylog graylog server.log

(Jochen) #8

Okay, so the system user graylog (which should also be the one running Graylog on that machine) is able to write into both files.

What’s the problem after using the Log4j 2 configuration file you’ve mentioned?


#9

Using this log4j2 file: https://gist.github.com/joschi/9c702417f81a8bf7037367d36bb20d4d

is giving the following in the server.log file:

2018-02-19T16:20:19.048+01:00 INFO  [Server] SIGNAL received. Shutting down.
2018-02-19T16:20:19.090+01:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2018-02-19T16:20:19.094+01:00 INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2018-02-19T16:20:23.120+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5a0c404a17fa7d3e380b2609> [Syslog UDP].
2018-02-19T16:20:23.128+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5a0c404a17fa7d3e380b2609> closed. Took [6ms]
2018-02-19T16:20:23.129+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.raw.tcp.RawTCPInput.59d7298b17fa7d6c48eea85a> [Raw/Plaintext TCP].
2018-02-19T16:20:23.137+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.raw.tcp.RawTCPInput.59d7298b17fa7d6c48eea85a> closed. Took [4ms]
2018-02-19T16:20:23.138+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.gelf.tcp.GELFTCPInput.59d7298b17fa7d6c48eea862> [GELF TCP].
2018-02-19T16:20:23.142+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.gelf.tcp.GELFTCPInput.59d7298b17fa7d6c48eea862> closed. Took [2ms]
2018-02-19T16:20:23.143+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5a0c434117fa7d3e380b2950> [Syslog UDP].
2018-02-19T16:20:23.146+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5a0c434117fa7d3e380b2950> closed. Took [2ms]
2018-02-19T16:20:23.150+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5a0c434117fa7d3e380b295a> [Syslog UDP].
2018-02-19T16:20:23.153+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5a0c434117fa7d3e380b295a> closed. Took [2ms]
2018-02-19T16:20:23.154+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.gelf.tcp.GELFTCPInput.59d7298b17fa7d6c48eea85f> [GELF TCP].
2018-02-19T16:20:23.157+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.gelf.tcp.GELFTCPInput.59d7298b17fa7d6c48eea85f> closed. Took [2ms]
2018-02-19T16:20:23.158+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog.plugins.internallogs.input.InternalLogsInput.5a7833f317fa7d76911bfc89> [Internal Logs].
2018-02-19T16:20:23.161+01:00 INFO  [InputSetupService] Input <org.graylog.plugins.internallogs.input.InternalLogsInput.5a7833f317fa7d76911bfc89> closed. Took [2ms]
2018-02-19T16:20:23.162+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.gelf.tcp.GELFTCPInput.59d7298b17fa7d6c48eea859> [GELF TCP].
2018-02-19T16:20:23.164+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.gelf.tcp.GELFTCPInput.59d7298b17fa7d6c48eea859> closed. Took [2ms]
2018-02-19T16:20:23.165+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.gelf.http.GELFHttpInput.5a27abed17fa7d55d3991195> [GELF HTTP].
2018-02-19T16:20:23.167+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.gelf.http.GELFHttpInput.5a27abed17fa7d55d3991195> closed. Took [2ms]
2018-02-19T16:20:23.185+01:00 INFO  [Buffers] Waiting until all buffers are empty.
2018-02-19T16:20:23.187+01:00 INFO  [Buffers] All buffers are empty. Continuing.
2018-02-19T16:20:23.188+01:00 INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2018-02-19T16:20:23.189+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.AlertScannerThread].
2018-02-19T16:20:23.189+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.AlertScannerThread] complete, took <0ms>.
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>.
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] complete, took <0ms>.
2018-02-19T16:20:23.192+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical].
2018-02-19T16:20:23.192+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] complete, took <0ms>.
2018-02-19T16:20:23.192+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2018-02-19T16:20:23.192+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2018-02-19T16:20:23.192+01:00 INFO  [GracefulShutdown] Goodbye.
2018-02-19T16:20:23.193+01:00 INFO  [JerseyService] Shutting down HTTP listener at <http://127.0.0.1:9000/api/>
2018-02-19T16:20:23.204+01:00 INFO  [LookupDataAdapterRefreshService] Stopping 0 jobs
2018-02-19T16:20:23.204+01:00 INFO  [LogManager] Shutting down.
2018-02-19T16:20:23.225+01:00 INFO  [JournalReader] Stopping.
2018-02-19T16:20:23.249+01:00 INFO  [LogManager] Shutdown complete.
2018-02-19T16:20:23.317+01:00 INFO  [NetworkListener] Stopped listener bound to [127.0.0.1:9000]
2018-02-19T16:20:23.318+01:00 INFO  [ServiceManagerListener] Services are now stopped.

and nothing is written anymore to the restaccess.log


(Jochen) #10

Please use a Log4j 2 configuration file for your setup and not some random log4j2.xml from the Internet.

For example, the log4j2.xml file you’ve mentioned intentionally doesn’t write the normal server logs into a file but only to STDOUT.


#11

You are right, I will stick to the one mentioned in the documentation, but do you have any suggestion on how to get the Rest API ip in the restaccess.log file? Thanks in advance.


(Jochen) #12

You can find the relevant snippets for the Log4j 2 configuration file at http://docs.graylog.org/en/2.4/pages/securing.html#configuring-the-access-log

Take note that there is intentionally isn’t a complete Log4j 2 configuration file you can copy & paste. You should consciously add the relevant configuration for your specific environment to your Log4j 2 configuration file.


#13

I already added this snippet, but It is missing the ip of the Rest API access log call.


(Jochen) #14

The format of the REST API access log is fixed and cannot be configured.


#15

Here’s a log from my restaccess.log:
2018-02-20 10:01:07,047 DEBUG: org.graylog2.rest.accesslog - 127.0.0.1 at [-] "POST api/cluster/metrics/multiple" Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:57.0) Gecko/20100101 Firefox/57.0 200 -1

As you can see, I am getting the ip of the localhost, but what I am actually interested in is getting the remote address as specified in the source code you provided. So, is there a specific configuration that allows me to get the remote address in the logs as well? Many thanks for your help.


(Jochen) #16

Are you using a proxy server or something comparable in front of Graylog?


#17

Yes, I am using nginx as a reverse proxy.


(Jochen) #18

Then you’ll have to add its IP address to the trusted_proxies setting in the Graylog configuration file and let nginx set the X-Forwarded-For HTTP request header.


#19

Thanks a lot for your help, now it is working like a charm.

Cheers


(system) #20

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.