Rest access log reformatting

hun · February 19, 2018, 12:19pm

Hi Community,

I have added the rest-access-log step to graylog using the following part of the documentation: http://docs.graylog.org/en/2.4/pages/securing.html#configuring-the-access-log

What I am actually interested in is getting the format of the rest-access-log to be like the following: https://gist.github.com/joschi/f1ae18b3a37891374ef9294f88e8c80a

I tried exchanging the log4j2.xml configuration with the one given in this repo: https://gist.github.com/joschi/9c702417f81a8bf7037367d36bb20d4d

which corrupted both my server.log and restaccess.log and no logs are written to both of them anymore.

It would be much appreciated If any one can assist me on how to do the reformatting in such a way that I would see this format
46.139.54.184 [Tue Feb 22 23:25:11 UTC 2017] GET /_js/master.js HTTP/1.1 200 6943 http://www.google.com/url?sa=t&rct=j&q=log%20management&source=web&cd=4 Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1467.0 Safari/537.36

instead of that one:
2018-02-19 14:51:07,558 DEBUG: org.graylog2.rest.accesslog - 127.0.0.1 admin [-] "POST api/cluster/metrics/multiple" Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 200 -1

Cheers,

jochen · February 19, 2018, 12:22pm

So, which version of Graylog are you running and what did you actually add to the Log4j 2 configuration of your Graylog nodes?

hun · February 19, 2018, 12:25pm

I am running graylog 2.3.2

Here’s my log file configuration:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.graylog2.log4j" shutdownHook="disable">
    <Appenders>
        <RollingFile name="rolling-file" fileName="/var/log/graylog-server/server.log" filePattern="/var/log/graylog-server/server.log.%i.gz">
            <PatternLayout pattern="%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5p [%c{1}] %m%n"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="50MB"/>
            </Policies>
            <DefaultRolloverStrategy max="10" fileIndex="min"/>
        </RollingFile>
        <Console name="STDOUT" target="SYSTEM_OUT">
            <PatternLayout pattern="%d %-5p: %c - %m%n"/>
        </Console>
        <!-- Simple appender that writes access log to specified file -->
        <File name="RestAccessLog" fileName="/var/log/graylog/server/restaccess.log" append="true">
            <PatternLayout pattern="%d %-5p: %c - %m%n"/>
        </File>
        <!-- Internal Graylog log appender. Please do not disable. This makes internal log messages available via REST calls. -->
        <Memory name="graylog-internal-logs" bufferSize="500"/>
    </Appenders>
    <Loggers>
        <!-- Application Loggers -->
        <Logger name="org.graylog2" level="info"/>
        <Logger name="com.github.joschi.jadconfig" level="warn"/>
        <!-- This emits a harmless warning for ActiveDirectory every time which we can't work around :( -->
        <Logger name="org.apache.directory.api.ldap.model.message.BindRequestImpl" level="error"/>
        <!-- Prevent DEBUG message about Lucene Expressions not found. -->
        <Logger name="org.elasticsearch.script" level="warn"/>
        <!-- Disable messages from the version check -->
        <Logger name="org.graylog2.periodical.VersionCheckThread" level="off"/>
        <!-- Suppress crazy byte array dump of Drools -->
        <Logger name="org.drools.compiler.kie.builder.impl.KieRepositoryImpl" level="warn"/>
        <!-- Silence chatty natty -->
        <Logger name="com.joestelmach.natty.Parser" level="warn"/>
        <!-- Silence Kafka log chatter -->
        <Logger name="kafka.log.Log" level="warn"/>
        <Logger name="kafka.log.OffsetIndex" level="warn"/>
        <!-- Silence useless session validation messages -->
        <Logger name="org.apache.shiro.session.mgt.AbstractValidatingSessionManager" level="warn"/>
        <!-- RestAccessLogFilter -->
        <Logger name="org.graylog2.rest.accesslog" level="debug" additivity="false">
                <AppenderRef ref="RestAccessLog"/>
        </Logger>
        <Root level="warn">
            <AppenderRef ref="rolling-file"/>
            <AppenderRef ref="STDOUT"/>
            <AppenderRef ref="graylog-internal-logs"/>
        </Root>
    </Loggers>
</Configuration>

jochen · February 19, 2018, 12:27pm

Does the system user running Graylog have sufficient permissions to write into the directory /var/log/graylog/server/ and into the file /var/log/graylog/server/restaccess.log?
You can check this with the following commands:

# namei -l /var/log/graylog/server/
# namei -l /var/log/graylog/server/restaccess.log

hun · February 19, 2018, 12:33pm

I guess so. Anyway here’s the outcome of the two commands:

f: /var/log/graylog/server/
drwxr-xr-x root    root    /
drwxr-xr-x root    root    var
drwxr-xr-x root    syslog  log
drwxrwxr-x graylog graylog graylog
drwxr-xr-x graylog graylog server

f: /var/log/graylog/server/restaccess.log
drwxr-xr-x root    root    /
drwxr-xr-x root    root    var
drwxr-xr-x root    syslog  log
drwxrwxr-x graylog graylog graylog
drwxr-xr-x graylog graylog server
-rw-r--r-- graylog graylog restaccess.log

jochen · February 19, 2018, 2:56pm

And what’s the output of the following commands?

# namei -l /var/log/graylog-server
# namei -l /var/log/graylog-server/server.log

hun · February 19, 2018, 3:13pm

f: /var/log/graylog-server
drwxr-xr-x root    root    /
drwxr-xr-x root    root    var
drwxr-xr-x root    syslog  log
drwxr-xr-x graylog graylog graylog-server

f: /var/log/graylog-server/server.log
drwxr-xr-x root    root    /
drwxr-xr-x root    root    var
drwxr-xr-x root    syslog  log
drwxr-xr-x graylog graylog graylog-server
-rw-r--r-- graylog graylog server.log

jochen · February 19, 2018, 3:17pm

Okay, so the system user graylog (which should also be the one running Graylog on that machine) is able to write into both files.

What’s the problem after using the Log4j 2 configuration file you’ve mentioned?

hun · February 19, 2018, 3:26pm

Using this log4j2 file: https://gist.github.com/joschi/9c702417f81a8bf7037367d36bb20d4d

is giving the following in the server.log file:

2018-02-19T16:20:19.048+01:00 INFO  [Server] SIGNAL received. Shutting down.
2018-02-19T16:20:19.090+01:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2018-02-19T16:20:19.094+01:00 INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2018-02-19T16:20:23.120+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5a0c404a17fa7d3e380b2609> [Syslog UDP].
2018-02-19T16:20:23.128+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5a0c404a17fa7d3e380b2609> closed. Took [6ms]
2018-02-19T16:20:23.129+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.raw.tcp.RawTCPInput.59d7298b17fa7d6c48eea85a> [Raw/Plaintext TCP].
2018-02-19T16:20:23.137+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.raw.tcp.RawTCPInput.59d7298b17fa7d6c48eea85a> closed. Took [4ms]
2018-02-19T16:20:23.138+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.gelf.tcp.GELFTCPInput.59d7298b17fa7d6c48eea862> [GELF TCP].
2018-02-19T16:20:23.142+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.gelf.tcp.GELFTCPInput.59d7298b17fa7d6c48eea862> closed. Took [2ms]
2018-02-19T16:20:23.143+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5a0c434117fa7d3e380b2950> [Syslog UDP].
2018-02-19T16:20:23.146+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5a0c434117fa7d3e380b2950> closed. Took [2ms]
2018-02-19T16:20:23.150+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5a0c434117fa7d3e380b295a> [Syslog UDP].
2018-02-19T16:20:23.153+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5a0c434117fa7d3e380b295a> closed. Took [2ms]
2018-02-19T16:20:23.154+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.gelf.tcp.GELFTCPInput.59d7298b17fa7d6c48eea85f> [GELF TCP].
2018-02-19T16:20:23.157+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.gelf.tcp.GELFTCPInput.59d7298b17fa7d6c48eea85f> closed. Took [2ms]
2018-02-19T16:20:23.158+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog.plugins.internallogs.input.InternalLogsInput.5a7833f317fa7d76911bfc89> [Internal Logs].
2018-02-19T16:20:23.161+01:00 INFO  [InputSetupService] Input <org.graylog.plugins.internallogs.input.InternalLogsInput.5a7833f317fa7d76911bfc89> closed. Took [2ms]
2018-02-19T16:20:23.162+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.gelf.tcp.GELFTCPInput.59d7298b17fa7d6c48eea859> [GELF TCP].
2018-02-19T16:20:23.164+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.gelf.tcp.GELFTCPInput.59d7298b17fa7d6c48eea859> closed. Took [2ms]
2018-02-19T16:20:23.165+01:00 INFO  [InputSetupService] Attempting to close input <org.graylog2.inputs.gelf.http.GELFHttpInput.5a27abed17fa7d55d3991195> [GELF HTTP].
2018-02-19T16:20:23.167+01:00 INFO  [InputSetupService] Input <org.graylog2.inputs.gelf.http.GELFHttpInput.5a27abed17fa7d55d3991195> closed. Took [2ms]
2018-02-19T16:20:23.185+01:00 INFO  [Buffers] Waiting until all buffers are empty.
2018-02-19T16:20:23.187+01:00 INFO  [Buffers] All buffers are empty. Continuing.
2018-02-19T16:20:23.188+01:00 INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2018-02-19T16:20:23.189+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.AlertScannerThread].
2018-02-19T16:20:23.189+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.AlertScannerThread] complete, took <0ms>.
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2018-02-19T16:20:23.190+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>.
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical].
2018-02-19T16:20:23.191+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] complete, took <0ms>.
2018-02-19T16:20:23.192+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical].
2018-02-19T16:20:23.192+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] complete, took <0ms>.
2018-02-19T16:20:23.192+01:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2018-02-19T16:20:23.192+01:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2018-02-19T16:20:23.192+01:00 INFO  [GracefulShutdown] Goodbye.
2018-02-19T16:20:23.193+01:00 INFO  [JerseyService] Shutting down HTTP listener at <http://127.0.0.1:9000/api/>
2018-02-19T16:20:23.204+01:00 INFO  [LookupDataAdapterRefreshService] Stopping 0 jobs
2018-02-19T16:20:23.204+01:00 INFO  [LogManager] Shutting down.
2018-02-19T16:20:23.225+01:00 INFO  [JournalReader] Stopping.
2018-02-19T16:20:23.249+01:00 INFO  [LogManager] Shutdown complete.
2018-02-19T16:20:23.317+01:00 INFO  [NetworkListener] Stopped listener bound to [127.0.0.1:9000]
2018-02-19T16:20:23.318+01:00 INFO  [ServiceManagerListener] Services are now stopped.

and nothing is written anymore to the restaccess.log

jochen · February 19, 2018, 3:44pm

Please use a Log4j 2 configuration file for your setup and not some random log4j2.xml from the Internet.

For example, the log4j2.xml file you’ve mentioned intentionally doesn’t write the normal server logs into a file but only to STDOUT.

hun · February 19, 2018, 3:50pm

You are right, I will stick to the one mentioned in the documentation, but do you have any suggestion on how to get the Rest API ip in the restaccess.log file? Thanks in advance.

jochen · February 19, 2018, 3:54pm

You can find the relevant snippets for the Log4j 2 configuration file at http://docs.graylog.org/en/2.4/pages/securing.html#configuring-the-access-log

Take note that there is intentionally isn’t a complete Log4j 2 configuration file you can copy & paste. You should consciously add the relevant configuration for your specific environment to your Log4j 2 configuration file.

hun · February 19, 2018, 3:58pm

I already added this snippet, but It is missing the ip of the Rest API access log call.

jochen · February 19, 2018, 4:23pm

The format of the REST API access log is fixed and cannot be configured.

github.com

Graylog2/graylog2-server/blob/2.4.3/graylog2-server/src/main/java/org/graylog2/shared/rest/RestAccessLogFilter.java#L60-L69


LOG.debug("{} {} [{}] \"{} {}{}\" {} {} {}",
        remoteAddress,
        userName == null ? "-" : userName,
        (requestDate == null ? "-" : requestDate),
        requestContext.getMethod(),
        requestContext.getUriInfo().getPath(),
        (rawQuery == null ? "" : "?" + rawQuery),
        (userAgent == null ? "-" : userAgent),
        responseContext.getStatus(),
        responseContext.getLength());

hun · February 20, 2018, 9:04am

Here’s a log from my restaccess.log:
2018-02-20 10:01:07,047 DEBUG: org.graylog2.rest.accesslog - 127.0.0.1 at [-] "POST api/cluster/metrics/multiple" Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:57.0) Gecko/20100101 Firefox/57.0 200 -1

As you can see, I am getting the ip of the localhost, but what I am actually interested in is getting the remote address as specified in the source code you provided. So, is there a specific configuration that allows me to get the remote address in the logs as well? Many thanks for your help.

jochen · February 20, 2018, 9:07am

Are you using a proxy server or something comparable in front of Graylog?

hun · February 20, 2018, 9:09am

Yes, I am using nginx as a reverse proxy.

jochen · February 20, 2018, 9:35am

Then you’ll have to add its IP address to the trusted_proxies setting in the Graylog configuration file and let nginx set the X-Forwarded-For HTTP request header.

hun · February 20, 2018, 10:00am

Thanks a lot for your help, now it is working like a charm.

Cheers

system · March 6, 2018, 10:00am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Log rotation of Graylog restaccess.log Graylog Central (peer support)	4	825	February 27, 2021
Graylog Access Logs - I'm lost Graylog Central (peer support)	8	337	February 25, 2023
After create USER using ReST API Login GUI does not work Graylog Central (peer support)	7	549	April 24, 2019
Graylog 2.5 REST API Failing Graylog Central (peer support)	2	406	March 18, 2021
New gray log install but won't start up Graylog Central (peer support)	5	8474	April 26, 2017

Rest access log reformatting

Related Topics