Log not Updating


(Omar Abdo) #1

Hi all,
For some reason my Graylog’s server.log file isn’t updating. I’ve been trying to troubleshoot it but the log file hasn’t been updated in months for some reason. I deleted the old log file and recreated it and tried to restart Graylog’s service. Apparently the restart worked, but still nothing in the log file. Any ideas?

/var/log/graylog-server/server.log is the path to my log.


(Jan Doberstein) #2

did you tried to configure something with the log4j.xml? Like creating API Access Logs or similar?


(Omar Abdo) #3

Hi Jan,
Here’s my XML file. I haven’t edited it except to change the xml version which for some reason was previously “0.0”.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.graylog2.log4j" shutdownHook="disable">
    <Appenders>
        <RollingFile name="rolling-file" fileName="/var/log/upstart/graylog-server.log" filePattern="/var/log/upstart/graylog-server,log.%i.gz">
            <PatternLayout pattern="%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5p [%c{1}] %m%n"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="2GB"/>
            </Policies>
            <DefaultRolloverStrategy max="5" fileIndex="min"/>
        </RollingFile>

        <!-- Internal Graylog log appender. Please do not disable. This makes internal log messages available via REST calls. -->
        <Memory name="graylog-internal-logs" bufferSize="500"/>
    </Appenders>
    <Loggers>
        <!-- Application Loggers -->
        <Logger name="org.graylog2" level="info"/>
        <Logger name="com.github.joschi.jadconfig" level="warn"/>
        <!-- This emits a harmless warning for ActiveDirectory every time which we can't work around :( -->
        <Logger name="org.apache.directory.api.ldap.model.message.BindRequestImpl" level="error"/>
        <!-- Prevent DEBUG message about Lucene Expressions not found. -->
        <Logger name="org.elasticsearch.script" level="warn"/>
        <!-- Disable messages from the version check -->
        <Logger name="org.graylog2.periodical.VersionCheckThread" level="off"/>
        <!-- Suppress crazy byte array dump of Drools -->
        <Logger name="org.drools.compiler.kie.builder.impl.KieRepositoryImpl" level="warn"/>
        <!-- Silence chatty natty -->
        <Logger name="com.joestelmach.natty.Parser" level="warn"/>
        <!-- Silence Kafka log chatter -->
        <Logger name="kafka.log.Log" level="warn"/>
        <Logger name="kafka.log.OffsetIndex" level="warn"/>
        <!-- Silence useless session validation messages -->
        <Logger name="org.apache.shiro.session.mgt.AbstractValidatingSessionManager" level="warn"/>
        <Root level="warn">
            <AppenderRef ref="rolling-file"/>
            <AppenderRef ref="graylog-internal-logs"/>
        </Root>
    </Loggers>
</Configuration>

(Jesse Hills) #4

It appears that your log file is configured to be: /var/log/upstart/graylog-server.log

I would check that location and possibly change it so that the logs are stored where you expect to find them :wink:


(Omar Abdo) #5

Thank you Ponet. So I looked at that log and when I view it in real time using tail, I keep seeing these messages:

2019-01-09 08:10:13,331 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:229)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:890)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:513)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:237)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:249)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:545)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:617)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:634)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:229)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:152)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:122)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
at org.graylog2.bootstrap.CmdLineTool.(CmdLineTool.java:88)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.github.rvesse.airline.parser.ParserUtil.createInstance(ParserUtil.java:39)
at com.github.rvesse.airline.DefaultCommandFactory.createInstance(DefaultCommandFactory.java:25)
at com.github.rvesse.airline.parser.ParserUtil.createInstance(ParserUtil.java:94)
at com.github.rvesse.airline.parser.ParseResult.getCommand(ParseResult.java:114)
at com.github.rvesse.airline.parser.command.CliParser.parse(CliParser.java:54)
at com.github.rvesse.airline.Cli.parse(Cli.java:107)
at com.github.rvesse.airline.Cli.parse(Cli.java:93)
at org.graylog2.bootstrap.Main.main(Main.java:43)

2019-01-09 08:10:13,339 main ERROR Null object returned for RollingFile in Appenders.
2019-01-09 08:10:13,364 main ERROR Unable to locate appender “rolling-file” for logger config "root"

Any ideas?
Thanks again.


(Jesse Hills) #6

How did you install Graylog? What version of Graylog are you running?

I’ve not come across that error while working with Graylog. That error makes it look pretty broken.

If I were you, I would do a fresh install - Is that an option for you?


(Omar Abdo) #7

I installed it a while ago but I kept some documentation about how I did it…
Installed it on a fresh Ubuntu 14.04 LTS.

wget http://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.deb
sudo dpkg –I graylog-2.4-repository_latest.deb
sudo apt-get update
sudo apt-get install graylog-server

sudo service graylog-server start


(Jesse Hills) #8

OK. I’ve only attempted running on Ubuntu 16.04 and Ubuntu 18.04 but, I can’t imagine there will be any issue with that.

Have you also installed Elasticsearch and MongoDB?

Please see: http://docs.graylog.org/en/2.4/pages/installation/os/ubuntu.html


(Jan Doberstein) #9

sorry dude - you fucked up your log4j configuration.

Take the default from github and you should have no problems.


(Omar Abdo) #10

@Ponet Yep, got those both running.
@jan Haha, I appreciate the bluntness. I’m not honestly too familiar though…can I use any log4j ‘default’ config, or is there one specifically for Graylog?


(Jesse Hills) #11

@KuboMD The log4j2.xml file I have that is working in my system is as below.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.graylog2.log4j" shutdownHook="disable">
    <Appenders>
        <RollingFile name="rolling-file" fileName="/var/log/graylog-server/server.log" filePattern="/var/log/graylog-server/server.log.%i.gz">
            <PatternLayout pattern="%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5p [%c{1}] %m%n"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="50MB"/>
            </Policies>
            <DefaultRolloverStrategy max="10" fileIndex="min"/>
        </RollingFile>

        <!-- Internal Graylog log appender. Please do not disable. This makes internal log messages available via REST calls. -->
        <Memory name="graylog-internal-logs" bufferSize="500"/>
    </Appenders>
    <Loggers>
        <!-- Application Loggers -->
        <Logger name="org.graylog2" level="info"/>
        <Logger name="com.github.joschi.jadconfig" level="warn"/>
        <!-- This emits a harmless warning for ActiveDirectory every time which we can't work around :( -->
        <Logger name="org.apache.directory.api.ldap.model.message.BindRequestImpl" level="error"/>
        <!-- Prevent DEBUG message about Lucene Expressions not found. -->
        <Logger name="org.elasticsearch.script" level="warn"/>
        <!-- Disable messages from the version check -->
        <Logger name="org.graylog2.periodical.VersionCheckThread" level="off"/>
        <!-- Suppress crazy byte array dump of Drools -->
        <Logger name="org.drools.compiler.kie.builder.impl.KieRepositoryImpl" level="warn"/>
        <!-- Silence chatty natty -->
        <Logger name="com.joestelmach.natty.Parser" level="warn"/>
        <!-- Silence Kafka log chatter -->
        <Logger name="kafka.log.Log" level="warn"/>
        <Logger name="kafka.log.OffsetIndex" level="warn"/>
        <!-- Silence useless session validation messages -->
        <Logger name="org.apache.shiro.session.mgt.AbstractValidatingSessionManager" level="warn"/>
        <Root level="warn">
            <AppenderRef ref="rolling-file"/>
            <AppenderRef ref="graylog-internal-logs"/>
        </Root>
    </Loggers>
</Configuration>

I checked the above configuration against the one you posted earlier and there were not many differences.

In your configuration, you have SizeBasedTriggeringPolicy size="2GB" whereas, in my configuration, I have SizeBasedTriggeringPolicy size="50MB".
The other major difference I saw was your configuration has DefaultRolloverStrategy max="5" and mine is DefaultRolloverStrategy max="10".

I believe the configuration I pasted above is default as I don’t remember ever making any changes to it. Try changing using the above.


(Omar Abdo) #12

Thanks for providing your config, Jesse. I copy-pasted yours and restarted the three services - but messages are still being logged in real-time to my /var/log/upstart/graylog-server.log. And they say the same error about the missing apache appender.


(Jesse Hills) #13

That is very strange.
Is there now a file at: /var/log/graylog-server/server.log ?

I’m not too sure how or why graylog would still be writing to /var/log/upstart/graylog-server.log if you have changed the configuration and restarted. (Maybe that is some missing knowledge on my end though.)

I would double/triple check the log4j2.xml and make sure it is definitely what you now expect it to be.

Failing that, I would just start fresh with a clean server and install from the docs, provided that is an option.


(Omar Abdo) #14

Thanks again for the reply. I guess I’ll just rebuild from scratch. For the most part Ubuntu LTS 14.04 worked fine for me, but do you recommend another version @jan?


(Jan Doberstein) #15

you should take a distribution that is not end of live april this year - if you want to run this environment longer. Take the LTS 18 release and you are save for some time …


(Omar Abdo) #16

Awesome. Thanks again guys.


(system) closed #17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.