After adding memory and storage an input not started


(atul) #1

I installed Gray-log 2.3.0 OVA installation on VM-Ware all working fine but yesterday i request server team to increase the memory and space they increase the memory and space. But they increase space just increase LVM of root partition after that one input (configured to receive switch and LB logs as plain/text UDP at 514 port) but now that input not running failed to start. below are the logs…

2017-09-22_07:42:44.91562 INFO  [InputStateListener] Input [Raw/Plaintext UDP/59c4be52896f3a039cddbbc6] is now FAILED%INFO  [InputStateListener] Input [Raw/Plaintext UDP/59c4be52896f3a039cddbbc6] is now STOPPING%INFO  [InputStateListener] Input [Raw/Plaintext UDP/59c4be52896f3a039cddbbc6] is now STOPPED%INFO  [InputStateListener] Input [Raw/Plaintext UDP/59c4be52896f3a039cddbbc6] is now TERMINATED%INFO  [InputStateListener] Input [Raw/Plaintext UDP/59c4be52896f3a039cddbbc6] is now STARTING%ERROR [NettyTransport] Error in Input [Raw/Plaintext UDP/59c4be52896f3a039cddbbc6] (channel [id: 0xa37e180c])% java.net.BindException: Address already in use
2017-09-22_07:42:44.91840       at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_131]
2017-09-22_07:42:44.91878       at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_131]
2017-09-22_07:42:44.92479       at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:691) ~[?:1.8.0_131]
2017-09-22_07:42:44.92548       at sun.nio.ch.DatagramSocketAdaptor.bind(DatagramSocketAdaptor.java:91) ~[?:1.8.0_131]
2017-09-22_07:42:44.92760       at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.bind(NioDatagramPipelineSink.java:129) [graylog.jar:?]
2017-09-22_07:42:44.93107       at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:77) [graylog.jar:?]
2017-09-22_07:42:44.93316       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [graylog.jar:?]
2017-09-22_07:42:44.93638       at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) [graylog.jar:?]
2017-09-22_07:42:44.93773       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) [graylog.jar:?]
2017-09-22_07:42:44.94010       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
2017-09-22_07:42:44.94320       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) [graylog.jar:?]
2017-09-22_07:42:44.94543       at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) [graylog.jar:?]
2017-09-22_07:42:44.94789       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) [graylog.jar:?]
2017-09-22_07:42:44.95014       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
2017-09-22_07:42:44.95229       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) [graylog.jar:?]
2017-09-22_07:42:44.95440       at org.jboss.netty.channel.Channels.bind(Channels.java:561) [graylog.jar:?]
2017-09-22_07:42:44.95729       at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:197) [graylog.jar:?]
2017-09-22_07:42:44.95979       at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198) [graylog.jar:?]
2017-09-22_07:42:44.96295       at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) [graylog.jar:?]
2017-09-22_07:42:44.96509       at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:153) [graylog.jar:?]
2017-09-22_07:42:44.96809       at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
2017-09-22_07:42:44.97056       at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
2017-09-22_07:42:44.97373       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
2017-09-22_07:42:44.97606       at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
2017-09-22_07:42:44.97903       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
2017-09-22_07:42:44.98122       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
2017-09-22_07:42:44.98410       at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2017-09-22_07:42:44.98594 ERROR [InputLauncher] The [org.graylog2.inputs.raw.udp.RawUDPInput] input with ID <59c4be52896f3a039cddbbc6> misfired. Reason: Address already in use.% org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:514
2017-09-22_07:42:44.98946       at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:156) ~[graylog.jar:?]
2017-09-22_07:42:44.99199       at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
2017-09-22_07:42:44.99552       at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
2017-09-22_07:42:44.99840       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
2017-09-22_07:42:45.00148       at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
2017-09-22_07:42:45.00387       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
2017-09-22_07:42:45.00677       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
2017-09-22_07:42:45.00877       at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2017-09-22_07:42:45.01275 Caused by: org.graylog2.plugin.inputs.MisfireException: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:514
2017-09-22_07:42:45.01506       at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:155) ~[graylog.jar:?]
2017-09-22_07:42:45.01770       at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:153) ~[graylog.jar:?]
2017-09-22_07:42:45.01978       ... 7 more
2017-09-22_07:42:45.02257 Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:514
2017-09-22_07:42:45.02421       at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:204) ~[graylog.jar:?]
2017-09-22_07:42:45.02717       at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) ~[graylog.jar:?]
2017-09-22_07:42:45.02879       at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:153) ~[graylog.jar:?]
2017-09-22_07:42:45.03216       ... 7 more
2017-09-22_07:42:45.03450 Caused by: java.net.BindException: Address already in use
2017-09-22_07:42:45.03848       at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_131]
2017-09-22_07:42:45.04071       at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_131]
2017-09-22_07:42:45.04395       at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:691) ~[?:1.8.0_131]
2017-09-22_07:42:45.04583       at sun.nio.ch.DatagramSocketAdaptor.bind(DatagramSocketAdaptor.java:91) ~[?:1.8.0_131]
2017-09-22_07:42:45.04884       at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.bind(NioDatagramPipelineSink.java:129) ~[graylog.jar:?]
2017-09-22_07:42:45.05063       at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:77) ~[graylog.jar:?]
2017-09-22_07:42:45.05365       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) ~[graylog.jar:?]
2017-09-22_07:42:45.05525       at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
2017-09-22_07:42:45.05854       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
2017-09-22_07:42:45.06087       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
2017-09-22_07:42:45.06471       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) ~[graylog.jar:?]
2017-09-22_07:42:45.06652       at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
2017-09-22_07:42:45.06928       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
2017-09-22_07:42:45.07153       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
2017-09-22_07:42:45.07460       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) ~[graylog.jar:?]
2017-09-22_07:42:45.07636       at org.jboss.netty.channel.Channels.bind(Channels.java:561) ~[graylog.jar:?]
2017-09-22_07:42:45.07903       at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:197) ~[graylog.jar:?]
2017-09-22_07:42:45.08085       at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198) ~[graylog.jar:?]
2017-09-22_07:42:45.08361       at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) ~[graylog.jar:?]
2017-09-22_07:42:45.08545       at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:153) ~[graylog.jar:?]
2017-09-22_07:42:45.08855       ... 7 more

root@graylog:~# graylog-ctl tail

==> /var/log/graylog/etcd/current <==
2017-09-21_12:35:09.15649 2017-09-21 12:35:09.155659 I | raft: 8e9e05c52164694d is starting a new election at term 2
2017-09-21_12:35:09.16082 2017-09-21 12:35:09.155850 I | raft: 8e9e05c52164694d became candidate at term 3
2017-09-21_12:35:09.16506 2017-09-21 12:35:09.155920 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 3
2017-09-21_12:35:09.16835 2017-09-21 12:35:09.155966 I | raft: 8e9e05c52164694d became leader at term 3
2017-09-21_12:35:09.17235 2017-09-21 12:35:09.155994 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 3
2017-09-21_12:35:09.17462 2017-09-21 12:35:09.158180 I | etcdserver: published {Name:default ClientURLs:[http://10.248.14.144:4001]} to cluster cdf818194e3a8c32
2017-09-21_12:35:09.18140 2017-09-21 12:35:09.159637 I | embed: ready to serve client requests
2017-09-21_12:35:09.18406 2017-09-21 12:35:09.162060 I | embed: ready to serve client requests
2017-09-21_12:35:09.19239 2017-09-21 12:35:09.165496 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!
2017-09-21_12:35:09.19490 2017-09-21 12:35:09.166482 N | embed: serving insecure client requests on [::]:4001, this is strongly discouraged!

==> /var/log/graylog/etcd/config <==
s209715200
n30
t86400
!gzip



==> /var/log/graylog/etcd/state <==

==> /var/log/graylog/elasticsearch/graylog-2017-09-15.log <==
[2017-09-15T14:55:12,188][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][426120] overhead, spent [310ms] collecting in the last [1s]
[2017-09-15T15:09:45,663][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][426993] overhead, spent [270ms] collecting in the last [1s]
[2017-09-15T15:15:27,874][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][427335] overhead, spent [276ms] collecting in the last [1s]
[2017-09-15T16:38:58,006][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][432343] overhead, spent [358ms] collecting in the last [1s]
[2017-09-15T17:18:57,341][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][434741] overhead, spent [290ms] collecting in the last [1s]
[2017-09-15T18:02:31,696][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][437354] overhead, spent [265ms] collecting in the last [1s]
[2017-09-15T18:08:54,108][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][437736] overhead, spent [292ms] collecting in the last [1.1s]
[2017-09-15T18:11:59,183][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][437921] overhead, spent [260ms] collecting in the last [1s]
[2017-09-15T18:46:11,018][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][439972] overhead, spent [252ms] collecting in the last [1s]
[2017-09-15T22:10:47,202][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][452239] overhead, spent [4.2s] collecting in the last [4.9s]

==> /var/log/graylog/elasticsearch/graylog-2017-09-20.log <==
[2017-09-20T12:35:07,005][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][849453] overhead, spent [6.5s] collecting in the last [7.2s]
[2017-09-20T12:35:12,772][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][young][849455][8488] duration [3.6s], collections [1]/[4.7s], total [3.6s]/[24.8m], memory [1.6gb]->[1.5gb]/[2.2gb], all_pools {[young] [135.4mb]->[2.6mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [1.4gb]->[1.5gb]/[1.9gb]}
[2017-09-20T12:35:12,774][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][849455] overhead, spent [3.9s] collecting in the last [4.7s]
[2017-09-20T12:35:17,525][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][young][849458][8489] duration [2.4s], collections [1]/[2.7s], total [2.4s]/[24.8m], memory [1.7gb]->[1.5gb]/[2.2gb], all_pools {[young] [201.2mb]->[3.3mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [1.5gb]->[1.5gb]/[1.9gb]}
[2017-09-20T12:35:17,527][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][849458] overhead, spent [2.4s] collecting in the last [2.7s]
[2017-09-20T12:35:20,235][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][young][849460][8490] duration [1.3s], collections [1]/[1.7s], total [1.3s]/[24.8m], memory [1.7gb]->[1.6gb]/[2.2gb], all_pools {[young] [188.9mb]->[248.8kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [1.5gb]->[1.6gb]/[1.9gb]}
[2017-09-20T12:35:20,236][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][849460] overhead, spent [1.3s] collecting in the last [1.7s]
[2017-09-20T12:35:25,288][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][young][849462][8491] duration [3.6s], collections [1]/[4s], total [3.6s]/[24.9m], memory [1.8gb]->[1.6gb]/[2.2gb], all_pools {[young] [182.1mb]->[2.1kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [1.6gb]->[1.6gb]/[1.9gb]}
[2017-09-20T12:35:25,289][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][849462] overhead, spent [3.6s] collecting in the last [4s]
[2017-09-20T12:35:39,371][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][849472] overhead, spent [4.7s] collecting in the last [5s]

==> /var/log/graylog/elasticsearch/graylog.log <==
[2017-09-22T06:47:25,172][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][65485] overhead, spent [668ms] collecting in the last [1s]
[2017-09-22T06:47:28,525][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][65488] overhead, spent [403ms] collecting in the last [1.3s]
[2017-09-22T06:47:35,277][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][65494] overhead, spent [917ms] collecting in the last [1.7s]
[2017-09-22T06:47:54,289][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][65513] overhead, spent [447ms] collecting in the last [1s]
[2017-09-22T06:47:58,665][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][65517] overhead, spent [469ms] collecting in the last [1.3s]
[2017-09-22T07:24:37,090][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][67714] overhead, spent [515ms] collecting in the last [1.3s]
[2017-09-22T07:24:41,092][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][67718] overhead, spent [404ms] collecting in the last [1s]
[2017-09-22T07:24:47,851][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][67724] overhead, spent [1.1s] collecting in the last [1.7s]
[2017-09-22T07:25:06,175][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][67742] overhead, spent [545ms] collecting in the last [1.3s]
[2017-09-22T07:25:11,186][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][67747] overhead, spent [492ms] collecting in the last [1s]

==> /var/log/graylog/elasticsearch/graylog-2017-09-14.log <==
[2017-09-14T15:19:49,148][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][341283] overhead, spent [4.3s] collecting in the last [4.7s]
[2017-09-14T15:20:44,346][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][young][341294][3888] duration [1.9s], collections [1]/[2.6s], total [1.9s]/[20.1m], memory [1.4gb]->[1.2gb]/[2.2gb], all_pools {[young] [266.2mb]->[4.3mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [1.1gb]->[1.2gb]/[1.9gb]}
[2017-09-14T15:20:44,371][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][341294] overhead, spent [1.9s] collecting in the last [2.6s]
[2017-09-14T15:22:29,747][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][young][341399][3889] duration [863ms], collections [1]/[1.2s], total [863ms]/[20.1m], memory [1.4gb]->[1.2gb]/[2.2gb], all_pools {[young] [236.9mb]->[550.9kb]/[266.2mb]}{[survivor] [33.2mb]->[30.8mb]/[33.2mb]}{[old] [1.2gb]->[1.2gb]/[1.9gb]}
[2017-09-14T15:22:29,750][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][341399] overhead, spent [863ms] collecting in the last [1.2s]
[2017-09-14T15:25:03,817][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][341553] overhead, spent [384ms] collecting in the last [1s]
[2017-09-14T19:38:13,307][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][356736] overhead, spent [266ms] collecting in the last [1s]
[2017-09-14T20:30:06,076][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][359847] overhead, spent [444ms] collecting in the last [1.3s]
[2017-09-14T20:37:43,264][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][360304] overhead, spent [309ms] collecting in the last [1s]
[2017-09-14T20:37:54,656][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][360311] overhead, spent [5.1s] collecting in the last [5.3s]

==> /var/log/graylog/elasticsearch/current <==
2017-09-22_06:47:25.17399 [WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][65485] overhead, spent [668ms] collecting in the last [1s]
2017-09-22_06:47:28.52568 [INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][65488] overhead, spent [403ms] collecting in the last [1.3s]
2017-09-22_06:47:35.27816 [WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][65494] overhead, spent [917ms] collecting in the last [1.7s]
2017-09-22_06:47:54.29383 [INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][65513] overhead, spent [447ms] collecting in the last [1s]
2017-09-22_06:47:58.66619 [INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][65517] overhead, spent [469ms] collecting in the last [1.3s]
2017-09-22_07:24:37.09135 [INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][67714] overhead, spent [515ms] collecting in the last [1.3s]
2017-09-22_07:24:41.09328 [INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][67718] overhead, spent [404ms] collecting in the last [1s]
2017-09-22_07:24:47.85234 [WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][67724] overhead, spent [1.1s] collecting in the last [1.7s]
2017-09-22_07:25:06.17620 [INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][67742] overhead, spent [545ms] collecting in the last [1.3s]
2017-09-22_07:25:11.18650 [INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][67747] overhead, spent [492ms] collecting in the last [1s]

==> /var/log/graylog/elasticsearch/graylog-2017-09-12.log <==
[2017-09-12T11:59:02,480][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][young][156633][1523] duration [3.5s], collections [1]/[3.9s], total [3.5s]/[16m], memory [1.6gb]->[1.4gb]/[2.2gb], all_pools {[young] [176.7mb]->[1.4mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [1.4gb]->[1.4gb]/[1.9gb]}
[2017-09-12T11:59:02,481][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][156633] overhead, spent [3.5s] collecting in the last [3.9s]
[2017-09-12T11:59:07,590][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][young][156635][1524] duration [3.5s], collections [1]/[4.1s], total [3.5s]/[16m], memory [1.6gb]->[1.5gb]/[2.2gb], all_pools {[young] [143.5mb]->[2.6mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [1.4gb]->[1.5gb]/[1.9gb]}
[2017-09-12T11:59:07,590][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][156635] overhead, spent [3.5s] collecting in the last [4.1s]
[2017-09-12T11:59:11,584][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][young][156638][1525] duration [1.8s], collections [1]/[1.9s], total [1.8s]/[16m], memory [1.8gb]->[1.6gb]/[2.2gb], all_pools {[young] [265.1mb]->[4.9mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [1.5gb]->[1.5gb]/[1.9gb]}
[2017-09-12T11:59:11,584][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][156638] overhead, spent [1.9s] collecting in the last [1.9s]
[2017-09-12T12:01:30,792][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][young][156777][1528] duration [757ms], collections [1]/[1s], total [757ms]/[16.1m], memory [1.4gb]->[1.2gb]/[2.2gb], all_pools {[young] [257.1mb]->[23.7mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [1.1gb]->[1.2gb]/[1.9gb]}
[2017-09-12T12:01:30,814][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][156777] overhead, spent [757ms] collecting in the last [1s]
[2017-09-12T16:48:52,585][INFO ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][174011] overhead, spent [332ms] collecting in the last [1s]
[2017-09-12T16:53:26,434][WARN ][o.e.m.j.JvmGcMonitorService] [F_UHUhn] [gc][174284] overhead, spent [1.1s] collecting in the last [1.7s]

==> /var/log/graylog/elasticsearch/graylog-2017-09-10.log <==
[2017-09-10T10:23:05,950][INFO ][i.n.u.i.PlatformDependent] Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
[2017-09-10T10:23:06,203][INFO ][o.e.t.TransportService   ] [F_UHUhn] publish_address {10.248.14.144:9300}, bound_addresses {10.248.14.144:9300}
[2017-09-10T10:23:06,233][INFO ][o.e.b.BootstrapChecks    ] [F_UHUhn] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-09-10T10:23:16,393][INFO ][o.e.c.s.ClusterService   ] [F_UHUhn] new_master {F_UHUhn}{F_UHUhn5QdCoQ_VQRiVffg}{k5pLwc7BTK2kPWLliq-VFw}{10.248.14.144}{10.248.14.144:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-09-10T10:23:16,496][INFO ][o.e.h.n.Netty4HttpServerTransport] [F_UHUhn] publish_address {10.248.14.144:9200}, bound_addresses {10.248.14.144:9200}
[2017-09-10T10:23:16,497][INFO ][o.e.n.Node               ] [F_UHUhn] started
[2017-09-10T10:23:16,548][INFO ][o.e.g.GatewayService     ] [F_UHUhn] recovered [0] indices into cluster_state
[2017-09-10T10:23:42,724][INFO ][o.e.c.m.MetaDataCreateIndexService] [F_UHUhn] [graylog_0] creating index, cause [api], templates [graylog-internal], shards [4]/[1], mappings [message]
[2017-09-10T10:24:03,192][INFO ][o.e.c.m.MetaDataMappingService] [F_UHUhn] [graylog_0/kp8tsfx9RRijM5oqO6C8BQ] update_mapping [message]
[2017-09-10T10:24:04,916][INFO ][o.e.c.m.MetaDataMappingService] [F_UHUhn] [graylog_0/kp8tsfx9RRijM5oqO6C8BQ] update_mapping [message]

==> /var/log/graylog/elasticsearch/graylog_deprecation.log <==
[2017-09-12T11:05:04,417][WARN ][o.e.d.c.ParseField       ] Deprecated field [include] used, expected [includes] instead
[2017-09-12T16:22:21,996][WARN ][o.e.d.c.ParseField       ] Deprecated field [include] used, expected [includes] instead
[2017-09-13T10:07:30,610][WARN ][o.e.d.c.ParseField       ] Deprecated field [include] used, expected [includes] instead
[2017-09-13T10:08:48,554][WARN ][o.e.d.c.ParseField       ] Deprecated field [include] used, expected [includes] instead
[2017-09-13T10:16:23,569][WARN ][o.e.d.c.ParseField       ] Deprecated field [include] used, expected [includes] instead
[2017-09-13T10:30:03,981][WARN ][o.e.d.c.ParseField       ] Deprecated field [include] used, expected [includes] instead
[2017-09-18T09:29:38,014][WARN ][o.e.d.c.ParseField       ] Deprecated field [include] used, expected [includes] instead
[2017-09-20T11:37:03,230][WARN ][o.e.d.r.a.a.i.RestAnalyzeAction] analyzer request parameter is deprecated and will be removed in the next major release. Please use the JSON in the request body instead request param
[2017-09-20T12:49:25,669][WARN ][o.e.d.r.a.a.i.RestAnalyzeAction] analyzer request parameter is deprecated and will be removed in the next major release. Please use the JSON in the request body instead request param
[2017-09-20T17:19:29,684][WARN ][o.e.d.c.ParseField       ] Deprecated field [include] used, expected [includes] instead

==> /var/log/graylog/elasticsearch/graylog_index_indexing_slowlog.log <==

==> /var/log/graylog/elasticsearch/config <==
s209715200
n30
t86400
!gzip

(Jochen) #2

Make sure that there is only 1 instance of Graylog running, that each input is using its own socker (bind address and port), and that there is no other process binding to port 514 on that system.


(atul) #3

@jochen Thanks for reply problem solved . Problem in syslog configuration. Actually I configured syslog to receive logs and store in different directory which is not working before memory and space increased but after that when server rebooted its working but the input which is configured to accept logs at 514 not working again i configured rsyslog as original now input is working but again not able to collect logs in different directory. can you help me solve this.

Current configuration

root@graylog:~# egrep -v '^#|^$' /etc/rsyslog.conf -v
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support
$KLogPermitNonKernelFacility on
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$RepeatedMsgReduction on
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
$WorkDirectory /var/spool/rsyslog
$IncludeConfig /etc/rsyslog.d/*.conf
*.* @10.248.14.144:5140;RSYSLOG_SyslogProtocol23Format
$template TmplAuth, "/var/log/rsyslog_custom/%FROMHOST-IP%/%PROGRAMNAME%.log"
$template TmplMsg, "/var/log/rsyslog_custom/%FROMHOST-IP%/%PROGRAMNAME%.log"
authpriv.*   ?TmplAuth
*.info,mail.none,authpriv.none,cron.none   ?TmplMsg

Previous configuration

root@graylog:~# egrep -v '^#|^$' /etc/rsyslog.conf-22-sep -v
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support
$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514
$KLogPermitNonKernelFacility on
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$RepeatedMsgReduction on
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
$WorkDirectory /var/spool/rsyslog
$IncludeConfig /etc/rsyslog.d/*.conf
*.* @10.248.14.144:5140;RSYSLOG_SyslogProtocol23Format
$template TmplAuth, "/var/log/rsyslog_custom/%FROMHOST-IP%/%PROGRAMNAME%.log"
$template TmplMsg, "/var/log/rsyslog_custom/%FROMHOST-IP%/%PROGRAMNAME%.log"
authpriv.*   ?TmplAuth
*.info,mail.none,authpriv.none,cron.none   ?TmplMsg

(system) #4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.