Where is the ES log located?
Thanks for taking a look at the logs. This is why its so annoying. the web interface and logs all appear correct but all I see if lots coming in and nothing out, nothings in search, no sources. massively frustrating as was all working fine when I left work yesterday.
[2017-03-24 11:04:48,008][INFO ][node ] [Brother Voodoo] stopping ...
[2017-03-24 11:04:48,112][INFO ][node ] [Brother Voodoo] stopped
[2017-03-24 11:04:48,112][INFO ][node ] [Brother Voodoo] closing ...
[2017-03-24 11:04:48,116][INFO ][node ] [Brother Voodoo] closed
[2017-03-24 11:06:22,083][INFO ][node ] [Mister One] version[2.4.4], pid[1377], build[fcbb46d/2017-01-03T11:33:16Z]
[2017-03-24 11:06:22,084][INFO ][node ] [Mister One] initializing ...
[2017-03-24 11:06:22,630][INFO ][plugins ] [Mister One] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-03-24 11:06:22,655][INFO ][env ] [Mister One] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [3.5gb], net total_space [15.7gb], spins? [possibly], types [ext4]
[2017-03-24 11:06:22,655][INFO ][env ] [Mister One] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-03-24 11:06:24,387][INFO ][node ] [Mister One] initialized
[2017-03-24 11:06:24,387][INFO ][node ] [Mister One] starting ...
[2017-03-24 11:06:24,510][INFO ][transport ] [Mister One] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-03-24 11:06:24,514][INFO ][discovery ] [Mister One] graylog/NHICnk5HSuWlSRyMSqDH4g
[2017-03-24 11:06:27,558][INFO ][cluster.service ] [Mister One] new_master {Mister One}{NHICnk5HSuWlSRyMSqDH4g}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-03-24 11:06:27,616][INFO ][http ] [Mister One] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-03-24 11:06:27,616][INFO ][node ] [Mister One] started
[2017-03-24 11:06:27,668][INFO ][gateway ] [Mister One] recovered [1] indices into cluster_state
[2017-03-24 11:06:29,265][INFO ][cluster.routing.allocation] [Mister One] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2017-03-24 11:06:35,596][INFO ][cluster.service ] [Mister One] added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{baJhZwvyTMulolwRAoegHA}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{baJhZwvyTMulolwRAoegHA}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}])
[2017-03-24 11:09:03,017][INFO ][node ] [Mister One] stopping ...
[2017-03-24 11:09:03,161][INFO ][node ] [Mister One] stopped
[2017-03-24 11:09:03,162][INFO ][node ] [Mister One] closing ...
[2017-03-24 11:09:03,168][INFO ][node ] [Mister One] closed
[2017-03-24 11:11:22,518][INFO ][node ] [Sub-Mariner] version[2.4.4], pid[1359], build[fcbb46d/2017-01-03T11:33:16Z]
[2017-03-24 11:11:22,519][INFO ][node ] [Sub-Mariner] initializing ...
[2017-03-24 11:11:23,092][INFO ][plugins ] [Sub-Mariner] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-03-24 11:11:23,124][INFO ][env ] [Sub-Mariner] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [3.5gb], net total_space [15.7gb], spins? [possibly], types [ext4]
[2017-03-24 11:11:23,125][INFO ][env ] [Sub-Mariner] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-03-24 11:11:25,062][INFO ][node ] [Sub-Mariner] initialized
[2017-03-24 11:11:25,062][INFO ][node ] [Sub-Mariner] starting ...
[2017-03-24 11:11:25,177][INFO ][transport ] [Sub-Mariner] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-03-24 11:11:25,182][INFO ][discovery ] [Sub-Mariner] graylog/vyfuTLYLSjqcWCArw0Oj5Q
[2017-03-24 11:11:28,228][INFO ][cluster.service ] [Sub-Mariner] new_master {Sub-Mariner}{vyfuTLYLSjqcWCArw0Oj5Q}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-03-24 11:11:28,452][INFO ][http ] [Sub-Mariner] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-03-24 11:11:28,454][INFO ][node ] [Sub-Mariner] started
[2017-03-24 11:11:28,480][INFO ][gateway ] [Sub-Mariner] recovered [1] indices into cluster_state
[2017-03-24 11:11:30,659][INFO ][cluster.routing.allocation] [Sub-Mariner] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2017-03-24 11:11:39,054][INFO ][cluster.service ] [Sub-Mariner] added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{g0nSr1TcR_WesewyOMWI0g}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{g0nSr1TcR_WesewyOMWI0g}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}])
[2017-03-24 11:14:39,232][INFO ][node ] [Sub-Mariner] stopping ...
[2017-03-24 11:14:39,297][INFO ][node ] [Sub-Mariner] stopped
[2017-03-24 11:14:39,297][INFO ][node ] [Sub-Mariner] closing ...
[2017-03-24 11:14:39,301][INFO ][node ] [Sub-Mariner] closed
[2017-03-24 11:14:39,950][INFO ][node ] [Solomon O'Sullivan] version[2.4.4], pid[3231], build[fcbb46d/2017-01-03T11:33:16Z]
[2017-03-24 11:14:39,951][INFO ][node ] [Solomon O'Sullivan] initializing ...
[2017-03-24 11:14:40,405][INFO ][plugins ] [Solomon O'Sullivan] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-03-24 11:14:40,422][INFO ][env ] [Solomon O'Sullivan] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.5gb], net total_space [49gb], spins? [possibly], types [ext4]
[2017-03-24 11:14:40,422][INFO ][env ] [Solomon O'Sullivan] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-03-24 11:14:41,797][INFO ][node ] [Solomon O'Sullivan] initialized
[2017-03-24 11:14:41,798][INFO ][node ] [Solomon O'Sullivan] starting ...
[2017-03-24 11:14:41,862][INFO ][transport ] [Solomon O'Sullivan] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-03-24 11:14:41,867][INFO ][discovery ] [Solomon O'Sullivan] graylog/nx8T5luRT6uCkCwhEUFAWA
[2017-03-24 11:14:44,921][INFO ][cluster.service ] [Solomon O'Sullivan] new_master {Solomon O'Sullivan}{nx8T5luRT6uCkCwhEUFAWA}{127.0.0.1}{127.0.0.1:9300}, added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{g0nSr1TcR_WesewyOMWI0g}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-03-24 11:14:44,950][INFO ][http ] [Solomon O'Sullivan] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-03-24 11:14:44,950][INFO ][node ] [Solomon O'Sullivan] started
[2017-03-24 11:14:44,989][INFO ][gateway ] [Solomon O'Sullivan] recovered [1] indices into cluster_state
[2017-03-24 11:14:45,765][INFO ][cluster.routing.allocation] [Solomon O'Sullivan] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2017-03-24 11:14:46,169][INFO ][cluster.service ] [Solomon O'Sullivan] removed {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{g0nSr1TcR_WesewyOMWI0g}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-node-left({graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{g0nSr1TcR_WesewyOMWI0g}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}), reason(left)
[2017-03-24 11:14:59,201][INFO ][cluster.service ] [Solomon O'Sullivan] added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{KnkG4hK3QJmQDxArz6OYxw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{KnkG4hK3QJmQDxArz6OYxw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}])
[2017-03-24 11:33:01,559][INFO ][node ] [Solomon O'Sullivan] stopping ...
[2017-03-24 11:33:01,699][INFO ][node ] [Solomon O'Sullivan] stopped
[2017-03-24 11:33:01,699][INFO ][node ] [Solomon O'Sullivan] closing ...
[2017-03-24 11:33:01,707][INFO ][node ] [Solomon O'Sullivan] closed
[2017-03-24 11:35:21,864][INFO ][node ] [Sam Sawyer] version[2.4.4], pid[1408], build[fcbb46d/2017-01-03T11:33:16Z]
[2017-03-24 11:35:21,866][INFO ][node ] [Sam Sawyer] initializing ...
[2017-03-24 11:35:22,392][INFO ][plugins ] [Sam Sawyer] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-03-24 11:35:22,414][INFO ][env ] [Sam Sawyer] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.5gb], net total_space [49gb], spins? [possibly], types [ext4]
[2017-03-24 11:35:22,414][INFO ][env ] [Sam Sawyer] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-03-24 11:35:24,269][INFO ][node ] [Sam Sawyer] initialized
[2017-03-24 11:35:24,270][INFO ][node ] [Sam Sawyer] starting ...
[2017-03-24 11:35:24,370][INFO ][transport ] [Sam Sawyer] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-03-24 11:35:24,378][INFO ][discovery ] [Sam Sawyer] graylog/wqPTeumHTUqNTtRt5Wk_Ig
[2017-03-24 11:35:27,475][INFO ][cluster.service ] [Sam Sawyer] new_master {Sam Sawyer}{wqPTeumHTUqNTtRt5Wk_Ig}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-03-24 11:35:27,517][INFO ][http ] [Sam Sawyer] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-03-24 11:35:27,517][INFO ][node ] [Sam Sawyer] started
[2017-03-24 11:35:27,553][INFO ][gateway ] [Sam Sawyer] recovered [1] indices into cluster_state
[2017-03-24 11:35:29,190][INFO ][cluster.routing.allocation] [Sam Sawyer] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2017-03-24 11:35:35,604][INFO ][cluster.service ] [Sam Sawyer] added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{m3c9m4fXTNeutbOIFROqNw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{m3c9m4fXTNeutbOIFROqNw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}])
[2017-03-24 11:46:08,508][INFO ][cluster.service ] [Sam Sawyer] removed {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{m3c9m4fXTNeutbOIFROqNw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-node-left({graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{m3c9m4fXTNeutbOIFROqNw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}), reason(left)
[2017-03-24 11:46:21,659][INFO ][cluster.service ] [Sam Sawyer] added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{1HzolHpiRoifH1RXrn-aLg}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{1HzolHpiRoifH1RXrn-aLg}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}])
[2017-03-24 11:52:18,072][INFO ][cluster.service ] [Sam Sawyer] removed {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{1HzolHpiRoifH1RXrn-aLg}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-node-left({graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{1HzolHpiRoifH1RXrn-aLg}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}), reason(left)
[2017-03-24 11:52:31,130][INFO ][cluster.service ] [Sam Sawyer] added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{zqmI8SWmRmyb4goCXa-sTg}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{zqmI8SWmRmyb4goCXa-sTg}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}])
[2017-03-24 11:53:50,900][INFO ][node ] [Sam Sawyer] stopping ...
[2017-03-24 11:53:51,014][INFO ][node ] [Sam Sawyer] stopped
[2017-03-24 11:53:51,014][INFO ][node ] [Sam Sawyer] closing ...
[2017-03-24 11:53:51,024][INFO ][node ] [Sam Sawyer] closed
[2017-03-24 11:56:10,777][INFO ][node ] [Ringmaster] version[2.4.4], pid[1384], build[fcbb46d/2017-01-03T11:33:16Z]
[2017-03-24 11:56:10,793][INFO ][node ] [Ringmaster] initializing ...
[2017-03-24 11:56:11,415][INFO ][plugins ] [Ringmaster] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-03-24 11:56:11,444][INFO ][env ] [Ringmaster] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.4gb], net total_space [49gb], spins? [possibly], types [ext4]
[2017-03-24 11:56:11,445][INFO ][env ] [Ringmaster] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-03-24 11:56:13,165][INFO ][node ] [Ringmaster] initialized
[2017-03-24 11:56:13,165][INFO ][node ] [Ringmaster] starting ...
[2017-03-24 11:56:13,257][INFO ][transport ] [Ringmaster] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-03-24 11:56:13,262][INFO ][discovery ] [Ringmaster] graylog/YLH4iiE-Tzy_i6EF7QkY2Q
[2017-03-24 11:56:16,302][INFO ][cluster.service ] [Ringmaster] new_master {Ringmaster}{YLH4iiE-Tzy_i6EF7QkY2Q}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-03-24 11:56:16,322][INFO ][http ] [Ringmaster] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-03-24 11:56:16,322][INFO ][node ] [Ringmaster] started
[2017-03-24 11:56:16,391][INFO ][gateway ] [Ringmaster] recovered [1] indices into cluster_state
[2017-03-24 11:56:18,220][INFO ][cluster.routing.allocation] [Ringmaster] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2017-03-24 11:56:24,339][INFO ][cluster.service ] [Ringmaster] added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{YwhFXpP5RjKzxo_Guea7uA}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{YwhFXpP5RjKzxo_Guea7uA}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}])
[2017-03-24 14:10:40,268][INFO ][node ] [Ringmaster] stopping ...
[2017-03-24 14:10:40,342][INFO ][node ] [Ringmaster] stopped
[2017-03-24 14:10:40,342][INFO ][node ] [Ringmaster] closing ...
[2017-03-24 14:10:40,348][INFO ][node ] [Ringmaster] closed
[2017-03-24 14:10:41,007][INFO ][node ] [Elf With A Gun] version[2.4.4], pid[3650], build[fcbb46d/2017-01-03T11:33:16Z]
[2017-03-24 14:10:41,007][INFO ][node ] [Elf With A Gun] initializing ...
[2017-03-24 14:10:41,476][INFO ][plugins ] [Elf With A Gun] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-03-24 14:10:41,493][INFO ][env ] [Elf With A Gun] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.4gb], net total_space [49gb], spins? [possibly], types [ext4]
[2017-03-24 14:10:41,493][INFO ][env ] [Elf With A Gun] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-03-24 14:10:42,988][INFO ][node ] [Elf With A Gun] initialized
[2017-03-24 14:10:42,989][INFO ][node ] [Elf With A Gun] starting ...
[2017-03-24 14:10:43,038][INFO ][transport ] [Elf With A Gun] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-03-24 14:10:43,043][INFO ][discovery ] [Elf With A Gun] graylog/tjaDweOnQzutgS_iSNXXjw
[2017-03-24 14:10:46,080][INFO ][cluster.service ] [Elf With A Gun] new_master {Elf With A Gun}{tjaDweOnQzutgS_iSNXXjw}{127.0.0.1}{127.0.0.1:9300}, added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{YwhFXpP5RjKzxo_Guea7uA}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-03-24 14:10:46,117][INFO ][http ] [Elf With A Gun] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-03-24 14:10:46,118][INFO ][node ] [Elf With A Gun] started
[2017-03-24 14:10:46,166][INFO ][gateway ] [Elf With A Gun] recovered [1] indices into cluster_state
[2017-03-24 14:10:47,013][INFO ][cluster.routing.allocation] [Elf With A Gun] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2017-03-24 14:11:02,581][INFO ][cluster.service ] [Elf With A Gun] removed {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{YwhFXpP5RjKzxo_Guea7uA}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-node-left({graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{YwhFXpP5RjKzxo_Guea7uA}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}), reason(left)
[2017-03-24 14:11:16,605][INFO ][cluster.service ] [Elf With A Gun] added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{CjM6Il9NRO2bm3mUpUk-Ew}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{CjM6Il9NRO2bm3mUpUk-Ew}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}])
[2017-03-24 14:19:03,668][INFO ][node ] [Elf With A Gun] stopping ...
[2017-03-24 14:19:03,769][INFO ][node ] [Elf With A Gun] stopped
[2017-03-24 14:19:03,769][INFO ][node ] [Elf With A Gun] closing ...
[2017-03-24 14:19:03,773][INFO ][node ] [Elf With A Gun] closed
[2017-03-24 14:21:23,336][INFO ][node ] [Mad Thinker] version[2.4.4], pid[1353], build[fcbb46d/2017-01-03T11:33:16Z]
[2017-03-24 14:21:23,341][INFO ][node ] [Mad Thinker] initializing ...
[2017-03-24 14:21:23,891][INFO ][plugins ] [Mad Thinker] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-03-24 14:21:23,926][INFO ][env ] [Mad Thinker] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.3gb], net total_space [49gb], spins? [possibly], types [ext4]
[2017-03-24 14:21:23,927][INFO ][env ] [Mad Thinker] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-03-24 14:21:25,796][INFO ][node ] [Mad Thinker] initialized
[2017-03-24 14:21:25,798][INFO ][node ] [Mad Thinker] starting ...
[2017-03-24 14:21:25,882][INFO ][transport ] [Mad Thinker] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-03-24 14:21:25,887][INFO ][discovery ] [Mad Thinker] graylog/apJpVEK5SQ-EU4E-GivJog
[2017-03-24 14:21:28,925][INFO ][cluster.service ] [Mad Thinker] new_master {Mad Thinker}{apJpVEK5SQ-EU4E-GivJog}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-03-24 14:21:29,018][INFO ][http ] [Mad Thinker] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-03-24 14:21:29,019][INFO ][node ] [Mad Thinker] started
[2017-03-24 14:21:29,022][INFO ][gateway ] [Mad Thinker] recovered [1] indices into cluster_state
[2017-03-24 14:21:30,809][INFO ][cluster.routing.allocation] [Mad Thinker] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2017-03-24 14:21:36,837][INFO ][cluster.service ] [Mad Thinker] added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{TmGmBC8jSbO0ix1FrrCJmw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{TmGmBC8jSbO0ix1FrrCJmw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}])
[2017-03-24 14:27:08,518][INFO ][cluster.service ] [Mad Thinker] removed {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{TmGmBC8jSbO0ix1FrrCJmw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-node-left({graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{TmGmBC8jSbO0ix1FrrCJmw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}), reason(left)
[2017-03-24 14:27:34,792][INFO ][node ] [Mad Thinker] stopping ...
[2017-03-24 14:27:34,820][INFO ][node ] [Mad Thinker] stopped
[2017-03-24 14:27:34,821][INFO ][node ] [Mad Thinker] closing ...
[2017-03-24 14:27:34,824][INFO ][node ] [Mad Thinker] closed
[2017-03-24 14:28:35,205][INFO ][node ] [Lancer] version[2.4.4], pid[3400], build[fcbb46d/2017-01-03T11:33:16Z]
[2017-03-24 14:28:35,206][INFO ][node ] [Lancer] initializing ...
[2017-03-24 14:28:35,635][INFO ][plugins ] [Lancer] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-03-24 14:28:35,651][INFO ][env ] [Lancer] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.4gb], net total_space [49gb], spins? [possibly], types [ext4]
[2017-03-24 14:28:35,652][INFO ][env ] [Lancer] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-03-24 14:28:37,052][INFO ][node ] [Lancer] initialized
[2017-03-24 14:28:37,052][INFO ][node ] [Lancer] starting ...
[2017-03-24 14:28:37,118][INFO ][transport ] [Lancer] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-03-24 14:28:37,122][INFO ][discovery ] [Lancer] graylog/RjWaKdXcTzyAEkzDRGZONA
[2017-03-24 14:28:40,147][INFO ][cluster.service ] [Lancer] new_master {Lancer}{RjWaKdXcTzyAEkzDRGZONA}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-03-24 14:28:40,178][INFO ][http ] [Lancer] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-03-24 14:28:40,178][INFO ][node ] [Lancer] started
[2017-03-24 14:28:40,191][INFO ][gateway ] [Lancer] recovered [1] indices into cluster_state
[2017-03-24 14:28:40,893][INFO ][cluster.routing.allocation] [Lancer] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2017-03-24 14:28:50,144][INFO ][cluster.service ] [Lancer] added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{gOgEooV6RAe28SwCCGUVPw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{gOgEooV6RAe28SwCCGUVPw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}])
[2017-03-24 14:32:58,390][INFO ][node ] [Lancer] stopping ...
[2017-03-24 14:32:58,484][INFO ][node ] [Lancer] stopped
[2017-03-24 14:32:58,484][INFO ][node ] [Lancer] closing ...
[2017-03-24 14:32:58,488][INFO ][node ] [Lancer] closed
[2017-03-24 14:32:59,099][INFO ][node ] [Franklin Hall] version[2.4.4], pid[5843], build[fcbb46d/2017-01-03T11:33:16Z]
[2017-03-24 14:32:59,099][INFO ][node ] [Franklin Hall] initializing ...
[2017-03-24 14:32:59,576][INFO ][plugins ] [Franklin Hall] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-03-24 14:32:59,593][INFO ][env ] [Franklin Hall] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.4gb], net total_space [49gb], spins? [possibly], types [ext4]
[2017-03-24 14:32:59,593][INFO ][env ] [Franklin Hall] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-03-24 14:33:01,069][INFO ][node ] [Franklin Hall] initialized
[2017-03-24 14:33:01,069][INFO ][node ] [Franklin Hall] starting ...
[2017-03-24 14:33:01,123][INFO ][transport ] [Franklin Hall] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-03-24 14:33:01,127][INFO ][discovery ] [Franklin Hall] graylog/N12A7E_LQ_GJRgGD1f6Zfw
[2017-03-24 14:33:04,170][INFO ][cluster.service ] [Franklin Hall] new_master {Franklin Hall}{N12A7E_LQ_GJRgGD1f6Zfw}{127.0.0.1}{127.0.0.1:9300}, added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{gOgEooV6RAe28SwCCGUVPw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-03-24 14:33:04,211][INFO ][http ] [Franklin Hall] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-03-24 14:33:04,212][INFO ][node ] [Franklin Hall] started
[2017-03-24 14:33:04,245][INFO ][gateway ] [Franklin Hall] recovered [1] indices into cluster_state
[2017-03-24 14:33:04,966][INFO ][cluster.routing.allocation] [Franklin Hall] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2017-03-24 14:33:12,976][INFO ][cluster.service ] [Franklin Hall] removed {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{gOgEooV6RAe28SwCCGUVPw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-node-left({graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{gOgEooV6RAe28SwCCGUVPw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}), reason(left)
[2017-03-24 14:33:26,244][INFO ][cluster.service ] [Franklin Hall] added {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{wZkTp1vwTx2C1T2b__ez6g}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{wZkTp1vwTx2C1T2b__ez6g}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false}])Above is the ES log, had to cut some off the top due to character limits in posts here.
And I love the meme, very true. Luckily this is a thing i am running solo to test at the moment. Was hoping to roll out to business but not if this issue cant be identified. Need things to be stable and maintainable.
Again, please use
```
< Log here >
```
Well, Graylog is actually really stable. I never had any problems that werenāt caused by myself derping up 
Sorry, thought I had 
Well, Iām out of ideas and gtg, Iām already late, sorry. ^^
no need to apologise, you have been great helping out.
If anyone else can help out it would be much appreciated.
Thanks,especially @derPhlipsi
To me it seems your ES log is from a different time than the Graylog log. Perhaps the log from the time when ES stopped working the first time would shed some light here.
It might happen when your system was out of space that the journal is broken.
Did you notice something in your current graylog log that is saying that Elasticsearch is not available?
Additional you might want to stop Graylog move the journal to another location and restart Graylog.
But be aware that your data in the journal then might not be recoverable.
To get the right locations we have that covered in the documentation in the configuration section called ādefault file locationā.
The journal is picky when you are out of space and become a grave for all messages that are in.
Their is a way to recover that but that is not an easy task at all and is comparable to recovery of files from a broken disk.
I have arrived at work this morning after the weekend to find a fully working Graylog system. I have no idea what needed that time to sort itself but my next stop now is working out how to stop the logs filling the disk.
I know I want to delete all informational windows logs after a set time, probably a number of weeks. Any tips? I have been suggested indexes but that looks very complex, guide?
Thanks for all the help on this thread.
Hej,
just create an index set for your windows logs, look here in the docs is that explained and here how to configure the data retention.
That is a very basic and easy part to keep your disk filling up.