Archive-Segment

Graylog Central (peer support)
1

I have created a archive and navigate to my graylog server at /var/lib/graylog-server/archives/

When I open up the archive logs only 4 logs and the rest is all repeated:

{"timestamp":"2023-11-23T09:42:22.297Z","type":"message","id":"99ea4501-89e4-11ee-84fe-000c295fc0af","message":"blackbox sfcb-ProviderManager[2104080]: restart_crashed_providers: restarting from crash class SMX_FanRedundancySet, provider: SMXFan, group: smx, location: SMXFanProvider, vendor: ,  last started 1700732542 seconds ago","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":351,"level":4,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":3,"gl2_message_id":"01HFXRY7AGSDWQRMQCGMJGXPJX","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"system daemon"}}
{"timestamp":"2023-11-23T23:11:03.148Z","type":"message","id":"929a67f0-8a55-11ee-84fe-000c295fc0af","message":"blackbox Hostd: warning hostd[2101558] [Originator@6876 sub=VigorStatsProvider(00000041dd92e6a0) opID=esxui-2a14-8ebc] AddVirtualMachine: VM '51' already registered","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":289,"level":4,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":20,"gl2_message_id":"01HFZ76YZF1AT0TW7PDSN0NJGM","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local4"}}
{"timestamp":"2023-11-23T23:11:03.148Z","type":"message","id":"929a67f1-8a55-11ee-84fe-000c295fc0af","message":"blackbox Hostd: warning hostd[2101558] [Originator@6876 sub=VigorStatsProvider(00000041dd92e6a0) opID=esxui-2a14-8ebc] AddVirtualMachine: VM '10' already registered","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":289,"level":4,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":20,"gl2_message_id":"01HFZ76YZFP6EZHSF6HMJ0DF3C","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local4"}}
{"timestamp":"2023-11-23T10:17:28.449Z","type":"message","id":"81496440-89e9-11ee-84fe-000c295fc0af","message":"blackbox healthd: info healthd[2100118] [Originator@6876 sub=PluginLauncher] Peak requested reservation (MB): 40, Peak requested reservation of children groups (MB): 40","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":293,"level":6,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":20,"gl2_message_id":"01HFXTYG448G4Z2SQ8YZBRQCTA","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local4"}}
{"timestamp":"2023-11-23T09:42:22.314Z","type":"message","id":"99ecdd12-89e4-11ee-84fe-000c295fc0af","message":"blackbox sfcb-ProviderManager[2104080]: restart_crashed_providers: restarting from crash class SMX_MPScopedConsolidatedStatusView, provider: SMXMPLite, group: smx, location: SMXMPLiteProvider, vendor: ,  last started 1700732542 seconds ago","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":371,"level":4,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":3,"gl2_message_id":"01HFXRY7B2WZ0HVVJE35995ME5","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"system daemon"}}
{"timestamp":"2023-11-23T23:11:03.148Z","type":"message","id":"929a67f3-8a55-11ee-84fe-000c295fc0af","message":"blackbox Hostd: warning hostd[2101558] [Originator@6876 sub=VigorStatsProvider(00000041dd92e6a0) opID=esxui-2a14-8ebc] AddVirtualMachine: VM '65' already registered","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":289,"level":4,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":20,"gl2_message_id":"01HFZ76YZFRDT6W8EWJ6Q28Q7R","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local4"}}
{"timestamp":"2023-11-23T23:11:03.148Z","type":"message","id":"929a67f4-8a55-11ee-84fe-000c295fc0af","message":"blackbox Hostd: warning hostd[2101558] [Originator@6876 sub=VigorStatsProvider(00000041dd92e6a0) opID=esxui-2a14-8ebc] AddVirtualMachine: VM '36' already registered","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":289,"level":4,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":20,"gl2_message_id":"01HFZ76YZFZX2VS691R5BAC31Z","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local4"}}
{"timestamp":"2023-11-23T10:17:28.449Z","type":"message","id":"81496442-89e9-11ee-84fe-000c295fc0af","message":"blackbox healthd: info healthd[2100118] [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/ssdStorage.py ++group=healthd-plugins,mem=40 -u http://localhost:9996","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":325,"level":6,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":20,"gl2_message_id":"01HFXTYG44CEG0T4DT7W9N46H2","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local4"}}
{"timestamp":"2023-11-23T09:42:22.314Z","type":"message","id":"99ecdd13-89e4-11ee-84fe-000c295fc0af","message":"blackbox sfcb-ProviderManager[2104080]: restart_crashed_providers: restarting from crash class SMX_FCHBAFCPortFCPortStatistics, provider: SMXFCHBA, group: smx, location: SMXFCHBAProvider, vendor: ,  last started 1700732542 seconds ago","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":366,"level":4,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":3,"gl2_message_id":"01HFXRY7B2HXRNJ4VW3X1QQ6ZJ","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"system daemon"}}
{"timestamp":"2023-11-23T23:11:03.148Z","type":"message","id":"929a67f2-8a55-11ee-84fe-000c295fc0af","message":"blackbox Hostd: warning hostd[2101558] [Originator@6876 sub=VigorStatsProvider(00000041dd92e6a0) opID=esxui-2a14-8ebc] AddVirtualMachine: VM '22' already registered","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":289,"level":4,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":20,"gl2_message_id":"01HFZ76YZF1TSXWWA8DSNBQAFP","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local4"}}
{"timestamp":"2023-11-23T23:11:07.386Z","type":"message","id":"952112d0-8a55-11ee-84fe-000c295fc0af","message":"blackbox kmxa: info kmxa[2098049] [Originator@6876 sub=Libs opID=LogState-52d84170-b69f-c4d4-bdb8-d92d8c18f4af-3389] STATE: Total number of tokens: 0","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":274,"level":6,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":20,"gl2_message_id":"01HFZ7733YCVNP6SQ9677BVDEK","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local4"}}
{"timestamp":"2023-11-23T10:17:56.722Z","type":"message","id":"92233431-89e9-11ee-84fe-000c295fc0af","message":"blackbox Hostd: info hostd[2101374] [Originator@6876 sub=Libs opID=60132b3d] NetstackInstanceImpl: congestion control algorithm: newreno","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":261,"level":6,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":20,"gl2_message_id":"01HFXTZBQM9JA2C7S35Y08KKMA","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local4"}}
{"timestamp":"2023-11-23T09:42:22.315Z","type":"message","id":"99ed5240-89e4-11ee-84fe-000c295fc0af","message":"blackbox sfcb-ProviderManager[2104080]: restart_crashed_providers: restarting from crash class SMX_SASEnclosureFirmware, provider: SMXSASHBA, group: smx, location: SMXSASHBAProvider, vendor: ,  last started 1700732542 seconds ago","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":361,"level":4,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":3,"gl2_message_id":"01HFXRY7B4MFY1BAT9WSE61FFG","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"system daemon"}}
{"timestamp":"2023-11-23T23:11:13.690Z","type":"message","id":"98e2fcd0-8a55-11ee-84fe-000c295fc0af","message":"blackbox vmkernel: cpu24:2558196)User: 3220: sfcb-smx: wantCoreDump:sfcb-smx signal:11 exitCode:0 coredump:enabled","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":239,"level":6,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":22,"gl2_message_id":"01HFZ7798XJYFENWYV991QA9N9","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local6"}}
{"timestamp":"2023-11-23T23:11:13.089Z","type":"message","id":"98874840-8a55-11ee-84fe-000c295fc0af","message":"blackbox sfcb-vmware_int[2105407]: Timeout (or other socket error) waiting for response from provider","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":233,"level":3,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":3,"gl2_message_id":"01HFZ778P53WQWPPB3FYJBW5BC","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"system daemon"}}
{"timestamp":"2023-11-23T10:17:58.245Z","type":"message","id":"930b9860-89e9-11ee-84fe-000c295fc0af","message":"blackbox healthd: verbose healthd[2100118] [Originator@6876 sub=HealthdHandler] Temporary report name: /var/lib/vmware/osdata/healthd/.vmw.memoryHealth-1.json","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":283,"level":7,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":20,"gl2_message_id":"01HFXTZD76V7RV6402B19GZCXA","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local4"}}
{"timestamp":"2023-11-23T09:42:22.318Z","type":"message","id":"99ed7951-89e4-11ee-84fe-000c295fc0af","message":"blackbox sfcb-ProviderManager[2104080]: restart_crashed_providers: restarting from crash class SMX_SASStorageEnclosureStorageEnclosureProcessor, provider: SMXSASHBA, group: smx, location: SMXSASHBAProvider, vendor: ,  last started 1700732542 seconds ago","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":385,"level":4,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":3,"gl2_message_id":"01HFXRY7B5KP869K6ZFZXAPWXM","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"system daemon"}}
{"timestamp":"2023-11-23T23:11:13.808Z","type":"message","id":"98f4fe30-8a55-11ee-84fe-000c295fc0af","message":"blackbox vmkernel: cpu24:2558196)UserDump: 2635: sfcb-smx: Dumping cartel 2558191 (from world 2558196) to file /var/core/sfcb-smx-zdump.000 ...","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":268,"level":6,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":22,"gl2_message_id":"01HFZ779CMVPMJKHB5HSW33CZS","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local6"}}
{"timestamp":"2023-11-23T23:11:13.227Z","type":"message","id":"989c2fd0-8a55-11ee-84fe-000c295fc0af","message":"blackbox sfcb-vmware_int[2105407]: Timeout (or other socket error) waiting for response from provider","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":233,"level":3,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":3,"gl2_message_id":"01HFZ778TDW9CRS5D29XSG3BKZ","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"system daemon"}}
{"timestamp":"2023-11-23T10:18:18.256Z","type":"message","id":"9ef90810-89e9-11ee-84fe-000c295fc0af","message":"blackbox Hostd: info hostd[2099741] [Originator@6876 sub=Vimsvc.TaskManager opID=60132b48 user=root] Task Completed : haTask--vim.ServiceInstance.currentTime-3645297727 Status success","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":308,"level":6,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":20,"gl2_message_id":"01HFXV00RH1J17J1RFTXQ8AGT0","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local4"}}
{"timestamp":"2023-11-23T09:42:22.318Z","type":"message","id":"99eda063-89e4-11ee-84fe-000c295fc0af","message":"blackbox sfcb-ProviderManager[2104080]: restart_crashed_providers: restarting from crash class SMX_SAPhysicalPackageArraySystemLocation, provider: SMXSmartArray, group: smx, location: SMXSmartArrayProvider, vendor: ,  last started 1700732542 seconds ago","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":385,"level":4,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":3,"gl2_message_id":"01HFXRY7B766JRY5AT6HETSDEF","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"system daemon"}}
{"timestamp":"2023-11-23T23:11:14.493Z","type":"message","id":"995d8400-8a55-11ee-84fe-000c295fc0af","message":"blackbox vobd:  [UserWorldCorrelator] 2034043423391us: [vob.uw.core.dumped] /bin/sfcbd(2558191) /var/core/sfcb-smx-zdump.000","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":253,"level":6,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":1,"gl2_message_id":"01HFZ77A20CARDQJJQR13S96QK","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"user-level"}}
{"timestamp":"2023-11-23T23:11:13.227Z","type":"message","id":"989c2fd1-8a55-11ee-84fe-000c295fc0af","message":"blackbox sfcb-vmware_int[2105407]: TIMEOUT DOING SHARED SOCKET RECV RESULT (2105407)","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":216,"level":3,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":3,"gl2_message_id":"01HFZ778TD2Y2MD9ZBVG0HAF6T","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"system daemon"}}
{"timestamp":"2023-11-23T10:18:28.103Z","type":"message","id":"a4d7b791-89e9-11ee-84fe-000c295fc0af","message":"blackbox Hostd: info hostd[2099556] [Originator@6876 sub=Libs] SOCKET connect failed, error 2: No such file or directory","full_message":null,"source":"blackbox","streams":["655ecc5dcc5f420ded7f48cd"],"fields":{"gl2_accounted_message_size":245,"level":6,"gl2_remote_ip":"192.168.1.230","gl2_remote_port":44674,"facility_num":20,"gl2_message_id":"01HFXV0ACAVXQP7MZ7AP75677N","gl2_source_node":"c5731138-c034-4aec-a4cd-dbbe48a1b719","gl2_source_input":"6532279cc010810d29282679","facility":"local4"}}

image
image1077×348 21.6 KB

When I double confirm with my graylog search, I have alot more logs coming in.
I have check all my other archive, it show repeated too.

2

The message IDs of those repeating logs are unique, so it looks like your log source is sending all of these.

Are you saying that you can see messages in search, that no longer show up after archiving and restoring the index?

3

Good day @patrickmann

Thank on noticing the message ID.
So sorry about it. Didn’t check on the, message ID before sending in. All the logs are in.

Understanding that when we set up out server we set our timezone, for me will be Singapore. Also understand from the archive logs the time shown T09:42:22.297Z is a UTC timing which is about 8 hours different. How will I be able to change the timing of it.

Just to answer your question too.
Yes, I am still able to see my logs in the search.
The reason is, my index was set with the following configuration:
Rotation period : P1D
Index retention strategy : Delete
Max number of indices: 20

So understanding from the configuration, the older index will be delete after it hit 20 indices.

Do correct me if I’m wrong.
Thanks

4

Here’s a nice review of time zone settings:

5

Good day @patrickmann

Thanks for your help.
You solve my issue.

1 Like
6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.