Filebeat multiline option

AmrAbdelFattah · July 9, 2019, 10:46am

I want to enable multi line option in filebeat.yml configuration for below log , i want to receive below response in one message in graylog ,any advise ?

2019-07-02 16:00:00.058 [SUBSCRIBER_PROFILE-1157917705-d73442b7-8d07-4aee-a850-09aa51ff37e2] Response [<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<subscriberProfileResponse>
    <errorCode>00000</errorCode>
    <errorMessage>SUCCESS</errorMessage>
    <responseStatus>SUCCESS</responseStatus>
    <customerDetails>
        <attributeList>
            <name>PERSONAL_BALANCE</name>
            <value>0.0</value>
        </attributeList>
        <paymentType>Prepaid</paymentType>
        <productDetailsList>
            <attributeList>
                <value>N/A</value>
            </attributeList>
            <attributeList>
                <arabicDescription>واتساب</arabicDescription>
                <arabicName>واتساب</arabicName>
                <arabicUnit>ميجا</arabicUnit>
                <bundleResourceID>10000125</bundleResourceID>
                <englishDescription>WhatsApp</englishDescription>
                <englishName>WhatsApp</englishName>
                <name>HEKAYA_REVAMP_WhatsApp_BUNDLE_DATA_CONSUMED_USAGE</name>
                <order>2</order>
                <type>METER</type>
                <unit>MB</unit>
                <value>2.73</value>
            </attributeList>
            <attributeList>
                <arabicDescription>رصيدك الحالي</arabicDescription>
                <arabicName>رصيدك الحالي</arabicName>
                <arabicUnit>دقائق</arabicUnit>
                <bundleResourceID>1000079</bundleResourceID>
                <englishDescription>My Balance</englishDescription>
                <englishName>My Balance</englishName>
                <name>PERSONAL_BALANCE</name>
                <type>METER</type>
                <unit>Mins</unit>
                <value>0.0</value>
            </attributeList>
            <attributeList>
                <arabicDescription>باقة ميكس</arabicDescription>
                <arabicName>باقة ميكس</arabicName>
                <arabicUnit>وحدات</arabicUnit>
                <bundleResourceID>10000124</bundleResourceID>
                <englishDescription>Mix Bundle</englishDescription>
                <englishName>Mix Bundle</englishName>
                <name>INQUIRY_GENERAL_UNIT_TOTAL_QUOTA</name>
                <order>1</order>
                <type>METER</type>
                <unit>Units</unit>
                <value>212</value>
            </attributeList>
            <attributeList>
                <name>INQUIRY_DATA_CONSUMED_PERCENTAGE</name>
                <value>N/A</value>
            </attributeList>
            <attributeList>
                <name>INQUIRY_RECURRING_DATE</name>
                <value>N/A</value>
            </attributeList>
            <attributeList>
                <name>PRODUCT_GROUP_TYPE</name>
                <value>Hybrids</value>
            </attributeList>
            <attributeList>
                <name>INQUIRY_QUOTA_MEASUREMENT_UNIT</name>
                <value>N/A</value>
            </attributeList>
            <attributeList>
                <name>PRODUCT_IS_SUSPENDED</name>
                <value>true</value>
            </attributeList>
            <attributeList>
                <arabicDescription>واتساب</arabicDescription>
                <arabicName>واتساب</arabicName>
                <arabicUnit>ميجا</arabicUnit>
                <bundleResourceID>10000125</bundleResourceID>
                <englishDescription>WhatsApp</englishDescription>
                <englishName>WhatsApp</englishName>
                <name>HEKAYA_REVAMP_WhatsApp_BUNDLE_DATA_REMAINING_QUOTA</name>
                <order>2</order>
                <type>METER</type>
                <unit>MB</unit>
                <value>47.27</value>
            </attributeList>
            <attributeList>
                <name>IS_TEMP_PRODUCT</name>
                <value>YES</value>
            </attributeList>
            <attributeList>
                <name>INQUIRY_GENERAL_UNIT_CONSUMED_PERCENTAGE</name>
                <value>0.8780392156862745</value>
            </attributeList>
            <attributeList>
                <name>INQUIRY_NEXT_RENEWAL_DATE_TIME</name>
                <value>PASSED</value>
            </attributeList>
            <attributeList>
                <arabicDescription>موبايل انترنت</arabicDescription>
                <arabicName>موبايل انترنت</arabicName>
                <arabicUnit>ميجا</arabicUnit>
                <bundleResourceID>10000102</bundleResourceID>
                <englishDescription>Mobile Internet</englishDescription>
                <englishName>Mobile Internet</englishName>
                <name>INQUIRY_DATA_TOTAL_QUOTA</name>
                <type>METER</type>
                <unit>MB</unit>
                <value>N/A</value>
            </attributeList>
            <attributeList>
                <arabicDescription>واتساب</arabicDescription>
                <arabicName>واتساب</arabicName>
                <arabicUnit>ميجا</arabicUnit>
                <bundleResourceID>10000125</bundleResourceID>
                <englishDescription>WhatsApp</englishDescription>
                <englishName>WhatsApp</englishName>
                <name>HEKAYA_REVAMP_WhatsApp_BUNDLE_DATA_TOTAL_QUOTA</name>
                <order>2</order>
                <type>METER</type>
                <unit>MB</unit>
                <value>50</value>
            </attributeList>
            <attributeList>
                <arabicDescription>باقة ميكس</arabicDescription>
                <arabicName>باقة ميكس</arabicName>
                <arabicUnit>وحدات</arabicUnit>
                <bundleResourceID>10000124</bundleResourceID>
                <englishDescription>Mix Bundle</englishDescription>
                <englishName>Mix Bundle</englishName>
                <name>INQUIRY_GENERAL_UNIT_CONSUMED_USAGE</name>
                <order>1</order>
                <type>METER</type>
                <unit>Units</unit>
                <value>186</value>
            </attributeList>
            <attributeList>
                <name>TEMP_NUMBER_OF_DAYS</name>
                <value>65</value>
            </attributeList>
            <attributeList>
                <arabicDescription>صالحه حتى</arabicDescription>
                <arabicName>صالحه حتى</arabicName>
                <bundleResourceID>1000078</bundleResourceID>
                <englishDescription>Validity</englishDescription>
                <englishName>Validity</englishName>
                <name>INQUIRY_NEXT_RENEWAL_DATE</name>
                <type>METER</type>
                <value>PASSED</value>
            </attributeList>
            <attributeList>
                <name>INQUIRY_FAF_LIST</name>
                <value>N/A</value>
            </attributeList>
            <attributeList>
                <arabicDescription>باقة ميكس</arabicDescription>
                <arabicName>باقة ميكس</arabicName>
                <arabicUnit>وحدات</arabicUnit>
                <bundleResourceID>10000124</bundleResourceID>
                <englishDescription>Mix Bundle</englishDescription>
                <englishName>Mix Bundle</englishName>
                <name>INQUIRY_GENERAL_UNIT_REMAINING_QUOTA</name>
                <order>1</order>
                <type>METER</type>
                <unit>Units</unit>
                <value>25</value>
            </attributeList>
            <attributeList>
                <name>HEKAYA_REVAMP_WhatsApp_BUNDLE_DATA_CONSUMED_PERCENTAGE</name>
                <value>0.0546</value>
            </attributeList>
            <attributeList>
                <arabicDescription>موبايل انترنت</arabicDescription>
                <arabicName>موبايل انترنت</arabicName>
                <arabicUnit>ميجا</arabicUnit>
                <bundleResourceID>10000101</bundleResourceID>
                <englishDescription>Mobile Internet</englishDescription>
                <englishName>Mobile Internet</englishName>
                <name>INQUIRY_DATA_CONSUMED_USAGE</name>
                <type>METER</type>
                <unit>MB</unit>
                <value>N/A</value>
            </attributeList>
            <attributeList>
                <arabicDescription>المتبقى</arabicDescription>
                <arabicName>المتبقى</arabicName>
                <arabicUnit>ميجا</arabicUnit>
                <bundleResourceID>1000027</bundleResourceID>
                <englishDescription>Remaining</englishDescription>
                <englishName>Remaining</englishName>
                <name>INQUIRY_DATA_REMAINING_QUOTA</name>
                <order>2</order>
                <type>METER</type>
                <unit>MB</unit>
                <value>N/A</value>
            </attributeList>
            <category>
                <categoryDescription>Susbended Hybird Rateplan</categoryDescription>
                <categoryName>HYBIRD_RATEPLAN_AND_ADDON</categoryName>
            </category>
            <category>
                <categoryDescription>IVR_RATEPLANS</categoryDescription>
                <categoryName>IVR_RATEPLANS</categoryName>
            </category>
            <category>
                <categoryDescription>WEB DASHBOARD</categoryDescription>
                <categoryName>PORTAL_RATEPLAN</categoryName>
            </category>
            <category>
                <categoryDescription>IPIPE_RATEPLAN</categoryDescription>
                <categoryName>IPIPE_RATEPLAN</categoryName>
            </category>
            <category>
                <categoryDescription>ICON_HYBRIDS</categoryDescription>
                <categoryName>ICON_HYBRIDS</categoryName>
            </category>
            <category>
                <categoryDescription>RTIM_RATEPLAN</categoryDescription>
                <categoryName>RTIM_RATEPLAN</categoryName>
            </category>
            <fees>24.5</fees>
            <mainBundle>Y</mainBundle>
            <overridenAttribute/>
            <productArabicName>حكاية ميكس 35</productArabicName>
            <productEnglishName>New Hekaya Mix 35</productEnglishName>
            <productName>Vulcan_Revamp_2</productName>
            <productOperations>
                <operationName>SET_LIMIT</operationName>
            </productOperations>
            <productOperations>
                <operationName>RESET_MI_CAPPING</operationName>
            </productOperations>
            <productOperations>
                <operationName>RESET_CROSSNET_CAPPING</operationName>
            </productOperations>
            <productOperations>
                <operationName>INTERNATIONAL_SET_LIMIT</operationName>
                <operationType>DUMMY</operationType>
            </productOperations>
            <productOperations>
                <operationName>SET_INTERNATIONAL_LIMIT</operationName>
            </productOperations>
            <productOperations>
                <operationName>RESET_INTERNATIONAL_LIMIT</operationName>
            </productOperations>
            <productOperations>
                <operationName>RESET_WHATSAPP_BUNDLE</operationName>
            </productOperations>
            <productOperations>
                <operationName>ACTIVATE</operationName>
            </productOperations>
            <productOperations>
                <arabicDescription>تجديد</arabicDescription>
                <arabicName>تجديد</arabicName>
                <bundleResourceID>1000203</bundleResourceID>
                <englishDescription>RENEW</englishDescription>
                <englishName>RENEW</englishName>
                <operationName>RENEW</operationName>
            </productOperations>
            <productOperations>
                <arabicDescription>خدمة الرقم المفضل</arabicDescription>
                <arabicName>خدمة الرقم المفضل</arabicName>
                <bundleResourceID>10008535</bundleResourceID>
                <englishDescription>By subscribing to the favorite number service you will be granted minutes dedicated to any Etisalat number you choose</englishD
escription>
                <englishName>Favorite Number Service</englishName>
                <operationName>HEKAYA_FAF_SERVICE</operationName>
                <operationType>DUMMY</operationType>
            </productOperations>
            <productOperations>
                <arabicDescription>تحويل</arabicDescription>
                <arabicName>تحويل</arabicName>
                <bundleResourceID>10008534</bundleResourceID>
                <englishDescription>You can transfer units to any Etisalat number you choose</englishDescription>
                <englishName>Transfer</englishName>
                <operationName>HEKAYA_TRANSFER_SERVICE</operationName>
                <operationType>DUMMY</operationType>
            </productOperations>
            <productOperations>
                <arabicDescription>تجديد نص شهر</arabicDescription>
                <arabicName>تجديد نص شهر</arabicName>
                <bundleResourceID>10008533</bundleResourceID>
                <englishDescription>You can still renew your bundle even if you don't have the full bundle fees by renewing the bundle for half a month with half t
he bundle fees</englishDescription>
                <englishName>Half Month Renewal</englishName>
                <operationName>HEKAYA_HALF_MONTH_RENEW</operationName>
                <paramList>
                    <isMandatory>N</isMandatory>
                    <paramName>HALF_MONTH_RENWAL_FEES</paramName>
                    <paramType>String</paramType>
                    <valueList>12.6</valueList>
                </paramList>
            </productOperations>
            <productOperations>
                <arabicDescription>شراء باقات اضافية</arabicDescription>
                <arabicName>شراء باقات اضافية</arabicName>
                <bundleResourceID>10008536</bundleResourceID>
                <englishDescription>If your bundle is not enough, you can choose an extra add-on bundle that suits your needs</englishDescription>
                <englishName>Purchase Add-ons</englishName>
                <operationName>VULK_BUY_ADDON</operationName>
                <operationType>DUMMY</operationType>
            </productOperations>
            <productOperations>
                <arabicDescription>تحديد استخدام الميكسات</arabicDescription>
                <arabicName>تحديد استخدام الميكسات</arabicName>
                <bundleResourceID>10008538</bundleResourceID>
                <englishDescription>Set limit to your usage from the Mix for the "Minutes to other networks" or Internet</englishDescription>
                <englishName>Set Limit</englishName>
                <operationName>SET_LIMIT_DUMMY</operationName>
                <operationType>DUMMY</operationType>
            </productOperations>
            <productOperations>
                <arabicDescription>خدمة السوبر نت</arabicDescription>
                <arabicName>خدمة السوبر نت</arabicName>
                <bundleResourceID>10008537</bundleResourceID>
                <englishDescription>You will take an extra mega on any mega you will use from your bundle with super-net service for 15LE per month only including
recharge taxes</englishDescription>
                <englishName>Super-Net Service</englishName>
                <operationName>SUPER_NET</operationName>
                <operationType>DUMMY</operationType>
            </productOperations>
        </productDetailsList>
        <productDetailsList>
            <attributeList>
                <name>INQUIRY_REMAINING_UNITS_AND_EXPIRY_DATE</name>
                <value>97:05/07/2019</value>
            </attributeList>
            <attributeList>
                <name>INQUIRY_GENERAL_UNIT_CONSUMED_PERCENTAGE</name>
                <value>N/A</value>
            </attributeList>
            <attributeList>
                <arabicDescription>وحدات</arabicDescription>
                <arabicName>وحدات</arabicName>
                <arabicUnit>وحدات</arabicUnit>
                <bundleResourceID>1000065</bundleResourceID>
                <englishDescription>Units</englishDescription>
                <englishName>Units</englishName>
                <name>INQUIRY_GENERAL_UNIT_TOTAL_QUOTA</name>
                <type>METER</type>
                <unit>Units</unit>
                <value>N/A</value>
            </attributeList>
            <attributeList>
                <arabicDescription>وحدات</arabicDescription>
                <arabicName>وحدات</arabicName>
                <arabicUnit>وحدات</arabicUnit>
                <bundleResourceID>10000135</bundleResourceID>
                <englishDescription>Units</englishDescription>
                <englishName>Units</englishName>
                <name>INQUIRY_GENERAL_UNIT_CONSUMED_USAGE</name>
                <type>METER</type>
                <unit>Units</unit>
                <value>N/A</value>
            </attributeList>
            <category>
                <categoryDescription>TESLA</categoryDescription>
                <categoryName>TESLA</categoryName>
            </category>
            <fees>0.0</fees>
            <mainBundle>N</mainBundle>
            <overridenAttribute/>
            <productArabicName>Tesla MIX bundle AR NAME</productArabicName>
            <productEnglishName>COMMERCIAL MIX Tesla bundle</productEnglishName>
            <productName>Tesla_Units_bundle</productName>
        </productDetailsList>
    </customerDetails>
</subscriberProfileResponse>
] in [187] Millies.

i tried below configuration but it didn’t work:

Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  multiline.pattern: ^\</

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: true

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  multiline.match: after

jan · July 9, 2019, 12:30pm

as you placed only one message it is hard to guess - but I see a date as first entry 2019-07-02 16:00:00.058. So I guess that the example from the elastic filebeat docs should word:

https://www.elastic.co/guide/en/beats/filebeat/master/_examples_of_multiline_configuration.html#_timestamps

multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after

system · July 23, 2019, 12:30pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Shipping multiline logs to Graylog Graylog Central (peer support)	4	292	September 14, 2022
Multiline logs in Graylog Graylog Central (peer support)	1	553	December 22, 2020
REST API present in syslog comes in distorted format Graylog Central (peer support)	4	440	February 16, 2020
Graylog multiline GROK issue Graylog Central (peer support)	2	1669	April 27, 2017
Graylog stream multiline logs Graylog Central (peer support) sidecar , nxlog , multi-linenx	1	857	November 5, 2020

Filebeat multiline option

Multiline options

Related Topics