GELF HTTP Input Dropping data

[sachin]

Hi,

I started a GELF HTTP Input with the default parameters.
Then I executed a script to pull 10000 records from a database and stream it to http://localhost:12201/gelf
But only around 5000 records are visible in the Streams.

If we do the streaming in smaller batches like 1000 then all messages are landing in the Streams.

Is there any parameter that I need to set to handle a larger streaming load?
Please guide.

[jochen]

How exactly did you send messages to Graylog?
What are the exact HTTP requests you were sending?

[sachin]

Let me explain the process.
I have a node.js script which connects to a database, executes a Sql query and stores the results in a variable recordSet in Json using requestData.query function.

Then, the below snippet pushes the data in a for loop:

var url = "http://localhost:12201/gelf"
for(var i=0; i<recordSet.length; i++){
request({
                   url: url,
                   method: "POST",
                   json: true,
                   headers: {
                   "content-type": "application/json",
                    },
                    body: recordSet[i]
                }, function (error, response, body) {
                })

}

Let me know if there is a better way to do this.
Thanks for your prompt response.

[jochen]

It seems like you’re not doing any error handling, so you don’t know whether the requests were successful.