We have an application stack where several different components hosted on different servers talk to each other.
The components are as below,
Apache Http Server - Web Server
IBM Websphere Portal - Web Application Server
Documentum Platform - Middleware Server
Oracle - Database Server
Below is a brief overview of how the system works.
User sends a http request to the web application
The web application receives the http request and sends a RPC call to the Middleware
Middleware receives the RPC call and sends a Database call to the Database
Middleware receives the response from the database
Webapplication receives the response from the middleware
Webapplication sends out an http resopne
FYI, RPC stands for Remote Procedure Call
Basically the entire flow would be something like HTTP request -> RPC call -> Database call/response -> RPC Response- > HTTP Response
All the 4 different components mentioned above are hosted on completely different servers.
As you can see, debugging becomes difficult and time consuming when an issue happens in one of these components.
Any suggestions you may have on how can we have a central logging/data flow system using graylog would be really helpful
Add a transaction-id to every request and write that id into the log files together with the action that is done at the part of the stack.
If you now put all your log files in a central location you are able to search for the transaction-id and get all log entries from all application parts that include this id.
Any take on this would be a very specific solution for your custom stack.
In general, try generating a unique ID for each request at the edge of your system (e. g. in Apache httpd) and propagate that unique ID throughout your stack to enable correlating log messages with actual user interaction.
Depending on how much effort you want to invest into instrumenting your applications, you might also want to take a look at tracing frameworks like OpenTracing which is a good complement for generic logging.
