License Violation Issues

Graylog Central (peer support)
1

Good morning. I have recently installed the < 5Gb per day enterprise license, and I keep running into a problems. Currently I have a red banner at the top of the window that states “Graylog Enterprise License Violation” and when I go to the licensing page, the messages I see are “Daily traffic limit 5.0Gb” and “Remote checks have failed too many times”.

My heaviest day since setting up the appliance has been 1.7Gb, so I am no where near the 5.0Gb daily limit. The other issue about remote checks seems to be the problem. I have had a little bit of back and forth with a graylog employee that emailed me when I requested my enterprise license, and I was told that I need to be able to connect to api[.]graylog[.]com in order for the license checks to work correctly. Initially the appliance was having issues hitting that URL, but I got it resolved after adding DNS servers to the interfaces config file. This appeared to fix the problem because the red banner went away.

Yesterday, I come in to the office and the banner message is back. The weird thing now is that I can’t ping the api[.]graylog[.]com URL from anywhere. Not from the appliance, not from any other workstations on the corporate network, and not even from my home computer, which is on a completely separate network.

api[.]graylog[.]com resolves to “api[.]graylog[.]com[.]herokudns[.]com” and a small list of IP addresses. Trace routes to api[.]graylog[.]com and to a few of the IP address (54.243.65.67, 54.243.175.62) both show the traffic leaving my network, but then just a whole bunch of timeouts and no response.

Is anyone else seeing or dealing with similar issues? Has anyone seen this before? Does anyone have a magic bullet for me? Thank you.

2

Hi,
I think it’s ok that the Heroku loadbalancer are not pingable. Could you try a couple of times to access the api by hand from your Graylog server? For example with curl: curl -v api.graylog.com
Should show at least a 404 if the connection is working.

Cheers,
Marius

3

Thank you for the reply. It looks like this may have been a false alarm. It is just weird that the other day I was able to get a ping response from the api URL and then a few days later no dice. I think that I just need to wait for the license violation to clear out. Not sure exactly how long that takes, but I have a temporary trial license now. My red banner has been replaced by a blue one…

4

Hi!

Did the warning disappear by itself then?

I’m experiencing the same issue, after installing the license a few days ago and with a workload that peaks at ~1 GiB a day.

Remote checks have failed too many times.
Requires remote checks: Yes (allowed consecutive check failures: 72)
License expiration warning: 30 days before

curl -v https://api.graylog.com gives 404 from the graylog-host.

I’m gonna demo our installation of Graylog for some colleagues & teamleader tomorrow, rather not having a big red warning while doing that.

Not sure what I should do next?

Edit; saw mccrollys answer now. So how would a temporary trial license affect a production environment, if you use auditing and archiving?

5

I’m still riding the trial license that I was provided. It is a little different that a typical trial license, it has everything enabled and remote checks are turned off, but it only lasts 30 days… so now I have a blue bar complaining about a trial license instead of the red bar… so I guess thats progress. The information that I was relayed was that the licenses recent themselves after a 30 day window. Which is kinda odd to me since the thing is checking in all the time, why not just fix it at next check-in, rather than wait a month??

I’m worried that when I issue a new enterprise license for myself that I’ll go back to having the big red warning message even though I’m not doing anything wrong or breaking any rules, but for now I have just decided to deal with that when it comes up.

I did have some initial connectivity problems (…DNS) that I’m almost positive caused my license to dip into the red, but I have resolved all of that and I spoke with support and confirmed that everything is working correctly. So I am assuming that I’ll be good after the 30 day cooldown, but I’m still a little hesitant.

6

Thanks for reply!

Yea it’s a little odd that behaviour. idk myself why I got that red box, seeing how we’ve had no network/dns issues what I know of.

I’ll ask support about this, hope you get it solved!

Have a nice day - Rikard

7

Just tried to import a fresh license and apparently I’m still committing some sort of license violation. I can’t seem to get rid of this message. I was told that the licenses reset after 30 days, I believe that I am outside of that 30 day refresh, but immediately upon importing the new license I’m back to having the licensing violation notification.

Based on other comments, it doesn’t look like I am the only one with this mysterious license violation problem. Is there any fix for this? I have been dealing with this for a month now.

8

Hi,

We quickly checked and the license service seems to be working as expected. From this side is hard to give you any other advise without more information, so please check you can reach the license service from your Graylog servers. If you find any errors or log messages that may contain more information on the issue we are happy to help.

Cheers,
Edmundo

9 
ubuntu@graylog:~$ date
Thu Mar 22 16:29:28 UTC 2018
ubuntu@graylog:~$ curl -v -XGET https://api.graylog.com
* Rebuilt URL to: https://api.graylog.com/
* Hostname was NOT found in DNS cache
*   Trying 50.19.87.162...
* Connected to api.graylog.com (50.19.87.162) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: CN=api.graylog.com
*        start date: 2018-02-22 23:36:53 GMT
*        expire date: 2018-05-23 23:36:53 GMT
*        subjectAltName: api.graylog.com matched
*        issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*        SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: api.graylog.com
> Accept: */*
> 
< HTTP/1.1 404 Not Found
* Server Cowboy is not blacklisted
< Server: Cowboy
< Connection: keep-alive
< Date: Thu, 22 Mar 2018 16:30:01 GMT
< Content-Type: application/json
< Content-Length: 43
< Via: 1.1 vegur
< 
* Connection #0 to host api.graylog.com left intact
{"code":404,"message":"HTTP 404 Not Found"}
ubuntu@graylog:~$ date
Thu Mar 22 16:29:36 UTC 2018
10

What other logs or information can I get for you? Let me know and I’ll be more than happy to provide. I have been dealing with this issue for a month now, even involving support at one point, and it still hasn’t been resolved.

Also, there have been a few people that have jumped on this post with the same or similar issues, so it doesn’t seem that I am an isolated incident. I really want this to work and for this non-legitimate warning message to go away. I really want to roll this into production, but can’t do that if my licensing status could blow up on me at any moment.

11

Looking at the output for curl you kindly provided, it looks like you can connect to the license service from that machine, but you still didn’t say how many Graylog servers you are using or if you executed the command on the same machine Graylog is running. Please also ensure that works consistently and not only a single time, since the request you do from the command line and the one Graylog does may end up going to a different server.

I would start by providing information on your setup and also include any Graylog server logs that may be related to the license checks, specially any warnings or failures. You should also ensure your Graylog servers have the right date and time set, as that may also affect license checks.

12

it is a single server appliance, and the command was run from that server.

I’ll run the command periodically and provide the output. here it is again.

Thu Mar 22 17:35:29 UTC 2018
ubuntu@graylog:~$ 
ubuntu@graylog:~$ 
ubuntu@graylog:~$ curl -v -XGET https://api.graylog.com
* Rebuilt URL to: https://api.graylog.com/
* Hostname was NOT found in DNS cache
*   Trying 50.16.237.173...
* Connected to api.graylog.com (50.16.237.173) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: CN=api.graylog.com
*        start date: 2018-02-22 23:36:53 GMT
*        expire date: 2018-05-23 23:36:53 GMT
*        subjectAltName: api.graylog.com matched
*        issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*        SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: api.graylog.com
> Accept: */*
> 
< HTTP/1.1 404 Not Found
* Server Cowboy is not blacklisted
< Server: Cowboy
< Connection: keep-alive
< Date: Thu, 22 Mar 2018 17:36:02 GMT
< Content-Type: application/json
< Content-Length: 43
< Via: 1.1 vegur
< 
* Connection #0 to host api.graylog.com left intact
{"code":404,"message":"HTTP 404 Not Found"}ubuntu@graylog:~$ 
ubuntu@graylog:~$ 
ubuntu@graylog:~$ 
ubuntu@graylog:~$ date
Thu Mar 22 17:35:43 UTC 2018

What logs from the server would you like for me to provide or look at? Which ones would have licensing events?

13

All of them:
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html#omnibus-graylog

14

You say “all of them” and while that seems all encompassing, it is a little vague. Do you mean every single file listed on that page under the Omnibus heading? All of the graylog, elasticsearch, and mongodb files? Or just the graylog logs? The /var/log/graylog/server/ folder has a bunch of files in it, do you want just the most recent one, or really all of them? Also, how can I get these two you? Do you guys have an upload site, can I send a sharefile/dropbox link?

15

Does this mean anything to you?

2018-03-22_21:44:51.70765 WARN [LicenseChecker] License violation - Failed to report license status to Graylog, Inc. - consecutive failures: 118, limit: 72
2018-03-22_21:44:51.70810 WARN [LicenseChecker] License violation - Detected irregular traffic records
2018-03-22_21:49:51.76372 WARN [LicenseChecker] License violation - Failed to report license status to Graylog, Inc. - consecutive failures: 118, limit: 72
2018-03-22_21:49:51.76430 WARN [LicenseChecker] License violation - Detected irregular traffic records

16

Please provide the complete logs and not just some arbitrary excerpts.

The logs of your Graylog node. Elasticsearch and MongoDB don’t have anything to do with the license management.

You can use a pastebin service like https://gist.github.com/ or https://0bin.net/.

17

You could always check you can reach the release API, which runs at the same host:

http GET http://api.graylog.com/releases/active

This will give you back the current stable release Version

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 217
Content-Type: application/json
Date: Fri, 23 Mar 2018 08:53:41 GMT
Server: Cowboy
Vary: Accept-Encoding
Via: 1.1 vegur

{
    "announcement_link": "https://www.graylog.org/blog/108-announcing-graylog-v2-4-3",
    "codename": "Wildwuchs",
    "published": true,
    "released_at": "2018-01-25T00:00:00.000Z",
    "suffix": "",
    "version": {
        "major": 2,
        "minor": 4,
        "patch": 3
    }
}

@mccrolly this way you could verify if from a network level everything is working. ( I have used httpie for this request)

18

Seems to be working fine.

ubuntu@graylog:~$ http GET http://api.graylog.com/releases/active
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 217
Content-Type: application/json
Date: Fri, 23 Mar 2018 12:11:59 GMT
Server: Cowboy
Vary: Accept-Encoding
Via: 1.1 vegur

{
    "announcement_link": "https://www.graylog.org/blog/108-announcing-graylog-v2-4-3", 
    "codename": "Wildwuchs", 
    "published": true, 
    "released_at": "2018-01-25T00:00:00.000Z", 
    "suffix": "", 
    "version": {
        "major": 2, 
        "minor": 4, 
        "patch": 3
    }
}
19

just checked it again this morning and everything appears fine connectivity-wise.

ubuntu@graylog:~$ date 
Fri Mar 23 08:12:17 EDT 2018
ubuntu@graylog:~$ 
ubuntu@graylog:~$ 
ubuntu@graylog:~$ curl -v -XGET https://api.graylog.com
* Rebuilt URL to: https://api.graylog.com/
* Hostname was NOT found in DNS cache
*   Trying 50.19.87.162...
* Connected to api.graylog.com (50.19.87.162) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: CN=api.graylog.com
*        start date: 2018-02-22 23:36:53 GMT
*        expire date: 2018-05-23 23:36:53 GMT
*        subjectAltName: api.graylog.com matched
*        issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*        SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: api.graylog.com
> Accept: */*
> 
< HTTP/1.1 404 Not Found
* Server Cowboy is not blacklisted
< Server: Cowboy
< Connection: keep-alive
< Date: Fri, 23 Mar 2018 12:12:53 GMT
< Content-Type: application/json
< Content-Length: 43
< Via: 1.1 vegur
< 
* Connection #0 to host api.graylog.com left intact
{"code":404,"message":"HTTP 404 Not Found"}
ubuntu@graylog:~$ date
Fri Mar 23 08:12:29 EDT 2018
20

2018-03-23_12:14:51.79565 WARN [LicenseChecker] License violation - Detected irregular traffic records
2018-03-23_12:14:55.40255 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2018-03-23_12:15:02.93175 WARN [LicenseChecker] License violation - Detected irregular traffic records
2018-03-23_12:15:02.94498 WARN [LicenseChecker] License violation - Failed to report license status to Graylog, Inc. - consecutive failures: 133, limit: 72
2018-03-23_12:15:02.94569 WARN [LicenseChecker] License violation - Detected irregular traffic records