Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error

[shijiant]

i have a problem with heka, need some help,thanks

my heka process get a continued error after run a while as follow .
it won't recover until restart the heka process. but my ES service is OK

2015/11/10 18:20:46 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:21:05 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:21:16 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:21:55 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:22:05 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:22:45 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:23:16 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:23:46 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:24:15 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:24:26 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:24:55 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:25:46 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:26:49 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:26:55 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:27:05 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal Server Error
2015/11/10 18:27:15 Plugin 'ElasticSearchOutput' error: HTTP response error status: 500 Internal

here is my heka config:
[ElasticSearchOutput]
server = "http://10.209.6.46:8080"
message_matcher = "Type == 'hdcp_apache_access'"
encoder = "ESJsonEncoder"
flush_interval = 10000
flush_count = 200
[ElasticSearchOutput.buffering]
max_file_size = 2684354560

[sathieu]

Take a look at your ES logs. There is hopefully some usefull info.

Alternatively, you can do a network capture, if you don't have too much bandwidth.

[tianchaijz]

I also encounter this problem, by printing the message which leads ES respond 500, I found some messages were truncated (or were overwritten) as showing:

...
{"index":{"_index":"heka-foo-2015.11.16","_type":"bar"}}
{"json":"this json is ok"}
on":"this json is truncated"}
...

...
foo-2015.11.16","_type":"bar"}}
...

After inspecting heka's source code, I suspect it is the ProcessMessage function of ES plugin(slice append) which is not thread safe that leads this bug.

[ghost]

We met this problem too, and still can't not find a good way to fix it.

[tianchaijz]

I'll open a PR which fixes this bug.