forked from darold/pgbadger
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ChangeLog
2896 lines (2558 loc) · 141 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2023-03-20 - v12.1
This is a maintenance release of pgBadger that fixes issues reported by users
since past six months.
Here is the complete list of changes and acknowledgments:
- Fix parsing of multiline parameters. Thanks to Bekir Niyaz for the report.
- Fix failure to normalize query with ::tsrange. Thanks to Philippe Griboval
for the report.
- Add logical decoding consistent point and start for slot log entries to
the events report.
- Handle other ns + timezone format in timestamp. Thanks to Ronan Dunklau
for the report.
- Fix detection of %m when notation with T is used. Thanks to Ronan Dunklau
for the report.
- Add parsing of CloudNativePG generated logs. Thanks to codrut panea for
the patch.
- Fix unused option --outdir in report generation. Thanks to Frederic Guiet
for the report.
- Update README with last documentation changes. Thanks to Manisankar for
the report.
- Fix a typo in pgbadger examples. Thanks to Shinichi Hashiba for the patch.
2022-09-13 - v12.0
This major release of pgBadger fixes some issues reported by users since
past five months. As usual there is also new features and improvements:
* Remove support to Tsung output.
* Improve pgbadger performances when there are hundred of bind parameters
to replace.
* Remove option -n | --nohighlight which is no more used since upgrade to
pgFormatter 4.
* Use POST method to send auto_explain plan to explain.depesz.com to avoid
GET length parameter limit.
* Apply --exclude-query and --include-query to bind/parse traces.
* Add link to pgBadger report examples to documentation.
Here is the complete list of changes and acknowledgments:
- Fix monthly reports that was failing on "log file ... must exists". Thanks
to Jaume Sabater for the report.
- Fix pgbouncer start parsing debug message when input is stdin. Thanks to
aleszeleny for the report.
- Remove support to Tsung output.
- Drastically improve pgbadger performances for bind parameters
replacement that could make pgbadger run infinitely when there was
hundred of parameters. Thanks to Monty Mobile for the report.
- Fix documentation about pgBadger return codes and also some wrong return
code at some places. Thanks to Jaume Sabater for the report.
- Fix several typo. Thanks to David Gilman for the patch.
- Remove option -n | --nohighlight which is no more used since upgrade to
pgFormatter 4. Thanks to Elena Indrupskaya for the report.
- Lot of pgbadger documentation fixes. Thanks to Elena Indrupskay from
Postgres Pro for the patch.
- Allow half hour in --log-timezone and --timezone, value can be an integer,
ex: 2 or a float, ex: 2.5. Thanks to Mujjamil-K for the feature request.
- Allow use of regexp for --exclude-app and --exclude-client. Thanks to
rdnkrkmz for the feature request.
- Allow use of --explain-url with previous commit and restore the limitation
to explain text format.
- Use POST method to send auto_explain plan to explain.depesz.com to avoid
GET length parameter limit. Thanks to hvisage for the report.
- Apply --exclude-query and --include-query to bind/parse traces. Thanks to
Alec Lazarescu for the report.
- Fix parsing of autovacuum stats from RDS logs. Thanks to David Gilman for
the report.
- Fix passing of log format when parsing remote log. Thanks to spookypeanut
the report.
- Add link to pgBadger report examples to documentation.
- Fix Session per user reports. Thanks to vitalca for the report.
- Fix jsonlog parsing from PG15 ouput
- Fix text-based error/events reporting. Thanks to Michael Banck for the patch
- Fix regexp typo in normalize_error(). Thanks to Michael Banck for the patch.
2022-04-08 - v11.8
This release of pgBadger fix some issues reported by users since past
three months and especially two fixes on new log entries detection in
incremental mode.
* Fix detection of new log entries with timestamp when millisecond (%m) or
epoch (%n) was used in log_line_prefix.
* Fix detection of new log entries in local file when multiprocess was not
used.
Here is the complete list of changes and acknowledgments:
- Full review and simplification of the log file change detection.
- Reports messages "could not (receive|send) data (from|to) client" in the
Events reports. Thanks to Adrien Nayrat for the report.
- Fix parsing issue when the name of a prepared query contain the ':'
character. Thanks to aleszeleny for the report.
- Fix detection of new log entries with timestamp when millisecond (%m) or
epoch (%n). Thanks to aleszeleny for the report.
- Fix detection of new log entries in local file when multiprocess was not
used. Thanks to aleszeleny for the report.
- Fix detection of new log entries in remote files through ssh. Thanks to
Luca Ferrari for the report
- Fix garbage in username of "Connections per user" report. Thanks to
caseyandgina for the report.
- Fix ssh command when using URI, the ssh options was missing. Thanks to Luca
Ferrari for the report.
- Handle queryid %Q placeholder. Thanks to Adrien Nayrat for the patch.
- Fix typo in error sentence. Thanks to Luca Ferrari for the patch
- Report message: "server process was terminated by signal" in the Events
report. Thanks to Avi Vallarapu for the report.
- doc: fix filename for incremental every week command. Thanks to Theophile
Helleboid for the patch.
- t/04_advanced.t: Fix syslog test. Thanks to Christoph Berg for the patch.
2022-01-23 - v11.7
This release of pgBadger fix some issues reported by users since past
five months as well as some improvements:
* Add new option --no-progressbar option to not display it but keep the
other outputs.
* Add new option --day-report that can be used to rebuild an HTML report
over the specified day. Like option --month-report but only for a day.
It requires the incremental output directories and the presence of all
necessary binary data files. The value is date in format: YYYY-MM-DD
* Improve parsing of Heroku logplex and cloudsql json logs.
Here is the complete list of changes and acknowledgments:
- Update contribution guidelines and Makefile.PL to improve consistency,
clarity, and dependencies. Thanks to diffuse for the patch.
- Fix use of last parse file (--last-parsed) with binary mode. Thanks to
wibrt for the report.
- Add regression test for --last-parsed use and fix regression test on
report for temporary files only.
- Fix title for session per host graph. Thanks to Norbert Bede for the
report.
- Fix week number when computing weeks reports when --iso-week-number
and --incremental options was enabled. Thanks to hansgv for the report.
- Add --no-progressbar option to not display it and keep the other outputs.
Thanks to seidlmic for the feature request.
- Prevent too much unknown format line prints in debug mode for multi-line
jsonlog.
- Fix parsing of single line cloudsql json log. Thanks to Thomas Leclaire
for the report.
- Fix temporary files summary with log_temp_files only.
- Print debug message with -v even if -q or --quiet is used.
- Fix autodetection of jsonlog file.
- Fix parsing of cloudsql log file. Thanks to Luc Lamarle for the report.
- Fixes pid extraction in parse_json_input. Thanks to Francois Scala for
the patch.
- Add new option --day-report with value as date in format: YYYY-MM-DD
that can be used to rebuild an HTML report over the specified day.
Thanks to Thomas Leclaire for the feature request.
- Fix query counter in progress bar. Thanks to Guillaume Lelarge for the
report.
- Fix incomplete queries stored for top bind and prepare reports.
- Fix normalization of object identifier, in some case the numbers was
replaced by a ?.
- Fix unformatted normalized queries when there is a comment at beginning.
- Fix multi-line in stderr format when --dbname is used. Thanks to
Guillaume Lelarge for the report.
- Fix not generated reports in incremental mode when --dbname is used.
Thanks to Dudley Perkins for the report.
- Do not die anymore if a binary file is not compatible, switch to next
file. Thanks to Thomas Leclaire for the suggestion.
- Fix Heroku logplex format change in pgbadger parser. Thanks to François
Pietka for the report.
2021-09-04 - v11.6
This release of pgBadger fix some issues reported by users since past
seven months as well as some improvements:
* Add detection of Query Id in log_line_prefix new in PG14. Thanks to
Florent Jardin for the report.
* Add advanced regression tests with db exclusion and the explode
feature. Thanks to MigOps Inc for the patch.
* Apply multiprocess to report generation when --explode is used.
Thanks to MigOps Inc for the patch and Thomas Leclaire for the
feature request.
* Add --iso-week-number in incremental mode, calendar's weeks start
on a Monday and respect the ISO 8601 week number, range 01 to 53,
where week 1 is the first week that has at least 4 days in the new
year. Thanks to Alex Muntada for the feature request.
* Add command line option --keep-comments to not remove comments from
normalized queries. It can be useful if you want to distinguish
between same normalized queries. Thanks to Stefan Corneliu Petrea
for the feature request.
* Skip INFO lines introduced in PostgreSQL log file by third parties
software. Thanks to David Piscitelli for the report.
* Add compatibility with PostgresPro log file including rows number
and size in bytes following the statement duration. Thanks to
panatamann for the report.
* Parse times with T's to allow using the timestamps from journalctl.
Thanks to Graham Christensen for the patch.
* Improve Windows port. Thanks to Bertrand Bourgier for the patches.
Important note:
* Expect that --iso-week-number will be the default in next major
release and that --start-monday option will be removed as the week
will always start a Monday. The possibility to have week reports
start a Sunday will be removed to simplify the code.
Here is the complete list of changes and acknowledgments:
- Fix duplicate of warning message:
"database ... must be vacuumed within ... transactions".
Thank to Christophe Courtois for the report.
- Fix use of uninitialized variable. Thanks to phiresky for the report.
- Improve query id detection, it can be negative, as well as read it
from csvlog.
- Fix case where last file in incremental mode is always parsed even if
it was already done. Thanks to Thomas Leclaire for the report.
- Update syslog format regex to handle where session line indicator
only contains one int vs two ints separated by dash. Thanks to
Timothy Alexander for the patch.
- Fix --exclude-db option to create anyway the related report with json
log. Thanks to MigOps Inc for the patch and Thomas Leclaire for the
report.
- Add regression test about Storable buggy version.
- Fix use of uninitialized value in substitution iterator in incremental
mode during the week report generation. Thanks to Thomas Leclaire,
Michael Vitale, Sumeet Shukla and Stefan Corneliu Petrea for the report.
- Add 'g' option to replace all bind parameters. Thanks to Nicolas Lutic
and Sebastien Lardiere for the patch.
- Documentation improvements. Thanks to Stefan Petrea for the patch.
- Fixes change log time zone calculation. Thanks to Stefan Petrea for
the patch.
- Fix log filter by begin/end time.
- Fix wrong association of orphan lines for multi-line queries with a
filter on database. Thanks to Abhishek Mehta for the report.
- Fix reports in incremental mode when --dbname parameter is partially
ignored with "explode" option (-E). Thanks to lrevest for the report.
- Update javascript resources.
- Fix display of menu before switching to hamburger mode when screen is
reduced. Thanks to Guillaume Lelarge for the report.
- Fix bind parameters values over multiple lines in the log that were
not well supported.
- Apply same fix for previous patch than in pgFormatter.
- Fix an other use of uninitialized value in substitution iterator from
pgFormatter code. Thanks to Christophe Courtois for the report.
- Fix query normalization. Thanks to Jeffrey Beale for the patch.
- Be sure that all statements end with a semicolon when --dump-all-queries
is used. Thanks to Christian for the report.
- Fix typo and init of EOL type with multiple log files.
- Add auto detection of EOL type to fix LAST_PARSED offset when OEL is on
2 bytes (Windows case). Thanks to Bertrand Bourgier for the patch.
- Fix get_day_of_week() port on Windows where strftime %u is not supported.
Thanks to Bertrand Bourgier for the patch.
- Fix Windows port that call pl2bat.bat perl utility to create a corrupted
pgbadger.bat du to the way __DATA__ was read in pgbadger. Thanks to
Bertrand Bourgier for the patch.
- Fix begin/end time filter and add regression test for timestamp filters.
Thanks to Alexis Lahouze and plmayekar for the report.
- Fix use of uninitialized value in pattern match introduced by pgFormatter
update. Thanks to arlt for the report.
2021-02-18 - v11.5
This release of pgBadger fix some issues reported by users since
past three months as well as some improvements:
* Add report about sessions idle time, computed using:
"total sessions time - total queries time / number of sessions
This require that log_connection and log disconnection have been
enabled and that log_min_duration_statement = 0 (all queries logged)
to have a reliable value. This can help to know how much idle time
is lost, and if a pooler transaction mode would be useful.
This report is available in the "Sessions" tab of "Global Stats"
and in the "Sessions" tab of "General Activity" reports (per hour).
* Add anonymization of numeric values, replaced by 4 random digits.
* Update SQL beautifier based on pgFormatter 5.0.
Here is the complete list of changes and acknowledgments:
- Fix parsing of cloudsql multi-line statement. Thanks to Jon Young for the report.
- Add regression test for anonymization.
- Fix anonymization broken by maxlength truncate. Thanks to artl for the report.
- Add anonymization of parameter in time consuming prepare and bind reports. Thanks to arlt for the report.
- Add support to microseconds in logplex log line prefix. Thanks to Ross Gardiner for the report.
- Add report about sessions idle time. Thanks to Guillaume Lelarge for the feature request.
- Complete patch to support multi-line in jsonlog format.
2020-11-24 - v11.4
This release of pgBadger fix some issues reported by users since
past four months. Improve support for PostgreSQL 13 log information
and adds some new features:
* Add full autovacuum information in "Vacuums per table" report for
buffer usage (hits, missed, dirtied), skipped due to pins, skipped
frozen and WAL usage (records, full page images, bytes). In report
"Tuples removed per table" additional autovacuum information are
tuples remaining, tuples not yet removable and pages remaining.
These information are only available on the "Table" tab.
* Add new repartition report about checkpoint starting causes.
* Add detection of application name from connection authorized traces.
Here is the complete list of changes and acknowledgments:
- Fix typo in an error message. Thanks to Vidar Tyldum for the patch.
- Fix Windows port with error: "can not load incompatible binary data".
Thanks to Eric Brawner for the report.
- Fix typo on option --html-outdir in pgbadger usage and documentation.
Thanks to Vidar Tyldum for the patch.
- Fix autodetection of jsonlog/cloudsql format. Thanks to Jon Young
for the report.
- Fix CSV log parsing with PG v13. Thanks to Kanwei Li for the report
and Kaarel Moppel for the patch.
- Fix sort of queries generating the most temporary files report.
Thanks to Sebastien Lardiere for the report.
- Add pgbadger version trace in debug mode.
2020-07-26 - v11.3
This release of pgBadger fix several issues reported by users since
past four months. It also adds some new features and new command line
options:
* Add autodetection of UTC timestamp to avoid applying timezone
for graphs.
* Add support to GCP CloudSQL json log format.
* Add new option --dump-all-queries to use pgBadger to dump all
queries to a text file, no report is generated just the full list
of statements found in the PostgreSQL log. Bind parameters are
inserted into the queries at their respective position.
* Add new option -Q | --query-numbering used to add numbering of
queries to the output when using options --dump-all-queries or
--normalized-only.
* Add new command line option --tempdir to set the directory where
temporary files will be written. Can be useful on system that do
not allow writing to /tmp.
* Add command line option --ssh-port used to set the ssh port if not
default to 22. The URI notation also adds support to ssh port
specification by using the form:
ssh://192.168.1.100:2222//var/log/postgresql-11.log
Here is the complete list of changes and acknowledgments:
- Fix incremental reports for jsonlog/cloudsql log format. Thanks
to Ryan DeShone for the report
- Add autodetection of UTC timestamp to avoid applying autodetected
timezone for graphs. With UTC time the javascript will apply the
local timezone. Thanks to Brett Stauner for the report.
- Fix incremental parsing of journalctl logs doesn't work from the
second run. Thanks to Paweł Koziol for the patch.
- Fix path to resources file when -X and -E are used. Thanks to Ryan
DeShone for the report.
- Fix General Activity report about read/write queries. Thanks to
alexandre-sk5 for the report.
- Add debug message when parallel mode is not use.
- Fix elsif logic in file size detection and extra space introduced
in the journalctl command when the --since option is added. Thanks
to Pawel Koziol for the patch.
- Fix "not a valid file descriptor" error. Thanks to Pawel Koziol
for the report.
- Fix incremental mode with RDS files. Thanks to Ildefonso Camargo,
nodje and John Walsh for the report.
- Add new option -Q | --query-numbering used to add numbering of
queries to the output when using options --dump-all-queries or
--normalized-only. This can be useful to extract multiline queries
in the output file from an external script. Thanks to Shantanu Oak
for the feature request.
- Fix parsing of cloudsql json logs when log_min_duration_statement
is enabled. Thanks to alexandre-sk5 for the report.
- Fix wrong hash key for users in RDS log. Thanks to vosmax for the
report.
- Fix error related to modification of non-creatable array value.
Thanks to John Walsh and Mark Fletcher for the report.
- Add support to GCP CloudSQL json log format, log format (-f) is
jsonlog. Thanks to Thomas Poindessous for the feature request.
- Add new option --dump-all-queries to use pgBadger to dump all
queries to a text file, no report is generated just the full list
of statements found in the PostgreSQL log. Bind parameters are
inserted into the queries at their respective position. There is
not sort on unique queries, all queries are logged. Thanks to
Shantanu Oak for the feature request.
- Add documentation for --dump-all-queries option.
- Fix vacuum report for new PG version. Thanks to Alexey Timanovsky
for the report.
- Add new command line option --no-process-info to disable change of
process title to help identify pgbadger process, some system do
not allow it. Thanks to Akshay2378 for the report.
- Add new command line option --tempdir to set the directory where
temporary files will be written. Default:
File::Spec->tmpdir() || '/tmp'
Can be useful on system that do not allow writing to /tmp. Thanks
to Akshay2378 for the report.
- Fix unsupported compressed filenames with spaces and/or brackets.
Thanks to Alexey Timanovsky for the report.
- Add command line option --ssh-port used to set the ssh port if not
default to 22. The URI notation also adds support to ssh port
specification by using the form:
ssh://192.168.1.100:2222//var/log/postgresql-11.log
Thanks to Augusto Murri for the feature request.
2020-03-11 - v11.2
This release of pgBadger fix several issues reported by users since
past six months. It also adds some new features:
* Add support and autodetection of AWS redshift log format.
* Add support to pgbouncer 1.11 new log format.
* Handle zstd and lz4 compression format
* Allow to fully separate statistics build and HTML report build in
incremental mode without having to read a log file. For example
it is possible to run pgbadger each hours as follow:
pgbadger -I -O "/out-dir/data" --noreport /var/log/postgresql*.log
It just creates the data binary files in "/out-dir/data" then
for example you can make reports each night for the next day in
a separate directory `/out-dir/reports`:
pgbadger -I -l "/out-dir/data/LAST_PARSED" -H "/out-dir/reports" /out-dir/data/2020/02/19/*.bin
This require to set the path to the last parsed information, the
path where HTML reports will be written and the binary data file
of the day.
There is also new command line options:
* Add new command line option --explain-url used to override the url
of the graphical explain tool. Default URL is:
http://explain.depesz.com/?is_public=0&is_anon=0&plan=
If you want to use a local install of PgExplain or an other tool.
pgBadger will add the plan in text format escaped at the end of
the URL.
* Add new option --no-week to instruct pgbadger to not build weekly
reports in incremental mode. Useful if it takes too much time and
resources.
* Add new command line option --command to be able to set a command
that pgBadger will execute to retrieve log entries on stdin.
pgBadger will open a pipe to the command and parse log entries
generated by the command. For example:
pgbadger -f stderr --command 'cat /var/log/postgresql.log'
which is the same as executing pgbadger with the log file directly
as argument. The interest of this option is obvious if you have to
modify the log file on the fly or that log entries are extracted
from a program or generated from a database. For example:
pgbadger -f csv --command 'psql dbname -c "COPY jrn_log TO STDOUT (FORMAT CSV)"'
* Add new command line option --noexplain to prevent pgBadger to
parse and report explain plan written to log by auto_explain
extension. This is useful if you have a PostgreSQL version < 9.0
where pgBadger generate broken reports when there is explain plan
in log.
Backward compatibility:
- By default pgBadger will truncate queries up to 100000 characters.
This arbitrary value and can be adjusted using option --maxlength.
Previous behavior was to not truncate queries but this could
lead in excessive resources usage. Limiting default size is safer
and the size limit might allow no truncate in most cases. However
queries will not be beautified if they exceed 25000 characters.
Here is the complete list of changes and acknowledgments:
- Fix non working --exclude-client option. Thanks to John Walsh
for the report.
- Add regression test for RDS log parsing and --exclude-client.
- Fix progress bar for pgbouncer log file. The "queries" label is
changed in "stats" for pgbouncer log files.
- Add command line option --explain-url used to override the url
of the graphical explain tool. Thanks to Christophe Courtois for
the feature request.
- Add support to pgbouncer 1.11 new log format. Thanks to Dan
Aksenov for the report.
- Handle zstd and lz4 compression format. Thanks to Adrien Nayrat
for the patch.
- Add support and autodetection of AWS redshift log format. Thanks
to Bhuvanesh for the reature request.
- Update documentation about redshift log format.
- Add new option --no-week to instruct pgbadger to not build weekly
reports in incremental mode. Thanks to cleverKermit17 for the
feature request.
- Fix a pattern match on file path that breaks pgBadger on Windows.
- Fix #554 about cyrillic and other encoded statement parameters
that was not reported properly in the HTML report even with custom
charset. The regression was introduced with a fix to the well
known Perl error message "Wide character in print". The patch have
been reverted and a new command line option: --wide-char is
available to recover this behavior. Add this option to your
pgbadger command if you have message "Wide character in print".
Add a regression test with Cyrillic and french encoding. Thanks
to 4815162342lost and yethee for the report.
- Update documentation to inform that lc_messages = 'en_US.UTF-8'
is valid too. Thanks to nodje for the report.
- Update documentation about --maxlength which default truncate size
is 100000 and no more default to no truncate. Thanks to nodje for
the report.
- Fix retention calculation at year overlap. Thanks to Fabio Pereira
for the patch.
- Fix parsing of rds log file format. Thanks to Kadaffy Talavera for
the report.
- Prevent generating empty index file in incremental mode when there
is no new log entries. Thanks to Kadaffy Talavera for the report.
- Fix non up to date documentation. Thanks to Eric Hanson for the
patch.
- Fixes the command line parameter from -no-explain to -noexplain.
Thanks to Indrek Toom for the patch.
- Fall back to default file size when totalsize can not be found.
Thanks to Adrien Nayrat for the patch.
- Fix some dates in examples. Thanks to Greg Clough for the patch.
- Use compressed file extension regexp in remaining test and extract
.bin extension in a separate condition.
- Handle zstd and lz4 compression format. Thanks to Adrien Nayrat
for the patch.
- Fix remaining call of SIGUSR2 on Windows. Thanks to inrap for the
report.
- Fix progress bar with log file of indetermined size.
- Add new command line option --command to be able to set a command
that pgBadger will execute to retrieve log entries on stdin.
Thanks to Justin Pryzby for the feature request.
- Add new command line option --noexplain to prevent pgBadger to
parse and report explain plan written to log by auto_explain
extension. This is useful if you have a PostgreSQL version < 9.0
where pgBadger generate broken reports when there is explain plan
in log. Thanks to Massimo Sala for the feature request.
- Fix RDS log parsing when the prefix is set at command line. Thanks
to Bing Zhao for the report.
- Fix incremental mode with rds log format. Thanks to Bing Zhao for
the report.
- Fix possible rds log parsing. Thanks to James van Lommel and Simon
Dobner for the report.
- Fix statement classification and add regression test. Thanks to
alexanderlaw for the report.
- Fix anonymization of single characters in IN clause. Thanks to
Massimo Sala for the report.
- Fix RDS log parsing for rows without client/user/db information.
Thanks to Konrad for the report.
2019-09-16 - v11.1
This release of pgBadger fix several issues reported by users since
three months. It also adds some new features and reports:
- Add report of top N queries that consume the most time in the
prepare or parse stage.
- Add report of top N queries that consume the most time in the
bind stage.
- Add report of timing for prepare/bind/execute queries parts.
Reported in a new "Duration" tab in Global Stats report. Example:
Total query duration: 6m16s
Prepare/parse total duration: 45s564ms
Bind total duration: 4m46s
Execute total duration: 44s71m
This also fix previous report of "Total query duration" that was
only reporting execute total duration.
- Add support to RDS and CloudWatch log format, they are detected
automatically. You can use -f rds if pgbadger is not able to
auto-detect the log format.
- Add new configuration option --month-report to be able to build
monthly incremental reports.
- Restore support to Windows operating system.
There's also some bugs fixes and features enhancements.
- Add auto-generated Markdown documentation in README.md using tool
pod2markdown. If the command is not present the file will just not
be generated. Thanks to Derek Yang for the patch.
- Translate action WITH into CTE, regression introduced in last release.
- Fix support of Windows Operating System
- Add support to RDS and CloudWatch log format, use -f rds if pgbadger is
not able to auto-detect this log format. Thanks to peruuparkar for the
feature request.
- Fix option -f | --format that was not applied on all files get from the
parameter list where log format auto-detection was failing, the format was
taken from the fist file parsed. Thanks to Levente Birta for the report.
- Update source documentation file to replace reference to pgBadger v7.x
with v11. Thanks to Will Buckner for the patch.
- Limit height display size of top queries to avoid taking the whole page
with huge queries. Thanks to ilias ilisepe1 for the patch.
- Fix overflow of queries and detail in Slowest individual queries.
- Fix SSH URIs for files, directories and wildcards. Thanks to tbussmann for
the patch.
- Fix URI samples in documentation. Thanks to tbussmann for the patch.
- Hide message of use of default out file when --rebuild is used.
- Add extra newline to usage() output to not bread POD documentation at
make time.
- Reapply --exclude-client option description in documentation. Thanks to
Christoph Berg for the report.
2019-06-25 - v11.0
This release of pgBadger adds some major new features and fixes some
issues reported by users since the last four months. New features:
- Regroup cursor related query (DECLARE,CLOSE,FETCH,MOVE) into new
query type CURSOR.
- Add top bind queries that generate the more temporary files.
Require log_connection and log_disconnection be activated.
- Add --exclude-client command line option to be able to exclude log
entries for the specified client ip. Can be used multiple time.
- Allow to use time only in --begin and --end filters.
- Add -H, --html-dir option to be able to set a different path where
HTML report must be written in incremental mode. Binary files stay
on directory defined with -O, --outdir option.
- Add -E | --explode option to explode the main report into one
report per database. Global information not related to a database
are added to the postgres database report.
- Add per database report to incremental mode. In this mode there
will be a sub directory per database with dedicated incremental
reports.
- Add support to Heroku's PostgreSQL logplex format. Log can be
parsed using:
heroku logs -p postgres | pgbadger -f logplex -o heroku.html -
- When a query is > 10Kb we first limit size of all constant string
parameters to 30 characters and then the query is truncated to 10Kb.
This prevent pgbadger to waste time/hang with very long queries
when inserting bytea for example. The 10Kb limit can be controlled
with the --maxlength command line parameter.
The query is normalized or truncated to maxlength value only after
this first attempt to limit size.
This new release breaks backward compatibility with old binary or JSON
files. This also mean that incremental mode will not be able to read
old binary file. If you want to update pgBadger and keep you old reports
take care to upgrade at start of a new week otherwise weekly report will
be broken. pgBadger will print a warning and just skip the old binary
file.
There's also some bugs fixes and features enhancements.
- Add a warning about version and skip loading incompatible binary file.
- Update code formatter to pgFormatter 4.0.
- Fix pgbadger hang on Windows OS. Thanks to JMLessard for the report.
- Update tools/pgbadger_tools script to be compatible with new binary
file format in pgBadger v11.
- Add top bind queries that generate the more temporary files. This
collect is possible only if log_connection and log_disconnection
are activated in postgresql.conf. Thanks to Ildefonso Camargo for
the feature request.
- Fix auto detection of timezone. Thanks to massimosala for the fix.
- Remove some remaining graph when --nograph is used
- Force use of .txt extension when --normalized-only is used.
- Fix report of auto vacuum/analyze in logplex format. Thanks to
Konrad zichul for the report.
- Fix use of progress bar on Windows operating system. Thanks to
JMLessard for the report.
- Use a `$prefix_vars{'t_time'} to store the log time. Thanks to Luca
Ferrari for the patch.
- Update usage and documentation to remove perl command from pgbadger
invocations. Thanks to Luca Ferrari for the patch.
- Use begin and end with times without date. Thanks to Luca Ferrari
for the patch.
- Added some very minor spelling and grammar fixes to the readme file.
Thanks to ofni yratilim for the patch.
- Fix remote paths using SSH. Thanks to Luca Ferrari for the patch.
- Update regression test to works with new structure introduced with
the per database report feature.
- Fix fractional seconds in all begin and end parameters. Thanks to
Luca Ferrari for the patch.
- Fix documentation URL. Thanks to Kara Mansel for the report.
- Fix parsing of auto_explain.
Add more information about -U option that can be used multiple time.
Thanks to Douglas J Hunley for the report.
- Lot of HTML / CSS report improvements. Thanks to Pierre Giraud for
the patches.
- Update resource file.
- Add regression test for logplex format.
- Add support to Heroku's PostgreSQL logplex format. You should be able
to parse these logs as follow:
heroku logs -p postgres | pgbadger -f logplex -o heroku.html -
or if you have already saved the output to a file:
pgbadger heroku.log
The logplex format is auto-dectected like any other supported format.
pgBadger understand the following default log_line_prefix:
database = %d connection_source = %r sql_error_code = %e
or simply:
sql_error_code = %e
Let me know if there's any other default log_line_prefix. The prefix
can always be set using the -p | --prefix pgbadger option:
pgbadger --p 'base = %d source = %r sql_state = %e' heroku.log
for example.
Thanks to Anthony Sosso for the feature request.
- Fix pgbadger help on URI use.
- Fix broken wildcard use in ssh URI introduced in previous patch.
Thanks to Tobias Bussmann for the report.
- Allow URI with space in path to log file. Thanks to Tobias Bussmann
for the report.
- Fix URI samples in documentation. Thanks to Tobias Bussmann for the
patch.
- Fix t/02_basics.t to don't fail if syslog test takes more than 10s.
Thanks to Christoph Berg for the patch.
2019-02-14 - v10.3
This release of pgBadger is a maintenance release that fixes some
log format autodetection issues another pgBouncer log parsing issue
reported by users. There is also a new feature:
The -o | --outfile option can now be used multiple time to dump
output in several format in a single command. For example:
pgbadger -o out.html -o out.json /log/pgsql-11.log
will create two reports in html and json format saved in the
two corresponding files.
There's also some bugs fixes and features enhancements.
- Fix statistics reports when there a filter on database, user,
client or application is requested. Some queries was not
reported.
- Fix autodetection of pg>=10 defauilt log line prefix.
- Fix autodetection of log file with "non standard" log line prefix.
If --prefix specify %t, %m, %n and %p or %c, set format to stderr.
Thanks to Alex Danvy for the report.
- Remove extra space at end of line.
- Add minimal test to syslog parser.
- Fix a call to autodetect_format().
- Truncate statement when maxlength is used. Thanks to Thibaud
Madelaine for the patch.
- Add test for multiple output format.
- The -o | --outfile option can now be used multiple time to dump
output in several format in a single command. For example:
pgbadger -o out.txt -o out.html -o - -x json /log/pgsql-11.log
Here pgbadger will create two reports in text and html format
saved in the two corresponding file. It will also output a json
report on standard output. Thanks to Nikolay for the feature
request.
- Move detection of output format and setting of out filename into
a dedicated function set_output_extension().
- Fix another pgBouncer log parsing issue. Thanks to Douglas J.
Hunley for the report.
2018-12-27 - v10.2
This release of pgBadger is a maintenance release that fixes issues
reported by users during last three months. There is also some new
features:
* Add support to pgbouncer 1.8 Stats log format.
* Auto adjust javascript graph timezone.
There is a new command line option:
* Add --exclude-db option to compute report about everything except
the specified database.
* Add support to http or ftp remote PostgreSQL log file download.
The log file is parsed during the download using curl command
and never saved to disk. With ssh remote log parsing you can use
uri as command line argument to specify the PostgreSQL log file.
ssh://localhost/postgresql-10-main.log
http://localhost/postgresql-10-main.log.gz
ftp://localhost/postgresql-10-main.log
with http and ftp protocol you need to specify the log file format
at end of the uri:
http://localhost/postgresql-10-main.log:stderr
You can specify multiple uri for log files to be parsed. This is
useful when you have pgbouncer log file on a remote host and
PostgreSQL logs in the local host.
With ssh protocol you can use wild card too like with remote
mode, ex: ssh://localhost/postgresql-10-main.log*
Old syntax to parse remote log file using -r option is still
working but is obsolete and might be removed in future versions.
There's also some bugs fixes and features enhancements.
- Adjust end of progress bar with files with estimate size (bz2
compressed files and remote compressed files.
- Update year in copyright.
- Add information about URI notation to parse remote log files.
- Force progress to reach 100% at end of parsing of compressed
remote file.
- Extract information about PL/pgSQL function call in queries of
temporary file reports. The information is append to the details
display block.
- Fix progress bar with csv files.
- Fix reading binary file as input file instead of log file.
- Encode html output of queries into UTF8 to avoid message "Wide
character in print". Thanks to Colin 't Hart for the report.
- Add Checkpoints distance key/value for distance peak.
- Fix pgbouncer parsing and request throughput reports. Thanks
to Levente Birta for the report.
- Fix use of csvlog instead of csv for input format.
- Add support to pgbouncer 1.8 Stats log format. Thanks to Levente
Birta for the report.
- Add warning about parallel processing disabled with csvlog. Thanks
to cstdenis for the report.
- Add information in usage output about single process forcing with
csvlog format in -j and -J options. Thanks to cstdenis for the
report.
- Fix unknown line format error for multi line log while incremental
analysis over ssh. Thanks to Wooyoung Cho for the report.
- Add -k (--insecure) option to curl command to be able to download
logs from server using a self signed certificate.
- Auto adjust javascript graph timezone. Thanks to Massimino Sala
for the feature request.
- Add support to HTTP logfile download by pgBadger, for example:
/usr/bin/pgbadger http://www.mydom.com/postgresql-10.log
- Will parse the file during download using curl command.
- Fix documentation. Thanks to 0xflotus for the patch.
- Reapply fix on missing replacement of bind parameters after some
extra code cleaning. Thanks to Bernhard J. M. Grun for the report.
- Add --exclude-db option to compute report about everything except
the specified database. The option can be used multiple time.
2018-09-12 - v10.1
This release of pgBadger is a maintenance release that fixes reports
in incremental mode and multiprocess with -j option. Log parsing from
standard input was also broken. If you are using v10.0 please upgrade
now.
- Add test on pgbouncer log parser.
- Some little performances improvment.
- Fix not a valid file descriptor at pgbadger line 12314.
- Fix unwanted newline in progressbar at startup.
- Remove circleci files from the project.
- Remove dependency of bats and jq for the test suite, they are
replaced with Test::Simple and JSON::XS.
- Add more tests especially for incremental mode and input from
stdin that was broken in release 10.0.
- Sync pgbadger, pod, and README, and fix some syntax errors.
Thanks to Christoph Berg for the patch.
- Add documentation on how to install Perl module JSON::XS from
apt and yum repositories.
- Fix URI for CSS in incremental mode. Thanks to Floris van Nee
for the report.
- Fix fatal error when looking for log from STDIN. Thanks to
Jacek Szpot for the report.
- Fixes SED use for OSX builds. Thanks to Steve Newson for the
patch.
- Fix illegal division by zero in incrental mode. Thanks to
aleszeleny for the report.
- Replace SQL::Beautify with v3.1 of pgFormatter::Beautify.
2018-09-09 - v10.0
This release of pgBadger is a major release that adds some new
features and fix all issues reported by users since last release.
* Add support of pgbouncer syslog log file format.
* Add support to all auto_explain format (text, xml, json and yaml).
* Add support to %q placeholder in log_line_prefix.
* Add jsonlog format of Michael Paquier extension, with -f jsonlog
pgbadger will be able to parse the log.
* Replace the SQL formatter/beautify with v3.0 of pgFormatter.
There is some new command line option:
- Add --prettify-json command line option to prettify JSON output.
- Add --log-timezone +/-XX command line option to set the number
of hours from GMT of the timezone that must be used to adjust
date/time read from log file before beeing parsed. Note that you
might still need to adjust the graph timezone using -Z when the
client has not the same timezone.
- Add --include-time option to add the ability to choose times that
you want to see, instead of excluding all the times you do not
want to see (--exclude-time).
The pgBadger project and copyrights has been transfered from Dalibo
to the author and official maintainer of the project. Please update
your links:
- Web site: http://pgbadger.darold.net/
- Source code: https://github.com/darold/pgbadger
I want to thanks the great guys at Dalibo for all their investments
into pgBadger during these years and especially Damien Clochard and
Jean-paul argudo for their help to promote pgBadger.
- Fix checkpoint distance and estimate not reported in incremental
mode. Thanks to aleszeleny for the report.
- Fix title of pgbouncer simultaneous session report. Thansks to
Jehan Guillaume De Rorthais for the report.
- Add support of pgbouncer syslog log file format. Thanks to djester
for the feature request.
- Fix error when a remote log is empty. Thanks to Parasit Hendersson
for the report.
- Fix test with binary format. Binary file must be generated as it
is dependent of the plateform. Thanks to Michal Nowak for the
report.
- Fix case where an empty explain plan is generated.
- Fix parsing of autodetected default format with a prefix in
command line.
- Remove dependency of git command in Makefile.PL.
- Update documentation about options changes and remove of the
[%l-1] part of the mandatory prefix.
- Fix parsing of vacuum / analyze system usage for PostgreSQL 10.
Thanks to Achilleas Mantzios for the patch.
- Fix Temporary File Activity table.
- Remove dependency to git during install.
- Add --log-timezone +/-XX command line option to set the number
of hours from GMT of the timezone that must be used to adjust
date/time read from log file before beeing parsed. Using this
option make more difficult log search with a date/time because the
time will not be the same in the log. Note that you might still
need to adjust the graph timezone using -Z when the client has not
the same timezone. Thanks to xdexter for the feature request and
Julien Tachoire for the patch.
- Add support to auto_explain json output format. Thanks to dmius
for the report.
- Fix auto_explain parser and queries that was counted twice.
Thanks to zam6ak for the report.
- Fix checkpoint regex to match PostgreSQL 10 log messages. Thanks
to Edmund Horner for the patch.
- Update description of -f | --format option by adding information
about jsonlog format.
- Fix query normalisation to not duplicate with bind queries.
Normalisation of values are now tranformed into a single ? and no
more 0 for numbers, two single quote for string. Thanks to vadv
for the report.
- Fix log level count. Thanks to Jean-Christophe Arnu for the report
- Make pgbadger more compliant with B::Lint bare sub name.
- Made perlcritic happy.
- Add --prettify-json command line option to prettify JSON output.
Default output is all in single line.
- Fix Events distribution report.
- Fix bug with --prefix when log_line_prefix contain multiple %%.
Thanks to svb007 for the report.
- Add --log-timezone +/-XX command line option to set the number
of hours from GMT of the timezone that must be used to adjust
date/time read from log file before beeing parsed. Using this
option make more difficult log search with a date/time because the
time will not be the same in the log. Note that you might still
need to adjust the graph timezone using -Z when the client has not
the same timezone. Thanks to xdexter for the feature request.
- Remove INDEXES from the keyword list and add BUFFERS to this list.
- Fix normalization of query using cursors.
- Remove Dockerfile and documentation about docker run. pgBadger
comes as a single Perl script without any dependence and it can
be used on any plateform. It is a non sens to use docker to run
pgbadger, if you don't want to install anything, just copy the
file pgbadger where you want and execute it.
- Fix broken grid when no temp files activity. Thanks to Pierre
Giraud for the patch
- Add doc warning about log_in_duration_statement vs log_duration +
log_statement. Thanks to Julien Tachoire for the patch.
- Apply timezone offset to bar charts. Thanks to Julien Tachoire
for the patch.
- Delete current temp file info if we meet an error for the same PID
Thanks to Julien Tachoire for the patch.
- Consistently use app= in examples, and support appname=
Some of the usage examples used appname= in the prefix, but the
code didn't recognize that token. Use app= in all examples, and
add appname= to the prefix parser. Thanks to Christoph Berg for
the patch
- Fix wrong long name for option -J that should be --Jobs intead
of --job_per_file. Thanks to Chad Trabant for the report and
Etienne Bersac for the patch.
- Ignore blib files. Thanks to Etienne Bersac for the patch.
- Add consistency tests. Thanks to damien clochard for the patch.
- doc update : stderr is not a default for -f. Thanks to Christophe
Courtois for the patch.
- Always update pod and README. Thanks to Etienne Bersac for
the patch.
- Add some regression tests. Thanks to Etienne Bersac for the patch.
- Add editorconfig configuration. Thanks to Etienne Bersac for the
patch.
- Drop vi temp files from gitignore. Thanks to Etienne Bersac for
the patch.
- Add --include-time option to add the ability to choose times that
you want to see, instead of excluding all the times you do not
want to see. This is handy when wanting to view only one or two
days from a week's worth of logs (simplifies down from multiple
--exlucde-time options to one --include-time). Thanks to Wesley
Bowman for the patch.
- Check pod syntax. Thanks to Etienne Bersac for the patch.
- Add HACKING to document tests. Thanks to Etienne Bersac for the
patch.
- Drop obsolete --bar-graph option. Thanks to Etienne Bersac for
the patch.
- Drop misleading .perltidyrc. This file date from 2012 and
pgbadger code is far from compliant. perltidy unified diff is
10k lines. Let's drop this. Thanks to Etienne Bersac for the
patch.
- Fix use of uninitialized value in SQL formatting. Thanks to John
Krugger for the report and Jean-paul Argudo for the report.
2017-07-27 - v9.2
This release of pgBadger is a maintenance release that adds some new
features.
* Add report of checkpoint distance and estimate.
* Add support of AWS Redshift keywords to SQL code beautifier.
* Add autodetection of log format in remote mode to allow remote
parsing of pgbouncer log file together with PostgreSQL log file.