diff --git a/SECURITY.md b/SECURITY.md index 33a273e..089eb21 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -12,7 +12,7 @@ Send an email to security@centreon.com. If possible, encrypt your message with o You should receive a response within 48 hours. If for some reason you do not, please follow up via email to ensure we received your original message. -To help us better understand the nature and scope of the possible issue, please describe as much as you can: +To help us better understand the nature and scope of the possible issue, please describe as much as you can: * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) * Full paths of source file(s) related to the manifestation of the issue @@ -22,10 +22,94 @@ To help us better understand the nature and scope of the possible issue, please * Proof-of-concept or exploit code (if possible) * Impact of the issue, including how an attacker might exploit the issue +## Bug bounty + +We don't have a bug bounty program but this is something we are thinking about. + ## PGP information ### Public key +| Tag | Value | +| -- | -- | +| ID | F92686A9EC269C1A | +| Type | RSA | +| Size | 4096 | +| Created | 2022-12-28 | +| Expires | 2023-01-22 | +| Fingerprint | 3552 91EA 7DAF 9E2A 192C 62B6 F926 86A9 EC26 9C1A | + +``` +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQINBGHLO4IBEADB5ZlFUNNH/Y5TEVHAAHIMjHEt63M5hA+C94EYv89R2+swz212 +Hla4f5sVl5wPNSwiIAed+bJNKnGiaDM/508aMcTHurGRu3x5/MyvuxpXmzOSY1Mt +JZxLBBkonL1iX0tCytWriOhgAty9gi58DPKA6f7sVDgt3Hm/NtIEULSbXy6xfDYo +m+Sz39+hb0PcKKEkacRGzOGDIR0UgOAUGDBEbDoLPxjM6flHjXcjs4fZNY2HHQXO +AB65qM9my4ALxxsrIbsKfu25HY52qSZoqZD90AxKdNtRFlnkXClWN0l26fVqiGjv +oCPMYGPp80OYvymE2QhtlD+jRAepwyWx1YY96VFIA9LsZtjmoRxw/KLghdeP4Q7p +/BUCVkT393OOTayNhoNa7iCqbK0lmB6mequi7KV3vNXn79WP0Hm8AQ9/9bEJaY8x +oNTKAxsR6gLP1fc7S/zg9iIHUuTj6XU9CbW45ADrCJRel5LoM+MZ3DWXh+kd0Iuw +yANU+XVgC1fXQOf76BJeYSalZS8Ln1vpYjDwEZBSmLdyefCYBjspxjDNzpCAy+wH +gc/vpQbjmFxgkbZ3AroGDaNu1JVhA3yy8oXAEwAxl8BzsYye2YbhAUb2RgZhIndd +TCcWkwhEWey3XYMCtnFWxsXnteA1cvWD8PvCiWy53yc/Ng59H3XyB3sJbQARAQAB +tJJDZW50cmVvbiBTZWN1cml0eSAoRW1haWwgdXNlZCB0byByZXBvcnQgYSB2dWxu +ZXJhYmlsaXR5IG9uIENlbnRyZW9uJ3MgcHJvZHVjdHMgb3IgdG8gY29udGFjdCB0 +aGUgQ2VudHJlb24ncyBzZWN1cml0eSB0ZWFtKSA8c2VjdXJpdHlAY2VudHJlb24u +Y29tPokCVAQTAQoAPhYhBPSiLqQfofMIpmrARo99hgxsMlQ6BQJhyzuCAhsDBQkC +AikABQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJEI99hgxsMlQ6Ms0P/1lJfoaj +5/mRIKvaWZnTZm6OCpJtRT/9WrBGxbVi0TfyFb8M5OHeoz2oXc7WEPDJNNW9aRat +i40oHfQExW0UBsMvpfGlhPc5nwIxzpvFuckPrSTU3Y9ZLQvCoyxPIWMsXlghwwHD +OFW5oYg+rAvtmFuLyM18MdkH3YmrUHs7wyZczvi5hqdv5yB92kEjDTRbCYG8k9ep +P1rr3U/WyQ6u5BEZLTFNceK//VQsEfN7l8QLQV5CtG7PtdnLh5V4lkDHV7DW3PCg +TXyy8A5IatLh7z+ODQ/GDwCw7NNQm5M8dB1T/tLZn7+kTf8KqjHz4Qbh2PFBu5R7 +sDPEM81EMSAN/dfuZyQKbABnJk0JalvZeuZYUQYZWTgK6mSFJ3Rac8aX4TX7fwfS +kjh0ivG/FWzKaDa/dGfpH0J69IE2SACLRF+022hPLz7dRmSyNQSFZ4McXC+ihQps +mDJrqLmBPSNIRkfTwczINPSAX051w3GsJDSoAID+X0iqmoZTPYuf9XJO689vx+5B +g4FwhFwETkCqq6sZLzBI8+8dHKCn7imCtmrJ7JN8uy4mPS4Y2yVTBr1sZiAuvHzN +4314o3N30OZeqG61OuE6XO67mlw5Rk5Kay7P/s3uu5wXEL01z5+VKH/hjzVV+GXq +9J+VKYfXOcvu9PDhVBzlXgtP5xNuhh/IlvoDuQINBGHLPM4BEADdN4QbeuBSFts7 +9iIJXBmYiwcfDUOyZYaam6tI1fi1MbWCTAwpDpR0e8wdAon5yrF5jF6f6PIzqSfV +jc1rtLfdftVlzCMobXyjPxO9LkwChSm+b1tR6R7FyxfkUu5Og7TdrTpzzbTPN6TA +0BReEy20NpU6b3xC96BdaxE2ePlDOm26C5ygmAWszGD29ztxVtbBq7w0M+q7MO5Q +UH286bNQXKL5C5wZuA18li/hk7Cri4XwtRyMMh0dlT7hYYuKKJN5d8swx793JdlP +uYxmL93bM7rlka+W9fHYEbW5Zr7KUKwGygju/R2kx567iZyNwhFdeNsMFXFRIxCL +sEqNBc6EexafaxoJc4Ms0FD5WXiy9Je5+c5ue5Qb3SzavRu8u/z4bnjmpXqodBOS +jB4KPPGp4iCZPIOD98HyQ4XlYPSFc4hFEDxM6JGjfiKzFGaSmroSqwoKYYUjLsI+ +SuFPciOnH73KvwGX0uDYOKKZp5kmhKwu+AWaOfjoMGC2j+aOCQswziYBwr32bgyf +S/CM4uha7XhY4vi5IaRgXrSSsCRFwskTRQwtAHtu8m17D0CH7K5blKXtx+pHB+6x +cYTX9u2N2rqnOTma6+KEXFzVj0JiXsT3OwFmAFRvDU+erGSlAF3bQkZcu0hoFZmh +XBeRV+vA3D/rkwXal8vMRcn2V+XZ5wARAQABiQI8BBgBCgAmFiEE9KIupB+h8wim +asBGj32GDGwyVDoFAmHLPM4CGwwFCQICKQAACgkQj32GDGwyVDqkbg/+NVEZw2A4 +Uk6h4Exo9T0+ttd/ywi8P5aGnoiJ9Fw92RHgmSNUwIwgdeGKrgBbhVaO/V4CDGJp +iiwIAxzU/xCNibEGTUkH79AZvFHXxXwRKf/vWW1w3gyh9ppRLBlUw3S2DdEkxlzJ +5R1ryYTeV4yFAVK1Ln1v/UCA2WHho3IN/PIgDt701zONUEn1OOxHrMlKsgHIBAAk +NA6IQ1Tr8RW9abK3uAtJxxnyOqEMkiE03sJfd1dAUtvirGxr7g3t1Gfi8BPnQR5T +ZNqDOblM6fiY05AngPOLtV0n6LazK/buNenvUUhT0R9noMX6ZcApGpS4fFhADw1q +vrFYSG/4rLSGKvLqw5pQ7PzLDHPfn/HIME//SPuBnYrTYjiupzdmgtjGOc1iMV0X +YVXuA1yj+aJFaObLnVD31v2GIKvVS4WMsG74Mf5vMiMkbc0Zg2ULGun3sXscW0Yh +2MnvI5oYQcKmzjmhPdKHrmkiy9QC4442PbE8Bn9KUpcVoxCtFr/Zsc18iUVHYyIG +rrmZBE8MF1tGGBsdFC4Aktujuj3EevBo26QLozyfOLXXATHhmGh4SWsH68iyzynw +ARzB/pCyvB1Y/QbRn3ClFIksAyjrMxiNkSQXgToc8Ph+vLnHS3Y4399c74WZCHCH +i51yIfcTAPmxOst/YN5WXOxWHZjZ/STVi0Y= +=8Gny +-----END PGP PUBLIC KEY BLOCK----- +``` + +### Revoked Public key + +**_Kindly use the new key instead_** + +| Tag | Value | +| -- | -- | +| ID | BEAF6EBF631106F9 | +| Type | RSA | +| Size | 4096 | +| Created | 2020-02-11 | +| Expires | 2022-01-13 | +| Cipher |AES-256| +| Fingerprint | C377 E9D5 2D5C 137D 3DD5 73B5 BEA F6EBF 6311 06F9 | + ``` -----BEGIN PGP PUBLIC KEY BLOCK----- @@ -80,17 +164,3 @@ SL4taAGY0BDuA3zhB7p5tP8= =YBvx -----END PGP PUBLIC KEY BLOCK----- ``` - -| Tag | Value | -| -- | -- | -| ID | BEAF6EBF631106F9 | -| Type | RSA | -| Size | 4096 | -| Created | 2020-02-11 | -| Expires | 2022-01-13 | -| Cipher |AES-256| -| Fingerprint | C377 E9D5 2D5C 137D 3DD5 73B5 BEA F6EBF 6311 06F9 | - -## Bug bounty - -We don't have a bug bounty program but this is something we are thinking about. diff --git a/ci/scripts/gorgone-deb-package.sh b/ci/scripts/gorgone-deb-package.sh index 5503225..aa510a6 100755 --- a/ci/scripts/gorgone-deb-package.sh +++ b/ci/scripts/gorgone-deb-package.sh @@ -11,6 +11,8 @@ echo "################################################## PACKAGING COLLECT ##### AUTHOR="Luiz Costa" AUTHOR_EMAIL="me@luizgustavo.pro.br" +apt-get update + pwd if [ -d /build ]; then diff --git a/contrib/gorgone_audit.pl b/contrib/gorgone_audit.pl index 75e82ea..f6d86fa 100644 --- a/contrib/gorgone_audit.pl +++ b/contrib/gorgone_audit.pl @@ -50,7 +50,7 @@ sub json_decode { my $decoded; eval { - $decoded = JSON::XS->new->utf8->decode($options{content}); + $decoded = JSON::XS->new->decode($options{content}); }; if ($@) { $self->{logger}->writeLogError("cannot decode json response: $@"); @@ -151,7 +151,7 @@ sub get_audit_log { } if (defined($self->{audit})) { - $self->{logger}->writeLogInfo("audit result: " . JSON::XS->new->utf8->encode($self->{audit})); + $self->{logger}->writeLogInfo("audit result: " . JSON::XS->new->encode($self->{audit})); if (defined($self->{markdown})) { $self->md_output(); } diff --git a/contrib/mbi/centreonBIETL b/contrib/mbi/centreonBIETL new file mode 100644 index 0000000..4e666a0 --- /dev/null +++ b/contrib/mbi/centreonBIETL @@ -0,0 +1,382 @@ +#!/usr/bin/perl + +use warnings; +use strict; +use FindBin; +use lib "$FindBin::Bin"; +# to be launched from contrib directory +use lib "$FindBin::Bin/../"; + +gorgone::script::centreonBIETL->new()->run(); + +package gorgone::script::centreonBIETL; + +use strict; +use warnings; +use Data::Dumper; +use gorgone::modules::centreon::mbi::libs::Utils; +use gorgone::standard::misc; +use gorgone::class::http::http; +use JSON::XS; + +use base qw(gorgone::class::script); + +sub new { + my $class = shift; + my $self = $class->SUPER::new( + 'centreonBIETL', + centreon_db_conn => 0, + centstorage_db_conn => 0, + noconfig => 0 + ); + + bless $self, $class; + + $self->{moptions}->{rebuild} = 0; + $self->{moptions}->{daily} = 0; + $self->{moptions}->{import} = 0; + $self->{moptions}->{dimensions} = 0; + $self->{moptions}->{event} = 0; + $self->{moptions}->{perfdata} = 0; + $self->{moptions}->{start} = ''; + $self->{moptions}->{end} = ''; + $self->{moptions}->{create_tables} = 0; + $self->{moptions}->{ignore_databin} = 0; + $self->{moptions}->{centreon_only} = 0; + $self->{moptions}->{nopurge} = 0; + + $self->add_options( + 'url:s' => \$self->{url}, + 'status' => \$self->{status}, + 'r' => \$self->{moptions}->{rebuild}, + 'd' => \$self->{moptions}->{daily}, + 'I' => \$self->{moptions}->{import}, + 'D' => \$self->{moptions}->{dimensions}, + 'E' => \$self->{moptions}->{event}, + 'P' => \$self->{moptions}->{perfdata}, + 's:s' => \$self->{moptions}->{start}, + 'e:s' => \$self->{moptions}->{end}, + 'c' => \$self->{moptions}->{create_tables}, + 'i' => \$self->{moptions}->{ignore_databin}, + 'C' => \$self->{moptions}->{centreon_only}, + 'p' => \$self->{moptions}->{nopurge} + ); + return $self; +} + +sub init { + my $self = shift; + $self->SUPER::init(); + + $self->{url} = 'http://127.0.0.1:8085' if (!defined($self->{url}) || $self->{url} eq ''); + $self->{http} = gorgone::class::http::http->new(logger => $self->{logger}); + + return if (defined($self->{status})); + + my $utils = gorgone::modules::centreon::mbi::libs::Utils->new($self->{logger}); + if ($utils->checkBasicOptions($self->{moptions}) == 1) { + exit(1); + } + + if ($self->{moptions}->{create_tables} == 0 && + $self->{moptions}->{import} == 0 && + $self->{moptions}->{dimensions} == 0 && + $self->{moptions}->{event} == 0 && + $self->{moptions}->{perfdata} == 0) { + $self->{moptions}->{import} = 1; + $self->{moptions}->{dimensions} = 1; + $self->{moptions}->{event} = 1; + $self->{moptions}->{perfdata} = 1; + } +} + +sub json_decode { + my ($self, %options) = @_; + + my $decoded; + eval { + $decoded = JSON::XS->new->decode($options{content}); + }; + if ($@) { + $self->{logger}->writeLogError("cannot decode json response: $@"); + exit(1); + } + + return $decoded; +} + +sub run_etl { + my ($self) = @_; + + my ($code, $content) = $self->{http}->request( + http_backend => 'curl', + method => 'POST', + hostname => '', + full_url => $self->{url} . '/api/centreon/mbietl/run', + query_form_post => JSON::XS->new->encode($self->{moptions}), + header => [ + 'Accept-Type: application/json; charset=utf-8', + 'Content-Type: application/json; charset=utf-8', + ], + curl_opt => ['CURLOPT_SSL_VERIFYPEER => 0', 'CURLOPT_POSTREDIR => CURL_REDIR_POST_ALL'], + warning_status => '', + unknown_status => '', + critical_status => '' + ); + + if ($self->{http}->get_code() < 200 || $self->{http}->get_code() >= 300) { + $self->{logger}->writeLogError("Login error [code: '" . $self->{http}->get_code() . "'] [message: '" . $self->{http}->get_message() . "']"); + exit(1); + } + + my $decoded = $self->json_decode(content => $content); + if (!defined($decoded->{token})) { + $self->{logger}->writeLogError('cannot get token'); + exit(1); + } + + $self->{token} = $decoded->{token}; +} + +sub display_messages { + my ($self, %options) = @_; + + if (defined($options{data}->{messages})) { + foreach (@{$options{data}->{messages}}) { + if ($_->[0] eq 'D') { + $self->{logger}->writeLogDebug($_->[1]) + } elsif ($_->[0] eq 'I') { + $self->{logger}->writeLogInfo($_->[1]); + } elsif ($_->[0] eq 'E') { + $self->{logger}->writeLogError($_->[1]); + } + } + } +} + +sub get_etl_log { + my ($self) = @_; + + my $log_id; + while (1) { + my $get_param = []; + if (defined($log_id)) { + $get_param = ['id=' . $log_id]; + } + + my ($code, $content) = $self->{http}->request( + http_backend => 'curl', + method => 'GET', + hostname => '', + full_url => $self->{url} . '/api/log/' . $self->{token}, + get_param => $get_param, + header => [ + 'Accept-Type: application/json; charset=utf-8' + ], + curl_opt => ['CURLOPT_SSL_VERIFYPEER => 0', 'CURLOPT_POSTREDIR => CURL_REDIR_POST_ALL'], + warning_status => '', + unknown_status => '', + critical_status => '' + ); + + if ($self->{http}->get_code() < 200 || $self->{http}->get_code() >= 300) { + $self->{logger}->writeLogError("Login error [code: '" . $self->{http}->get_code() . "'] [message: '" . $self->{http}->get_message() . "']"); + exit(1); + } + + my $decoded = $self->json_decode(content => $content); + if (!defined($decoded->{data})) { + $self->{logger}->writeLogError("Cannot get log information"); + exit(1); + } + + my $stop = 0; + foreach (@{$decoded->{data}}) { + my $data = $self->json_decode(content => $_->{data}); + next if (defined($log_id) && $log_id >= $_->{id}); + $log_id = $_->{id}; + + if ($_->{code} == 600) { + $self->display_messages(data => $data); + } elsif ($_->{code} == 1) { + $self->display_messages(data => $data); + $stop = 1; + } elsif ($_->{code} == 2) { + $self->display_messages(data => $data); + $stop = 1; + } + } + + last if ($stop == 1); + sleep(2); + } +} + +sub get_etl_status { + my ($self) = @_; + + my ($code, $content) = $self->{http}->request( + http_backend => 'curl', + method => 'GET', + hostname => '', + full_url => $self->{url} . '/api/centreon/mbietl/status', + header => [ + 'Accept-Type: application/json; charset=utf-8', + 'Content-Type: application/json; charset=utf-8', + ], + curl_opt => ['CURLOPT_SSL_VERIFYPEER => 0', 'CURLOPT_POSTREDIR => CURL_REDIR_POST_ALL'], + warning_status => '', + unknown_status => '', + critical_status => '' + ); + + if ($self->{http}->get_code() < 200 || $self->{http}->get_code() >= 300) { + $self->{logger}->writeLogError("Login error [code: '" . $self->{http}->get_code() . "'] [message: '" . $self->{http}->get_message() . "']"); + exit(1); + } + + my $decoded = $self->json_decode(content => $content); + if (!defined($decoded->{token})) { + $self->{logger}->writeLogError('cannot get token'); + exit(1); + } + + my $token = $decoded->{token}; + my $log_id; + my $result; + + while (1) { + my $get_param = []; + if (defined($log_id)) { + $get_param = ['id=' . $log_id]; + } + + my ($code, $content) = $self->{http}->request( + http_backend => 'curl', + method => 'GET', + hostname => '', + full_url => $self->{url} . '/api/log/' . $token, + get_param => $get_param, + header => [ + 'Accept-Type: application/json; charset=utf-8' + ], + curl_opt => ['CURLOPT_SSL_VERIFYPEER => 0', 'CURLOPT_POSTREDIR => CURL_REDIR_POST_ALL'], + warning_status => '', + unknown_status => '', + critical_status => '' + ); + + if ($self->{http}->get_code() < 200 || $self->{http}->get_code() >= 300) { + $self->{logger}->writeLogError("Login error [code: '" . $self->{http}->get_code() . "'] [message: '" . $self->{http}->get_message() . "']"); + exit(1); + } + + my $decoded = $self->json_decode(content => $content); + if (!defined($decoded->{data})) { + $self->{logger}->writeLogError("Cannot get log information"); + exit(1); + } + + my $stop = 0; + foreach (@{$decoded->{data}}) { + my $data = $self->json_decode(content => $_->{data}); + next if (defined($log_id) && $log_id >= $_->{id}); + $log_id = $_->{id}; + + if ($_->{code} == 1) { + $self->{logger}->writeLogError('cannot get etl status'); + exit(1); + } elsif ($_->{code} == 2) { + $result = $data; + $stop = 1; + } + } + + last if ($stop == 1); + sleep(2); + } + + print "ETL status: $result->{statusStr}\n"; + if ($result->{statusStr} ne 'ready') { + print "planning: $result->{planningStr}\n"; + foreach ('import', 'dimensions', 'event', 'perfdata') { + next if (!defined($result->{sections}->{$_})); + + print " $_ status: $result->{sections}->{$_}->{statusStr}"; + if (defined($result->{sections}->{$_}->{steps_total})) { + print " ($result->{sections}->{$_}->{steps_executed}/$result->{sections}->{$_}->{steps_total})"; + } + print "\n"; + } + } +} + +sub run { + my $self = shift; + + $self->SUPER::run(); + + if (defined($self->{status})) { + $self->get_etl_status(); + } else { + $self->run_etl(); + $self->get_etl_log(); + } +} + +__END__ + +=head1 NAME + +centreonBIETL - script to execute mbi etl + +=head1 SYNOPSIS + +centreonBIETL [options] + +=head1 OPTIONS + +=over 8 + +=item B<--url> + +Specify the api url (default: 'http://127.0.0.1:8085'). + +=item B<--severity> + +Set the script log severity (default: 'info'). + +=item B<--help> + +Print a brief help message and exits. + +Execution modes + + -c Create the reporting database model + -d Daily execution to calculate statistics on yesterday + -r Rebuild mode to calculate statitics on a historical period. Can be used with: + Extra arguments for options -d and -r (if none of the following is specified, these one are selected by default: -IDEP): + -I Extract data from the monitoring server + Extra arguments for option -I: + -C Extract only Centreon configuration database only. Works with option -I. + -i Ignore perfdata extraction from monitoring server + -o Extract only perfdata from monitoring server + + -D Calculate dimensions + -E Calculate event and availability statistics + -P Calculate perfdata statistics + Common options for -rIDEP: + -s Start date in format YYYY-MM-DD. + By default, the program uses the data retention period from Centreon BI configuration + -e End date in format YYYY-MM-DD. + By default, the program uses the data retention period from Centreon BI configuration + -p Do not empty statistic tables, delete only entries for the processed period. + Does not work on raw data tables, only on Centreon BI statistics tables. + +=back + +=head1 DESCRIPTION + +B + +=cut diff --git a/contrib/mbi/dimensionBuilder.pl b/contrib/mbi/dimensionBuilder.pl new file mode 100644 index 0000000..1e81760 --- /dev/null +++ b/contrib/mbi/dimensionBuilder.pl @@ -0,0 +1,237 @@ +#!/usr/bin/perl + +use warnings; +use strict; +use FindBin; +use lib "$FindBin::Bin"; +# to be launched from contrib directory +use lib "$FindBin::Bin/../"; + +gorgone::script::dimensionsBuilder->new()->run(); + +package gorgone::script::dimensionsBuilder; + +use strict; +use warnings; +use Data::Dumper; +use gorgone::modules::centreon::mbi::libs::Utils; +use gorgone::standard::misc; +use gorgone::class::http::http; +use JSON::XS; + +use base qw(gorgone::class::script); + +sub new { + my $class = shift; + my $self = $class->SUPER::new( + 'dimensionsBuilder', + centreon_db_conn => 0, + centstorage_db_conn => 0, + noconfig => 0 + ); + + bless $self, $class; + + $self->{moptions}->{rebuild} = 0; + $self->{moptions}->{daily} = 0; + $self->{moptions}->{import} = 0; + $self->{moptions}->{dimensions} = 1; + $self->{moptions}->{event} = 0; + $self->{moptions}->{perfdata} = 0; + $self->{moptions}->{start} = ''; + $self->{moptions}->{end} = ''; + $self->{moptions}->{nopurge} = 0; + $self->{moptions}->{centile} = 0; + + $self->add_options( + 'url:s' => \$self->{url}, + 'r|rebuild' => \$self->{moptions}->{rebuild}, + 'd|daily' => \$self->{moptions}->{daily}, + 'centile' => \$self->{moptions}->{centile}, + 'p|no-purge' => \$self->{moptions}->{nopurge} + ); + return $self; +} + +sub init { + my $self = shift; + $self->SUPER::init(); + + $self->{url} = 'http://127.0.0.1:8085' if (!defined($self->{url}) || $self->{url} eq ''); + $self->{http} = gorgone::class::http::http->new(logger => $self->{logger}); + my $utils = gorgone::modules::centreon::mbi::libs::Utils->new($self->{logger}); + if ($utils->checkBasicOptions($self->{moptions}) == 1) { + exit(1); + } +} + +sub json_decode { + my ($self, %options) = @_; + + my $decoded; + eval { + $decoded = JSON::XS->new->decode($options{content}); + }; + if ($@) { + $self->{logger}->writeLogError("cannot decode json response: $@"); + exit(1); + } + + return $decoded; +} + +sub run_etl { + my ($self) = @_; + + my ($code, $content) = $self->{http}->request( + http_backend => 'curl', + method => 'POST', + hostname => '', + full_url => $self->{url} . '/api/centreon/mbietl/run', + query_form_post => JSON::XS->new->encode($self->{moptions}), + header => [ + 'Accept-Type: application/json; charset=utf-8', + 'Content-Type: application/json; charset=utf-8', + ], + curl_opt => ['CURLOPT_SSL_VERIFYPEER => 0', 'CURLOPT_POSTREDIR => CURL_REDIR_POST_ALL'], + warning_status => '', + unknown_status => '', + critical_status => '' + ); + + if ($self->{http}->get_code() < 200 || $self->{http}->get_code() >= 300) { + $self->{logger}->writeLogError("Login error [code: '" . $self->{http}->get_code() . "'] [message: '" . $self->{http}->get_message() . "']"); + exit(1); + } + + my $decoded = $self->json_decode(content => $content); + if (!defined($decoded->{token})) { + $self->{logger}->writeLogError('cannot get token'); + exit(1); + } + + $self->{token} = $decoded->{token}; +} + +sub display_messages { + my ($self, %options) = @_; + + if (defined($options{data}->{messages})) { + foreach (@{$options{data}->{messages}}) { + if ($_->[0] eq 'D') { + $self->{logger}->writeLogDebug($_->[1]) + } elsif ($_->[0] eq 'I') { + $self->{logger}->writeLogInfo($_->[1]); + } elsif ($_->[0] eq 'E') { + $self->{logger}->writeLogError($_->[1]); + } + } + } +} + +sub get_etl_log { + my ($self) = @_; + + my $log_id; + while (1) { + my $get_param = []; + if (defined($log_id)) { + $get_param = ['id=' . $log_id]; + } + + my ($code, $content) = $self->{http}->request( + http_backend => 'curl', + method => 'GET', + hostname => '', + full_url => $self->{url} . '/api/log/' . $self->{token}, + get_param => $get_param, + header => [ + 'Accept-Type: application/json; charset=utf-8' + ], + curl_opt => ['CURLOPT_SSL_VERIFYPEER => 0', 'CURLOPT_POSTREDIR => CURL_REDIR_POST_ALL'], + warning_status => '', + unknown_status => '', + critical_status => '' + ); + + if ($self->{http}->get_code() < 200 || $self->{http}->get_code() >= 300) { + $self->{logger}->writeLogError("Login error [code: '" . $self->{http}->get_code() . "'] [message: '" . $self->{http}->get_message() . "']"); + exit(1); + } + + my $decoded = $self->json_decode(content => $content); + if (!defined($decoded->{data})) { + $self->{logger}->writeLogError("Cannot get log information"); + exit(1); + } + + my $stop = 0; + foreach (@{$decoded->{data}}) { + my $data = $self->json_decode(content => $_->{data}); + next if (defined($log_id) && $log_id >= $_->{id}); + $log_id = $_->{id}; + + if ($_->{code} == 600) { + $self->display_messages(data => $data); + } elsif ($_->{code} == 1) { + $self->display_messages(data => $data); + $stop = 1; + } elsif ($_->{code} == 2) { + $self->display_messages(data => $data); + $stop = 1; + } + } + + last if ($stop == 1); + sleep(2); + } +} + +sub run { + my $self = shift; + + $self->SUPER::run(); + $self->run_etl(); + $self->get_etl_log(); +} + +__END__ + +=head1 NAME + +dimensionsBuilder.pl - script to compute dimensions + +=head1 SYNOPSIS + +dimensionsBuilder.pl [options] + +=head1 OPTIONS + +=over 8 + +=item B<--url> + +Specify the api url (default: 'http://127.0.0.1:8085'). + +=item B<--severity> + +Set the script log severity (default: 'info'). + +=item B<--help> + +Print a brief help message and exits. + +=back + + Rebuild options: + [-r|--rebuild] : Rebuild dimensions + [--no-purge] : Do not delete previous dimensions while rebuilding + [--centile] : import only centile dimensions without deleting other dimensions + Daily run options: + [-d|--daily] + +=head1 DESCRIPTION + +B + +=cut diff --git a/contrib/mbi/eventStatisticsBuilder.pl b/contrib/mbi/eventStatisticsBuilder.pl new file mode 100644 index 0000000..992d454 --- /dev/null +++ b/contrib/mbi/eventStatisticsBuilder.pl @@ -0,0 +1,247 @@ +#!/usr/bin/perl + +use warnings; +use strict; +use FindBin; +use lib "$FindBin::Bin"; +# to be launched from contrib directory +use lib "$FindBin::Bin/../"; + +gorgone::script::eventStatisticsBuilder->new()->run(); + +package gorgone::script::eventStatisticsBuilder; + +use strict; +use warnings; +use Data::Dumper; +use gorgone::modules::centreon::mbi::libs::Utils; +use gorgone::standard::misc; +use gorgone::class::http::http; +use JSON::XS; + +use base qw(gorgone::class::script); + +sub new { + my $class = shift; + my $self = $class->SUPER::new( + 'eventStatisticsBuilder', + centreon_db_conn => 0, + centstorage_db_conn => 0, + noconfig => 0 + ); + + bless $self, $class; + + $self->{moptions}->{rebuild} = 0; + $self->{moptions}->{daily} = 0; + $self->{moptions}->{import} = 0; + $self->{moptions}->{dimensions} = 0; + $self->{moptions}->{event} = 1; + $self->{moptions}->{perfdata} = 0; + $self->{moptions}->{start} = ''; + $self->{moptions}->{end} = ''; + $self->{moptions}->{nopurge} = 0; + $self->{moptions}->{host_only} = 0; + $self->{moptions}->{service_only} = 0; + $self->{moptions}->{availability_only} = 0; + $self->{moptions}->{events_only} = 0; + + $self->add_options( + 'url:s' => \$self->{url}, + 'r|rebuild' => \$self->{moptions}->{rebuild}, + 'd|daily' => \$self->{moptions}->{daily}, + 's:s' => \$self->{moptions}->{start}, + 'e:s' => \$self->{moptions}->{end}, + 'host-only' => \$self->{moptions}->{host_only}, + 'service-only' => \$self->{moptions}->{service_only}, + 'availability-only' => \$self->{moptions}->{availability_only}, + 'events-only' => \$self->{moptions}->{events_only} + ); + return $self; +} + +sub init { + my $self = shift; + $self->SUPER::init(); + + $self->{url} = 'http://127.0.0.1:8085' if (!defined($self->{url}) || $self->{url} eq ''); + $self->{http} = gorgone::class::http::http->new(logger => $self->{logger}); + my $utils = gorgone::modules::centreon::mbi::libs::Utils->new($self->{logger}); + if ($utils->checkBasicOptions($self->{moptions}) == 1) { + exit(1); + } +} + +sub json_decode { + my ($self, %options) = @_; + + my $decoded; + eval { + $decoded = JSON::XS->new->decode($options{content}); + }; + if ($@) { + $self->{logger}->writeLogError("cannot decode json response: $@"); + exit(1); + } + + return $decoded; +} + +sub run_etl { + my ($self) = @_; + + my ($code, $content) = $self->{http}->request( + http_backend => 'curl', + method => 'POST', + hostname => '', + full_url => $self->{url} . '/api/centreon/mbietl/run', + query_form_post => JSON::XS->new->encode($self->{moptions}), + header => [ + 'Accept-Type: application/json; charset=utf-8', + 'Content-Type: application/json; charset=utf-8', + ], + curl_opt => ['CURLOPT_SSL_VERIFYPEER => 0', 'CURLOPT_POSTREDIR => CURL_REDIR_POST_ALL'], + warning_status => '', + unknown_status => '', + critical_status => '' + ); + + if ($self->{http}->get_code() < 200 || $self->{http}->get_code() >= 300) { + $self->{logger}->writeLogError("Login error [code: '" . $self->{http}->get_code() . "'] [message: '" . $self->{http}->get_message() . "']"); + exit(1); + } + + my $decoded = $self->json_decode(content => $content); + if (!defined($decoded->{token})) { + $self->{logger}->writeLogError('cannot get token'); + exit(1); + } + + $self->{token} = $decoded->{token}; +} + +sub display_messages { + my ($self, %options) = @_; + + if (defined($options{data}->{messages})) { + foreach (@{$options{data}->{messages}}) { + if ($_->[0] eq 'D') { + $self->{logger}->writeLogDebug($_->[1]) + } elsif ($_->[0] eq 'I') { + $self->{logger}->writeLogInfo($_->[1]); + } elsif ($_->[0] eq 'E') { + $self->{logger}->writeLogError($_->[1]); + } + } + } +} + +sub get_etl_log { + my ($self) = @_; + + my $log_id; + while (1) { + my $get_param = []; + if (defined($log_id)) { + $get_param = ['id=' . $log_id]; + } + + my ($code, $content) = $self->{http}->request( + http_backend => 'curl', + method => 'GET', + hostname => '', + full_url => $self->{url} . '/api/log/' . $self->{token}, + get_param => $get_param, + header => [ + 'Accept-Type: application/json; charset=utf-8' + ], + curl_opt => ['CURLOPT_SSL_VERIFYPEER => 0', 'CURLOPT_POSTREDIR => CURL_REDIR_POST_ALL'], + warning_status => '', + unknown_status => '', + critical_status => '' + ); + + if ($self->{http}->get_code() < 200 || $self->{http}->get_code() >= 300) { + $self->{logger}->writeLogError("Login error [code: '" . $self->{http}->get_code() . "'] [message: '" . $self->{http}->get_message() . "']"); + exit(1); + } + + my $decoded = $self->json_decode(content => $content); + if (!defined($decoded->{data})) { + $self->{logger}->writeLogError("Cannot get log information"); + exit(1); + } + + my $stop = 0; + foreach (@{$decoded->{data}}) { + my $data = $self->json_decode(content => $_->{data}); + next if (defined($log_id) && $log_id >= $_->{id}); + $log_id = $_->{id}; + + if ($_->{code} == 600) { + $self->display_messages(data => $data); + } elsif ($_->{code} == 1) { + $self->display_messages(data => $data); + $stop = 1; + } elsif ($_->{code} == 2) { + $self->display_messages(data => $data); + $stop = 1; + } + } + + last if ($stop == 1); + sleep(2); + } +} + +sub run { + my $self = shift; + + $self->SUPER::run(); + $self->run_etl(); + $self->get_etl_log(); +} + +__END__ + +=head1 NAME + +eventStatisticsBuilder.pl - script to calculate events and availbility statistics + +=head1 SYNOPSIS + +eventStatisticsBuilder.pl [options] + +=head1 OPTIONS + +=over 8 + +=item B<--url> + +Specify the api url (default: 'http://127.0.0.1:8085'). + +=item B<--severity> + +Set the script log severity (default: 'info'). + +=item B<--help> + +Print a brief help message and exits. + +=back + + Rebuild options: + [-s|--start] [-e|--end] [-r|--rebuild] [--no-purge] + Daily run options: + [-d|--daily] + Other options:\n"; + --host-only Process only host events and availability statistics + --service-only Process only service events and availability statistics + --availability-only Build only availability statistics + --events-only Build only event statistics + +=head1 DESCRIPTION + +B + +=cut diff --git a/contrib/mbi/importData.pl b/contrib/mbi/importData.pl new file mode 100644 index 0000000..82e429c --- /dev/null +++ b/contrib/mbi/importData.pl @@ -0,0 +1,250 @@ +#!/usr/bin/perl + +use warnings; +use strict; +use FindBin; +use lib "$FindBin::Bin"; +# to be launched from contrib directory +use lib "$FindBin::Bin/../"; + +gorgone::script::importData->new()->run(); + +package gorgone::script::importData; + +use strict; +use warnings; +use Data::Dumper; +use gorgone::modules::centreon::mbi::libs::Utils; +use gorgone::standard::misc; +use gorgone::class::http::http; +use JSON::XS; + +use base qw(gorgone::class::script); + +sub new { + my $class = shift; + my $self = $class->SUPER::new( + 'importData', + centreon_db_conn => 0, + centstorage_db_conn => 0, + noconfig => 0 + ); + + bless $self, $class; + + $self->{moptions}->{rebuild} = 0; + $self->{moptions}->{daily} = 0; + $self->{moptions}->{import} = 1; + $self->{moptions}->{dimensions} = 0; + $self->{moptions}->{event} = 0; + $self->{moptions}->{perfdata} = 0; + $self->{moptions}->{start} = ''; + $self->{moptions}->{end} = ''; + $self->{moptions}->{create_tables} = 0; + $self->{moptions}->{databin_only} = 0; + $self->{moptions}->{ignore_databin} = 0; + $self->{moptions}->{centreon_only} = 0; + $self->{moptions}->{nopurge} = 0; + $self->{moptions}->{bam_only} = 0; + + $self->add_options( + 'url:s' => \$self->{url}, + 'r|rebuild' => \$self->{moptions}->{rebuild}, + 'd|daily' => \$self->{moptions}->{daily}, + 's:s' => \$self->{moptions}->{start}, + 'e:s' => \$self->{moptions}->{end}, + 'c|create-tables' => \$self->{moptions}->{create_tables}, + 'databin-only' => \$self->{moptions}->{databin_only}, + 'i|ignore-databin' => \$self->{moptions}->{ignore_databin}, + 'C|centreon-only' => \$self->{moptions}->{centreon_only}, + 'p|no-purge' => \$self->{moptions}->{nopurge}, + 'bam-only' => \$self->{moptions}->{bam_only} + ); + return $self; +} + +sub init { + my $self = shift; + $self->SUPER::init(); + + $self->{url} = 'http://127.0.0.1:8085' if (!defined($self->{url}) || $self->{url} eq ''); + $self->{http} = gorgone::class::http::http->new(logger => $self->{logger}); + my $utils = gorgone::modules::centreon::mbi::libs::Utils->new($self->{logger}); + if ($utils->checkBasicOptions($self->{moptions}) == 1) { + exit(1); + } +} + +sub json_decode { + my ($self, %options) = @_; + + my $decoded; + eval { + $decoded = JSON::XS->new->decode($options{content}); + }; + if ($@) { + $self->{logger}->writeLogError("cannot decode json response: $@"); + exit(1); + } + + return $decoded; +} + +sub run_etl { + my ($self) = @_; + + my ($code, $content) = $self->{http}->request( + http_backend => 'curl', + method => 'POST', + hostname => '', + full_url => $self->{url} . '/api/centreon/mbietl/run', + query_form_post => JSON::XS->new->encode($self->{moptions}), + header => [ + 'Accept-Type: application/json; charset=utf-8', + 'Content-Type: application/json; charset=utf-8', + ], + curl_opt => ['CURLOPT_SSL_VERIFYPEER => 0', 'CURLOPT_POSTREDIR => CURL_REDIR_POST_ALL'], + warning_status => '', + unknown_status => '', + critical_status => '' + ); + + if ($self->{http}->get_code() < 200 || $self->{http}->get_code() >= 300) { + $self->{logger}->writeLogError("Login error [code: '" . $self->{http}->get_code() . "'] [message: '" . $self->{http}->get_message() . "']"); + exit(1); + } + + my $decoded = $self->json_decode(content => $content); + if (!defined($decoded->{token})) { + $self->{logger}->writeLogError('cannot get token'); + exit(1); + } + + $self->{token} = $decoded->{token}; +} + +sub display_messages { + my ($self, %options) = @_; + + if (defined($options{data}->{messages})) { + foreach (@{$options{data}->{messages}}) { + if ($_->[0] eq 'D') { + $self->{logger}->writeLogDebug($_->[1]) + } elsif ($_->[0] eq 'I') { + $self->{logger}->writeLogInfo($_->[1]); + } elsif ($_->[0] eq 'E') { + $self->{logger}->writeLogError($_->[1]); + } + } + } +} + +sub get_etl_log { + my ($self) = @_; + + my $log_id; + while (1) { + my $get_param = []; + if (defined($log_id)) { + $get_param = ['id=' . $log_id]; + } + + my ($code, $content) = $self->{http}->request( + http_backend => 'curl', + method => 'GET', + hostname => '', + full_url => $self->{url} . '/api/log/' . $self->{token}, + get_param => $get_param, + header => [ + 'Accept-Type: application/json; charset=utf-8' + ], + curl_opt => ['CURLOPT_SSL_VERIFYPEER => 0', 'CURLOPT_POSTREDIR => CURL_REDIR_POST_ALL'], + warning_status => '', + unknown_status => '', + critical_status => '' + ); + + if ($self->{http}->get_code() < 200 || $self->{http}->get_code() >= 300) { + $self->{logger}->writeLogError("Login error [code: '" . $self->{http}->get_code() . "'] [message: '" . $self->{http}->get_message() . "']"); + exit(1); + } + + my $decoded = $self->json_decode(content => $content); + if (!defined($decoded->{data})) { + $self->{logger}->writeLogError("Cannot get log information"); + exit(1); + } + + my $stop = 0; + foreach (@{$decoded->{data}}) { + my $data = $self->json_decode(content => $_->{data}); + next if (defined($log_id) && $log_id >= $_->{id}); + $log_id = $_->{id}; + + if ($_->{code} == 600) { + $self->display_messages(data => $data); + } elsif ($_->{code} == 1) { + $self->display_messages(data => $data); + $stop = 1; + } elsif ($_->{code} == 2) { + $self->display_messages(data => $data); + $stop = 1; + } + } + + last if ($stop == 1); + sleep(2); + } +} + +sub run { + my $self = shift; + + $self->SUPER::run(); + $self->run_etl(); + $self->get_etl_log(); +} + +__END__ + +=head1 NAME + +importData.pl - script to execute import centreon datas + +=head1 SYNOPSIS + +importData.pl [options] + +=head1 OPTIONS + +=over 8 + +=item B<--url> + +Specify the api url (default: 'http://127.0.0.1:8085'). + +=item B<--severity> + +Set the script log severity (default: 'info'). + +=item B<--help> + +Print a brief help message and exits. + +=back + + First run + [-c|--create-tables] + Rebuild options: + [-r|--rebuild] [--databin-only] [--centreon-only] [--ignore-databin] [--bam-only] + [-s|--start] [-e|--end] Not mandatory : if you don't use these options, the retention parameters will be taken into account + [--no-purge] Only use this mode with rebuild mode to import missing data. + This command may create duplicate entries if executed on a non appropriate period + Daily run options: + [-d|--daily] + +=head1 DESCRIPTION + +B + +=cut diff --git a/contrib/mbi/perfdataStatisticsBuilder.pl b/contrib/mbi/perfdataStatisticsBuilder.pl new file mode 100644 index 0000000..801e004 --- /dev/null +++ b/contrib/mbi/perfdataStatisticsBuilder.pl @@ -0,0 +1,240 @@ +#!/usr/bin/perl + +use warnings; +use strict; +use FindBin; +use lib "$FindBin::Bin"; +# to be launched from contrib directory +use lib "$FindBin::Bin/../"; + +gorgone::script::perfdataStatisticsBuilder->new()->run(); + +package gorgone::script::perfdataStatisticsBuilder; + +use strict; +use warnings; +use Data::Dumper; +use gorgone::modules::centreon::mbi::libs::Utils; +use gorgone::standard::misc; +use gorgone::class::http::http; +use JSON::XS; + +use base qw(gorgone::class::script); + +sub new { + my $class = shift; + my $self = $class->SUPER::new( + 'perfdataStatisticsBuilder', + centreon_db_conn => 0, + centstorage_db_conn => 0, + noconfig => 0 + ); + + bless $self, $class; + + $self->{moptions}->{rebuild} = 0; + $self->{moptions}->{daily} = 0; + $self->{moptions}->{import} = 0; + $self->{moptions}->{dimensions} = 0; + $self->{moptions}->{event} = 0; + $self->{moptions}->{perfdata} = 1; + $self->{moptions}->{start} = ''; + $self->{moptions}->{end} = ''; + $self->{moptions}->{nopurge} = 0; + $self->{moptions}->{month_only} = 0; + $self->{moptions}->{centile_only} = 0; + $self->{moptions}->{no_centile} = 0; + + $self->add_options( + 'url:s' => \$self->{url}, + 'r|rebuild' => \$self->{moptions}->{rebuild}, + 'd|daily' => \$self->{moptions}->{daily}, + 's:s' => \$self->{moptions}->{start}, + 'e:s' => \$self->{moptions}->{end}, + 'month-only' => \$self->{moptions}->{month_only}, + 'centile-only' => \$self->{moptions}->{centile_only}, + 'no-centile' => \$self->{moptions}->{no_centile} + ); + return $self; +} + +sub init { + my $self = shift; + $self->SUPER::init(); + + $self->{url} = 'http://127.0.0.1:8085' if (!defined($self->{url}) || $self->{url} eq ''); + $self->{http} = gorgone::class::http::http->new(logger => $self->{logger}); + my $utils = gorgone::modules::centreon::mbi::libs::Utils->new($self->{logger}); + if ($utils->checkBasicOptions($self->{moptions}) == 1) { + exit(1); + } +} + +sub json_decode { + my ($self, %options) = @_; + + my $decoded; + eval { + $decoded = JSON::XS->new->decode($options{content}); + }; + if ($@) { + $self->{logger}->writeLogError("cannot decode json response: $@"); + exit(1); + } + + return $decoded; +} + +sub run_etl { + my ($self) = @_; + + my ($code, $content) = $self->{http}->request( + http_backend => 'curl', + method => 'POST', + hostname => '', + full_url => $self->{url} . '/api/centreon/mbietl/run', + query_form_post => JSON::XS->new->encode($self->{moptions}), + header => [ + 'Accept-Type: application/json; charset=utf-8', + 'Content-Type: application/json; charset=utf-8', + ], + curl_opt => ['CURLOPT_SSL_VERIFYPEER => 0', 'CURLOPT_POSTREDIR => CURL_REDIR_POST_ALL'], + warning_status => '', + unknown_status => '', + critical_status => '' + ); + + if ($self->{http}->get_code() < 200 || $self->{http}->get_code() >= 300) { + $self->{logger}->writeLogError("Login error [code: '" . $self->{http}->get_code() . "'] [message: '" . $self->{http}->get_message() . "']"); + exit(1); + } + + my $decoded = $self->json_decode(content => $content); + if (!defined($decoded->{token})) { + $self->{logger}->writeLogError('cannot get token'); + exit(1); + } + + $self->{token} = $decoded->{token}; +} + +sub display_messages { + my ($self, %options) = @_; + + if (defined($options{data}->{messages})) { + foreach (@{$options{data}->{messages}}) { + if ($_->[0] eq 'D') { + $self->{logger}->writeLogDebug($_->[1]) + } elsif ($_->[0] eq 'I') { + $self->{logger}->writeLogInfo($_->[1]); + } elsif ($_->[0] eq 'E') { + $self->{logger}->writeLogError($_->[1]); + } + } + } +} + +sub get_etl_log { + my ($self) = @_; + + my $log_id; + while (1) { + my $get_param = []; + if (defined($log_id)) { + $get_param = ['id=' . $log_id]; + } + + my ($code, $content) = $self->{http}->request( + http_backend => 'curl', + method => 'GET', + hostname => '', + full_url => $self->{url} . '/api/log/' . $self->{token}, + get_param => $get_param, + header => [ + 'Accept-Type: application/json; charset=utf-8' + ], + curl_opt => ['CURLOPT_SSL_VERIFYPEER => 0', 'CURLOPT_POSTREDIR => CURL_REDIR_POST_ALL'], + warning_status => '', + unknown_status => '', + critical_status => '' + ); + + if ($self->{http}->get_code() < 200 || $self->{http}->get_code() >= 300) { + $self->{logger}->writeLogError("Login error [code: '" . $self->{http}->get_code() . "'] [message: '" . $self->{http}->get_message() . "']"); + exit(1); + } + + my $decoded = $self->json_decode(content => $content); + if (!defined($decoded->{data})) { + $self->{logger}->writeLogError("Cannot get log information"); + exit(1); + } + + my $stop = 0; + foreach (@{$decoded->{data}}) { + my $data = $self->json_decode(content => $_->{data}); + next if (defined($log_id) && $log_id >= $_->{id}); + $log_id = $_->{id}; + + if ($_->{code} == 600) { + $self->display_messages(data => $data); + } elsif ($_->{code} == 1) { + $self->display_messages(data => $data); + $stop = 1; + } elsif ($_->{code} == 2) { + $self->display_messages(data => $data); + $stop = 1; + } + } + + last if ($stop == 1); + sleep(2); + } +} + +sub run { + my $self = shift; + + $self->SUPER::run(); + $self->run_etl(); + $self->get_etl_log(); +} + +__END__ + +=head1 NAME + +perfdataStatisticsBuilder.pl - script to calculate perfdata statistics + +=head1 SYNOPSIS + +perfdataStatisticsBuilder.pl [options] + +=head1 OPTIONS + +=over 8 + +=item B<--url> + +Specify the api url (default: 'http://127.0.0.1:8085'). + +=item B<--severity> + +Set the script log severity (default: 'info'). + +=item B<--help> + +Print a brief help message and exits. + +=back + + Rebuild options: + [-r | --rebuild] [-s|--start] [-e|--end] [--no-purge] [--month-only] [--centile-only] [--no-centile] + Daily run options: + [-d | --daily] + +=head1 DESCRIPTION + +B + +=cut diff --git a/docs/configuration.md b/docs/configuration.md index 0b0fa45..0789429 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -42,12 +42,11 @@ configuration: | :-------------------- | :---------------------------------------------------------------------- | :--------------------------------------------- | | internal_com_type | Type of the internal ZMQ socket | `ipc` | | internal_com_path | Path to the internal ZMQ socket | `/tmp/gorgone/routing.ipc` | -| internal_com_crypt | Internal communication crypt enabled | `false` | +| internal_com_crypt | Crypt internal communication | `true` | | internal_com_cipher | Internal communication cipher | `AES` | | internal_com_padding | Internal communication padding | `1` (mean: PKCS5) | | internal_com_keysize | Internal communication key size | `32` (bytes) | | internal_com_rotation | Internal communication time before key rotation | `1440` (minutes) | -| internal_com_crypt | Crypt internal communication | `false` | | external_com_type | Type of the external ZMQ socket | `tcp` | | external_com_path | Path to the external ZMQ socket | `*:5555` | | external_com_cipher | Cipher used for encryption | `AES` | diff --git a/docs/modules.md b/docs/modules.md index 56ee20e..fb1f1ee 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -12,7 +12,7 @@ List of the available modules: * [Register](../docs/modules/core/register.md) * Centreon * [Autodiscovery](../docs/modules/centreon/autodiscovery.md) - * [Broker](../docs/modules/centreon/broker.md) + * [Broker](../docs/modules/centreon/statistics.md) * [Engine](../docs/modules/centreon/engine.md) * [Legacy Cmd](../docs/modules/centreon/legacycmd.md) * [Nodes](../docs/modules/centreon/nodes.md) diff --git a/docs/modules/core/action.md b/docs/modules/core/action.md index 79a362b..fc1e298 100644 --- a/docs/modules/core/action.md +++ b/docs/modules/core/action.md @@ -6,11 +6,12 @@ This module aims to execute actions on the server running the Gorgone daemon or ## Configuration -| Directive | Description | Default value | -| :-------------- | :------------------------------------------------------- | :------------ | -| command_timeout | Time in seconds before a command is considered timed out | `30` | -| whitelist_cmds | Boolean to enable commands whitelist | `false` | -| allowed_cmds | Regexp list of allowed commands | | +| Directive | Description | Default value | +| :--------------- | :------------------------------------------------------------- | :------------ | +| command_timeout | Time in seconds before a command is considered timed out | `30` | +| whitelist_cmds | Boolean to enable commands whitelist | `false` | +| allowed_cmds | Regexp list of allowed commands | | +| paranoid_plugins | Block centengine restart/reload if plugin dependencies missing | `false` | #### Example diff --git a/gorgone/class/core.pm b/gorgone/class/core.pm index d8c8f89..ecf89fc 100644 --- a/gorgone/class/core.pm +++ b/gorgone/class/core.pm @@ -92,7 +92,7 @@ sub init_server_keys { $self->{keys_loaded} = 0; $self->{config} = { configuration => {} } if (!defined($self->{config}->{configuration})); $self->{config}->{configuration} = { gorgone => {} } if (!defined($self->{config}->{configuration}->{gorgone})); - $self->{config}->{configuration}->{gorgone} = { gorgonecore => {} } if (!defined($self->{config}->{configuration}->{gorgone}->{gorgonecore})); + $self->{config}->{configuration}->{gorgone}->{gorgonecore} = {} if (!defined($self->{config}->{configuration}->{gorgone}->{gorgonecore})); $self->{config}->{configuration}->{gorgone}->{gorgonecore}->{privkey} = '/var/lib/centreon-gorgone/.keys/rsakey.priv.pem' if (!defined($self->{config}->{configuration}->{gorgone}->{gorgonecore}->{privkey}) || $self->{config}->{configuration}->{gorgone}->{gorgonecore}->{privkey} eq ''); @@ -179,10 +179,10 @@ sub init { $self->{config}->{configuration}->{gorgone}->{gorgonecore}->{internal_com_path} = '/tmp/gorgone/routing-' . $time_hi . '.ipc' if (!defined($self->{config}->{configuration}->{gorgone}->{gorgonecore}->{internal_com_path}) || $self->{config}->{configuration}->{gorgone}->{gorgonecore}->{internal_com_path} eq ''); - if (defined($self->{config}->{configuration}->{gorgone}->{gorgonecore}->{internal_com_crypt}) && $self->{config}->{configuration}->{gorgone}->{gorgonecore}->{internal_com_crypt} =~ /^(?:true|1)$/i) { - $self->{config}->{configuration}->{gorgone}->{gorgonecore}->{internal_com_crypt} = 1; - } else { + if (defined($self->{config}->{configuration}->{gorgone}->{gorgonecore}->{internal_com_crypt}) && $self->{config}->{configuration}->{gorgone}->{gorgonecore}->{internal_com_crypt} =~ /^(?:false|0)$/i) { $self->{config}->{configuration}->{gorgone}->{gorgonecore}->{internal_com_crypt} = 0; + } else { + $self->{config}->{configuration}->{gorgone}->{gorgonecore}->{internal_com_crypt} = 1; } $self->{internal_crypt} = { enabled => 0 }; diff --git a/gorgone/class/db.pm b/gorgone/class/db.pm index f35acdf..0fc2cc8 100644 --- a/gorgone/class/db.pm +++ b/gorgone/class/db.pm @@ -46,6 +46,7 @@ sub new { $self->{dsn} =~ s/"\s*$//; } + $self->{die} = defined($options{die}) ? 1 : 0; $self->{instance} = undef; $self->{transaction_begin} = 0; $self->{args} = []; @@ -62,6 +63,12 @@ sub type { return $self->{type}; } +sub getInstance { + my ($self) = @_; + + return $self->{instance}; +} + # Getter/Setter DB name sub db { my $self = shift; @@ -71,6 +78,28 @@ sub db { return $self->{db}; } +sub sameParams { + my ($self, %options) = @_; + + my $params = ''; + if (defined($self->{dsn})) { + $params = $self->{dsn}; + } else { + $params = $self->{host} . ':' . $self->{port} . ':' . $self->{db}; + } + $params .= ':' . $self->{user} . ':' . $self->{password}; + + my $paramsNew = ''; + if (defined($options{dsn})) { + $paramsNew = $options{dsn}; + } else { + $paramsNew = $options{host} . ':' . $options{port} . ':' . $options{db}; + } + $params .= ':' . $options{user} . ':' . $options{password}; + + return ($paramsNew eq $params) ? 1 : 0; +} + # Getter/Setter DB host sub host { my $self = shift; @@ -254,6 +283,8 @@ sub connect() { (defined($self->{db}) ? $self->{db} : $self->{dsn}) . "': " . $DBI::errstr . " (caller: $package:$filename:$line) (try: $count)" ); if ($self->{force} == 0 || ($self->{force} == 2 && $count == 1)) { + $self->{lastError} = "MySQL error : cannot connect to database '" . + (defined($self->{db}) ? $self->{db} : $self->{dsn}) . "': " . $DBI::errstr; $status = -1; last; } @@ -293,10 +324,10 @@ sub error { my ($package, $filename, $line) = caller 1; chomp($query); - $self->{logger}->writeLogError(<<"EOE"); -SQL error: $error (caller: $package:$filename:$line) + $self->{lastError} = "SQL error: $error (caller: $package:$filename:$line) Query: $query -EOE +"; + $self->{logger}->writeLogError($error); if ($self->{transaction_begin} == 1) { $self->rollback(); } @@ -304,6 +335,12 @@ EOE $self->{instance} = undef; } +sub prepare { + my ($self, $query) = @_; + + return $self->query($query, prepare_only => 1); +} + sub query { my $self = shift; my $query = shift; @@ -338,6 +375,7 @@ sub query { } if (defined($options{prepare_only})) { + return $statement_handle if ($self->{die} == 1); return ($status, $statement_handle); } @@ -353,6 +391,11 @@ sub query { last; } + if ($self->{die} == 1) { + die $self->{lastError} if ($status == -1); + return $statement_handle; + } + return ($status, $statement_handle); } diff --git a/gorgone/class/http/backend/curl.pm b/gorgone/class/http/backend/curl.pm index 3c12e82..f2801ba 100644 --- a/gorgone/class/http/backend/curl.pm +++ b/gorgone/class/http/backend/curl.pm @@ -93,12 +93,14 @@ my $http_code_explained = { 415 => 'Unsupported Media Type', 416 => 'Requested Range Not Satisfiable', 417 => 'Expectation Failed', + 450 => 'Timeout reached', # custom code + 451 => 'Failed Connection Host', # custom code 500 => 'Internal Server Error', 501 => 'Not Implemented', 502 => 'Bad Gateway', 503 => 'Service Unavailable', 504 => 'Gateway Timeout', - 505 => 'HTTP Version Not Supported', + 505 => 'HTTP Version Not Supported' }; sub cb_debug { @@ -342,10 +344,25 @@ sub request { $self->curl_setopt(option => $self->{constant_cb}->(name => 'CURLOPT_HEADERFUNCTION'), parameter => \&cb_get_header); eval { + $SIG{__DIE__} = sub {}; + $self->{curl_easy}->perform(); }; if ($@) { - $self->{logger}->writeLogError('curl perform error : ' . $@); + my $err = $@; + if (ref($@) eq "Net::Curl::Easy::Code") { + my $num = $@; + if ($num == $self->{constant_cb}->(name => 'CURLE_OPERATION_TIMEDOUT')) { + $self->{response_code} = 450; + } elsif ($num == $self->{constant_cb}->(name => 'CURLE_COULDNT_CONNECT')) { + $self->{response_code} = 451; + } + } + + if (!defined($self->{response_code})) { + $self->{logger}->writeLogError('curl perform error : ' . $err); + } + return 1; } diff --git a/gorgone/class/listener.pm b/gorgone/class/listener.pm index 5047199..4c52364 100644 --- a/gorgone/class/listener.pm +++ b/gorgone/class/listener.pm @@ -24,7 +24,6 @@ use strict; use warnings; use gorgone::standard::constants qw(:all); use gorgone::standard::library; -use Encode; sub new { my ($class, %options) = @_; @@ -49,16 +48,9 @@ sub event_log { delete $self->{tokens}->{ $options{token} }; } - my $encoded = 0; foreach (keys %{$events->{events}}) { $self->{logger}->writeLogDebug("[listener] trigger event '$options{token}'"); - # it will be decoded by module (hooks.pm). So in some case, we want to avoid double utf8 decode - if ($encoded == 0 && defined($options{encode_utf8}) && $options{encode_utf8} == 1) { - $options{data} = Encode::encode('UTF-8', $options{data}); - $encoded = 1; - } - $self->{gorgone_core}->message_run( message => '[' . $_ . '] [' . $options{token} . '] [] { "code": ' . $options{code} . ', "data": ' . $options{data} . ' }', router_type => 'internal' diff --git a/gorgone/class/logger.pm b/gorgone/class/logger.pm index c169033..f5ea65e 100644 --- a/gorgone/class/logger.pm +++ b/gorgone/class/logger.pm @@ -210,7 +210,6 @@ sub writeLog { return; } - $newmsg = encode('UTF-8', $newmsg); chomp($newmsg); if ($self->{log_mode} == 0) { print "$newmsg\n"; diff --git a/gorgone/class/module.pm b/gorgone/class/module.pm index fc9835e..1672197 100644 --- a/gorgone/class/module.pm +++ b/gorgone/class/module.pm @@ -247,7 +247,7 @@ sub json_encode { my $encoded_arguments; eval { - $encoded_arguments = JSON::XS->new->utf8->encode($options{argument}); + $encoded_arguments = JSON::XS->new->encode($options{argument}); }; if ($@) { $self->{logger}->writeLogError("[$self->{module_id}]$self->{container} $options{method} - cannot encode json: $@"); @@ -262,7 +262,7 @@ sub json_decode { my $decoded_arguments; eval { - $decoded_arguments = JSON::XS->new->utf8->decode($options{argument}); + $decoded_arguments = JSON::XS->new->decode($options{argument}); }; if ($@) { $self->{logger}->writeLogError("[$self->{module_id}]$self->{container} $options{method} - cannot decode json: $@"); diff --git a/gorgone/class/tpapi/centreonv2.pm b/gorgone/class/tpapi/centreonv2.pm index d3939b1..9bd5d84 100644 --- a/gorgone/class/tpapi/centreonv2.pm +++ b/gorgone/class/tpapi/centreonv2.pm @@ -42,7 +42,7 @@ sub json_decode { my $decoded; eval { - $decoded = JSON::XS->new->utf8->decode($options{content}); + $decoded = JSON::XS->new->decode($options{content}); }; if ($@) { $self->{is_error} = 1; diff --git a/gorgone/modules/centreon/audit/metrics/centreon/packages.pm b/gorgone/modules/centreon/audit/metrics/centreon/packages.pm index 753aaf5..a8bacc1 100644 --- a/gorgone/modules/centreon/audit/metrics/centreon/packages.pm +++ b/gorgone/modules/centreon/audit/metrics/centreon/packages.pm @@ -79,7 +79,7 @@ sub metrics { if ($options{os} =~ /Debian|Ubuntu/i) { dpkg_list(metrics => $metrics); - } elsif ($options{os} =~ /CentOS|Redhat|rhel|almalinux/i) { + } elsif ($options{os} =~ /CentOS|Redhat|rhel|almalinux|rocky/i) { rpm_list(metrics => $metrics); } elsif ($options{os} eq 'ol' || $options{os} =~ /Oracle Linux/i) { rpm_list(metrics => $metrics); diff --git a/gorgone/modules/centreon/legacycmd/class.pm b/gorgone/modules/centreon/legacycmd/class.pm index f5f9add..463eef1 100644 --- a/gorgone/modules/centreon/legacycmd/class.pm +++ b/gorgone/modules/centreon/legacycmd/class.pm @@ -615,9 +615,9 @@ sub action_addimporttaskwithparent { sub move_cmd_file { my ($self, %options) = @_; - my $operator = '+<:encoding(UTF-8)'; + my $operator = '+<'; if ($self->{config}->{dirty_mode} == 1) { - $operator = '<:encoding(UTF-8)'; + $operator = '<'; } my $handle; if (-e $options{dst}) { diff --git a/gorgone/modules/centreon/mbi/etl/class.pm b/gorgone/modules/centreon/mbi/etl/class.pm new file mode 100644 index 0000000..d957546 --- /dev/null +++ b/gorgone/modules/centreon/mbi/etl/class.pm @@ -0,0 +1,902 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::etl::class; + +use base qw(gorgone::class::module); + +use strict; +use warnings; +use gorgone::standard::library; +use gorgone::standard::constants qw(:all); +use gorgone::class::sqlquery; +use gorgone::class::http::http; +use ZMQ::LibZMQ4; +use ZMQ::Constants qw(:all); +use XML::LibXML::Simple; +use JSON::XS; +use gorgone::modules::centreon::mbi::libs::Messages; +use gorgone::modules::centreon::mbi::etl::import::main; +use gorgone::modules::centreon::mbi::etl::event::main; +use gorgone::modules::centreon::mbi::etl::perfdata::main; +use gorgone::modules::centreon::mbi::libs::centreon::ETLProperties; +use Try::Tiny; + +use constant NONE => 0; +use constant RUNNING => 1; +use constant STOP => 2; + +use constant NOTDONE => 0; +use constant DONE => 1; + +use constant UNPLANNED => -1; +use constant PLANNED => 0; +#use constant RUNNING => 1; +use constant FINISHED => 2; + +my %handlers = (TERM => {}, HUP => {}); +my ($connector); + +sub new { + my ($class, %options) = @_; + $connector = $class->SUPER::new(%options); + bless $connector, $class; + + $connector->{cbis_profile} = (defined($connector->{config}->{cbis_profile}) && $connector->{config}->{cbis_profile} ne '') ? + $connector->{config}->{cbis_profile} : '/etc/centreon-bi/cbis-profile.xml'; + $connector->{reports_profile} = (defined($connector->{config}->{reports_profile}) && $connector->{config}->{reports_profile} ne '') ? + $connector->{config}->{reports_profile} : '/etc/centreon-bi/reports-profile.xml'; + + $connector->{run} = { status => NONE }; + + $connector->set_signal_handlers(); + return $connector; +} + +sub set_signal_handlers { + my $self = shift; + + $SIG{TERM} = \&class_handle_TERM; + $handlers{TERM}->{$self} = sub { $self->handle_TERM() }; + $SIG{HUP} = \&class_handle_HUP; + $handlers{HUP}->{$self} = sub { $self->handle_HUP() }; +} + +sub handle_HUP { + my $self = shift; + $self->{reload} = 0; +} + +sub handle_TERM { + my $self = shift; + $self->{logger}->writeLogDebug("[nodes] $$ Receiving order to stop..."); + $self->{stop} = 1; +} + +sub class_handle_TERM { + foreach (keys %{$handlers{TERM}}) { + &{$handlers{TERM}->{$_}}(); + } +} + +sub class_handle_HUP { + foreach (keys %{$handlers{HUP}}) { + &{$handlers{HUP}->{$_}}(); + } +} + +sub reset { + my ($self, %options) = @_; + + $self->{run} = { status => NONE }; +} + +sub runko { + my ($self, %options) = @_; + + $self->send_log( + code => GORGONE_ACTION_FINISH_KO, + token => defined($options{token}) ? $options{token} : $self->{run}->{token}, + data => { + messages => [ ['E', $options{msg} ] ] + } + ); + + $self->check_stopped_ko(); + return 1; +} + +sub db_parse_xml { + my ($self, %options) = @_; + + my ($rv, $message, $content) = gorgone::standard::misc::slurp(file => $options{file}); + return (0, $message) if (!$rv); + eval { + $SIG{__WARN__} = sub {}; + $content = XMLin($content, ForceArray => [], KeyAttr => []); + }; + if ($@) { + die 'cannot read xml file: ' . $@; + } + + my $dbcon = {}; + if (!defined($content->{profile})) { + die 'no profile'; + } + foreach my $profile (@{$content->{profile}}) { + my $name = lc($profile->{name}); + $name =~ s/censtorage/centstorage/; + $dbcon->{$name} = { port => 3306 }; + foreach my $prop (@{$profile->{baseproperties}->{property}}) { + if ($prop->{name} eq 'odaURL' && $prop->{value} =~ /jdbc\:[a-z]+\:\/\/([^:]*)(\:\d+)?\/(.*)/) { + $dbcon->{$name}->{host} = $1; + $dbcon->{$name}->{db} = $3; + if (defined($2) && $2 ne '') { + $dbcon->{$name}->{port} = $2; + $dbcon->{$name}->{port} =~ s/\://; + } + $dbcon->{$name}->{db} =~ s/\?autoReconnect\=true//; + } elsif ($prop->{name} eq 'odaUser') { + $dbcon->{$name}->{user} = $prop->{value}; + } elsif ($prop->{name} eq 'odaPassword') { + $dbcon->{$name}->{password} = $prop->{value}; + } + } + } + foreach my $profile ('centreon', 'centstorage') { + die 'cannot find profile ' . $profile if (!defined($dbcon->{$profile})); + foreach ('host', 'db', 'port', 'user', 'password') { + die "property $_ for profile $profile must be defined" + if (!defined($dbcon->{$profile}->{$_}) || $dbcon->{$profile}->{$_} eq ''); + } + } + + return $dbcon; +} + +sub execute_action { + my ($self, %options) = @_; + + $self->send_internal_action( + action => 'ADDLISTENER', + data => [ + { + identity => 'gorgone-' . $self->{module_id}, + event => 'CENTREONMBIETLLISTENER', + token => $self->{module_id} . '-' . $self->{run}->{token} . '-' . $options{substep}, + timeout => 43200 + } + ] + ); + + my $content = { + dbmon => $self->{run}->{dbmon}, + dbbi => $self->{run}->{dbbi}, + params => $options{params} + }; + if (defined($options{etlProperties})) { + $content->{etlProperties} = $self->{run}->{etlProperties}; + } + if (defined($options{dataRetention})) { + $content->{dataRetention} = $self->{run}->{dataRetention}; + } + if (defined($options{options})) { + $content->{options} = $self->{run}->{options}; + } + + $self->send_internal_action( + action => $options{action}, + token => $self->{module_id} . '-' . $self->{run}->{token} . '-' . $options{substep}, + data => { + instant => 1, + content => $content + } + ); +} + +sub watch_etl_event { + my ($self, %options) = @_; + + if (defined($options{indexes})) { + $self->{run}->{schedule}->{event}->{substeps_executed}++; + my ($idx, $idx2) = split(/-/, $options{indexes}); + $self->{run}->{schedule}->{event}->{stages}->[$idx]->[$idx2]->{status} = FINISHED; + } + + return if (!$self->check_stopped_ko()); + + if ($self->{run}->{schedule}->{event}->{substeps_executed} >= $self->{run}->{schedule}->{event}->{substeps_total}) { + $self->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $self->{run}->{token}, data => { messages => [ ['I', '[SCHEDULER][EVENT] <<<<<<< end'] ] }); + $self->{run}->{schedule}->{event}->{status} = FINISHED; + $self->check_stopped_ok(); + return ; + } + + my $stage = $self->{run}->{schedule}->{event}->{current_stage}; + my $stage_finished = 0; + while ($stage <= 2) { + while (my ($idx, $val) = each(@{$self->{run}->{schedule}->{event}->{stages}->[$stage]})) { + if (!defined($val->{status})) { + $self->{logger}->writeLogDebug("[mbi-etl] execute substep event-$stage-$idx"); + $self->{run}->{schedule}->{event}->{substeps_execute}++; + $self->execute_action( + action => 'CENTREONMBIETLWORKERSEVENT', + substep => "event-$stage-$idx", + etlProperties => 1, + options => 1, + params => $self->{run}->{schedule}->{event}->{stages}->[$stage]->[$idx] + ); + $self->{run}->{schedule}->{event}->{stages}->[$stage]->[$idx]->{status} = RUNNING; + } elsif ($val->{status} == FINISHED) { + $stage_finished++; + } + } + + if ($stage_finished >= scalar(@{$self->{run}->{schedule}->{event}->{stages}->[$stage]})) { + $self->{run}->{schedule}->{event}->{current_stage}++; + $stage = $self->{run}->{schedule}->{event}->{current_stage}; + } else { + last; + } + } +} + +sub watch_etl_perfdata { + my ($self, %options) = @_; + + if (defined($options{indexes})) { + $self->{run}->{schedule}->{perfdata}->{substeps_executed}++; + my ($idx, $idx2) = split(/-/, $options{indexes}); + $self->{run}->{schedule}->{perfdata}->{stages}->[$idx]->[$idx2]->{status} = FINISHED; + } + + return if (!$self->check_stopped_ko()); + + if ($self->{run}->{schedule}->{perfdata}->{substeps_executed} >= $self->{run}->{schedule}->{perfdata}->{substeps_total}) { + $self->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $self->{run}->{token}, data => { messages => [ ['I', '[SCHEDULER][PERFDATA] <<<<<<< end'] ] }); + $self->{run}->{schedule}->{perfdata}->{status} = FINISHED; + $self->check_stopped_ok(); + return ; + } + + my $stage = $self->{run}->{schedule}->{perfdata}->{current_stage}; + my $stage_finished = 0; + while ($stage <= 2) { + while (my ($idx, $val) = each(@{$self->{run}->{schedule}->{perfdata}->{stages}->[$stage]})) { + if (!defined($val->{status})) { + $self->{logger}->writeLogDebug("[mbi-etl] execute substep perfdata-$stage-$idx"); + $self->{run}->{schedule}->{perfdata}->{substeps_execute}++; + $self->execute_action( + action => 'CENTREONMBIETLWORKERSPERFDATA', + substep => "perfdata-$stage-$idx", + etlProperties => 1, + options => 1, + params => $self->{run}->{schedule}->{perfdata}->{stages}->[$stage]->[$idx] + ); + $self->{run}->{schedule}->{perfdata}->{stages}->[$stage]->[$idx]->{status} = RUNNING; + } elsif ($val->{status} == FINISHED) { + $stage_finished++; + } + } + + if ($stage_finished >= scalar(@{$self->{run}->{schedule}->{perfdata}->{stages}->[$stage]})) { + $self->{run}->{schedule}->{perfdata}->{current_stage}++; + $stage = $self->{run}->{schedule}->{perfdata}->{current_stage}; + } else { + last; + } + } +} + +sub watch_etl_dimensions { + my ($self, %options) = @_; + + if (defined($options{indexes})) { + $self->{run}->{schedule}->{dimensions}->{substeps_executed}++; + } + + return if (!$self->check_stopped_ko()); + + if ($self->{run}->{schedule}->{dimensions}->{substeps_executed} >= $self->{run}->{schedule}->{dimensions}->{substeps_total}) { + $self->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $self->{run}->{token}, data => { messages => [ ['I', '[SCHEDULER][DIMENSIONS] <<<<<<< end'] ] }); + $self->{run}->{schedule}->{dimensions}->{status} = FINISHED; + $self->run_etl(); + $self->check_stopped_ok(); + return ; + } + + $self->{run}->{schedule}->{dimensions}->{substeps_execute}++; + $self->execute_action( + action => 'CENTREONMBIETLWORKERSDIMENSIONS', + substep => 'dimensions-1', + etlProperties => 1, + options => 1, + params => {} + ); +} + +sub watch_etl_import { + my ($self, %options) = @_; + + if (defined($options{indexes})) { + $self->{run}->{schedule}->{import}->{substeps_executed}++; + my ($idx, $idx2) = split(/-/, $options{indexes}); + if (defined($idx) && defined($idx2)) { + $self->{run}->{schedule}->{import}->{actions}->[$idx]->{actions}->[$idx2]->{status} = FINISHED; + } else { + $self->{run}->{schedule}->{import}->{actions}->[$idx]->{status} = FINISHED; + } + } + + return if (!$self->check_stopped_ko()); + + if ($self->{run}->{schedule}->{import}->{substeps_executed} >= $self->{run}->{schedule}->{import}->{substeps_total}) { + $self->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $self->{run}->{token}, data => { messages => [ ['I', '[SCHEDULER][IMPORT] <<<<<<< end'] ] }); + $self->{run}->{schedule}->{import}->{status} = FINISHED; + $self->run_etl(); + $self->check_stopped_ok(); + return ; + } + + while (my ($idx, $val) = each(@{$self->{run}->{schedule}->{import}->{actions}})) { + if (!defined($val->{status})) { + $self->{logger}->writeLogDebug("[mbi-etl] execute substep import-$idx"); + $self->{run}->{schedule}->{import}->{substeps_execute}++; + $self->{run}->{schedule}->{import}->{actions}->[$idx]->{status} = RUNNING; + $self->execute_action( + action => 'CENTREONMBIETLWORKERSIMPORT', + substep => "import-$idx", + params => { + type => $val->{type}, + db => $val->{db}, + sql => $val->{sql}, + command => $val->{command}, + message => $val->{message} + } + ); + } elsif ($val->{status} == FINISHED) { + while (my ($idx2, $val2) = each(@{$val->{actions}})) { + next if (defined($val2->{status})); + + $self->{logger}->writeLogDebug("[mbi-etl] execute substep import-$idx-$idx2"); + $self->{run}->{schedule}->{import}->{substeps_execute}++; + $self->{run}->{schedule}->{import}->{actions}->[$idx]->{actions}->[$idx2]->{status} = RUNNING; + $self->execute_action( + action => 'CENTREONMBIETLWORKERSIMPORT', + substep => "import-$idx-$idx2", + params => $val2 + ); + } + } + } +} + +sub run_etl_import { + my ($self, %options) = @_; + + if ((defined($self->{run}->{etlProperties}->{'host.dedicated'}) && $self->{run}->{etlProperties}->{'host.dedicated'} eq 'false') + || ($self->{run}->{dbbi}->{centstorage}->{host} . ':' . $self->{run}->{dbbi}->{centstorage}->{port} eq $self->{run}->{dbmon}->{centstorage}->{host} . ':' . $self->{run}->{dbmon}->{centstorage}->{port}) + || ($self->{run}->{dbbi}->{centreon}->{host} . ':' . $self->{run}->{dbbi}->{centreon}->{port} eq $self->{run}->{dbmon}->{centreon}->{host} . ':' . $self->{run}->{dbmon}->{centreon}->{port})) { + die 'Do not execute this script if the reporting engine is installed on the monitoring server. In case of "all in one" installation, do not consider this message'; + } + + $self->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $self->{run}->{token}, data => { messages => [ ['I', '[SCHEDULER][IMPORT] >>>>>>> start' ] ] }); + + gorgone::modules::centreon::mbi::etl::import::main::prepare($self); + + $self->{run}->{schedule}->{import}->{status} = RUNNING; + + $self->{run}->{schedule}->{import}->{substeps_execute} = 0; + $self->{run}->{schedule}->{import}->{substeps_executed} = 0; + $self->{run}->{schedule}->{import}->{substeps_total} = 0; + foreach (@{$self->{run}->{schedule}->{import}->{actions}}) { + $self->{run}->{schedule}->{import}->{substeps_total}++; + my $num = defined($_->{actions}) ? scalar(@{$_->{actions}}) : 0; + $self->{run}->{schedule}->{import}->{substeps_total} += $num if ($num > 0); + } + + $self->{logger}->writeLogDebug("[mbi-etl] import substeps " . $self->{run}->{schedule}->{import}->{substeps_total}); + + $self->watch_etl_import(); +} + +sub run_etl_dimensions { + my ($self, %options) = @_; + + $self->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $self->{run}->{token}, data => { messages => [ ['I', '[SCHEDULER][DIMENSIONS] >>>>>>> start' ] ] }); + $self->{run}->{schedule}->{dimensions}->{status} = RUNNING; + $self->{run}->{schedule}->{dimensions}->{substeps_execute} = 0; + $self->{run}->{schedule}->{dimensions}->{substeps_executed} = 0; + $self->{run}->{schedule}->{dimensions}->{substeps_total} = 1; + $self->watch_etl_dimensions(); +} + +sub run_etl_event { + my ($self, %options) = @_; + + $self->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $self->{run}->{token}, data => { messages => [ ['I', '[SCHEDULER][EVENT] >>>>>>> start' ] ] }); + + gorgone::modules::centreon::mbi::etl::event::main::prepare($self); + + $self->{run}->{schedule}->{event}->{status} = RUNNING; + $self->{run}->{schedule}->{event}->{current_stage} = 0; + $self->{run}->{schedule}->{event}->{substeps_execute} = 0; + $self->{run}->{schedule}->{event}->{substeps_executed} = 0; + $self->{run}->{schedule}->{event}->{substeps_total} = + scalar(@{$self->{run}->{schedule}->{event}->{stages}->[0]}) + scalar(@{$self->{run}->{schedule}->{event}->{stages}->[1]}) + scalar(@{$self->{run}->{schedule}->{event}->{stages}->[2]}); + + $self->{logger}->writeLogDebug("[mbi-etl] event substeps " . $self->{run}->{schedule}->{event}->{substeps_total}); + + $self->watch_etl_event(); +} + +sub run_etl_perfdata { + my ($self, %options) = @_; + + $self->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $self->{run}->{token}, data => { messages => [ ['I', '[SCHEDULER][PERFDATA] >>>>>>> start' ] ] }); + + gorgone::modules::centreon::mbi::etl::perfdata::main::prepare($self); + + $self->{run}->{schedule}->{perfdata}->{status} = RUNNING; + $self->{run}->{schedule}->{perfdata}->{current_stage} = 0; + $self->{run}->{schedule}->{perfdata}->{substeps_execute} = 0; + $self->{run}->{schedule}->{perfdata}->{substeps_executed} = 0; + $self->{run}->{schedule}->{perfdata}->{substeps_total} = + scalar(@{$self->{run}->{schedule}->{perfdata}->{stages}->[0]}) + scalar(@{$self->{run}->{schedule}->{perfdata}->{stages}->[1]}) + scalar(@{$self->{run}->{schedule}->{perfdata}->{stages}->[2]}); + + $self->{logger}->writeLogDebug("[mbi-etl] perfdata substeps " . $self->{run}->{schedule}->{perfdata}->{substeps_total}); + + $self->watch_etl_perfdata(); +} + +sub run_etl { + my ($self, %options) = @_; + + if ($self->{run}->{schedule}->{import}->{status} == PLANNED) { + $self->run_etl_import(); + return ; + } elsif ($self->{run}->{schedule}->{dimensions}->{status} == PLANNED) { + $self->run_etl_dimensions(); + return ; + } + if ($self->{run}->{schedule}->{event}->{status} == PLANNED) { + $self->run_etl_event(); + } + if ($self->{run}->{schedule}->{perfdata}->{status} == PLANNED) { + $self->run_etl_perfdata(); + } +} + +sub check_stopped_ko_import { + my ($self, %options) = @_; + + return 0 if ($self->{run}->{schedule}->{import}->{substeps_executed} >= $self->{run}->{schedule}->{import}->{substeps_execute}); + + return 1; +} + +sub check_stopped_ko_dimensions { + my ($self, %options) = @_; + + return 0 if ($self->{run}->{schedule}->{dimensions}->{substeps_executed} >= $self->{run}->{schedule}->{dimensions}->{substeps_execute}); + + return 1; +} + +sub check_stopped_ko_event { + my ($self, %options) = @_; + + return 0 if ($self->{run}->{schedule}->{event}->{substeps_executed} >= $self->{run}->{schedule}->{event}->{substeps_execute}); + + return 1; +} + +sub check_stopped_ko_perfdata { + my ($self, %options) = @_; + + return 0 if ($self->{run}->{schedule}->{perfdata}->{substeps_executed} >= $self->{run}->{schedule}->{perfdata}->{substeps_execute}); + + return 1; +} + +sub check_stopped_ko { + my ($self, %options) = @_; + + # if nothing planned. we stop + if ($self->{run}->{schedule}->{planned} == NOTDONE) { + $self->reset(); + return 0; + } + + return 1 if ($self->{run}->{status} != STOP); + + my $stopped = 0; + $stopped += $self->check_stopped_ko_import() + if ($self->{run}->{schedule}->{import}->{status} == RUNNING); + $stopped += $self->check_stopped_ko_dimensions() + if ($self->{run}->{schedule}->{dimensions}->{status} == RUNNING); + $stopped += $self->check_stopped_ko_event() + if ($self->{run}->{schedule}->{event}->{status} == RUNNING); + $stopped += $self->check_stopped_ko_perfdata() + if ($self->{run}->{schedule}->{perfdata}->{status} == RUNNING); + + if ($stopped == 0) { + $self->reset(); + return 0; + } + + return 1; +} + +sub check_stopped_ok_import { + my ($self, %options) = @_; + + return 0 if ($self->{run}->{schedule}->{import}->{substeps_executed} >= $self->{run}->{schedule}->{import}->{substeps_total}); + + return 1; +} + +sub check_stopped_ok_dimensions { + my ($self, %options) = @_; + + return 0 if ($self->{run}->{schedule}->{dimensions}->{substeps_executed} >= $self->{run}->{schedule}->{dimensions}->{substeps_total}); + + return 1; +} + +sub check_stopped_ok_event { + my ($self, %options) = @_; + + return 0 if ($self->{run}->{schedule}->{event}->{substeps_executed} >= $self->{run}->{schedule}->{event}->{substeps_total}); + + return 1; +} + +sub check_stopped_ok_perfdata { + my ($self, %options) = @_; + + return 0 if ($self->{run}->{schedule}->{perfdata}->{substeps_executed} >= $self->{run}->{schedule}->{perfdata}->{substeps_total}); + + return 1; +} + +sub check_stopped_ok { + my ($self, %options) = @_; + + return 1 if ($self->{run}->{status} == STOP); + + my $stopped = 0; + $stopped += $self->check_stopped_ok_import() + if ($self->{run}->{schedule}->{import}->{status} == RUNNING); + $stopped += $self->check_stopped_ok_dimensions() + if ($self->{run}->{schedule}->{dimensions}->{status} == RUNNING); + $stopped += $self->check_stopped_ok_event() + if ($self->{run}->{schedule}->{event}->{status} == RUNNING); + $stopped += $self->check_stopped_ok_perfdata() + if ($self->{run}->{schedule}->{perfdata}->{status} == RUNNING); + + if ($stopped == 0) { + $self->send_log( + code => GORGONE_ACTION_FINISH_OK, + token => $self->{run}->{token}, + data => { + messages => [ ['I', '[SCHEDULER] <<<<<<< end' ] ] + } + ); + $self->reset(); + return 0; + } + + return 1; +} + +sub planning { + my ($self, %options) = @_; + + if ($self->{run}->{options}->{import} == 1) { + $self->{run}->{schedule}->{import}->{status} = PLANNED; + $self->{run}->{schedule}->{steps_total}++; + } + if ($self->{run}->{options}->{dimensions} == 1) { + $self->{run}->{schedule}->{dimensions}->{status} = PLANNED; + $self->{run}->{schedule}->{steps_total}++; + } + if ($self->{run}->{options}->{event} == 1) { + $self->{run}->{schedule}->{event}->{status} = PLANNED; + $self->{run}->{schedule}->{steps_total}++; + } + if ($self->{run}->{options}->{perfdata} == 1) { + $self->{run}->{schedule}->{perfdata}->{status} = PLANNED; + $self->{run}->{schedule}->{steps_total}++; + } + + if ($self->{run}->{schedule}->{steps_total} == 0) { + die "[SCHEDULING] nothing planned"; + } + + $self->{run}->{schedule}->{steps_executed} = 0; + $self->{run}->{schedule}->{planned} = DONE; +} + +sub check_basic_options { + my ($self, %options) = @_; + + if (($options{daily} == 0 && $options{rebuild} == 0 && $options{create_tables} == 0 && !defined($options{centile})) + || ($options{daily} == 1 && $options{rebuild} == 1)) { + die "Specify one execution method"; + } + if (($options{rebuild} == 1 || $options{create_tables} == 1) + && (($options{start} ne '' && $options{end} eq '') + || ($options{start} eq '' && $options{end} ne ''))) { + die "Specify both options start and end or neither of them to use default data retention options"; + } + if ($options{rebuild} == 1 && $options{start} ne '' && $options{end} ne '' + && ($options{start} !~ /[1-2][0-9]{3}\-[0-1][0-9]\-[0-3][0-9]/ || $options{end} !~ /[1-2][0-9]{3}\-[0-1][0-9]\-[0-3][0-9]/)) { + die "Verify period start or end date format"; + } +} + +sub action_centreonmbietlrun { + my ($self, %options) = @_; + + try { + $options{token} = $self->generate_token() if (!defined($options{token})); + + return $self->runko(token => $options{token}, msg => '[SCHEDULER] already running') if ($self->{run}->{status} == RUNNING); + return $self->runko(token => $options{token}, msg => '[SCHEDULER] currently wait previous execution finished - can restart gorgone mbi process') if ($self->{run}->{status} == STOP); + + $self->{run}->{token} = $options{token}; + $self->{run}->{messages} = gorgone::modules::centreon::mbi::libs::Messages->new(); + + $self->check_basic_options(%{$options{data}->{content}}); + + $self->{run}->{schedule} = { + steps_total => 0, + steps_executed => 0, + planned => NOTDONE, + import => { status => UNPLANNED, actions => [] }, + dimensions => { status => UNPLANNED }, + event => { status => UNPLANNED, stages => [ [], [], [] ] }, + perfdata => { status => UNPLANNED, stages => [ [], [], [] ] } + }; + $self->{run}->{status} = RUNNING; + + $self->{run}->{options} = $options{data}->{content}; + + $self->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $self->{run}->{token}, data => { messages => [ ['I', '[SCHEDULER] >>>>>>> start' ] ] }); + + $self->{run}->{dbmon} = $self->db_parse_xml(file => $self->{cbis_profile}); + $self->{run}->{dbbi} = $self->db_parse_xml(file => $self->{reports_profile}); + + $self->{run}->{dbmon_centreon_con} = gorgone::class::db->new( + type => 'mysql', + force => 2, + logger => $self->{logger}, + die => 1, + %{$self->{run}->{dbmon}->{centreon}} + ); + $self->{run}->{dbmon_centstorage_con} = gorgone::class::db->new( + type => 'mysql', + force => 2, + logger => $self->{logger}, + die => 1, + %{$self->{run}->{dbmon}->{centstorage}} + ); + $self->{run}->{dbbi_centstorage_con} = gorgone::class::db->new( + type => 'mysql', + force => 2, + logger => $self->{logger}, + die => 1, + %{$self->{run}->{dbbi}->{centstorage}} + ); + + $self->{etlProp} = gorgone::modules::centreon::mbi::libs::centreon::ETLProperties->new($self->{logger}, $self->{run}->{dbmon_centreon_con}); + ($self->{run}->{etlProperties}, $self->{run}->{dataRetention}) = $self->{etlProp}->getProperties(); + + $self->planning(); + $self->run_etl(); + } catch { + $self->runko(msg => $_); + $self->reset(); + }; + + return 0; +} + +sub action_centreonmbietllistener { + my ($self, %options) = @_; + + return 0 if (!defined($options{token}) || $options{token} !~ /^$self->{module_id}-$self->{run}->{token}-(.*?)-(.*)$/); + my ($type, $indexes) = ($1, $2); + + if ($options{data}->{code} == GORGONE_ACTION_FINISH_KO) { + $self->{run}->{status} = STOP; + $self->send_log(code => GORGONE_ACTION_FINISH_KO, token => $self->{run}->{token}, data => $options{data}->{data}); + } elsif ($options{data}->{code} == GORGONE_ACTION_FINISH_OK) { + $self->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $self->{run}->{token}, data => $options{data}->{data}); + } else { + return 0; + } + + if ($type eq 'import') { + $self->watch_etl_import(indexes => $indexes); + } elsif ($type eq 'dimensions') { + $self->watch_etl_dimensions(indexes => $indexes); + } elsif ($type eq 'event') { + $self->watch_etl_event(indexes => $indexes); + } elsif ($type eq 'perfdata') { + $self->watch_etl_perfdata(indexes => $indexes); + } + + return 1; +} + +sub action_centreonmbietlkill { + my ($self, %options) = @_; + + $options{token} = $self->generate_token() if (!defined($options{token})); + + if ($self->{run}->{status} == NONE) { + $self->{logger}->writeLogDebug('[mbi-etl] kill action - etl not running'); + $self->send_log( + code => GORGONE_ACTION_FINISH_OK, + token => $options{token}, + data => { + messages => 'etl not running' + } + ); + return 0; + } + + $self->{logger}->writeLogDebug('[mbi-etl] kill sent to the module etlworkers'); + + $self->send_internal_action( + action => 'KILL', + token => $options{token}, + data => { + content => { + package => 'gorgone::modules::centreon::mbi::etlworkers::hooks' + } + } + ); + + # RUNNING or STOP + $self->send_log( + code => GORGONE_ACTION_CONTINUE, + token => $options{token}, + data => { + messages => 'kill sent to the module etlworkers' + } + ); + + $self->reset(); + + return 0; +} + +sub action_centreonmbietlstatus { + my ($self, %options) = @_; + + $options{token} = $self->generate_token() if (!defined($options{token})); + + my $map_etl_status = { + 0 => 'ready', + 1 => 'running', + 2 => 'stopping' + }; + + my $map_planning_status = { + 0 => 'running', + 1 => 'ok' + }; + + my $map_section_status = { + -1 => 'unplanned', + 0 => 'planned', + 1 => 'running', + 2 => 'ok' + }; + + my $section = {}; + foreach ('import', 'dimensions', 'event', 'perfdata') { + next if (!defined($self->{run}->{schedule})); + + $section->{$_} = { + status => $self->{run}->{schedule}->{$_}->{status}, + statusStr => $map_section_status->{ $self->{run}->{schedule}->{$_}->{status} } + }; + if ($self->{run}->{schedule}->{$_}->{status} == RUNNING) { + $section->{$_}->{steps_total} = $self->{run}->{schedule}->{$_}->{substeps_total}; + $section->{$_}->{steps_executed} = $self->{run}->{schedule}->{$_}->{substeps_executed}; + } + } + + $self->send_log( + code => GORGONE_ACTION_FINISH_OK, + token => $options{token}, + data => { + token => defined($self->{run}->{token}) ? $self->{run}->{token} : undef, + + status => $self->{run}->{status}, + statusStr => $map_etl_status->{ $self->{run}->{status} }, + + planning => defined($self->{run}->{schedule}->{planned}) ? $self->{run}->{schedule}->{planned} : undef, + planningStr => defined($self->{run}->{schedule}->{planned}) ? $map_planning_status->{ $self->{run}->{schedule}->{planned} } : undef, + + sections => $section + } + ); + + return 0; +} + +sub event { + while (1) { + my $message = $connector->read_message(); + last if (!defined($message)); + + $connector->{logger}->writeLogDebug("[mbi-etl] Event: $message"); + if ($message =~ /^\[(.*?)\]/) { + if ((my $method = $connector->can('action_' . lc($1)))) { + $message =~ /^\[(.*?)\]\s+\[(.*?)\]\s+\[.*?\]\s+(.*)$/m; + my ($action, $token) = ($1, $2); + my ($rv, $data) = $connector->json_decode(argument => $3, token => $token); + next if ($rv); + + $method->($connector, token => $token, data => $data); + } + } + } +} + +sub run { + my ($self, %options) = @_; + + # Connect internal + $connector->{internal_socket} = gorgone::standard::library::connect_com( + zmq_type => 'ZMQ_DEALER', + name => 'gorgone-' . $self->{module_id}, + logger => $self->{logger}, + type => $self->get_core_config(name => 'internal_com_type'), + path => $self->get_core_config(name => 'internal_com_path') + ); + $connector->send_internal_action( + action => 'CENTREONMBIETLREADY', + data => {} + ); + $self->{poll} = [ + { + socket => $connector->{internal_socket}, + events => ZMQ_POLLIN, + callback => \&event + } + ]; + + while (1) { + my $rev = scalar(zmq_poll($self->{poll}, 5000)); + if (defined($rev) && $rev == 0 && $self->{stop} == 1) { + $self->{logger}->writeLogInfo("[" . $self->{module_id} . "] $$ has quit"); + zmq_close($connector->{internal_socket}); + exit(0); + } + } +} + +1; diff --git a/gorgone/modules/centreon/mbi/etl/event/main.pm b/gorgone/modules/centreon/mbi/etl/event/main.pm new file mode 100644 index 0000000..bc59ac6 --- /dev/null +++ b/gorgone/modules/centreon/mbi/etl/event/main.pm @@ -0,0 +1,283 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::etl::event::main; + +use strict; +use warnings; + +use gorgone::modules::centreon::mbi::libs::bi::Time; +use gorgone::modules::centreon::mbi::libs::bi::LiveService; +use gorgone::modules::centreon::mbi::libs::bi::MySQLTables; +use gorgone::modules::centreon::mbi::libs::Utils; + +my ($biTables, $utils, $liveService, $time); +my ($start, $end); + +sub initVars { + my ($etl) = @_; + + $biTables = gorgone::modules::centreon::mbi::libs::bi::MySQLTables->new($etl->{run}->{messages}, $etl->{run}->{dbbi_centstorage_con}); + $utils = gorgone::modules::centreon::mbi::libs::Utils->new($etl->{run}->{messages}); + $liveService = gorgone::modules::centreon::mbi::libs::bi::LiveService->new($etl->{run}->{messages}, $etl->{run}->{dbbi_centstorage_con}); + $time = gorgone::modules::centreon::mbi::libs::bi::Time->new($etl->{run}->{messages}, $etl->{run}->{dbbi_centstorage_con}); +} + +sub emptyTableForRebuild { + my ($etl, %options) = @_; + + my $sql = [ [ '[CREATE] Deleting table [' . $options{name} . ']', 'DROP TABLE IF EXISTS `' . $options{name} . '`' ] ]; + + my $structure = $biTables->dumpTableStructure($options{name}); + $structure =~ s/KEY.*\(\`$options{column}\`\)\,//g; + $structure =~ s/KEY.*\(\`$options{column}\`\)//g; + $structure =~ s/\,[\n\s+]+\)/\n\)/g; + + if (defined($options{start})) { + $structure =~ s/\n.*PARTITION.*//g; + $structure =~ s/\,[\n\s]+\)/\)/; + $structure .= ' PARTITION BY RANGE(`' . $options{column} . '`) ('; + + my $partitionsPerf = $utils->getRangePartitionDate($options{start}, $options{end}); + + my $append = ''; + foreach (@$partitionsPerf) { + $structure .= $append . "PARTITION p" . $_->{name} . " VALUES LESS THAN (" . $_->{epoch} . ")"; + $append = ','; + } + $structure .= ');'; + } + + push @$sql, + [ '[CREATE] Add table [' . $options{name} . ']', $structure ], + [ "[INDEXING] Adding index [idx_$options{name}_$options{column}] on table [$options{name}]", "ALTER TABLE `$options{name}` ADD INDEX `idx_$options{name}_$options{column}` (`$options{column}`)" ]; + + push @{$etl->{run}->{schedule}->{event}->{stages}->[0]}, { type => 'sql', db => 'centstorage', sql => $sql }; +} + +sub deleteEntriesForRebuild { + my ($etl, %options) = @_; + + my $sql = []; + if (!$biTables->isTablePartitioned($options{name})) { + push @$sql, + [ + "[PURGE] Delete table [$options{name}] from $options{start} to $options{end}", + "DELETE FROM $options{name} WHERE time_id >= " . $utils->getDateEpoch($options{start}) . " AND time_id < " . $utils->getDateEpoch($options{end}) + ]; + } else { + my $partitionsPerf = $utils->getRangePartitionDate($options{start}, $options{end}); + foreach (@$partitionsPerf) { + push @$sql, + [ + "[PURGE] Truncate partition $_->{name} on table [$options{name}]", + "ALTER TABLE $options{name} TRUNCATE PARTITION p$_->{name}" + ]; + } + } + + push @{$etl->{run}->{schedule}->{event}->{stages}->[0]}, { type => 'sql', db => 'centstorage', sql => $sql }; +} + +sub purgeAvailabilityTables { + my ($etl, $start, $end) = @_; + + my $firstDayOfMonth = $start; + $firstDayOfMonth =~ s/([1-2][0-9]{3})\-([0-1][0-9])\-[0-3][0-9]/$1\-$2\-01/; + + if ($etl->{run}->{options}->{nopurge} == 0) { + if (!defined($etl->{run}->{options}->{service_only}) || $etl->{run}->{options}->{service_only} == 0) { + if (!defined($etl->{run}->{options}->{month_only}) || $etl->{run}->{options}->{month_only} == 0) { + emptyTableForRebuild($etl, name => 'mod_bi_hostavailability', column => 'time_id', start => $start, end => $end); + } + + emptyTableForRebuild($etl, name => 'mod_bi_hgmonthavailability', column => 'time_id'); + } + if (!defined($etl->{run}->{options}->{host_only}) || $etl->{run}->{options}->{host_only} == 0) { + if (!defined($etl->{run}->{options}->{month_only}) || $etl->{run}->{options}->{month_only} == 0) { + emptyTableForRebuild($etl, name => 'mod_bi_serviceavailability', column => 'time_id', start => $start, end => $end); + } + + emptyTableForRebuild($etl, name => 'mod_bi_hgservicemonthavailability', column => 'time_id'); + } + } else { + if (!defined($etl->{run}->{options}->{service_only}) || $etl->{run}->{options}->{service_only} == 0) { + if (!defined($etl->{run}->{options}->{month_only}) || $etl->{run}->{options}->{month_only} == 0) { + deleteEntriesForRebuild($etl, name => 'mod_bi_hostavailability', start => $start, end => $end); + } + + deleteEntriesForRebuild($etl, name => 'mod_bi_hgmonthavailability', start => $firstDayOfMonth, end => $end); + } + if (!defined($etl->{run}->{options}->{host_only}) || $etl->{run}->{options}->{host_only} == 0) { + if (!defined($etl->{run}->{options}->{month_only}) || $etl->{run}->{options}->{month_only} == 0) { + deleteEntriesForRebuild($etl, name => 'mod_bi_serviceavailability', start => $start, end => $end); + } + deleteEntriesForRebuild($etl, name => 'mod_bi_hgservicemonthavailability', start => $firstDayOfMonth, end => $end); + } + } +} + +sub processByDay { + my ($etl, $liveServices, $start, $end) = @_; + + while (my ($liveserviceName, $liveserviceId) = each (%$liveServices)) { + if (!defined($etl->{run}->{options}->{service_only}) || $etl->{run}->{options}->{service_only} == 0) { + push @{$etl->{run}->{schedule}->{event}->{stages}->[1]}, { + type => 'availability_day_hosts', + liveserviceName => $liveserviceName, + liveserviceId => $liveserviceId, + start => $start, + end => $end + }; + } + + if (!defined($etl->{run}->{options}->{host_only}) || $etl->{run}->{options}->{host_only} == 0) { + push @{$etl->{run}->{schedule}->{event}->{stages}->[1]}, { + type => 'availability_day_services', + liveserviceName => $liveserviceName, + liveserviceId => $liveserviceId, + start => $start, + end => $end + }; + } + } +} + +sub processHostgroupAvailability { + my ($etl, $start, $end) = @_; + + $time->insertTimeEntriesForPeriod($start, $end); + if (!defined($etl->{run}->{options}->{service_only}) || $etl->{run}->{options}->{service_only} == 0) { + push @{$etl->{run}->{schedule}->{event}->{stages}->[2]}, { + type => 'availability_month_services', + start => $start, + end => $end + }; + } + if (!defined($etl->{run}->{options}->{host_only}) || $etl->{run}->{options}->{host_only} == 0) { + push @{$etl->{run}->{schedule}->{event}->{stages}->[2]}, { + type => 'availability_month_hosts', + start => $start, + end => $end + }; + } +} + +sub dailyProcessing { + my ($etl, $liveServices) = @_; + + # getting yesterday start and end date to process yesterday data + my ($start, $end) = $utils->getYesterdayTodayDate(); + # daily mod_bi_time table filling + $time->insertTimeEntriesForPeriod($start, $end); + + my ($epoch, $partName) = $utils->getDateEpoch($end); + push @{$etl->{run}->{schedule}->{event}->{stages}->[0]}, { + type => 'sql', + db => 'centstorage', + sql => [ + [ + '[PARTITIONS] Add partition [p' . $partName . '] on table [mod_bi_hostavailability]', + "ALTER TABLE `mod_bi_hostavailability` ADD PARTITION (PARTITION `p$partName` VALUES LESS THAN(" . $epoch . "))" + ] + ] + }; + push @{$etl->{run}->{schedule}->{event}->{stages}->[0]}, { + type => 'sql', + db => 'centstorage', + sql => [ + [ + '[PARTITIONS] Add partition [p' . $partName . '] on table [mod_bi_serviceavailability]', + "ALTER TABLE `mod_bi_serviceavailability` ADD PARTITION (PARTITION `p$partName` VALUES LESS THAN(" . $epoch . "))" + ] + ] + }; + + # Calculating availability of hosts and services for the current day + processByDay($etl, $liveServices, $start, $end); + + # Calculating statistics for last month if day of month si 1 + my ($year, $mon, $day) = split('-', $end); + if ($day == 1) { + processHostgroupAvailability($etl, $utils->subtractDateMonths($end, 1), $utils->subtractDateDays($end, 1)); + } + + push @{$etl->{run}->{schedule}->{event}->{stages}->[0]}, + { type => 'events', services => 1, start => $start, end => $end }, { type => 'events', hosts => 1, start => $start, end => $end }; +} + +# rebuild availability statistics +sub rebuildAvailability { + my ($etl, $start, $end, $liveServices) = @_; + + my $days = $utils->getRangePartitionDate($start, $end); + foreach (@$days) { + $end = $_->{date}; + processByDay($etl, $liveServices, $start, $end); + + my ($year, $mon, $day) = split('-', $end); + if ($day == 1) { + processHostgroupAvailability($etl, $utils->subtractDateMonths($end, 1), $utils->subtractDateDays($end, 1)); + } + + $start = $end; + } +} + +sub rebuildProcessing { + my ($etl, $liveServices) = @_; + + if ($etl->{run}->{options}->{start} ne '' && $etl->{run}->{options}->{end} ne '') { + # setting manually start and end dates for each granularity of perfdata + ($start, $end) = ($etl->{run}->{options}->{start}, $etl->{run}->{options}->{end}); + }else { + # getting max perfdata retention period to fill mod_bi_time + my $periods = $etl->{etlProp}->getRetentionPeriods(); + ($start, $end) = ($periods->{'availability.daily'}->{start}, $periods->{'availability.daily'}->{end}); + } + + # insert entries into table mod_bi_time + $time->insertTimeEntriesForPeriod($start, $end); + if (!defined($etl->{run}->{options}->{events_only}) || $etl->{run}->{options}->{events_only} == 0) { + purgeAvailabilityTables($etl, $start, $end); + rebuildAvailability($etl, $start, $end, $liveServices); + } + + if (!defined($etl->{run}->{options}->{availability_only}) || $etl->{run}->{options}->{availability_only} == 0) { + push @{$etl->{run}->{schedule}->{event}->{stages}->[0]}, + { type => 'events', services => 1, start => $start, end => $end }, { type => 'events', hosts => 1, start => $start, end => $end }; + } +} + +sub prepare { + my ($etl) = @_; + + initVars($etl); + + my $liveServiceList = $liveService->getLiveServicesByNameForTpIds($etl->{run}->{etlProperties}->{'liveservices.availability'}); + + if ($etl->{run}->{options}->{daily} == 1) { + dailyProcessing($etl, $liveServiceList); + } elsif ($etl->{run}->{options}->{rebuild} == 1) { + rebuildProcessing($etl, $liveServiceList); + } +} + +1; diff --git a/gorgone/modules/centreon/mbi/etl/hooks.pm b/gorgone/modules/centreon/mbi/etl/hooks.pm new file mode 100644 index 0000000..48d8471 --- /dev/null +++ b/gorgone/modules/centreon/mbi/etl/hooks.pm @@ -0,0 +1,156 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::etl::hooks; + +use warnings; +use strict; +use gorgone::class::core; +use gorgone::modules::centreon::mbi::etl::class; +use gorgone::standard::constants qw(:all); + +use constant NAMESPACE => 'centreon'; +use constant NAME => 'mbietl'; +use constant EVENTS => [ + { event => 'CENTREONMBIETLRUN', uri => '/run', method => 'POST' }, + { event => 'CENTREONMBIETLKILL', uri => '/kill', method => 'GET' }, + { event => 'CENTREONMBIETLSTATUS', uri => '/status', method => 'GET' }, + { event => 'CENTREONMBIETLLISTENER' }, + { event => 'CENTREONMBIETLREADY' } +]; + +my $config_core; +my $config; +my $run = {}; +my $stop = 0; + +sub register { + my (%options) = @_; + + $config = $options{config}; + $config_core = $options{config_core}; + return (1, NAMESPACE, NAME, EVENTS); +} + +sub init { + my (%options) = @_; + + create_child(logger => $options{logger}); +} + +sub routing { + my (%options) = @_; + + if ($options{action} eq 'CENTREONMBIETLREADY') { + $run->{ready} = 1; + return undef; + } + + if (gorgone::class::core::waiting_ready(ready => \$run->{ready}) == 0) { + gorgone::standard::library::add_history( + dbh => $options{dbh}, + code => GORGONE_ACTION_FINISH_KO, + token => $options{token}, + data => { message => 'gorgone-' . NAME . ': still no ready' }, + json_encode => 1 + ); + return undef; + } + + $options{gorgone}->send_internal_message( + identity => 'gorgone-' . NAME, + action => $options{action}, + data => $options{data}, + token => $options{token} + ); +} + +sub gently { + my (%options) = @_; + + $stop = 1; + if (defined($run->{running}) && $run->{running} == 1) { + $options{logger}->writeLogDebug("[" . NAME . "] Send TERM signal $run->{pid}"); + CORE::kill('TERM', $run->{pid}); + } +} + +sub kill { + my (%options) = @_; + + if ($run->{running} == 1) { + $options{logger}->writeLogDebug("[" . NAME . "] Send KILL signal $run->{pid}"); + CORE::kill('KILL', $run->{pid}); + } +} + +sub kill_internal { + my (%options) = @_; + +} + +sub check { + my (%options) = @_; + + my $count = 0; + foreach my $pid (keys %{$options{dead_childs}}) { + # Not me + next if (!defined($run->{pid}) || $run->{pid} != $pid); + + $run = {}; + delete $options{dead_childs}->{$pid}; + if ($stop == 0) { + create_child(logger => $options{logger}); + } + } + + $count++ if (defined($run->{running}) && $run->{running} == 1); + + return $count; +} + +sub broadcast { + my (%options) = @_; + + routing(%options); +} + +# Specific functions +sub create_child { + my (%options) = @_; + + $options{logger}->writeLogInfo("[" . NAME . "] Create module '" . NAME . "' process"); + my $child_pid = fork(); + if ($child_pid == 0) { + $0 = 'gorgone-' . NAME; + my $module = gorgone::modules::centreon::mbi::etl::class->new( + logger => $options{logger}, + module_id => NAME, + config_core => $config_core, + config => $config + ); + $module->run(); + exit(0); + } + $options{logger}->writeLogDebug("[" . NAME . "] PID $child_pid (gorgone-" . NAME . ")"); + $run = { pid => $child_pid, ready => 0, running => 1 }; +} + +1; diff --git a/gorgone/modules/centreon/mbi/etl/import/main.pm b/gorgone/modules/centreon/mbi/etl/import/main.pm new file mode 100644 index 0000000..b316368 --- /dev/null +++ b/gorgone/modules/centreon/mbi/etl/import/main.pm @@ -0,0 +1,412 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::etl::import::main; + +use strict; +use warnings; + +use gorgone::modules::centreon::mbi::libs::bi::MySQLTables; +use gorgone::modules::centreon::mbi::libs::Utils; + +my ($biTables, $monTables, $utils); +my ($argsMon, $argsBi); + +sub initVars { + my ($etl) = @_; + + $biTables = gorgone::modules::centreon::mbi::libs::bi::MySQLTables->new($etl->{run}->{messages}, $etl->{run}->{dbbi_centstorage_con}); + $monTables = gorgone::modules::centreon::mbi::libs::bi::MySQLTables->new($etl->{run}->{messages}, $etl->{run}->{dbmon_centstorage_con}); + $utils = gorgone::modules::centreon::mbi::libs::Utils->new($etl->{run}->{messages}); + $argsMon = $utils->buildCliMysqlArgs($etl->{run}->{dbmon}->{centstorage}); + $argsBi = $utils->buildCliMysqlArgs($etl->{run}->{dbbi}->{centstorage}); +} + +# Create tables for centstorage database on reporting server +sub createTables { + my ($etl, $periods, $options, $notTimedTables) = @_; + + #Creating all centreon bi tables exept the one already created + my $sth = $etl->{run}->{dbmon_centstorage_con}->query("SHOW TABLES LIKE 'mod_bi_%'"); + while (my @row = $sth->fetchrow_array()) { + my $name = $row[0]; + if (!$biTables->tableExists($name)) { + my $structure = $monTables->dumpTableStructure($name); + push @{$etl->{run}->{schedule}->{import}->{actions}}, + { + type => 1, db => 'centstorage', sql => [ ["[CREATE] add table [$name]", $structure] ], actions => [] + }; + } + } + + # Manage centreonAcl + my $action; + if ($options->{create_tables} == 0) { + #Update centreon_acl table each time centreon-only is started - not the best way but need for Widgets + my $cmd = sprintf( + "mysqldump --replace --no-create-info --skip-add-drop-table --skip-add-locks --skip-comments %s '%s' %s | mysql %s '%s'", + $argsMon, + $etl->{run}->{dbmon}->{centstorage}->{db}, + 'centreon_acl', + $argsBi, + $etl->{run}->{dbbi}->{centstorage}->{db} + ); + $action = { type => 2, message => '[LOAD] import table [centreon_acl]', command => $cmd }; + } + + if (!$biTables->tableExists('centreon_acl')) { + my $structure = $monTables->dumpTableStructure('centreon_acl'); + push @{$etl->{run}->{schedule}->{import}->{actions}}, + { + type => 1, db => 'centstorage', sql => [ ["[CREATE] add table [centreon_acl]", $structure] ], actions => defined($action) ? [$action] : [] + }; + } elsif (defined($action)) { + push @{$etl->{run}->{schedule}->{import}->{actions}}, $action; + } + + my $tables = join('|', @$notTimedTables); + $sth = $etl->{run}->{dbmon_centstorage_con}->query("SHOW TABLES LIKE 'mod_bam_reporting_%'"); + while (my @row = $sth->fetchrow_array()) { + my $name = $row[0]; + next if ($name =~ /^(?:$tables)$/); + + if (!$biTables->tableExists($name)) { + my $structure = $monTables->dumpTableStructure($name); + push @{$etl->{run}->{schedule}->{import}->{actions}}, + { + type => 1, db => 'centstorage', sql => [ ["[CREATE] Add table [$name]", $structure] ], actions => [] + }; + } + } +} + +# Extract data from Centreon DB server +sub extractData { + my ($etl, $options, $notTimedTables) = @_; + + foreach my $name (@$notTimedTables) { + my $action = { type => 1, db => 'centstorage', sql => [], actions => [] }; + + push @{$action->{sql}}, [ '[CREATE] Deleting table [' . $name . ']', 'DROP TABLE IF EXISTS `' . $name . '`' ]; + + my $structure = $monTables->dumpTableStructure($name); + $structure =~ s/(CONSTRAINT.*\n)//g; + $structure =~ s/(\,\n\s+\))/\)/g; + $structure =~ s/auto_increment\=[0-9]+//i; + $structure =~ s/auto_increment//i; + + push @{$action->{sql}}, [ "[CREATE] Add table [$name]", $structure ]; + if ($name eq 'hoststateevents' || $name eq 'servicestateevents') { + # add drop indexes + my $indexes = $etl->{run}->{dbmon_centstorage_con}->query("SHOW INDEX FROM " . $name); + my $previous = ''; + while (my $row = $indexes->fetchrow_hashref()) { + if ($row->{Key_name} ne $previous) { + if (lc($row->{Key_name}) eq lc('PRIMARY')) { + push @{$action->{sql}}, + [ + "[INDEXING] Deleting index [PRIMARY KEY] on table [".$name."]", + "ALTER TABLE `" . $name . "` DROP PRIMARY KEY" + ]; + } else { + push @{$action->{sql}}, + [ + "[INDEXING] Deleting index [$row->{Key_name}] on table [".$name."]", + "ALTER TABLE `" . $name . "` DROP INDEX " . $row->{Key_name} + ]; + } + } + $previous = $row->{Key_name}; + } + + push @{$action->{sql}}, + [ + "[INDEXING] Adding index [in_downtime, start_time, end_time] on table [" . $name . "]", + "ALTER TABLE `" . $name . "` ADD INDEX `idx_" . $name . "_downtime_start_end_time` (in_downtime, start_time, end_time)" + ], + [ + "[INDEXING] Adding index [end_time] on table [" . $name . "]", + "ALTER TABLE `" . $name . "` ADD INDEX `idx_" . $name . "_end_time` (`end_time`)" + ]; + if ($name eq 'servicestateevents') { + push @{$action->{sql}}, + [ + "[INDEXING] Adding index [host_id, service_id, start_time, end_time, ack_time, state, last_update] on table [servicestateevents]", + "ALTER TABLE `servicestateevents` ADD INDEX `idx_servicestateevents_multi` (host_id, service_id, start_time, end_time, ack_time, state, last_update)" + ]; + } + } + + my $cmd = sprintf( + "mysqldump --no-create-info --skip-add-drop-table --skip-add-locks --skip-comments %s '%s' %s | mysql %s '%s'", + $argsMon, + $etl->{run}->{dbmon}->{centstorage}->{db}, + $name, + $argsBi, + $etl->{run}->{dbbi}->{centstorage}->{db} + ); + push @{$action->{actions}}, { type => 2, message => '[LOAD] import table [' . $name . ']', command => $cmd }; + push @{$etl->{run}->{schedule}->{import}->{actions}}, $action; + } +} + +# load data into the reporting server from files copied from the monitoring server +sub extractCentreonDB { + my ($etl, $etlProperties) = @_; + + my $tables = 'host hostgroup_relation hostgroup hostcategories_relation hostcategories ' . + 'host_service_relation service service_categories service_categories_relation ' . + 'timeperiod mod_bi_options servicegroup mod_bi_options_centiles servicegroup_relation contact contactgroup_service_relation '. + 'host_template_relation command contact_host_relation contactgroup_host_relation contactgroup contact_service_relation'; + + my $mon = $utils->buildCliMysqlArgs($etl->{run}->{dbmon}->{centreon}); + my $bi = $utils->buildCliMysqlArgs($etl->{run}->{dbbi}->{centreon}); + + my $cmd = sprintf( + "mysqldump --replace --skip-add-drop-table --skip-add-locks --skip-comments %s '%s' %s | mysql --force %s '%s'", + $mon, + $etl->{run}->{dbmon}->{centreon}->{db}, + $tables, + $bi, + $etl->{run}->{dbbi}->{centreon}->{db} + ); + + push @{$etl->{run}->{schedule}->{import}->{actions}}, + { type => 2, message => '[LOAD] import table [' . $tables . ']', command => $cmd }; +} + +sub dataBin { + my ($etl, $etlProperties, $options, $periods) = @_; + + return if ($options->{ignore_databin} == 1 || $options->{centreon_only} == 1 || (defined($options->{bam_only}) && $options->{bam_only} == 1)); + + my $action = { type => 1, db => 'centstorage', sql => [], actions => [] }; + + my $drop = 0; + if ($options->{rebuild} == 1 && $options->{nopurge} == 0) { + push @{$action->{sql}}, [ '[CREATE] Deleting table [data_bin]', 'DROP TABLE IF EXISTS `data_bin`' ]; + $drop = 1; + } + + my $isExists = 0; + $isExists = 1 if ($biTables->tableExists('data_bin')); + + my $partitionsPerf = $utils->getRangePartitionDate($periods->{raw_perfdata}->{start}, $periods->{raw_perfdata}->{end}); + + if ($isExists == 0 || $drop == 1) { + $action->{create} = 1; + + my $structure = $monTables->dumpTableStructure('data_bin'); + $structure =~ s/KEY.*\(\`id_metric\`\)\,//g; + $structure =~ s/KEY.*\(\`id_metric\`\)//g; + $structure =~ s/\n.*PARTITION.*//g; + $structure =~ s/\,[\n\s]+\)/\)/; + $structure .= " PARTITION BY RANGE(`ctime`) ("; + + my $append = ''; + foreach (@$partitionsPerf) { + $structure .= $append . "PARTITION p" . $_->{name} . " VALUES LESS THAN (" . $_->{epoch} . ")"; + $append = ','; + } + $structure .= ');'; + + push @{$action->{sql}}, + [ '[CREATE] Add table [data_bin]', $structure ], + [ '[INDEXING] Adding index [ctime] on table [data_bin]', "ALTER TABLE `data_bin` ADD INDEX `idx_data_bin_ctime` (`ctime`)" ], + [ '[INDEXING] Adding index [id_metric_id, ctime] on table [data_bin]', "ALTER TABLE `data_bin` ADD INDEX `idx_data_bin_idmetric_ctime` (`id_metric`,`ctime`)" ]; + } + + if ($isExists == 1 && $drop == 0) { + my $start = $biTables->getLastPartRange('data_bin'); + my $partitions = $utils->getRangePartitionDate($start, $periods->{raw_perfdata}->{end}); + foreach (@$partitions) { + push @{$action->{sql}}, + [ '[PARTITIONS] Add partition [' . $_->{name} . '] on table [data_bin]', "ALTER TABLE `data_bin` ADD PARTITION (PARTITION `p$_->{name}` VALUES LESS THAN($_->{epoch}))"]; + } + } + + if ($etl->{run}->{options}->{create_tables} == 0 && ($etlProperties->{'statistics.type'} eq 'all' || $etlProperties->{'statistics.type'} eq 'perfdata')) { + my $epoch = $utils->getDateEpoch($periods->{raw_perfdata}->{start}); + + my $overCond = 'ctime >= ' . $epoch . ' AND '; + foreach (@$partitionsPerf) { + my $cmd = sprintf( + "mysqldump --insert-ignore --single-transaction --no-create-info --skip-add-drop-table --skip-disable-keys --skip-add-locks --skip-comments %s --databases '%s' --tables %s --where=\"%s\" | mysql --init-command='SET SESSION unique_checks=0' %s '%s'", + $argsMon, + $etl->{run}->{dbmon}->{centstorage}->{db}, + 'data_bin', + $overCond . 'ctime < ' . $_->{epoch}, + $argsBi, + $etl->{run}->{dbbi}->{centstorage}->{db} + ); + $overCond = 'ctime >= ' . $_->{epoch} . ' AND '; + push @{$action->{actions}}, { type => 2, message => '[LOAD] partition [' . $_->{name} . '] on table [data_bin]', command => $cmd }; + } + + #my $file = $etlProperties->{'reporting.storage.directory'} . '/data_bin.sql'; + #push @{$action->{actions}}, { + # type => 3, + # message => '[LOAD] table [data_bin]', + # table => 'data_bin', + # db => 'centstorage', + # dump => $cmd, + # file => $file, + # load => "LOAD DATA LOCAL INFILE '" . $file . "' INTO TABLE `data_bin` CHARACTER SET UTF8 IGNORE 1 LINES" + #}; + } + + push @{$etl->{run}->{schedule}->{import}->{actions}}, $action; +} + +sub selectTables { + my ($etl, $etlProperties, $options) = @_; + + my @notTimedTables = (); + my %timedTables = (); + + my @ctime = ('ctime', 'ctime'); + my @startEnd = ('date_start', 'date_end'); + my @timeId = ('time_id', 'time_id'); + my $importComment = $etlProperties->{'import.comments'}; + my $importDowntimes = $etlProperties->{'import.downtimes'}; + + if (!defined($etlProperties->{'statistics.type'})) { + die 'cannot determine statistics type or compatibility mode for data integration'; + } + + if (!defined($options->{databin_only}) || $options->{databin_only} == 0) { + if (!defined($options->{bam_only}) || $options->{bam_only} == 0) { + if ($etlProperties->{'statistics.type'} eq 'all') { + push @notTimedTables, 'index_data'; + push @notTimedTables, 'metrics'; + push @notTimedTables, 'hoststateevents'; + push @notTimedTables, 'servicestateevents'; + push @notTimedTables, 'instances'; + push @notTimedTables, 'hosts'; + + if ($importComment eq 'true'){ + push @notTimedTables, 'comments'; + } + if ($importDowntimes eq 'true'){ + push @notTimedTables, 'downtimes'; + } + + push @notTimedTables, 'acknowledgements'; + } + if ($etlProperties->{'statistics.type'} eq 'availability') { + push @notTimedTables, 'hoststateevents'; + push @notTimedTables, 'servicestateevents'; + push @notTimedTables, 'instances'; + push @notTimedTables, 'hosts'; + if ($importComment eq 'true'){ + push @notTimedTables, 'comments'; + } + push @notTimedTables, 'acknowledgements'; + } + if ($etlProperties->{'statistics.type'} eq "perfdata") { + push @notTimedTables, 'index_data'; + push @notTimedTables, 'metrics'; + push @notTimedTables, 'instances'; + push @notTimedTables, 'hosts'; + push @notTimedTables, 'acknowledgements'; + + } + } + + my $sth = $etl->{run}->{dbmon_centreon_con}->query("SELECT id FROM modules_informations WHERE name='centreon-bam-server'"); + if (my $row = $sth->fetchrow_array() && $etlProperties->{'statistics.type'} ne 'perfdata') { + push @notTimedTables, "mod_bam_reporting_ba_availabilities"; + push @notTimedTables, "mod_bam_reporting_ba"; + push @notTimedTables, "mod_bam_reporting_ba_events"; + push @notTimedTables, "mod_bam_reporting_ba_events_durations"; + push @notTimedTables, "mod_bam_reporting_bv"; + push @notTimedTables, "mod_bam_reporting_kpi"; + push @notTimedTables, "mod_bam_reporting_kpi_events"; + push @notTimedTables, "mod_bam_reporting_relations_ba_bv"; + push @notTimedTables, "mod_bam_reporting_relations_ba_kpi_events"; + push @notTimedTables, "mod_bam_reporting_timeperiods"; + } + } + + return (\@notTimedTables, \%timedTables); +} + +sub prepare { + my ($etl) = @_; + + initVars($etl); + + # define data extraction period based on program options --start & --end or on data retention period + my %periods; + if ($etl->{run}->{options}->{rebuild} == 1 || $etl->{run}->{options}->{create_tables}) { + if ($etl->{run}->{options}->{start} eq '' && $etl->{run}->{options}->{end} eq '') { + # get max values for retention by type of statistics in order to be able to rebuild hourly and daily stats + my ($start, $end) = $etl->{etlProp}->getMaxRetentionPeriodFor('perfdata'); + + $periods{raw_perfdata} = { start => $start, end => $end }; + ($start, $end) = $etl->{etlProp}->getMaxRetentionPeriodFor('availability'); + $periods{raw_availabilitydata} = { start => $start, end => $end}; + } elsif ($etl->{run}->{options}->{start} ne '' && $etl->{run}->{options}->{end} ne '') { + # set period defined manually + my %dates = (start => $etl->{run}->{options}->{start}, end => $etl->{run}->{options}->{end}); + $periods{raw_perfdata} = \%dates; + $periods{raw_availabilitydata} = \%dates; + } + } else { + # set yesterday start and end dates as period (--daily) + my %dates; + ($dates{start}, $dates{end}) = $utils->getYesterdayTodayDate(); + $periods{raw_perfdata} = \%dates; + $periods{raw_availabilitydata} = \%dates; + } + + # identify the Centreon Storage DB tables to extract based on ETL properties + my ($notTimedTables, $timedTables) = selectTables( + $etl, + $etl->{run}->{etlProperties}, + $etl->{run}->{options} + ); + + dataBin( + $etl, + $etl->{run}->{etlProperties}, + $etl->{run}->{options}, + \%periods + ); + + # create non existing tables + createTables($etl, \%periods, $etl->{run}->{options}, $notTimedTables); + + # If we only need to create empty tables, create them then exit program + return if ($etl->{run}->{options}->{create_tables} == 1); + + # extract raw availability and perfdata from monitoring server and insert it into reporting server + if ($etl->{run}->{options}->{centreon_only} == 0) { + extractData($etl, $etl->{run}->{options}, $notTimedTables); + } + + # extract Centreon configuration DB from monitoring server and insert it into reporting server + if ((!defined($etl->{run}->{options}->{databin_only}) || $etl->{run}->{options}->{databin_only} == 0) + && (!defined($etl->{run}->{options}->{bam_only}) || $etl->{run}->{options}->{bam_only} == 0)) { + extractCentreonDB($etl, $etl->{run}->{etlProperties}); + } +} + +1; diff --git a/gorgone/modules/centreon/mbi/etl/perfdata/main.pm b/gorgone/modules/centreon/mbi/etl/perfdata/main.pm new file mode 100644 index 0000000..c5fbd0a --- /dev/null +++ b/gorgone/modules/centreon/mbi/etl/perfdata/main.pm @@ -0,0 +1,416 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::etl::perfdata::main; + +use strict; +use warnings; + +use gorgone::modules::centreon::mbi::libs::bi::Time; +use gorgone::modules::centreon::mbi::libs::bi::LiveService; +use gorgone::modules::centreon::mbi::libs::bi::MySQLTables; +use gorgone::modules::centreon::mbi::libs::Utils; +use gorgone::standard::constants qw(:all); + +my ($biTables, $utils, $liveService, $time); + +sub initVars { + my ($etl) = @_; + + $biTables = gorgone::modules::centreon::mbi::libs::bi::MySQLTables->new($etl->{run}->{messages}, $etl->{run}->{dbbi_centstorage_con}); + $utils = gorgone::modules::centreon::mbi::libs::Utils->new($etl->{run}->{messages}); + $liveService = gorgone::modules::centreon::mbi::libs::bi::LiveService->new($etl->{run}->{messages}, $etl->{run}->{dbbi_centstorage_con}); + $time = gorgone::modules::centreon::mbi::libs::bi::Time->new($etl->{run}->{messages}, $etl->{run}->{dbbi_centstorage_con}); +} + +sub emptyTableForRebuild { + my ($etl, %options) = @_; + + my $sql = [ [ '[CREATE] Deleting table [' . $options{name} . ']', 'DROP TABLE IF EXISTS `' . $options{name} . '`' ] ]; + + my $structure = $biTables->dumpTableStructure($options{name}); + $structure =~ s/KEY.*\(\`$options{column}\`\)\,//g; + $structure =~ s/KEY.*\(\`$options{column}\`\)//g; + $structure =~ s/\,[\n\s+]+\)/\n\)/g; + + if (defined($options{start})) { + $structure =~ s/\n.*PARTITION.*//g; + $structure =~ s/\,[\n\s]+\)/\)/; + $structure .= ' PARTITION BY RANGE(`' . $options{column} . '`) ('; + + my $partitionsPerf = $utils->getRangePartitionDate($options{start}, $options{end}); + + my $append = ''; + foreach (@$partitionsPerf) { + $structure .= $append . "PARTITION p" . $_->{name} . " VALUES LESS THAN (" . $_->{epoch} . ")"; + $append = ','; + } + $structure .= ');'; + } + + push @$sql, + [ '[CREATE] Add table [' . $options{name} . ']', $structure ], + [ "[INDEXING] Adding index [idx_$options{name}_$options{column}] on table [$options{name}]", "ALTER TABLE `$options{name}` ADD INDEX `idx_$options{name}_$options{column}` (`$options{column}`)" ]; + + push @{$etl->{run}->{schedule}->{perfdata}->{stages}->[0]}, { type => 'sql', db => 'centstorage', sql => $sql }; +} + +sub deleteEntriesForRebuild { + my ($etl, %options) = @_; + + my $sql = []; + if (!$biTables->isTablePartitioned($options{name})) { + push @$sql, + [ + "[PURGE] Delete table [$options{name}] from $options{start} to $options{end}", + "DELETE FROM $options{name} WHERE time_id >= " . $utils->getDateEpoch($options{start}) . " AND time_id < " . $utils->getDateEpoch($options{end}) + ]; + } else { + my $partitionsPerf = $utils->getRangePartitionDate($options{start}, $options{end}); + foreach (@$partitionsPerf) { + push @$sql, + [ + "[PURGE] Truncate partition $_->{name} on table [$options{name}]", + "ALTER TABLE $options{name} TRUNCATE PARTITION p$_->{name}" + ]; + } + } + + push @{$etl->{run}->{schedule}->{perfdata}->{stages}->[0]}, { type => 'sql', db => 'centstorage', sql => $sql }; +} + +sub purgeTables { + my ($etl, $periods) = @_; + + my ($daily_start, $daily_end) = ($periods->{'perfdata.daily'}->{'start'}, $periods->{'perfdata.daily'}->{'end'}); + my ($hourly_start, $hourly_end) = ($periods->{'perfdata.hourly'}->{'start'}, $periods->{'perfdata.hourly'}->{'end'}); + + #To prevent from purging monthly data when the no-purge rebuild is made inside one month + my $firstDayOfMonth = $daily_start; + my $firstDayOfMonthEnd = $daily_end; + my $startAndEndSameMonth = 0; + $firstDayOfMonth =~ s/([1-2][0-9]{3})\-([0-1][0-9])\-[0-3][0-9]/$1\-$2\-01/; + $firstDayOfMonthEnd =~ s/([1-2][0-9]{3})\-([0-1][0-9])\-[0-3][0-9]/$1\-$2\-01/; + + if ($firstDayOfMonth eq $firstDayOfMonthEnd) { + $startAndEndSameMonth = 1; + } + + if ($etl->{run}->{options}->{nopurge} == 1) { + # deleting data that will be rewritten + if ($etl->{run}->{etlProperties}->{'perfdata.granularity'} ne 'hour' && (!defined($etl->{run}->{options}->{month_only}) || $etl->{run}->{options}->{month_only} == 0)) { + if ((!defined($etl->{run}->{options}->{centile_only}) || $etl->{run}->{options}->{centile_only} == 0)) { + deleteEntriesForRebuild($etl, name => 'mod_bi_metricdailyvalue', start => $daily_start, end => $daily_end); + + if ($etl->{run}->{etlProperties}->{'perfdata.granularity'} ne "day" && (!defined($etl->{run}->{options}->{month_only}) || $etl->{run}->{options}->{month_only} == 0)) { + deleteEntriesForRebuild($etl, name => 'mod_bi_metrichourlyvalue', start => $hourly_start, end => $hourly_end); + } + + #Deleting monthly data only if start and end are not in the same month + if (!$startAndEndSameMonth) { + deleteEntriesForRebuild($etl, name => 'mod_bi_metricmonthcapacity', start => $firstDayOfMonth, end => $daily_end); + } + } + + if ((!defined($etl->{run}->{options}->{no_centile}) || $etl->{run}->{options}->{no_centile} == 0)) { + if (defined($etl->{run}->{etlProperties}->{'centile.day'}) && $etl->{run}->{etlProperties}->{'centile.day'} eq '1') { + deleteEntriesForRebuild($etl, name => 'mod_bi_metriccentiledailyvalue', start => $daily_start, end => $daily_end); + } + if (defined($etl->{run}->{etlProperties}->{'centile.week'}) && $etl->{run}->{etlProperties}->{'centile.week'} eq '1') { + deleteEntriesForRebuild($etl, name => 'mod_bi_metriccentileweeklyvalue', start => $daily_start, end => $daily_end); + } + + if (defined($etl->{run}->{etlProperties}->{'centile.month'}) && $etl->{run}->{etlProperties}->{'centile.month'} eq '1' && !$startAndEndSameMonth) { + deleteEntriesForRebuild($etl, name => 'mod_bi_metriccentilemonthlyvalue', start => $firstDayOfMonth, end => $daily_end); + } + } + } + } else { + # deleting and recreating tables, recreating partitions for daily and hourly tables + if ($etl->{run}->{etlProperties}->{'perfdata.granularity'} ne "hour" && (!defined($etl->{run}->{options}->{month_only}) || $etl->{run}->{options}->{month_only} == 0)) { + if ((!defined($etl->{run}->{options}->{centile_only}) || $etl->{run}->{options}->{centile_only} == 0)) { + emptyTableForRebuild($etl, name => 'mod_bi_metricdailyvalue', column => 'time_id', start => $daily_start, end => $daily_end); + + emptyTableForRebuild($etl, name => 'mod_bi_metricmonthcapacity', column => 'time_id'); + } + + if ((!defined($etl->{run}->{options}->{no_centile}) || $etl->{run}->{options}->{no_centile} == 0)) { + #Managing Daily Centile table + if (defined($etl->{run}->{etlProperties}->{'centile.day'}) && $etl->{run}->{etlProperties}->{'centile.day'} eq '1') { + emptyTableForRebuild($etl, name => 'mod_bi_metriccentiledailyvalue', column => 'time_id', start => $daily_start, end => $daily_end); + } + #Managing Weekly Centile table + if (defined($etl->{run}->{etlProperties}->{'centile.week'}) && $etl->{run}->{etlProperties}->{'centile.week'} eq '1') { + emptyTableForRebuild($etl, name => 'mod_bi_metriccentileweeklyvalue', column => 'time_id', start => $daily_start, end => $daily_end); + } + #Managing Monthly Centile table + if (defined($etl->{run}->{etlProperties}->{'centile.month'}) && $etl->{run}->{etlProperties}->{'centile.month'} eq '1') { + emptyTableForRebuild($etl, name => 'mod_bi_metriccentilemonthlyvalue', column => 'time_id', start => $daily_start, end => $daily_end); + } + } + } + + if ($etl->{run}->{etlProperties}->{'perfdata.granularity'} ne "day" && + (!defined($etl->{run}->{options}->{month_only}) || $etl->{run}->{options}->{month_only} == 0) && + (!defined($etl->{run}->{options}->{no_centile}) || $etl->{run}->{options}->{no_centile} == 0)) { + emptyTableForRebuild($etl, name => 'mod_bi_metrichourlyvalue', column => 'time_id', start => $hourly_start, end => $hourly_end); + } + } +} + +sub processDay { + my ($etl, $liveServices, $start, $end) = @_; + + if ($etl->{run}->{etlProperties}->{'perfdata.granularity'} eq 'hour' || + (defined($etl->{run}->{options}->{month_only}) && $etl->{run}->{options}->{month_only} == 1)) { + return 1; + } + + my ($currentDayId, $currentDayUtime) = $time->getEntryID($start); + + if ((!defined($etl->{run}->{options}->{centile_only}) || $etl->{run}->{options}->{centile_only} == 0)) { + while (my ($liveServiceName, $liveServiceId) = each (%$liveServices)) { + push @{$etl->{run}->{schedule}->{perfdata}->{stages}->[1]}, { + type => 'perfdata_day', + liveserviceName => $liveServiceName, + liveserviceId => $liveServiceId, + start => $start, + end => $end + }; + } + } + + if ((!defined($etl->{run}->{options}->{no_centile}) || $etl->{run}->{options}->{no_centile} == 0)) { + if (defined($etl->{run}->{etlProperties}->{'centile.include.servicecategories'}) && $etl->{run}->{etlProperties}->{'centile.include.servicecategories'} ne '') { + if (defined($etl->{run}->{etlProperties}->{'centile.day'}) && $etl->{run}->{etlProperties}->{'centile.day'} eq '1') { + push @{$etl->{run}->{schedule}->{perfdata}->{stages}->[2]}, { + type => 'centile_day', + start => $start, + end => $end + }; + } + if (defined($etl->{run}->{etlProperties}->{'centile.week'}) && $etl->{run}->{etlProperties}->{'centile.week'} eq '1') { + if ($utils->getDayOfWeek($end) eq $etl->{run}->{etlProperties}->{'centile.weekFirstDay'}) { + processWeek($etl, $end); + } + } + } + } +} + +sub processWeek { + my ($etl, $date) = @_; + + my $start = $utils->subtractDateDays($date, 7); + my $end = $utils->subtractDateDays($date, 1); + + $time->insertTimeEntriesForPeriod($start, $end); + + push @{$etl->{run}->{schedule}->{perfdata}->{stages}->[2]}, { + type => 'centile_week', + start => $start, + end => $end + }; +} + +sub processMonth { + my ($etl, $liveServices, $date) = @_; + + my $start = $utils->subtractDateMonths($date, 1); + my $end = $utils->subtractDateDays($date, 1); + + $time->insertTimeEntriesForPeriod($start, $end); + + my ($previousMonthStartTimeId, $previousMonthStartUtime) = $time->getEntryID($start); + my ($previousMonthEndTimeId, $previousMonthEndUtime) = $time->getEntryID($end); + + if (!defined($etl->{run}->{etlProperties}->{'capacity.include.servicecategories'}) || $etl->{run}->{etlProperties}->{'capacity.include.servicecategories'} eq "" + || !defined($etl->{run}->{etlProperties}->{'capacity.include.liveservices'}) || $etl->{run}->{etlProperties}->{'capacity.include.liveservices'} eq "") { + $etl->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $etl->{run}->{token}, data => { messages => [ ['I', "[SCHEDULER][PERFDATA] Skipping month: [" . $start . "] to [" . $end . "]" ] ] }); + return ; + } + + if ((!defined($etl->{run}->{options}->{centile_only}) || $etl->{run}->{options}->{centile_only} == 0) && + $etl->{run}->{etlProperties}->{'perfdata.granularity'} ne 'hour') { + push @{$etl->{run}->{schedule}->{perfdata}->{stages}->[2]}, { + type => 'perfdata_month', + start => $start, + end => $end + }; + } + + if ((!defined($etl->{run}->{options}->{no_centile}) || $etl->{run}->{options}->{no_centile} == 0) && + $etl->{run}->{etlProperties}->{'centile.month'} && $etl->{run}->{etlProperties}->{'perfdata.granularity'} ne 'hour') { + if (defined($etl->{run}->{etlProperties}->{'centile.include.servicecategories'}) && $etl->{run}->{etlProperties}->{'centile.include.servicecategories'} ne '') { + push @{$etl->{run}->{schedule}->{perfdata}->{stages}->[2]}, { + type => 'centile_month', + start => $start, + end => $end + }; + } + } +} + +sub processHours { + my ($etl, $start, $end) = @_; + + if ($etl->{run}->{etlProperties}->{'perfdata.granularity'} eq 'day' || + (defined($etl->{run}->{options}->{month_only}) && $etl->{run}->{options}->{month_only} == 1) || + (defined($etl->{run}->{options}->{centile_only}) && $etl->{run}->{options}->{centile_only} == 1)) { + return 1; + } + + push @{$etl->{run}->{schedule}->{perfdata}->{stages}->[2]}, { + type => 'perfdata_hour', + start => $start, + end => $end + }; +} + +sub processDayAndMonthAgregation { + my ($etl, $liveServices, $start, $end) = @_; + + processDay($etl, $liveServices, $start, $end); + my ($year, $mon, $day) = split ("-", $end); + if ($day == 1) { + processMonth($etl, $liveServices, $end); + } +} + +sub dailyProcessing { + my ($etl, $liveServices) = @_; + + # getting yesterday start and end date to process yesterday data + my ($start, $end) = $utils->getYesterdayTodayDate(); + # daily mod_bi_time table filling + $time->insertTimeEntriesForPeriod($start, $end); + + my ($epoch, $partName) = $utils->getDateEpoch($end); + push @{$etl->{run}->{schedule}->{perfdata}->{stages}->[0]}, { + type => 'sql', + db => 'centstorage', + sql => [ + [ + '[PARTITIONS] Add partition [p' . $partName . '] on table [mod_bi_metricdailyvalue]', + "ALTER TABLE `mod_bi_metricdailyvalue` ADD PARTITION (PARTITION `p$partName` VALUES LESS THAN(" . $epoch . "))" + ] + ] + }; + if ($etl->{run}->{etlProperties}->{'perfdata.granularity'} ne 'day') { + push @{$etl->{run}->{schedule}->{perfdata}->{stages}->[0]}, { + type => 'sql', + db => 'centstorage', + sql => [ + [ + '[PARTITIONS] Add partition [p' . $partName . '] on table [mod_bi_metrichourlyvalue]', + "ALTER TABLE `mod_bi_metrichourlyvalue` ADD PARTITION (PARTITION `p$partName` VALUES LESS THAN(" . $epoch . "))" + ] + ] + }; + } + if (defined($etl->{run}->{etlProperties}->{'centile.day'}) && $etl->{run}->{etlProperties}->{'centile.day'} eq '1') { + push @{$etl->{run}->{schedule}->{perfdata}->{stages}->[0]}, { + type => 'sql', + db => 'centstorage', + sql => [ + [ + '[PARTITIONS] Add partition [p' . $partName . '] on table [mod_bi_metriccentiledailyvalue]', + "ALTER TABLE `mod_bi_metriccentiledailyvalue` ADD PARTITION (PARTITION `p$partName` VALUES LESS THAN(" . $epoch . "))" + ] + ] + }; + } + + # processing agregation by month. If the day is the first day of the month, also processing agregation by month + processDayAndMonthAgregation($etl, $liveServices, $start, $end); + + # processing agregation by hour + processHours($etl, $start, $end); +} + +sub rebuildProcessing { + my ($etl, $liveServices) = @_; + + # getting rebuild period by granularity of perfdata from data retention rules + my $periods = $etl->{etlProp}->getRetentionPeriods(); + + my ($start, $end); + if ($etl->{run}->{options}->{start} ne '' && $etl->{run}->{options}->{end} ne '') { + ($start, $end) = ($etl->{run}->{options}->{start}, $etl->{run}->{options}->{end}); + while (my ($key, $values) = each %$periods) { + $values->{start} = $etl->{run}->{options}->{start}; + $values->{end} = $etl->{run}->{options}->{end}; + } + } else { + # getting max perfdata retention period to fill mod_bi_time + ($start, $end) = $etl->{etlProp}->getMaxRetentionPeriodFor('perfdata'); + } + + # insert entries into table mod_bi_time + $time->insertTimeEntriesForPeriod($start, $end); + + purgeTables($etl, $periods); + + # rebuilding statistics by day and by month + ($start, $end) = ($periods->{'perfdata.daily'}->{start}, $periods->{'perfdata.daily'}->{end}); + + my $days = $utils->getRangePartitionDate($start, $end); + foreach (@$days) { + $end = $_->{date}; + processDayAndMonthAgregation($etl, $liveServices, $start, $end); + $start = $end; + } + + # rebuilding statistics by hour + ($start, $end) = ($periods->{'perfdata.hourly'}->{start}, $periods->{'perfdata.hourly'}->{'end'}); + + $days = $utils->getRangePartitionDate($start, $end); + foreach (@$days) { + $end = $_->{date}; + processHours($etl, $start, $end); + $start = $end; + } +} + +sub prepare { + my ($etl) = @_; + + initVars($etl); + + if (!defined($etl->{run}->{etlProperties}->{'statistics.type'}) || $etl->{run}->{etlProperties}->{'statistics.type'} eq "availability") { + $etl->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $etl->{run}->{token}, data => { messages => [ ['I', '[SCHEDULER][PERFDATA] Performance statistics calculation disabled' ] ] }); + return ; + } + + if ((!defined($etl->{run}->{options}->{no_centile}) || $etl->{run}->{options}->{no_centile} == 0) && + defined($etl->{run}->{etlProperties}->{'centile.include.servicecategories'}) and $etl->{run}->{etlProperties}->{'centile.include.servicecategories'} eq '') { + $etl->send_log(code => GORGONE_MODULE_CENTREON_MBIETL_PROGRESS, token => $etl->{run}->{token}, data => { messages => [ ['I', '[SCHEDULER][PERFDATA] No service categories selected for centile calculation - centile agregation will not be calculated' ] ] }); + } + + my $liveServiceList = $liveService->getLiveServicesByNameForTpIds($etl->{run}->{etlProperties}->{'liveservices.perfdata'}); + + if ($etl->{run}->{options}->{daily} == 1) { + dailyProcessing($etl, $liveServiceList); + } elsif ($etl->{run}->{options}->{rebuild} == 1) { + rebuildProcessing($etl, $liveServiceList); + } +} + +1; diff --git a/gorgone/modules/centreon/mbi/etlworkers/class.pm b/gorgone/modules/centreon/mbi/etlworkers/class.pm new file mode 100644 index 0000000..3bbadca --- /dev/null +++ b/gorgone/modules/centreon/mbi/etlworkers/class.pm @@ -0,0 +1,349 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::etlworkers::class; + +use base qw(gorgone::class::module); + +use strict; +use warnings; +use gorgone::standard::library; +use gorgone::standard::constants qw(:all); +use gorgone::class::http::http; +use ZMQ::LibZMQ4; +use ZMQ::Constants qw(:all); +use JSON::XS; +use Try::Tiny; +use gorgone::modules::centreon::mbi::etlworkers::import::main; +use gorgone::modules::centreon::mbi::etlworkers::dimensions::main; +use gorgone::modules::centreon::mbi::etlworkers::event::main; +use gorgone::modules::centreon::mbi::etlworkers::perfdata::main; +use gorgone::modules::centreon::mbi::libs::Messages; + +my %handlers = (TERM => {}, HUP => {}); +my ($connector); + +sub new { + my ($class, %options) = @_; + $connector = $class->SUPER::new(%options); + bless $connector, $class; + + $connector->{pool_id} = $options{pool_id}; + + $connector->set_signal_handlers(); + return $connector; +} + +sub set_signal_handlers { + my $self = shift; + + $SIG{TERM} = \&class_handle_TERM; + $handlers{TERM}->{$self} = sub { $self->handle_TERM() }; + $SIG{HUP} = \&class_handle_HUP; + $handlers{HUP}->{$self} = sub { $self->handle_HUP() }; +} + +sub handle_HUP { + my $self = shift; + $self->{reload} = 0; +} + +sub handle_TERM { + my $self = shift; + $self->{logger}->writeLogDebug("[nodes] $$ Receiving order to stop..."); + $self->{stop} = 1; +} + +sub class_handle_TERM { + foreach (keys %{$handlers{TERM}}) { + &{$handlers{TERM}->{$_}}(); + } +} + +sub class_handle_HUP { + foreach (keys %{$handlers{HUP}}) { + &{$handlers{HUP}->{$_}}(); + } +} + +sub db_connections { + my ($self, %options) = @_; + + if (!defined($self->{dbmon_centstorage_con}) || $self->{dbmon_centstorage_con}->sameParams(%{$options{dbmon}->{centstorage}}) == 0) { + $self->{dbmon_centstorage_con} = gorgone::class::db->new( + type => 'mysql', + force => 2, + logger => $self->{logger}, + die => 1, + %{$options{dbmon}->{centstorage}} + ); + } + if (!defined($self->{dbbi_centstorage_con}) || $self->{dbbi_centstorage_con}->sameParams(%{$options{dbbi}->{centstorage}}) == 0) { + $self->{dbbi_centstorage_con} = gorgone::class::db->new( + type => 'mysql', + force => 2, + logger => $self->{logger}, + die => 1, + %{$options{dbbi}->{centstorage}} + ); + } + + if (!defined($self->{dbmon_centreon_con}) || $self->{dbmon_centreon_con}->sameParams(%{$options{dbmon}->{centreon}}) == 0) { + $self->{dbmon_centreon_con} = gorgone::class::db->new( + type => 'mysql', + force => 2, + logger => $self->{logger}, + die => 1, + %{$options{dbmon}->{centreon}} + ); + } + if (!defined($self->{dbbi_centreon_con}) || $self->{dbbi_centreon_con}->sameParams(%{$options{dbbi}->{centreon}}) == 0) { + $self->{dbbi_centreon_con} = gorgone::class::db->new( + type => 'mysql', + force => 2, + logger => $self->{logger}, + die => 1, + %{$options{dbbi}->{centreon}} + ); + } +} + +sub action_centreonmbietlworkersimport { + my ($self, %options) = @_; + + $options{token} = $self->generate_token() if (!defined($options{token})); + + $self->{messages} = gorgone::modules::centreon::mbi::libs::Messages->new(); + my $code = GORGONE_ACTION_FINISH_OK; + + try { + $self->db_connections( + dbmon => $options{data}->{content}->{dbmon}, + dbbi => $options{data}->{content}->{dbbi} + ); + if ($options{data}->{content}->{params}->{type} == 1) { + gorgone::modules::centreon::mbi::etlworkers::import::main::sql($self, params => $options{data}->{content}->{params}); + } elsif ($options{data}->{content}->{params}->{type} == 2) { + gorgone::modules::centreon::mbi::etlworkers::import::main::command($self, params => $options{data}->{content}->{params}); + } elsif ($options{data}->{content}->{params}->{type} == 3) { + gorgone::modules::centreon::mbi::etlworkers::import::main::load($self, params => $options{data}->{content}->{params}); + } + } catch { + $code = GORGONE_ACTION_FINISH_KO; + $self->{messages}->writeLog('ERROR', $_, 1); + }; + + $self->send_log( + code => $code, + token => $options{token}, + data => { + messages => $self->{messages}->getLogs() + } + ); +} + +sub action_centreonmbietlworkersdimensions { + my ($self, %options) = @_; + + $options{token} = $self->generate_token() if (!defined($options{token})); + + $self->{messages} = gorgone::modules::centreon::mbi::libs::Messages->new(); + my $code = GORGONE_ACTION_FINISH_OK; + + try { + $self->db_connections( + dbmon => $options{data}->{content}->{dbmon}, + dbbi => $options{data}->{content}->{dbbi} + ); + + gorgone::modules::centreon::mbi::etlworkers::dimensions::main::execute( + $self, + dbmon => $options{data}->{content}->{dbmon}, + dbbi => $options{data}->{content}->{dbbi}, + params => $options{data}->{content}->{params}, + etlProperties => $options{data}->{content}->{etlProperties}, + options => $options{data}->{content}->{options} + ); + } catch { + $code = GORGONE_ACTION_FINISH_KO; + $self->{messages}->writeLog('ERROR', $_, 1); + }; + + $self->send_log( + code => $code, + token => $options{token}, + data => { + messages => $self->{messages}->getLogs() + } + ); +} + +sub action_centreonmbietlworkersevent { + my ($self, %options) = @_; + + $options{token} = $self->generate_token() if (!defined($options{token})); + + $self->{messages} = gorgone::modules::centreon::mbi::libs::Messages->new(); + my $code = GORGONE_ACTION_FINISH_OK; + + try { + $self->db_connections( + dbmon => $options{data}->{content}->{dbmon}, + dbbi => $options{data}->{content}->{dbbi} + ); + if ($options{data}->{content}->{params}->{type} eq 'sql') { + gorgone::modules::centreon::mbi::etlworkers::event::main::sql($self, params => $options{data}->{content}->{params}); + } elsif ($options{data}->{content}->{params}->{type} eq 'events') { + gorgone::modules::centreon::mbi::etlworkers::event::main::events( + $self, + dbmon => $options{data}->{content}->{dbmon}, + dbbi => $options{data}->{content}->{dbbi}, + etlProperties => $options{data}->{content}->{etlProperties}, + params => $options{data}->{content}->{params}, + options => $options{data}->{content}->{options} + ); + } elsif ($options{data}->{content}->{params}->{type} =~ /^availability_/) { + gorgone::modules::centreon::mbi::etlworkers::event::main::availability( + $self, + dbmon => $options{data}->{content}->{dbmon}, + dbbi => $options{data}->{content}->{dbbi}, + etlProperties => $options{data}->{content}->{etlProperties}, + params => $options{data}->{content}->{params} + ); + } + } catch { + $code = GORGONE_ACTION_FINISH_KO; + $self->{messages}->writeLog('ERROR', $_, 1); + }; + + $self->send_log( + code => $code, + token => $options{token}, + data => { + messages => $self->{messages}->getLogs() + } + ); +} + +sub action_centreonmbietlworkersperfdata { + my ($self, %options) = @_; + + $options{token} = $self->generate_token() if (!defined($options{token})); + + $self->{messages} = gorgone::modules::centreon::mbi::libs::Messages->new(); + my $code = GORGONE_ACTION_FINISH_OK; + + try { + $self->db_connections( + dbmon => $options{data}->{content}->{dbmon}, + dbbi => $options{data}->{content}->{dbbi} + ); + + if ($options{data}->{content}->{params}->{type} eq 'sql') { + gorgone::modules::centreon::mbi::etlworkers::perfdata::main::sql($self, params => $options{data}->{content}->{params}); + } elsif ($options{data}->{content}->{params}->{type} =~ /^perfdata_/) { + gorgone::modules::centreon::mbi::etlworkers::perfdata::main::perfdata( + $self, + dbmon => $options{data}->{content}->{dbmon}, + dbbi => $options{data}->{content}->{dbbi}, + etlProperties => $options{data}->{content}->{etlProperties}, + params => $options{data}->{content}->{params}, + options => $options{data}->{content}->{options}, + pool_id => $self->{pool_id} + ); + } elsif ($options{data}->{content}->{params}->{type} =~ /^centile_/) { + gorgone::modules::centreon::mbi::etlworkers::perfdata::main::centile( + $self, + dbmon => $options{data}->{content}->{dbmon}, + dbbi => $options{data}->{content}->{dbbi}, + etlProperties => $options{data}->{content}->{etlProperties}, + params => $options{data}->{content}->{params}, + pool_id => $self->{pool_id} + ); + } + } catch { + $code = GORGONE_ACTION_FINISH_KO; + $self->{messages}->writeLog('ERROR', $_, 1); + }; + + $self->send_log( + code => $code, + token => $options{token}, + data => { + messages => $self->{messages}->getLogs() + } + ); +} + +sub event { + while (1) { + my $message = $connector->read_message(); + last if (!defined($message)); + + $connector->{logger}->writeLogDebug("[mbi-etlworkers] Event: $message"); + if ($message =~ /^\[(.*?)\]/) { + if ((my $method = $connector->can('action_' . lc($1)))) { + $message =~ /^\[(.*?)\]\s+\[(.*?)\]\s+\[.*?\]\s+(.*)$/m; + my ($action, $token) = ($1, $2); + my ($rv, $data) = $connector->json_decode(argument => $3, token => $token); + next if ($rv); + + $method->($connector, token => $token, data => $data); + } + } + } +} + +sub run { + my ($self, %options) = @_; + + # Connect internal + $connector->{internal_socket} = gorgone::standard::library::connect_com( + zmq_type => 'ZMQ_DEALER', + name => 'gorgone-' . $self->{module_id} . '-' . $self->{pool_id}, + logger => $self->{logger}, + type => $self->get_core_config(name => 'internal_com_type'), + path => $self->get_core_config(name => 'internal_com_path') + ); + $connector->send_internal_action( + action => 'CENTREONMBIETLWORKERSREADY', + data => { + pool_id => $self->{pool_id} + } + ); + $self->{poll} = [ + { + socket => $connector->{internal_socket}, + events => ZMQ_POLLIN, + callback => \&event + } + ]; + + while (1) { + my $rev = scalar(zmq_poll($self->{poll}, 5000)); + if (defined($rev) && $rev == 0 && $self->{stop} == 1) { + $self->{logger}->writeLogInfo("[" . $self->{module_id} . "] $$ has quit"); + zmq_close($connector->{internal_socket}); + exit(0); + } + } +} + +1; diff --git a/gorgone/modules/centreon/mbi/etlworkers/dimensions/main.pm b/gorgone/modules/centreon/mbi/etlworkers/dimensions/main.pm new file mode 100644 index 0000000..ec9c5dc --- /dev/null +++ b/gorgone/modules/centreon/mbi/etlworkers/dimensions/main.pm @@ -0,0 +1,263 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::etlworkers::dimensions::main; + +use strict; +use warnings; + +use IO::Socket::INET; + +use gorgone::modules::centreon::mbi::libs::centreon::Host; +use gorgone::modules::centreon::mbi::libs::centreon::HostGroup; +use gorgone::modules::centreon::mbi::libs::centreon::HostCategory; +use gorgone::modules::centreon::mbi::libs::centreon::ServiceCategory; +use gorgone::modules::centreon::mbi::libs::centreon::Service; +use gorgone::modules::centreon::mbi::libs::centreon::Timeperiod; +use gorgone::modules::centreon::mbi::libs::bi::BIHost; +use gorgone::modules::centreon::mbi::libs::bi::BIHostGroup; +use gorgone::modules::centreon::mbi::libs::bi::BIHostCategory; +use gorgone::modules::centreon::mbi::libs::bi::BIServiceCategory; +use gorgone::modules::centreon::mbi::libs::bi::BIService; +use gorgone::modules::centreon::mbi::libs::bi::BIMetric; +use gorgone::modules::centreon::mbi::libs::bi::Time; +use gorgone::modules::centreon::mbi::libs::bi::LiveService; +use gorgone::modules::centreon::mbi::libs::bi::DataQuality; + +my ($time, $liveService, $host, $service); +my ($hostBI, $biHost, $hostCentreon, $biService, $timePeriod, $biMetric); +my ($biHostgroup, $biServicecategory, $biHostcategory, $hostgroup, $servicecategory, $hostcategory, $biDataQuality); + +# Initialize objects for program +sub initVars { + my ($etlwk, %options) = @_; + + # instance of + $host = gorgone::modules::centreon::mbi::libs::centreon::Host->new($etlwk->{messages}, $etlwk->{dbbi_centreon_con}); + $hostcategory = gorgone::modules::centreon::mbi::libs::centreon::HostCategory->new($etlwk->{messages}, $etlwk->{dbbi_centreon_con}); + $servicecategory = gorgone::modules::centreon::mbi::libs::centreon::ServiceCategory->new($etlwk->{messages}, $etlwk->{dbbi_centreon_con}); + $hostgroup = gorgone::modules::centreon::mbi::libs::centreon::HostGroup->new($etlwk->{messages}, $etlwk->{dbbi_centreon_con}); + $service = gorgone::modules::centreon::mbi::libs::centreon::Service->new($etlwk->{messages}, $etlwk->{dbbi_centreon_con}); + $timePeriod = gorgone::modules::centreon::mbi::libs::centreon::Timeperiod->new($etlwk->{messages}, $etlwk->{dbbi_centreon_con}); + $biHost = gorgone::modules::centreon::mbi::libs::bi::BIHost->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $biHostgroup = gorgone::modules::centreon::mbi::libs::bi::BIHostGroup->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $biHostcategory = gorgone::modules::centreon::mbi::libs::bi::BIHostCategory->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $biServicecategory = gorgone::modules::centreon::mbi::libs::bi::BIServiceCategory->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $biService = gorgone::modules::centreon::mbi::libs::bi::BIService->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $time = gorgone::modules::centreon::mbi::libs::bi::Time->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $liveService = gorgone::modules::centreon::mbi::libs::bi::LiveService->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $biMetric = gorgone::modules::centreon::mbi::libs::bi::BIMetric->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $biDataQuality = gorgone::modules::centreon::mbi::libs::bi::DataQuality->new($etlwk->{messages}, $etlwk->{dbbi_centreon_con}); +} + +# temporary method to list liveservices for job configuration in Centreon +sub copyLiveServicesToMonitoringDB { + my ($etlwk, %options) = @_; + + return if ($etlwk->{dbmon_centstorage_con}->sameParams(%{$options{dbbi}->{centstorage}}) == 1); + + $etlwk->{dbmon_centstorage_con}->query("TRUNCATE TABLE mod_bi_liveservice"); + my $sth = $etlwk->{dbbi_centstorage_con}->query("SELECT id, name, timeperiod_id FROM mod_bi_liveservice"); + while (my $row = $sth->fetchrow_hashref()) { + my $insertQuery = "INSERT INTO mod_bi_liveservice (id, name, timeperiod_id) VALUES (". + $row->{'id'} . ",'" . $row->{name} . "'," . $row->{timeperiod_id} . ")"; + $etlwk->{dbmon_centstorage_con}->query($insertQuery); + } +} + +sub truncateDimensionTables { + my ($etlwk, %options) = @_; + + if ($options{options}->{rebuild} == 1 && $options{options}->{nopurge} == 0) { + $biHostgroup->truncateTable(); + $biHostcategory->truncateTable(); + $biServicecategory->truncateTable(); + $biHost->truncateTable(); + $biService->truncateTable(); + $biMetric->truncateTable(); + $time->truncateTable(); + $liveService->truncateTable(); + } +} + +sub denormalizeDimensionsFromCentreon { + my ($etlwk, %options) = @_; + + #set etlProperties for all dimensions object to be able to use it when filtering on hg/hc/sc + $host->setEtlProperties($options{etlProperties}); + $hostcategory->setEtlProperties($options{etlProperties}); + $servicecategory->setEtlProperties($options{etlProperties}); + $hostgroup->setEtlProperties($options{etlProperties}); + $service->setEtlProperties($options{etlProperties}); + + $etlwk->{messages}->writeLog("INFO", "Getting host properties from Centreon database"); + my $rows = $host->getHostGroupAndCategories(); + $etlwk->{messages}->writeLog("INFO", "Updating host dimension in Centstorage"); + if ($options{options}->{rebuild} == 1 && $options{options}->{nopurge} == 0) { + $biHost->insert($rows); + } else { + $biHost->update($rows, $options{etlProperties}->{'tmp.storage.memory'}); + } + + $etlwk->{messages}->writeLog("INFO", "Getting hostgroup properties from Centreon database"); + $rows = $hostgroup->getAllEntries(); + $etlwk->{messages}->writeLog("INFO", "Updating hostgroup dimension in Centstorage"); + $biHostgroup->insert($rows); + + $etlwk->{messages}->writeLog("INFO", "Getting hostcategories properties from Centreon database"); + $rows = $hostcategory->getAllEntries(); + $etlwk->{messages}->writeLog("INFO", "Updating hostcategories dimension in Centstorage"); + $biHostcategory->insert($rows); + + $etlwk->{messages}->writeLog("INFO", "Getting servicecategories properties from Centreon database"); + $rows = $servicecategory->getAllEntries(); + $etlwk->{messages}->writeLog("INFO", "Updating servicecategories dimension in Centstorage"); + $biServicecategory->insert($rows); + $etlwk->{messages}->writeLog("INFO", "Getting service properties from Centreon database"); + + my $hostRows = $biHost->getHostsInfo(); + my $serviceRows = $service->getServicesWithHostAndCategory($hostRows); + $etlwk->{messages}->writeLog("INFO", "Updating service dimension in Centstorage"); + if ($options{options}->{rebuild} == 1 && $options{options}->{nopurge} == 0) { + $biService->insert($serviceRows); + } else { + $biService->update($serviceRows, $options{etlProperties}->{'tmp.storage.memory'}); + } + + if (!defined($options{etlProperties}->{'statistics.type'}) || $options{etlProperties}->{'statistics.type'} ne 'availability') { + $etlwk->{messages}->writeLog("INFO", "Updating metric dimension in Centstorage"); + if ($options{options}->{rebuild} == 1 && $options{options}->{nopurge} == 0) { + $biMetric->insert(); + } else { + $biMetric->update($options{etlProperties}->{'tmp.storage.memory'}); + } + } + + # Getting live services to calculate reporting by time range + $etlwk->{messages}->writeLog("INFO", "Updating liveservice dimension in Centstorage"); + + my $timeperiods = $timePeriod->getPeriods($options{etlProperties}->{'liveservices.availability'}); + $liveService->insertList($timeperiods); + $timeperiods = $timePeriod->getPeriods($options{etlProperties}->{'liveservices.perfdata'}); + $liveService->insertList($timeperiods); + $timeperiods = $timePeriod->getCentilePeriods(); + $liveService->insertList($timeperiods); +} + +sub insertCentileParamToBIStorage{ + my ($etlwk, %options) = @_; + + my %result; + my $sth; + + #Insert potential missing time periods related to centile calculation in mod_bi_liveservices + $sth = $etlwk->{dbbi_centreon_con}->query("SELECT tp_id, tp_name FROM timeperiod WHERE tp_id IN (SELECT timeperiod_id FROM mod_bi_options_centiles)"); + while (my $row = $sth->fetchrow_hashref()) { + $result{$row->{tp_id}} = $row->{tp_name}; + } + + #If not time period is found in centile configuration, exit the function + if (%result eq 0){ + $etlwk->{messages}->writeLog("INFO", "No configuration found for centile calculation"); + return; + } + $etlwk->{messages}->writeLog("INFO", "Updating centile properties"); + + my $timeperiods = $timePeriod->getPeriods(\%result); + $liveService->insertList($timeperiods); + + #In case of rebuild, delete all centile parameters + if ($options{options}->{rebuild} == 1){ + $etlwk->{dbbi_centstorage_con}->query("TRUNCATE TABLE mod_bi_centiles"); + } + $sth = $etlwk->{dbbi_centreon_con}->query("select * from mod_bi_options_centiles"); + while (my $row = $sth->fetchrow_hashref()) { + my ($tpName,$liveServiceId) = $liveService->getLiveServicesByNameForTpId($row->{'timeperiod_id'}); + my $insertQuery = "INSERT IGNORE INTO mod_bi_centiles (id, centile_param, liveservice_id,tp_name) VALUES (".$row->{'id'}.",'".$row->{'centile_param'}."',".$liveServiceId.",'".$tpName."')"; + $etlwk->{dbbi_centstorage_con}->query($insertQuery); + } +} + +sub copyCentileToMonitoringDB { + my ($etlwk, %options) = @_; + + return if ($etlwk->{dbmon_centstorage_con}->sameParams(%{$options{dbbi}->{centstorage}}) == 1); + + $etlwk->{dbmon_centstorage_con}->query("TRUNCATE TABLE mod_bi_centiles"); + my $sth = $etlwk->{dbbi_centstorage_con}->query("SELECT id, centile_param, liveservice_id, tp_name FROM mod_bi_centiles"); + while (my $row = $sth->fetchrow_hashref()) { + my $insertQuery = "INSERT INTO mod_bi_centiles (id, centile_param, liveservice_id,tp_name) VALUES (". + $row->{id} . ",'" . $row->{centile_param} . "'," . $row->{liveservice_id} . ",'" . $row->{tp_name} . "')"; + $etlwk->{dbmon_centstorage_con}->query($insertQuery); + } +} + +sub startCbisAclSync{ + my ($etlwk, %options) = @_; + + # create a connecting socket + my $socket = new IO::Socket::INET( + PeerHost => 'localhost', + PeerPort => '1234', + Proto => 'tcp' + ); + + if (!$socket){ + $etlwk->{messages}->writeLog("WARNING", "Can't start ACL synchronization, make sure CBIS is started on port 1234"); + return 0; + } + #die "[ERROR] Cannot connect to CBIS on port 1234" unless $socket; + # XML ACL request + my $req = "\n". + "\n". + " \n". + " \n". + " \n". + "\n"; + $etlwk->{messages}->writeLog("INFO", "Send ACL synchronization signal to CBIS"); + my $size = $socket->send($req); + + # notify server that request has been sent + shutdown($socket, 1); + + # receive a response of up to 1024 characters from server + my $response = ""; + $socket->recv($response, 1024); + $socket->close(); +} + +sub execute { + my ($etlwk, %options) = @_; + + initVars($etlwk, %options); + + $biDataQuality->searchAndDeleteDuplicateEntries(); + if (!defined($options{options}->{centile}) || $options{options}->{centile} == 0) { + truncateDimensionTables($etlwk, %options); + denormalizeDimensionsFromCentreon($etlwk, %options); + copyLiveServicesToMonitoringDB($etlwk, %options); + } + + insertCentileParamToBIStorage($etlwk, %options); + copyCentileToMonitoringDB($etlwk, %options); + startCbisAclSync($etlwk, %options); +} + +1; diff --git a/gorgone/modules/centreon/mbi/etlworkers/event/main.pm b/gorgone/modules/centreon/mbi/etlworkers/event/main.pm new file mode 100644 index 0000000..68fab49 --- /dev/null +++ b/gorgone/modules/centreon/mbi/etlworkers/event/main.pm @@ -0,0 +1,259 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::etlworkers::event::main; + +use strict; +use warnings; +use gorgone::modules::centreon::mbi::libs::centreon::Timeperiod; +use gorgone::modules::centreon::mbi::libs::bi::HostAvailability; +use gorgone::modules::centreon::mbi::libs::bi::ServiceAvailability; +use gorgone::modules::centreon::mbi::libs::bi::HGMonthAvailability; +use gorgone::modules::centreon::mbi::libs::bi::HGServiceMonthAvailability; +use gorgone::modules::centreon::mbi::libs::bi::Time; +use gorgone::modules::centreon::mbi::libs::bi::MySQLTables; +use gorgone::modules::centreon::mbi::libs::bi::BIHostStateEvents; +use gorgone::modules::centreon::mbi::libs::bi::BIServiceStateEvents; +use gorgone::modules::centreon::mbi::libs::bi::LiveService; +use gorgone::modules::centreon::mbi::libs::centstorage::HostStateEvents; +use gorgone::modules::centreon::mbi::libs::centstorage::ServiceStateEvents; +use gorgone::modules::centreon::mbi::libs::Utils; +use gorgone::standard::misc; + +my ($utils, $time, $tablesManager, $timePeriod); +my ($hostAv, $serviceAv); +my ($hgAv, $hgServiceAv); +my ($biHostEvents, $biServiceEvents); +my ($hostEvents, $serviceEvents); +my ($liveService); + +sub initVars { + my ($etlwk, %options) = @_; + + $utils = gorgone::modules::centreon::mbi::libs::Utils->new($etlwk->{messages}); + $timePeriod = gorgone::modules::centreon::mbi::libs::centreon::Timeperiod->new($etlwk->{messages}, $etlwk->{dbbi_centreon_con}); + $time = gorgone::modules::centreon::mbi::libs::bi::Time->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $tablesManager = gorgone::modules::centreon::mbi::libs::bi::MySQLTables->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $biHostEvents = gorgone::modules::centreon::mbi::libs::bi::BIHostStateEvents->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}, $timePeriod); + $biServiceEvents = gorgone::modules::centreon::mbi::libs::bi::BIServiceStateEvents->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}, $timePeriod); + $liveService = gorgone::modules::centreon::mbi::libs::bi::LiveService->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $hostEvents = gorgone::modules::centreon::mbi::libs::centstorage::HostStateEvents->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}, $biHostEvents, $timePeriod); + $serviceEvents = gorgone::modules::centreon::mbi::libs::centstorage::ServiceStateEvents->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}, $biServiceEvents, $timePeriod); + $hostAv = gorgone::modules::centreon::mbi::libs::bi::HostAvailability->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $serviceAv = gorgone::modules::centreon::mbi::libs::bi::ServiceAvailability->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $hgAv = gorgone::modules::centreon::mbi::libs::bi::HGMonthAvailability->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $hgServiceAv = gorgone::modules::centreon::mbi::libs::bi::HGServiceMonthAvailability->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); +} + +sub sql { + my ($etlwk, %options) = @_; + + return if (!defined($options{params}->{sql})); + + foreach (@{$options{params}->{sql}}) { + $etlwk->{messages}->writeLog('INFO', $_->[0]); + if ($options{params}->{db} eq 'centstorage') { + $etlwk->{dbbi_centstorage_con}->query($_->[1]); + } elsif ($options{params}->{db} eq 'centreon') { + $etlwk->{dbbi_centreon_con}->query($_->[1]); + } + } +} + +sub processEventsHosts { + my ($etlwk, %options) = @_; + + my $mode = 'daily'; + if ($options{options}->{rebuild} == 1) { + $tablesManager->emptyTableForRebuild($biHostEvents->getName(), $tablesManager->dumpTableStructure($biHostEvents->getName()), $biHostEvents->getTimeColumn()); + $mode = 'rebuild'; + } else { + $biHostEvents->deleteUnfinishedEvents(); + } + + if ($options{options}->{rebuild} == 1) { + $tablesManager->dropIndexesFromReportingTable('mod_bi_hoststateevents'); + } + + #Agreggate events by TP and store them into a temporary table (mod_bi_hoststateevents_tmp) + $etlwk->{messages}->writeLog("INFO", "[HOST] Processing host events"); + $hostEvents->agreggateEventsByTimePeriod( + $options{etlProperties}->{'liveservices.availability'}, + $options{start}, + $options{end}, + $options{liveServices}, + $mode + ); + + #Dump the result of aggregated data join to dimensions and load this to the final mod_bi_hoststateevents table + my $request = "INSERT INTO mod_bi_hoststateevents "; + $request .= " SELECT id, t1.modbiliveservice_id, t1.state, t1.start_time, t1.end_time, t1.duration, t1.sla_duration,"; + $request .= " t1.ack_time, t1.last_update from mod_bi_hoststateevents_tmp t1"; + $request .= " INNER JOIN mod_bi_tmp_today_hosts t2 on t1.host_id = t2.host_id"; + + $etlwk->{messages}->writeLog("INFO", "[HOST] Loading calculated events in reporting table"); + $etlwk->{dbbi_centstorage_con}->query($request); + + if ($options{options}->{rebuild} == 1 && $options{options}->{rebuild} == 0) { + $etlwk->{messages}->writeLog("DEBUG", "[HOST] Creating index"); + $etlwk->{dbbi_centstorage_con}->query('ALTER TABLE mod_bi_hoststateevents ADD INDEX `modbihost_id` (`modbihost_id`,`modbiliveservice_id`,`state`,`start_time`,`end_time`)'); + $etlwk->{dbbi_centstorage_con}->query('ALTER TABLE mod_bi_hoststateevents ADD INDEX `state` (`state`,`modbiliveservice_id`,`start_time`,`end_time`)'); + $etlwk->{dbbi_centstorage_con}->query('ALTER TABLE mod_bi_hoststateevents ADD INDEX `idx_mod_bi_hoststateevents_end_time` (`end_time`)'); + } +} + +sub processEventsServices { + my ($etlwk, %options) = @_; + + my $mode = 'daily'; + if ($options{options}->{rebuild} == 1) { + $tablesManager->emptyTableForRebuild($biServiceEvents->getName(), $tablesManager->dumpTableStructure($biServiceEvents->getName()), $biServiceEvents->getTimeColumn()); + $mode = 'rebuild'; + } else { + $biServiceEvents->deleteUnfinishedEvents(); + } + + if ($options{options}->{rebuild} == 1) { + $tablesManager->dropIndexesFromReportingTable('mod_bi_servicestateevents'); + } + + #Agreggate events by TP and store them into a temporary table (mod_bi_hoststateevents_tmp) + $etlwk->{messages}->writeLog("INFO", "[SERVICE] Processing service events"); + $serviceEvents->agreggateEventsByTimePeriod( + $options{etlProperties}->{'liveservices.availability'}, + $options{start}, + $options{end}, + $options{liveServices}, + $mode + ); + + #Dump the result of aggregated data join to dimensions and load this to the final mod_bi_hoststateevents table + my $request = "INSERT INTO mod_bi_servicestateevents "; + $request .= " SELECT id,t1.modbiliveservice_id,t1.state,t1.start_time,t1.end_time,t1.duration,t1.sla_duration,"; + $request .= " t1.ack_time,t1.last_update FROM mod_bi_servicestateevents_tmp t1 INNER JOIN mod_bi_tmp_today_services t2 "; + $request .= " ON t1.host_id = t2.host_id AND t1.service_id = t2.service_id"; + + $etlwk->{messages}->writeLog("INFO", "[SERVICE] Loading calculated events in reporting table"); + $etlwk->{dbbi_centstorage_con}->query($request); + + if ($options{options}->{rebuild} == 1 && $options{options}->{rebuild} == 0) { + $etlwk->{messages}->writeLog("DEBUG", "[SERVICE] Creating index"); + $etlwk->{dbbi_centstorage_con}->query('ALTER TABLE mod_bi_servicestateevents ADD INDEX `modbiservice_id` (`modbiservice_id`,`modbiliveservice_id`,`state`,`start_time`,`end_time`)'); + $etlwk->{dbbi_centstorage_con}->query('ALTER TABLE mod_bi_servicestateevents ADD INDEX `state` (`state`,`modbiliveservice_id`,`start_time`,`end_time`)'); + $etlwk->{dbbi_centstorage_con}->query('ALTER TABLE mod_bi_servicestateevents ADD INDEX `idx_mod_bi_servicestateevents_end_time` (`end_time`)'); + } +} + +sub events { + my ($etlwk, %options) = @_; + + initVars($etlwk, %options); + + my ($startTimeId, $startUtime) = $time->getEntryID($options{params}->{start}); + my ($endTimeId, $endUtime) = $time->getEntryID($options{params}->{end}); + + my $liveServices = $liveService->getLiveServicesByTpId(); + + if (defined($options{params}->{hosts}) && $options{params}->{hosts} == 1) { + processEventsHosts($etlwk, start => $startUtime, end => $endUtime, liveServices => $liveServices, %options); + } elsif (defined($options{params}->{services}) && $options{params}->{services} == 1) { + processEventsServices($etlwk, start => $startUtime, end => $endUtime, liveServices => $liveServices, %options); + } +} + +sub availabilityDayHosts { + my ($etlwk, %options) = @_; + + $etlwk->{messages}->writeLog("INFO", "[AVAILABILITY] Processing hosts day: $options{params}->{start} => $options{params}->{end} [$options{params}->{liveserviceName}]"); + my $ranges = $timePeriod->getTimeRangesForDay($options{startWeekDay}, $options{params}->{liveserviceName}, $options{startUtime}); + my $dayEvents = $biHostEvents->getDayEvents($options{startUtime}, $options{endUtime}, $options{params}->{liveserviceId}, $ranges); + $hostAv->insertStats($dayEvents, $options{startTimeId}, $options{params}->{liveserviceId}); +} + +sub availabilityDayServices { + my ($etlwk, %options) = @_; + + $etlwk->{messages}->writeLog("INFO", "[AVAILABILITY] Processing services day: $options{params}->{start} => $options{params}->{end} [$options{params}->{liveserviceName}]"); + my $ranges = $timePeriod->getTimeRangesForDay($options{startWeekDay}, $options{params}->{liveserviceName}, $options{startUtime}); + my $dayEvents = $biServiceEvents->getDayEvents($options{startUtime}, $options{endUtime}, $options{params}->{liveserviceId}, $ranges); + $serviceAv->insertStats($dayEvents, $options{startTimeId}, $options{params}->{liveserviceId}); +} + +sub availabilityMonthHosts { + my ($etlwk, %options) = @_; + + $etlwk->{messages}->writeLog("INFO", "[AVAILABILITY] Processing services month: $options{params}->{start} => $options{params}->{end}"); + my $data = $hostAv->getHGMonthAvailability($options{params}->{start}, $options{params}->{end}, $biHostEvents); + $hgAv->insertStats($options{startTimeId}, $data); +} + +sub availabilityMonthServices { + my ($etlwk, %options) = @_; + + $etlwk->{messages}->writeLog("INFO", "[AVAILABILITY] Processing hosts month: $options{params}->{start} => $options{params}->{end}"); + my $data = $serviceAv->getHGMonthAvailability_optimised($options{params}->{start}, $options{params}->{end}, $biServiceEvents); + $hgServiceAv->insertStats($options{startTimeId}, $data); +} + +sub availability { + my ($etlwk, %options) = @_; + + initVars($etlwk, %options); + + my ($startTimeId, $startUtime) = $time->getEntryID($options{params}->{start}); + my ($endTimeId, $endUtime) = $time->getEntryID($options{params}->{end}); + my $startWeekDay = $utils->getDayOfWeek($options{params}->{start}); + + if ($options{params}->{type} eq 'availability_day_hosts') { + availabilityDayHosts( + $etlwk, + startTimeId => $startTimeId, + startUtime => $startUtime, + endTimeId => $endTimeId, + endUtime => $endUtime, + startWeekDay => $startWeekDay, + %options + ); + } elsif ($options{params}->{type} eq 'availability_day_services') { + availabilityDayServices( + $etlwk, + startTimeId => $startTimeId, + startUtime => $startUtime, + endTimeId => $endTimeId, + endUtime => $endUtime, + startWeekDay => $startWeekDay, + %options + ); + } elsif ($options{params}->{type} eq 'availability_month_services') { + availabilityMonthServices( + $etlwk, + startTimeId => $startTimeId, + %options + ); + } elsif ($options{params}->{type} eq 'availability_month_hosts') { + availabilityMonthHosts( + $etlwk, + startTimeId => $startTimeId, + %options + ); + } +} + +1; diff --git a/gorgone/modules/centreon/mbi/etlworkers/hooks.pm b/gorgone/modules/centreon/mbi/etlworkers/hooks.pm new file mode 100644 index 0000000..c75b4a2 --- /dev/null +++ b/gorgone/modules/centreon/mbi/etlworkers/hooks.pm @@ -0,0 +1,239 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::etlworkers::hooks; + +use warnings; +use strict; +use JSON::XS; +use gorgone::class::core; +use gorgone::standard::library; +use gorgone::standard::constants qw(:all); +use gorgone::modules::centreon::mbi::etlworkers::class; + +use constant NAMESPACE => 'centreon'; +use constant NAME => 'mbi-etlworkers'; +use constant EVENTS => [ + { event => 'CENTREONMBIETLWORKERSIMPORT' }, + { event => 'CENTREONMBIETLWORKERSDIMENSIONS' }, + { event => 'CENTREONMBIETLWORKERSEVENT' }, + { event => 'CENTREONMBIETLWORKERSPERFDATA' }, + { event => 'CENTREONMBIETLWORKERSREADY' } +]; + +my $config_core; +my $config; + +my $pools = {}; +my $pools_pid = {}; +my $rr_current = 0; +my $stop = 0; + +sub register { + my (%options) = @_; + + $config = $options{config}; + $config_core = $options{config_core}; + + $config->{pool} = defined($config->{pool}) && $config->{pool} =~ /(\d+)/ ? $1 : 8; + return (1, NAMESPACE, NAME, EVENTS); +} + +sub init { + my (%options) = @_; + + for my $pool_id (1..$config->{pool}) { + create_child(dbh => $options{dbh}, pool_id => $pool_id, logger => $options{logger}); + } +} + +sub routing { + my (%options) = @_; + + my $data; + eval { + $data = JSON::XS->new->decode($options{data}); + }; + if ($@) { + $options{logger}->writeLogError("[proxy] Cannot decode json data: $@"); + gorgone::standard::library::add_history( + dbh => $options{dbh}, + code => GORGONE_ACTION_FINISH_KO, + token => $options{token}, + data => { message => NAME . ' - cannot decode json' }, + json_encode => 1 + ); + return undef; + } + + if ($options{action} eq 'CENTREONMBIETLWORKERSREADY') { + if (defined($data->{pool_id})) { + $pools->{ $data->{pool_id} }->{ready} = 1; + } + return undef; + } + + my $pool_id = rr_pool(); + if (!defined($pool_id)) { + gorgone::standard::library::add_history( + dbh => $options{dbh}, + code => GORGONE_ACTION_FINISH_KO, + token => $options{token}, + data => { message => NAME . ' - no pool ready' }, + json_encode => 1 + ); + return undef; + } + + my $identity = 'gorgone-' . NAME . '-' . $pool_id; + + $options{gorgone}->send_internal_message( + identity => $identity, + action => $options{action}, + data => $options{data}, + token => $options{token} + ); +} + +sub gently { + my (%options) = @_; + + $stop = 1; + foreach my $pool_id (keys %$pools) { + if (defined($pools->{$pool_id}->{running}) && $pools->{$pool_id}->{running} == 1) { + $options{logger}->writeLogDebug("[" . NAME . "] Send TERM signal for pool '" . $pool_id . "'"); + CORE::kill('TERM', $pools->{$pool_id}->{pid}); + } + } +} + +sub kill { + my (%options) = @_; + + foreach (keys %$pools) { + if ($pools->{$_}->{running} == 1) { + $options{logger}->writeLogDebug("[" . NAME . "] Send KILL signal for pool '" . $_ . "'"); + CORE::kill('KILL', $pools->{$_}->{pid}); + } + } +} + +sub kill_internal { + my (%options) = @_; + +} + +sub check_create_child { + my (%options) = @_; + + return if ($stop == 1); + + # Check if we need to create a child + for my $pool_id (1..$config->{pool}) { + if (!defined($pools->{$pool_id})) { + create_child(dbh => $options{dbh}, pool_id => $pool_id, logger => $options{logger}); + } + } +} + +sub check { + my (%options) = @_; + + my $count = 0; + foreach my $pid (keys %{$options{dead_childs}}) { + # Not me + next if (!defined($pools_pid->{$pid})); + + # If someone dead, we recreate + my $pool_id = $pools_pid->{$pid}; + delete $pools->{$pools_pid->{$pid}}; + delete $pools_pid->{$pid}; + delete $options{dead_childs}->{$pid}; + if ($stop == 0) { + create_child(dbh => $options{dbh}, pool_id => $pool_id, logger => $options{logger}); + } + } + + check_create_child(dbh => $options{dbh}, logger => $options{logger}); + + foreach (keys %$pools) { + $count++ if ($pools->{$_}->{running} == 1); + } + + return ($count, 1); +} + +sub broadcast { + my (%options) = @_; + + foreach my $pool_id (keys %$pools) { + next if ($pools->{$pool_id}->{ready} != 1); + + $options{gorgone}->send_internal_message( + identity => 'gorgone-' . NAME . '-' . $pool_id, + action => $options{action}, + data => $options{data}, + token => $options{token} + ); + } +} + +# Specific functions +sub rr_pool { + my (%options) = @_; + + my ($loop, $i) = ($config->{pool}, 0); + while ($i <= $loop) { + $rr_current = $rr_current % $config->{pool}; + if ($pools->{$rr_current + 1}->{ready} == 1) { + $rr_current++; + return $rr_current; + } + $rr_current++; + $i++; + } + + return undef; +} + +sub create_child { + my (%options) = @_; + + $options{logger}->writeLogInfo("[" . NAME . "] Create module '" . NAME . "' child process for pool id '" . $options{pool_id} . "'"); + my $child_pid = fork(); + if ($child_pid == 0) { + $0 = 'gorgone-' . NAME; + my $module = gorgone::modules::centreon::mbi::etlworkers::class->new( + logger => $options{logger}, + module_id => NAME, + config_core => $config_core, + config => $config, + pool_id => $options{pool_id}, + container_id => $options{pool_id} + ); + $module->run(); + exit(0); + } + $options{logger}->writeLogDebug("[" . NAME . "] PID $child_pid (gorgone-" . NAME . ") for pool id '" . $options{pool_id} . "'"); + $pools->{$options{pool_id}} = { pid => $child_pid, ready => 0, running => 1 }; + $pools_pid->{$child_pid} = $options{pool_id}; +} + +1; diff --git a/gorgone/modules/centreon/mbi/etlworkers/import/main.pm b/gorgone/modules/centreon/mbi/etlworkers/import/main.pm new file mode 100644 index 0000000..c3dc86a --- /dev/null +++ b/gorgone/modules/centreon/mbi/etlworkers/import/main.pm @@ -0,0 +1,86 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::etlworkers::import::main; + +use strict; +use warnings; +use gorgone::standard::misc; +use File::Basename; + +sub sql { + my ($etlwk, %options) = @_; + + return if (!defined($options{params}->{sql})); + + foreach (@{$options{params}->{sql}}) { + $etlwk->{messages}->writeLog('INFO', $_->[0]); + if ($options{params}->{db} eq 'centstorage') { + $etlwk->{dbbi_centstorage_con}->query($_->[1]); + } elsif ($options{params}->{db} eq 'centreon') { + $etlwk->{dbbi_centreon_con}->query($_->[1]); + } + } +} + +sub command { + my ($etlwk, %options) = @_; + + return if (!defined($options{params}->{command}) || $options{params}->{command} eq ''); + + my ($error, $stdout, $return_code) = gorgone::standard::misc::backtick( + command => $options{params}->{command}, + timeout => 7200, + wait_exit => 1, + redirect_stderr => 1, + logger => $options{logger} + ); + + if ($error != 0) { + die $options{params}->{message} . ": execution failed: $stdout"; + } + + $etlwk->{messages}->writeLog('INFO', $options{params}->{message}); + $etlwk->{logger}->writeLogDebug("[mbi-etlworkers] succeeded command (code: $return_code): $stdout"); +} + +sub load { + my ($etlwk, %options) = @_; + + return if (!defined($options{params}->{file})); + + my ($file, $dir) = File::Basename::fileparse($options{params}->{file}); + + if (! -d "$dir" && ! -w "$dir") { + $etlwk->{messages}->writeLog('ERROR', "Cannot write into directory " . $dir); + } + + command($etlwk, params => { command => $options{params}->{dump}, message => $options{params}->{message} }); + + if ($options{params}->{db} eq 'centstorage') { + $etlwk->{dbbi_centstorage_con}->query($options{params}->{load}); + } elsif ($options{params}->{db} eq 'centreon') { + $etlwk->{dbbi_centreon_con}->query($options{params}->{load}); + } + + unlink($options{params}->{file}); +} + +1; diff --git a/gorgone/modules/centreon/mbi/etlworkers/perfdata/main.pm b/gorgone/modules/centreon/mbi/etlworkers/perfdata/main.pm new file mode 100644 index 0000000..d82c6c4 --- /dev/null +++ b/gorgone/modules/centreon/mbi/etlworkers/perfdata/main.pm @@ -0,0 +1,190 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::etlworkers::perfdata::main; + +use strict; +use warnings; + +use gorgone::modules::centreon::mbi::libs::centreon::Timeperiod; +use gorgone::modules::centreon::mbi::libs::centreon::CentileProperties; +use gorgone::modules::centreon::mbi::libs::bi::LiveService; +use gorgone::modules::centreon::mbi::libs::bi::Time; +use gorgone::modules::centreon::mbi::libs::Utils; +use gorgone::modules::centreon::mbi::libs::centstorage::Metrics; +use gorgone::modules::centreon::mbi::libs::bi::MetricDailyValue; +use gorgone::modules::centreon::mbi::libs::bi::MetricHourlyValue; +use gorgone::modules::centreon::mbi::libs::bi::MetricCentileValue; +use gorgone::modules::centreon::mbi::libs::bi::MetricMonthCapacity; +use gorgone::standard::misc; + +my ($utils, $time, $timePeriod, $centileProperties, $liveService); +my ($metrics); +my ($dayAgregates, $hourAgregates, $centileAgregates, $metricMonthCapacity); + +sub initVars { + my ($etlwk, %options) = @_; + + $timePeriod = gorgone::modules::centreon::mbi::libs::centreon::Timeperiod->new($etlwk->{messages}, $etlwk->{dbbi_centreon_con}); + $centileProperties = gorgone::modules::centreon::mbi::libs::centreon::CentileProperties->new($etlwk->{messages}, $etlwk->{dbbi_centreon_con}); + $liveService = gorgone::modules::centreon::mbi::libs::bi::LiveService->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $time = gorgone::modules::centreon::mbi::libs::bi::Time->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + $utils = gorgone::modules::centreon::mbi::libs::Utils->new($etlwk->{messages}); + $metrics = gorgone::modules::centreon::mbi::libs::centstorage::Metrics->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}, $options{pool_id}); + $dayAgregates = gorgone::modules::centreon::mbi::libs::bi::MetricDailyValue->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}, $options{pool_id}); + $hourAgregates = gorgone::modules::centreon::mbi::libs::bi::MetricHourlyValue->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}, $options{pool_id}); + $metricMonthCapacity = gorgone::modules::centreon::mbi::libs::bi::MetricMonthCapacity->new($etlwk->{messages}, $etlwk->{dbbi_centstorage_con}); + + $centileAgregates = gorgone::modules::centreon::mbi::libs::bi::MetricCentileValue->new( + logger => $etlwk->{messages}, + centstorage => $etlwk->{dbbi_centstorage_con}, + centreon => $etlwk->{dbbi_centreon_con}, + time => $time, + centileProperties => $centileProperties, + timePeriod => $timePeriod, + liveService => $liveService + ); +} + +sub sql { + my ($etlwk, %options) = @_; + + return if (!defined($options{params}->{sql})); + + foreach (@{$options{params}->{sql}}) { + $etlwk->{messages}->writeLog('INFO', $_->[0]); + if ($options{params}->{db} eq 'centstorage') { + $etlwk->{dbbi_centstorage_con}->query($_->[1]); + } elsif ($options{params}->{db} eq 'centreon') { + $etlwk->{dbbi_centreon_con}->query($_->[1]); + } + } +} + +sub perfdataDay { + my ($etlwk, %options) = @_; + + my ($currentDayId, $currentDayUtime) = $time->getEntryID($options{params}->{start}); + my $ranges = $timePeriod->getTimeRangesForDayByDateTime( + $options{params}->{liveserviceName}, + $options{params}->{start}, + $utils->getDayOfWeek($options{params}->{start}) + ); + if (scalar(@$ranges)) { + $etlwk->{messages}->writeLog("INFO", "[PERFDATA] Processing day: $options{params}->{start} => $options{params}->{end} [$options{params}->{liveserviceName}]"); + $metrics->getMetricsValueByDay($ranges, $options{etlProperties}->{'tmp.storage.memory'}); + $dayAgregates->insertValues($options{params}->{liveserviceId}, $currentDayId); + } +} + +sub perfdataMonth { + my ($etlwk, %options) = @_; + + my ($previousMonthStartTimeId, $previousMonthStartUtime) = $time->getEntryID($options{params}->{start}); + my ($previousMonthEndTimeId, $previousMonthEndUtime) = $time->getEntryID($options{params}->{end}); + + $etlwk->{messages}->writeLog("INFO", "[PERFDATA] Processing month: $options{params}->{start} => $options{params}->{end}"); + my $data = $dayAgregates->getMetricCapacityValuesOnPeriod($previousMonthStartTimeId, $previousMonthEndTimeId, $options{etlProperties}); + $metricMonthCapacity->insertStats($previousMonthStartTimeId, $data); +} + +sub perfdataHour { + my ($etlwk, %options) = @_; + + $etlwk->{messages}->writeLog("INFO", "[PERFDATA] Processing hours: $options{params}->{start} => $options{params}->{end}"); + + $metrics->getMetricValueByHour($options{params}->{start}, $options{params}->{end}, $options{etlProperties}->{'tmp.storage.memory'}); + $hourAgregates->insertValues(); +} + +sub perfdata { + my ($etlwk, %options) = @_; + + initVars($etlwk, %options); + + if ($options{params}->{type} eq 'perfdata_day') { + perfdataDay($etlwk, %options); + } elsif ($options{params}->{type} eq 'perfdata_month') { + perfdataMonth($etlwk, %options); + } elsif ($options{params}->{type} eq 'perfdata_hour') { + perfdataHour($etlwk, %options); + } +} + +sub centileDay { + my ($etlwk, %options) = @_; + + my ($currentDayId) = $time->getEntryID($options{params}->{start}); + + my $metricsId = $centileAgregates->getMetricsCentile(etlProperties => $options{etlProperties}); + $centileAgregates->calcMetricsCentileValueMultipleDays( + metricsId => $metricsId, + start => $options{params}->{start}, + end => $options{params}->{end}, + granularity => 'day', + timeId => $currentDayId + ); +} + +sub centileMonth { + my ($etlwk, %options) = @_; + + my ($previousMonthStartTimeId) = $time->getEntryID($options{params}->{start}); + + my $metricsId = $centileAgregates->getMetricsCentile(etlProperties => $options{etlProperties}); + $centileAgregates->calcMetricsCentileValueMultipleDays( + metricsId => $metricsId, + start => $options{params}->{start}, + end => $options{params}->{end}, + granularity => 'month', + timeId => $previousMonthStartTimeId + ); +} + +sub centileWeek { + my ($etlwk, %options) = @_; + + my ($currentDayId) = $time->getEntryID($options{params}->{start}); + + my $metricsId = $centileAgregates->getMetricsCentile(etlProperties => $options{etlProperties}); + $centileAgregates->calcMetricsCentileValueMultipleDays( + metricsId => $metricsId, + start => $options{params}->{start}, + end => $options{params}->{end}, + granularity => 'week', + timeId => $currentDayId + ); +} + +sub centile { + my ($etlwk, %options) = @_; + + initVars($etlwk, %options); + + if ($options{params}->{type} eq 'centile_day') { + centileDay($etlwk, %options); + } elsif ($options{params}->{type} eq 'centile_month') { + centileMonth($etlwk, %options); + } elsif ($options{params}->{type} eq 'centile_week') { + centileWeek($etlwk, %options); + } +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/Messages.pm b/gorgone/modules/centreon/mbi/libs/Messages.pm new file mode 100644 index 0000000..52fa032 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/Messages.pm @@ -0,0 +1,55 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::Messages; + +sub new { + my $class = shift; + my $self = {}; + + $self->{messages} = []; + + bless $self, $class; + return $self; +} + +sub writeLog { + my ($self, $severity, $message, $nodie) = @_; + + $severity = lc($severity); + + my %severities = ('debug' => 'D', 'info' => 'I', 'warning' => 'I', 'error' => 'E', 'fatal' => 'F'); + if ($severities{$severity} eq 'E' || $severities{$severity} eq 'F') { + die $message if (!defined($nodie) || $nodie == 0); + } + + push @{$self->{messages}}, [$severities{$severity}, $message]; +} + +sub getLogs { + my ($self) = @_; + + return $self->{messages}; +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/Utils.pm b/gorgone/modules/centreon/mbi/libs/Utils.pm new file mode 100644 index 0000000..ef2a404 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/Utils.pm @@ -0,0 +1,251 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; +use POSIX; +use Time::Local; +use Tie::File; +use DateTime; + +package gorgone::modules::centreon::mbi::libs::Utils; + +sub new { + my $class = shift; + my $self = {}; + bless $self, $class; + + $self->{logger} = shift; + $self->{tz} = DateTime::TimeZone->new(name => 'local')->name(); + return $self; +} + +sub checkBasicOptions { + my ($self, $options) = @_; + + # check execution mode daily to extract yesterday data or rebuild to get more historical data + if (($options->{daily} == 0 && $options->{rebuild} == 0 && (!defined($options->{create_tables}) || $options->{create_tables} == 0) && (!defined($options->{centile}) || $options->{centile} == 0)) + || ($options->{daily} == 1 && $options->{rebuild} == 1)) { + $self->{logger}->writeLogError("Specify one execution method. Check program help for more informations"); + return 1; + } + + # check if options are set correctly for rebuild mode + if (($options->{rebuild} == 1 || (defined($options->{create_tables}) && $options->{create_tables} == 1)) + && ($options->{start} ne '' && $options->{end} eq '') + || ($options->{start} eq '' && $options->{end} ne '')) { + $self->{logger}->writeLogError("Specify both options --start and --end or neither of them to use default data retention options"); + return 1; + } + # check start and end dates format + if ($options->{rebuild} == 1 && $options->{start} ne '' && $options->{end} ne '' + && !$self->checkDateFormat($options->{start}, $options->{end})) { + $self->{logger}->writeLogError("Verify period start or end date format"); + return 1; + } + + return 0; +} + +sub buildCliMysqlArgs { + my ($self, $con) = @_; + + my $args = '-u "' . $con->{user} . '" ' . + '-p"' . $con->{password} . '" ' . + '-h "' . $con->{host} . '" ' . + '-P ' . $con->{port}; + return $args; +} + +sub getYesterdayTodayDate { + my ($self) = @_; + + my $dt = DateTime->from_epoch( + epoch => time(), + time_zone => $self->{tz} + ); + + my $month = $dt->month(); + $month = '0' . $month if ($month < 10); + my $day = $dt->day(); + $day = '0' . $day if ($day < 10); + my $today = $dt->year() . '-' . $month . '-' . $day; + + $dt->subtract(days => 1); + $month = $dt->month(); + $month = '0' . $month if ($month < 10); + $day = $dt->day(); + $day = '0' . $day if ($day < 10); + my $yesterday = $dt->year() . '-' . $month . '-' . $day; + + return ($yesterday, $today); +} + +sub subtractDateMonths { + my ($self, $date, $num) = @_; + + if ($date !~ /(\d{4})-(\d{2})-(\d{2})/) { + $self->{logger}->writeLog('ERROR', "Verify date format"); + } + + my $dt = DateTime->new(year => $1, month => $2, day => $3, hour => 0, minute => 0, second => 0, time_zone => $self->{tz})->subtract(months => $num); + + my $month = $dt->month(); + $month = '0' . $month if ($month < 10); + my $day = $dt->day(); + $day = '0' . $day if ($day < 10); + return $dt->year() . '-' . $month . '-' . $day; +} + +sub subtractDateDays { + my ($self, $date, $num) = @_; + + if ($date !~ /(\d{4})-(\d{2})-(\d{2})/) { + $self->{logger}->writeLog('ERROR', "Verify date format"); + } + + my $dt = DateTime->new(year => $1, month => $2, day => $3, hour => 0, minute => 0, second => 0, time_zone => $self->{tz})->subtract(days => $num); + + my $month = $dt->month(); + $month = '0' . $month if ($month < 10); + my $day = $dt->day(); + $day = '0' . $day if ($day < 10); + return $dt->year() . '-' . $month . '-' . $day; +} + +sub getDayOfWeek { + my ($self, $date) = @_; + + if ($date !~ /(\d{4})-(\d{2})-(\d{2})/) { + $self->{logger}->writeLog('ERROR', "Verify date format"); + } + + return lc(DateTime->new(year => $1, month => $2, day => $3, hour => 0, minute => 0, second => 0, time_zone => $self->{tz})->day_name()); +} + +sub getDateEpoch { + my ($self, $date) = @_; + + if ($date !~ /(\d{4})-(\d{2})-(\d{2})/) { + $self->{logger}->writeLog('ERROR', "Verify date format"); + } + + my $epoch = DateTime->new(year => $1, month => $2, day => $3, hour => 0, minute => 0, second => 0, time_zone => $self->{tz})->epoch(); + $date =~ s/-//g; + + return wantarray ? ($epoch, $date) : $epoch; +} + +sub getRangePartitionDate { + my ($self, $start, $end) = @_; + + if ($start !~ /(\d{4})-(\d{2})-(\d{2})/) { + $self->{logger}->writeLog('ERROR', "Verify period start format"); + } + my $dt1 = DateTime->new(year => $1, month => $2, day => $3, hour => 0, minute => 0, second => 0, time_zone => $self->{tz}); + + if ($end !~ /(\d{4})-(\d{2})-(\d{2})/) { + $self->{logger}->writeLog('ERROR', "Verify period end format"); + } + my $dt2 = DateTime->new(year => $1, month => $2, day => $3, hour => 0, minute => 0, second => 0, time_zone => $self->{tz}); + + my $epoch = $dt1->epoch(); + my $epoch_end = $dt2->epoch(); + if ($epoch_end <= $epoch) { + $self->{logger}->writeLog('ERROR', "Period end date is older"); + } + + my $partitions = []; + while ($epoch < $epoch_end) { + $dt1->add(days => 1); + + $epoch = $dt1->epoch(); + my $month = $dt1->month(); + $month = '0' . $month if ($month < 10); + my $day = $dt1->day(); + $day = '0' . $day if ($day < 10); + + push @$partitions, { + name => $dt1->year() . $month . $day, + date => $dt1->year() . '-' . $month . '-' . $day, + epoch => $epoch + }; + } + + return $partitions; +} + +sub checkDateFormat { + my ($self, $start, $end) = @_; + + if (defined($start) && $start =~ /[1-2][0-9]{3}\-[0-1][0-9]\-[0-3][0-9]/ + && defined($end) && $end =~ /[1-2][0-9]{3}\-[0-1][0-9]\-[0-3][0-9]/) { + return 1; + } + return 0; +} + +sub getRebuildPeriods { + my ($self, $start, $end) = @_; + + my ($day,$month,$year) = (localtime($start))[3,4,5]; + $start = POSIX::mktime(0,0,0,$day,$month,$year,0,0,-1); + my $previousDay = POSIX::mktime(0,0,0,$day - 1,$month,$year,0,0,-1); + my @days = (); + while ($start < $end) { + # if there is few hour gap (time change : winter/summer), we also readjust it + if ($start == $previousDay) { + $start = POSIX::mktime(0,0,0, ++$day, $month, $year,0,0,-1); + } + my $dayEnd = POSIX::mktime(0, 0, 0, ++$day, $month, $year, 0, 0, -1); + + my %period = ("start" => $start, "end" => $dayEnd); + $days[scalar(@days)] = \%period; + $previousDay = $start; + $start = $dayEnd; + } + return (\@days); +} + +#parseFlatFile (file, key,value) : replace a line with a key by a value (entire line) to the specified file +sub parseAndReplaceFlatFile{ + my $self = shift; + my $file = shift; + my $key = shift; + my $value = shift; + + if (!-e $file) { + $self->{logger}->writeLog('ERROR', "File missing [".$file."]. Make sure you installed all the pre-requisites before executing this script"); + } + + tie my @flatfile, 'Tie::File', $file or die $!; + + foreach my $line(@flatfile) + { + if( $line =~ m/$key/ ) { + my $previousLine = $line; + $line =~ s/$key/$value/g; + $self->{logger}->writeLog('DEBUG', "[".$file."]"); + $self->{logger}->writeLog('DEBUG', "Replacing [".$previousLine."] by [".$value."]"); + } + } +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/BIHost.pm b/gorgone/modules/centreon/mbi/libs/bi/BIHost.pm new file mode 100644 index 0000000..114783f --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/BIHost.pm @@ -0,0 +1,233 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::BIHost; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + $self->{"today_table"} = "mod_bi_tmp_today_hosts"; + $self->{"tmp_comp"} = "mod_bi_tmp_hosts"; + $self->{"tmp_comp_storage"} = "mod_bi_tmp_hosts_storage"; + $self->{"table"} = "mod_bi_hosts"; + bless $self, $class; + return $self; +} + +sub getHostsInfo { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "SELECT `id`, `host_id`, `host_name`, `hc_id`, `hc_name`, `hg_id`, `hg_name`"; + $query .= " FROM `".$self->{"today_table"}."`"; + my $sth = $db->query($query); + my %result = (); + while (my $row = $sth->fetchrow_hashref()) { + if (defined($result{$row->{'host_id'}})) { + my $tab_ref = $result{$row->{'host_id'}}; + my @tab = @$tab_ref; + push @tab , $row->{"host_id"}.";".$row->{"host_name"}.";". + $row->{"hg_id"}.";".$row->{"hg_name"}.";". + $row->{"hc_id"}.";".$row->{"hc_name"}; + $result{$row->{'host_id'}} = \@tab; + }else { + my @tab = ($row->{"host_id"}.";".$row->{"host_name"}.";". + $row->{"hg_id"}.";".$row->{"hg_name"}.";". + $row->{"hc_id"}.";".$row->{"hc_name"}); + $result{$row->{'host_id'}} = \@tab; + } + } + $sth->finish(); + return (\%result); +} + +sub insert { + my $self = shift; + my $data = shift; + my $db = $self->{"centstorage"}; + $self->insertIntoTable("".$self->{"table"}."", $data); + $self->createTempTodayTable("false"); + my $fields = "id, host_name, host_id, hc_id, hc_name, hg_id, hg_name"; + my $query = "INSERT INTO ".$self->{"today_table"}." (".$fields.")"; + $query .= " SELECT ".$fields." FROM ".$self->{"table"}." "; + $db->query($query); +} + +sub update { + my ($self, $data, $useMemory) = @_; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + $self->createTempComparisonTable($useMemory); + $self->insertIntoTable($self->{"tmp_comp"}, $data); + $self->createTempStorageTable($useMemory); + $self->joinNewAndCurrentEntries(); + $self->insertNewEntries(); + $db->query("DROP TABLE `".$self->{"tmp_comp_storage"}."`"); + $self->createTempTodayTable("false"); + $self->insertTodayEntries(); + $db->query("DROP TABLE `".$self->{"tmp_comp"}."`"); +} + +sub insertIntoTable { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my $table = shift; + my $data = shift; + my $query = "INSERT INTO `".$table."`". + " (`host_id`, `host_name`, `hg_id`, `hg_name`, `hc_id`, `hc_name`)". + " VALUES (?,?,?,?,?,?)"; + my $sth = $db->prepare($query); + my $inst = $db->getInstance; + $inst->begin_work; + my $counter = 0; + + foreach (@$data) { + my ($host_id, $host_name, $hg_id, $hg_name, $hc_id, $hc_name) = split(";", $_); + $sth->bind_param(1, $host_id); + $sth->bind_param(2, $host_name); + $sth->bind_param(3, $hg_id); + $sth->bind_param(4, $hg_name); + $sth->bind_param(5, $hc_id); + $sth->bind_param(6, $hc_name); + $sth->execute; + if (defined($inst->errstr)) { + $logger->writeLog("FATAL", $self->{"table"}." insertion execute error : ".$inst->errstr); + } + if ($counter >= 1000) { + $counter = 0; + $inst->commit; + if (defined($inst->errstr)) { + $logger->writeLog("FATAL", $self->{"table"}." insertion commit error : ".$inst->errstr); + } + $inst->begin_work; + } + $counter++; + } + $inst->commit; +} + +sub createTempComparisonTable { + my ($self, $useMemory) = @_; + my $db = $self->{"centstorage"}; + $db->query("DROP TABLE IF EXISTS `".$self->{"tmp_comp"}."`"); + my $query = "CREATE TABLE `".$self->{"tmp_comp"}."` ("; + $query .= "`host_id` int(11) NOT NULL,`host_name` varchar(255) NOT NULL,"; + $query .= "`hc_id` int(11) DEFAULT NULL, `hc_name` varchar(255) NOT NULL,"; + $query .= "`hg_id` int(11) DEFAULT NULL, `hg_name` varchar(255) NOT NULL"; + if (defined($useMemory) && $useMemory eq "true") { + $query .= ") ENGINE=MEMORY DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + }else { + $query .= ") ENGINE=INNODB DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + } + $db->query($query); +} + +sub createTempStorageTable { + my ($self,$useMemory) = @_; + my $db = $self->{"centstorage"}; + + $db->query("DROP TABLE IF EXISTS `".$self->{"tmp_comp_storage"}."`"); + my $query = "CREATE TABLE `".$self->{"tmp_comp_storage"}."` ("; + $query .= "`id` INT NOT NULL,"; + $query .= "`host_id` int(11) NOT NULL,`host_name` varchar(255) NOT NULL,"; + $query .= "`hc_id` int(11) DEFAULT NULL, `hc_name` varchar(255) NOT NULL,"; + $query .= "`hg_id` int(11) DEFAULT NULL, `hg_name` varchar(255) NOT NULL,"; + $query .= " KEY `id` (`id`)"; + if (defined($useMemory) && $useMemory eq "true") { + $query .= ") ENGINE=MEMORY DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + }else { + $query .= ") ENGINE=INNODB DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + } + $db->query($query); +} + +sub createTempTodayTable { + my ($self,$useMemory) = @_; + my $db = $self->{"centstorage"}; + + $db->query("DROP TABLE IF EXISTS `".$self->{"today_table"}."`"); + my $query = "CREATE TABLE `".$self->{"today_table"}."` ("; + $query .= "`id` INT NOT NULL,"; + $query .= "`host_id` int(11) NOT NULL,`host_name` varchar(255) NOT NULL,"; + $query .= "`hc_id` int(11) DEFAULT NULL, `hc_name` varchar(255) NOT NULL,"; + $query .= "`hg_id` int(11) DEFAULT NULL, `hg_name` varchar(255) NOT NULL,"; + $query .= " KEY `id` (`host_id`)"; + if (defined($useMemory) && $useMemory eq "true") { + $query .= ") ENGINE=MEMORY DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + }else { + $query .= ") ENGINE=INNODB DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + } + $db->query($query); +} + +sub joinNewAndCurrentEntries { + my ($self) = @_; + my $db = $self->{"centstorage"}; + + my $query = "INSERT INTO ".$self->{"tmp_comp_storage"}. " (id, host_name, host_id, hc_id, hc_name, hg_id, hg_name)"; + $query .= " SELECT IFNULL(h.id, 0), t.host_name, t.host_id, t.hc_id, t.hc_name, t.hg_id, t.hg_name FROM ".$self->{"tmp_comp"}." t"; + $query .= " LEFT JOIN ".$self->{"table"}." h USING (host_name, host_id, hc_id, hc_name, hg_id, hg_name)"; + $db->query($query); +} + +sub insertNewEntries { + my ($self) = @_; + my $db = $self->{"centstorage"}; + my $fields = "host_name, host_id, hc_id, hc_name, hg_id, hg_name"; + my $query = " INSERT INTO `".$self->{"table"}."` (".$fields.") "; + $query .= " SELECT ".$fields." FROM ".$self->{"tmp_comp_storage"}; + $query .= " WHERE id = 0"; + $db->query($query); +} + +sub insertTodayEntries { + my ($self) = @_; + my $db = $self->{"centstorage"}; + my $fields = "host_name, host_id, hc_id, hc_name, hg_id, hg_name"; + my $query = "INSERT INTO ".$self->{"today_table"}." (id, host_name, host_id, hc_id, hc_name, hg_id, hg_name)"; + $query .= " SELECT h.id, t.host_name, t.host_id, t.hc_id, t.hc_name, t.hg_id, t.hg_name FROM ".$self->{"tmp_comp"}." t"; + $query .= " JOIN ".$self->{"table"}." h USING (host_name, host_id, hc_id, hc_name, hg_id, hg_name)"; + $db->query($query); +} + +sub truncateTable { + my $self = shift; + my $db = $self->{"centstorage"}; + + $db->query("TRUNCATE TABLE `".$self->{"table"}."`"); + $db->query("ALTER TABLE `".$self->{"table"}."` AUTO_INCREMENT=1"); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/BIHostCategory.pm b/gorgone/modules/centreon/mbi/libs/bi/BIHostCategory.pm new file mode 100644 index 0000000..dec1d24 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/BIHostCategory.pm @@ -0,0 +1,129 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::BIHostCategory; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + bless $self, $class; + return $self; +} + +sub getAllEntries { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "SELECT `hc_id`, `hc_name`"; + $query .= " FROM `mod_bi_hostcategories`"; + my $sth = $db->query($query); + my @entries = (); + while (my $row = $sth->fetchrow_hashref()) { + push @entries, $row->{"hc_id"}.";".$row->{"hc_name"}; + } + $sth->finish(); + return (\@entries); +} + +sub getEntryIds { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "SELECT `id`, `hc_id`, `hc_name`"; + $query .= " FROM `mod_bi_hostcategories`"; + my $sth = $db->query($query); + my %entries = (); + while (my $row = $sth->fetchrow_hashref()) { + $entries{$row->{"hc_id"}.";".$row->{"hc_name"}} = $row->{"id"}; + } + $sth->finish(); + return (\%entries); +} + +sub entryExists { + my $self = shift; + my ($value, $entries) = (shift, shift); + foreach(@$entries) { + if ($value eq $_) { + return 1; + } + } + return 0; +} +sub insert { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my $data = shift; + my $query = "INSERT INTO `mod_bi_hostcategories`". + " (`hc_id`, `hc_name`)". + " VALUES (?,?)"; + my $sth = $db->prepare($query); + my $inst = $db->getInstance; + $inst->begin_work; + my $counter = 0; + + my $existingEntries = $self->getAllEntries; + foreach (@$data) { + if (!$self->entryExists($_, $existingEntries)) { + my ($hc_id, $hc_name) = split(";", $_); + $sth->bind_param(1, $hc_id); + $sth->bind_param(2, $hc_name); + $sth->execute; + if (defined($inst->errstr)) { + $logger->writeLog("FATAL", "hostcategories insertion execute error : ".$inst->errstr); + } + if ($counter >= 1000) { + $counter = 0; + $inst->commit; + if (defined($inst->errstr)) { + $logger->writeLog("FATAL", "hostcategories insertion commit error : ".$inst->errstr); + } + $inst->begin_work; + } + $counter++; + } + } + $inst->commit; +} + +sub truncateTable { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "TRUNCATE TABLE `mod_bi_hostcategories`"; + $db->query($query); + $db->query("ALTER TABLE `mod_bi_hostcategories` AUTO_INCREMENT=1"); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/BIHostGroup.pm b/gorgone/modules/centreon/mbi/libs/bi/BIHostGroup.pm new file mode 100644 index 0000000..acc6e2d --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/BIHostGroup.pm @@ -0,0 +1,131 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::BIHostGroup; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + bless $self, $class; + return $self; +} + + +sub getAllEntries { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "SELECT `id`, `hg_id`, `hg_name`"; + $query .= " FROM `mod_bi_hostgroups`"; + my $sth = $db->query($query); + my @entries = (); + while (my $row = $sth->fetchrow_hashref()) { + push @entries, $row->{"hg_id"}.";".$row->{"hg_name"}; + } + $sth->finish(); + return (\@entries); +} + +sub getEntryIds { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "SELECT `id`, `hg_id`, `hg_name`"; + $query .= " FROM `mod_bi_hostgroups`"; + my $sth = $db->query($query); + my %entries = (); + while (my $row = $sth->fetchrow_hashref()) { + $entries{$row->{"hg_id"}.";".$row->{"hg_name"}} = $row->{"id"}; + } + $sth->finish(); + return (\%entries); +} + +sub entryExists { + my $self = shift; + my ($value, $entries) = (shift, shift); + foreach(@$entries) { + if ($value eq $_) { + return 1; + } + } + return 0; +} +sub insert { + my $self = shift; + + my $db = $self->{centstorage}; + my $logger = $self->{logger}; + my $data = shift; + my $query = "INSERT INTO `mod_bi_hostgroups`". + " (`hg_id`, `hg_name`)". + " VALUES (?,?)"; + my $sth = $db->prepare($query); + my $inst = $db->getInstance(); + $inst->begin_work(); + my $counter = 0; + + my $existingEntries = $self->getAllEntries(); + foreach (@$data) { + if (!$self->entryExists($_, $existingEntries)) { + my ($hg_id, $hg_name) = split(";", $_); + $sth->bind_param(1, $hg_id); + $sth->bind_param(2, $hg_name); + $sth->execute; + if (defined($inst->errstr)) { + $logger->writeLog("FATAL", "hostgroups insertion execute error : ".$inst->errstr); + } + if ($counter >= 1000) { + $counter = 0; + $inst->commit; + if (defined($inst->errstr)) { + $logger->writeLog("FATAL", "hostgroups insertion commit error : ".$inst->errstr); + } + $inst->begin_work; + } + $counter++; + } + } + $inst->commit(); +} + +sub truncateTable { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "TRUNCATE TABLE `mod_bi_hostgroups`"; + $db->query($query); + $db->query("ALTER TABLE `mod_bi_hostgroups` AUTO_INCREMENT=1"); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/BIHostStateEvents.pm b/gorgone/modules/centreon/mbi/libs/bi/BIHostStateEvents.pm new file mode 100644 index 0000000..2593910 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/BIHostStateEvents.pm @@ -0,0 +1,243 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::BIHostStateEvents; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + $self->{'timeperiod'} = shift; + $self->{'bind_counter'} = 0; + $self->{'statement'} = undef; + $self->{'name'} = "mod_bi_hoststateevents"; + $self->{'tmp_name'} = "mod_bi_hoststateevents_tmp"; + $self->{'timeColumn'} = "end_time"; + bless $self, $class; + return $self; +} + +sub getName() { + my $self = shift; + return $self->{'name'}; +} + +sub getTimeColumn() { + my $self = shift; + return $self->{'timeColumn'}; +} + +sub createTempBIEventsTable{ + my ($self) = @_; + my $db = $self->{"centstorage"}; + $db->query("DROP TABLE IF EXISTS `mod_bi_hoststateevents_tmp`"); + my $createTable = " CREATE TABLE `mod_bi_hoststateevents_tmp` ("; + $createTable .= " `host_id` int(11) NOT NULL,"; + $createTable .= " `modbiliveservice_id` tinyint(4) NOT NULL,"; + $createTable .= " `state` tinyint(4) NOT NULL,"; + $createTable .= " `start_time` int(11) NOT NULL,"; + $createTable .= " `end_time` int(11) DEFAULT NULL,"; + $createTable .= " `duration` int(11) NOT NULL,"; + $createTable .= " `sla_duration` int(11) NOT NULL,"; + $createTable .= " `ack_time` int(11) DEFAULT NULL,"; + $createTable .= " `last_update` tinyint(4) NOT NULL DEFAULT '0',"; + $createTable .= " KEY `modbihost_id` (`host_id`)"; + $createTable .= " ) ENGINE=InnoDB DEFAULT CHARSET=utf8"; + $db->query($createTable); +} + +sub prepareTempQuery { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "INSERT INTO `".$self->{'tmp_name'}."`". + " (`host_id`, `modbiliveservice_id`,". + " `state`, `start_time`, `sla_duration`,". + " `end_time`, `ack_time`, `last_update`, `duration`) ". + " VALUES (?,?,?,?,?,?,?,?, TIMESTAMPDIFF(SECOND, FROM_UNIXTIME(?), FROM_UNIXTIME(?)))"; + $self->{'statement'} = $db->prepare($query); + $self->{'dbinstance'} = $db->getInstance; + ($self->{'dbinstance'})->begin_work; +} + +sub prepareQuery { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "INSERT INTO `".$self->{'name'}."`". + " (`modbihost_id`, `modbiliveservice_id`,". + " `state`, `start_time`, `sla_duration`,". + " `end_time`, `ack_time`, `last_update`, `duration`) ". + " VALUES (?,?,?,?,?,?,?,?, TIMESTAMPDIFF(SECOND, FROM_UNIXTIME(?), FROM_UNIXTIME(?)))"; + $self->{'statement'} = $db->prepare($query); + $self->{'dbinstance'} = $db->getInstance; + ($self->{'dbinstance'})->begin_work; +} + +sub bindParam { + my ($self, $row) = @_; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my $size = scalar(@$row); + my $sth = $self->{'statement'}; + for (my $i = 0; $i < $size; $i++) { + $sth->bind_param($i + 1, $row->[$i]); + } + $sth->bind_param($size+1, $row->[3]); + $sth->bind_param($size+2, $row->[5]); + ($self->{'statement'})->execute; + if (defined(($self->{'dbinstance'})->errstr)) { + $logger->writeLog("FATAL", $self->{'name'}." insertion execute error : ".($self->{'dbinstance'})->errstr); + } + if ($self->{'bind_counter'} >= 1000) { + $self->{'bind_counter'} = 0; + ($self->{'dbinstance'})->commit; + if (defined(($self->{'dbinstance'})->errstr)) { + $logger->writeLog("FATAL", $self->{'name'}." insertion commit error : ".($self->{'dbinstance'})->errstr); + } + ($self->{'dbinstance'})->begin_work; + } + $self->{'bind_counter'} += 1; + +} + +sub getDayEvents { + my $self = shift; + my $db = $self->{"centstorage"}; + my $timeperiod = $self->{'timeperiod'}; + my ($start, $end, $liveserviceId, $ranges) = @_; + my %results = (); + + my $query = "SELECT start_time, end_time, state, modbihost_id"; + $query .= " FROM `" . $self->{name} . "`"; + $query .= " WHERE `start_time` < ".$end.""; + $query .= " AND `end_time` > ".$start.""; + $query .= " AND `state` in (0,1,2)"; + $query .= " AND modbiliveservice_id = ".$liveserviceId; + my $sth = $db->query($query); + + #For each events, for the current day, calculate statistics for the day + my $rows = []; + while (my $row = ( + shift(@$rows) || + shift(@{$rows = $sth->fetchall_arrayref(undef,10_000) || []}) ) + ) { + my $entryID = $row->[3]; + + my ($started, $ended) = (0, 0); + my $rangeSize = scalar(@$ranges); + my $eventDuration = 0; + for(my $count = 0; $count < $rangeSize; $count++) { + my $currentStart = $row->[0]; + my $currentEnd = $row->[1]; + + my $range = $ranges->[$count]; + my ($rangeStart, $rangeEnd) = ($range->[0], $range->[1]); + if ($currentStart < $rangeEnd && $currentEnd > $rangeStart) { + if ($currentStart < $rangeStart) { + $currentStart = $rangeStart; + }elsif ($count == 0) { + $started = 1; + } + if ($currentEnd > $rangeEnd) { + $currentEnd = $rangeEnd; + }elsif ($count == $rangeSize - 1) { + $ended = 1; + } + $eventDuration += $currentEnd - $currentStart; + } + } + if (!defined($results{$entryID})) { + my @tab = (0, 0, 0, 0, 0, 0, 0); + + #New version - sync with tables in database + # 0: UP, 1: DOWN time, 2: Unreachable time , 3 : DOWN alerts opened + # 4: Down time alerts closed, 5: unreachable alerts started, 6 : unreachable alerts ended + $results{$entryID} = \@tab; + } + + my $stats = $results{$entryID}; + my $state = $row->[2]; + + if ($state == 0) { + $stats->[0] += $eventDuration; + }elsif ($state == 1) { + $stats->[1] += $eventDuration; + $stats->[3] += $started; + $stats->[4] += $ended; + }elsif ($state == 2) { + $stats->[2] += $eventDuration; + $stats->[5] += $started; + $stats->[6] += $ended; + } + + $results{$entryID} = $stats; + } + + return (\%results); +} + +#Deprecated +sub getNbEvents { + my ($self, $start, $end, $groupId, $catId, $liveServiceID) = @_; + my $db = $self->{"centstorage"}; + + my $query = "SELECT count(state) as nbEvents, state"; + $query .= " FROM mod_bi_hosts h, ".$self->{'name'}." e"; + $query .= " WHERE h.hg_id = ".$groupId." AND h.hc_id=".$catId; + $query .= " AND h.id = e.modbihost_id"; + $query .= " AND e.modbiliveservice_id=".$liveServiceID; + $query .= " AND start_time < UNIX_TIMESTAMP('".$end."')"; + $query .= " AND end_time > UNIX_TIMESTAMP('".$start."')"; + $query .= " AND state in (1,2)"; + $query .= " GROUP BY state"; + my $sth = $db->query($query); + + my ($downEvents, $unrEvents) = (undef, undef); + while (my $row = $sth->fetchrow_hashref()) { + if ($row->{'state'} == 1) { + $downEvents = $row->{'nbEvents'}; + }else { + $unrEvents = $row->{'nbEvents'}; + } + } + return ($downEvents, $unrEvents); +} + +sub deleteUnfinishedEvents { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "DELETE FROM `mod_bi_hoststateevents`"; + $query .= " WHERE last_update = 1 OR end_time is null"; + $db->query($query); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/BIMetric.pm b/gorgone/modules/centreon/mbi/libs/bi/BIMetric.pm new file mode 100644 index 0000000..4462aa3 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/BIMetric.pm @@ -0,0 +1,199 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::BIMetric; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + + $self->{logger} = shift; + $self->{centstorage} = shift; + if (@_) { + $self->{centreon} = shift; + } + $self->{today_table} = "mod_bi_tmp_today_servicemetrics"; + $self->{tmpTable} = "mod_bi_tmp_servicemetrics"; + $self->{CRC32} = "mod_bi_tmp_servicemetrics_crc32"; + $self->{table} = "mod_bi_servicemetrics"; + + bless $self, $class; + return $self; +} + +sub insert { + my $self = shift; + my $db = $self->{centstorage}; + + $self->insertMetricsIntoTable("mod_bi_servicemetrics"); + $self->createTodayTable("false"); + my $query = "INSERT INTO ".$self->{today_table}. " (id, metric_id, metric_name, sc_id,hg_id,hc_id)"; + $query .= " SELECT id, metric_id, metric_name,sc_id,hg_id,hc_id FROM " . $self->{table} . " "; + $db->query($query); +} + +sub update { + my ($self,$useMemory) = @_; + + my $db = $self->{centstorage}; + + $self->createTempTable($useMemory); + $self->insertMetricsIntoTable($self->{tmpTable}); + $self->createCRC32Table(); + $self->insertNewEntries(); + $self->createCRC32Table(); + $self->createTodayTable("false"); + $self->insertTodayEntries(); + $db->query("DROP TABLE `".$self->{"tmpTable"}."`"); + $db->query("DROP TABLE `".$self->{"CRC32"}."`"); +} + +sub insertMetricsIntoTable { + my $self = shift; + my $db = $self->{"centstorage"}; + my $table = shift; + my $query = "INSERT INTO `".$table."` (`metric_id`, `metric_name`, `metric_unit`, `service_id`, `service_description`,"; + $query .= " `sc_id`, `sc_name`, `host_id`, `host_name`, `hc_id`, `hc_name`, `hg_id`, `hg_name`)"; + $query .= " SELECT `metric_id`, `metric_name`, `unit_name`, s.`service_id`, s.`service_description`, "; + $query .= " s.`sc_id`, s.`sc_name`, s.`host_id`, s.`host_name`, `hc_id`, `hc_name`, `hg_id`, `hg_name`"; + $query .= " FROM `mod_bi_tmp_today_services` s, `metrics` m, `index_data` i"; + $query .= " WHERE i.id = m.index_id and i.host_id=s.host_id and i.service_id=s.service_id"; + $query .= " group by s.hg_id, s.hc_id, s.sc_id, m.index_id, m.metric_id"; + my $sth = $db->query($query); + return $sth; +} + +sub createTempTable { + my ($self, $useMemory) = @_; + + my $db = $self->{"centstorage"}; + $db->query("DROP TABLE IF EXISTS `".$self->{"tmpTable"}."`"); + my $query = "CREATE TABLE `".$self->{"tmpTable"}."` ("; + $query .= "`metric_id` int(11) NOT NULL,`metric_name` varchar(255) NOT NULL,`metric_unit` char(10) DEFAULT NULL,"; + $query .= "`service_id` int(11) NOT NULL,`service_description` varchar(255) DEFAULT NULL,"; + $query .= "`sc_id` int(11) DEFAULT NULL,`sc_name` varchar(255) DEFAULT NULL,"; + $query .= "`host_id` int(11) DEFAULT NULL,`host_name` varchar(255) DEFAULT NULL,"; + $query .= "`hc_id` int(11) DEFAULT NULL,`hc_name` varchar(255) DEFAULT NULL,"; + $query .= "`hg_id` int(11) DEFAULT NULL,`hg_name` varchar(255) DEFAULT NULL"; + if (defined($useMemory) && $useMemory eq "true") { + $query .= ") ENGINE=MEMORY DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + }else { + $query .= ") ENGINE=INNODB DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + } + $db->query($query); +} + +sub createCRC32Table { + my ($self) = @_; + my $db = $self->{"centstorage"}; + + $db->query("DROP TABLE IF EXISTS `".$self->{"CRC32"}."`"); + my $query = "CREATE TABLE `".$self->{"CRC32"}."` CHARSET=utf8 COLLATE=utf8_general_ci"; + $query .= " SELECT `id`, CRC32(CONCAT_WS('-', COALESCE(metric_id, '?'),"; + $query .= " COALESCE(service_id, '?'),COALESCE(service_description, '?'),"; + $query .= " COALESCE(host_id, '?'),COALESCE(host_name, '?'), COALESCE(sc_id, '?'),COALESCE(sc_name, '?'),"; + $query .= " COALESCE(hc_id, '?'),COALESCE(hc_name, '?'), COALESCE(hg_id, '?'),COALESCE(hg_name, '?'))) as mycrc"; + $query .= " FROM ".$self->{"table"}; + $db->query($query); + $query = "ALTER TABLE `".$self->{"CRC32"}."` ADD INDEX (`mycrc`)"; + $db->query($query); +} + +sub insertNewEntries { + my ($self) = @_; + my $db = $self->{"centstorage"}; + my $fields = "metric_id, metric_name, metric_unit, service_id, service_description, host_name, host_id, sc_id, sc_name, hc_id, hc_name, hg_id, hg_name"; + my $tmpTableFields = "tmpTable.metric_id, tmpTable.metric_name,tmpTable.metric_unit,"; + $tmpTableFields .= " tmpTable.service_id, tmpTable.service_description, tmpTable.host_name, tmpTable.host_id, tmpTable.sc_id,"; + $tmpTableFields .= "tmpTable.sc_name, tmpTable.hc_id, tmpTable.hc_name, tmpTable.hg_id, tmpTable.hg_name"; + my $query = " INSERT INTO `".$self->{"table"}."` (".$fields.") "; + $query .= " SELECT ".$tmpTableFields." FROM ".$self->{"tmpTable"}." as tmpTable"; + $query .= " LEFT JOIN (".$self->{"CRC32"}. " INNER JOIN ".$self->{"table"}." as finalTable using (id))"; + $query .= " ON CRC32(CONCAT_WS('-', COALESCE(tmpTable.metric_id, '?'), COALESCE(tmpTable.service_id, '?'),COALESCE(tmpTable.service_description, '?'),"; + $query .= " COALESCE(tmpTable.host_id, '?'),COALESCE(tmpTable.host_name, '?'), COALESCE(tmpTable.sc_id, '?'),COALESCE(tmpTable.sc_name, '?'),"; + $query .= " COALESCE(tmpTable.hc_id, '?'),COALESCE(tmpTable.hc_name, '?'), COALESCE(tmpTable.hg_id, '?'),COALESCE(tmpTable.hg_name, '?'))) = mycrc"; + $query .= " AND tmpTable.metric_id=finalTable.metric_id"; + $query .= " AND tmpTable.service_id=finalTable.service_id AND tmpTable.service_description=finalTable.service_description"; + $query .= " AND tmpTable.host_id=finalTable.host_id AND tmpTable.host_name=finalTable.host_name"; + $query .= " AND tmpTable.sc_id=finalTable.sc_id AND tmpTable.sc_name=finalTable.sc_name"; + $query .= " AND tmpTable.hc_id=finalTable.hc_id AND tmpTable.hc_name=finalTable.hc_name"; + $query .= " AND tmpTable.hg_id=finalTable.hg_id AND tmpTable.hg_name=finalTable.hg_name"; + $query .= " WHERE finalTable.id is null"; + $db->query($query); +} + +sub createTodayTable { + my ($self,$useMemory) = @_; + my $db = $self->{"centstorage"}; + + $db->query("DROP TABLE IF EXISTS `".$self->{"today_table"}."`"); + my $query = "CREATE TABLE `" . $self->{"today_table"} . "` ("; + $query .= "`id` INT NOT NULL,"; + $query .= "`metric_id` int(11) NOT NULL,"; + $query .= "`metric_name` varchar(255) NOT NULL,"; + $query .= "`sc_id` int(11) NOT NULL,"; + $query .= "`hg_id` int(11) NOT NULL,"; + $query .= "`hc_id` int(11) NOT NULL,"; + $query .= " KEY `metric_id` (`metric_id`),"; + $query .= " KEY `schghc_id` (`sc_id`,`hg_id`,`hc_id`)"; + if (defined($useMemory) && $useMemory eq "true") { + $query .= ") ENGINE=MEMORY DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + }else { + $query .= ") ENGINE=INNODB DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + } + $db->query($query); +} + +sub insertTodayEntries { + my ($self) = @_; + my $db = $self->{"centstorage"}; + my $query = "INSERT INTO ".$self->{"today_table"}. " (id, metric_id, metric_name, sc_id,hg_id,hc_id)"; + $query .= " SELECT finalTable.id, finalTable.metric_id, finalTable.metric_name, finalTable.sc_id, finalTable.hg_id, finalTable.hc_id FROM ".$self->{"tmpTable"}." t"; + $query .= " LEFT JOIN (".$self->{"CRC32"}." INNER JOIN ".$self->{"table"}." finalTable USING (id))"; + $query .= " ON CRC32(CONCAT_WS('-', COALESCE(t.metric_id, '?'), COALESCE(t.service_id, '?'),COALESCE(t.service_description, '?'),"; + $query .= " COALESCE(t.host_id, '?'),COALESCE(t.host_name, '?'), COALESCE(t.sc_id, '?'),COALESCE(t.sc_name, '?'),"; + $query .= " COALESCE(t.hc_id, '?'),COALESCE(t.hc_name, '?'), COALESCE(t.hg_id, '?'),COALESCE(t.hg_name, '?'))) = mycrc"; + $query .= " AND finalTable.metric_id=t.metric_id"; + $query .= " AND finalTable.service_id=t.service_id AND finalTable.service_description=t.service_description "; + $query .= " AND finalTable.host_id=t.host_id AND finalTable.host_name=t.host_name "; + $query .= " AND finalTable.sc_id=t.sc_id AND finalTable.sc_name=t.sc_name "; + $query .= " AND finalTable.hc_id=t.hc_id AND finalTable.hc_name=t.hc_name "; + $query .= " AND finalTable.hg_id=t.hg_id AND finalTable.hg_name=t.hg_name "; + $db->query($query); +} + +sub truncateTable { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "TRUNCATE TABLE `".$self->{"table"}."`"; + $db->query($query); + $db->query("ALTER TABLE `".$self->{"table"}."` AUTO_INCREMENT=1"); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/BIService.pm b/gorgone/modules/centreon/mbi/libs/bi/BIService.pm new file mode 100644 index 0000000..df60737 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/BIService.pm @@ -0,0 +1,221 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::BIService; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + $self->{"today_table"} = "mod_bi_tmp_today_services"; + $self->{"tmpTable"} = "mod_bi_tmp_services"; + $self->{"CRC32"} = "mod_bi_tmp_services_crc32"; + $self->{"table"} = "mod_bi_services"; + + bless $self, $class; + return $self; +} + +sub insert { + my $self = shift; + my $data = shift; + my $db = $self->{"centstorage"}; + $self->insertIntoTable($self->{"table"}, $data); + $self->createTodayTable("false"); + my $fields = "id, service_id, service_description, host_name, host_id, sc_id, sc_name, hc_id, hc_name, hg_id, hg_name"; + my $query = "INSERT INTO ".$self->{"today_table"}. "(".$fields.")"; + $query .= " SELECT ".$fields." FROM ".$self->{"table"}; + $db->query($query); +} + +sub update { + my ($self, $data, $useMemory) = @_; + my $db = $self->{"centstorage"}; + + $self->createTempTable($useMemory); + $self->insertIntoTable($self->{"tmpTable"}, $data); + $self->createCRC32Table(); + $self->insertNewEntries(); + $self->createCRC32Table(); + $self->createTodayTable("false"); + $self->insertTodayEntries(); + $db->query("DROP TABLE `".$self->{"tmpTable"}."`"); + $db->query("DROP TABLE `".$self->{"CRC32"}."`"); +} + +sub insertIntoTable { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my $table = shift; + my $data = shift; + my $name = shift; + my $id = shift; + my $query = "INSERT INTO `".$table."`". + " (`service_id`, `service_description`, `sc_id`, `sc_name`,". + " `host_id`, `host_name`,`hg_id`, `hg_name`, `hc_id`, `hc_name`)". + " VALUES (?,?,?,?,?,?,?,?,?,?)"; + my $sth = $db->prepare($query); + my $inst = $db->getInstance; + $inst->begin_work; + my $counter = 0; + + foreach (@$data) { + my ($service_id, $service_description, $sc_id, $sc_name, $host_id, $host_name, $hg_id, $hg_name, $hc_id, $hc_name) = split(";", $_); + $sth->bind_param(1, $service_id); + $sth->bind_param(2, $service_description); + $sth->bind_param(3, $sc_id); + $sth->bind_param(4, $sc_name); + $sth->bind_param(5, $host_id); + $sth->bind_param(6, $host_name); + $sth->bind_param(7, $hg_id); + $sth->bind_param(8, $hg_name); + $sth->bind_param(9, $hc_id); + $sth->bind_param(10, $hc_name); + $sth->execute; + if (defined($inst->errstr)) { + $logger->writeLog("FATAL", $table." insertion execute error : ".$inst->errstr); + } + if ($counter >= 1000) { + $counter = 0; + $inst->commit; + if (defined($inst->errstr)) { + $logger->writeLog("FATAL", $table." insertion commit error : ".$inst->errstr); + } + $inst->begin_work; + } + $counter++; + } + $inst->commit; +} +sub createTempTable { + my ($self, $useMemory) = @_; + my $db = $self->{"centstorage"}; + $db->query("DROP TABLE IF EXISTS `".$self->{"tmpTable"}."`"); + my $query = "CREATE TABLE `".$self->{"tmpTable"}."` ("; + $query .= "`service_id` int(11) NOT NULL,`service_description` varchar(255) NOT NULL,"; + $query .= "`sc_id` int(11) NOT NULL,`sc_name` varchar(255) NOT NULL,"; + $query .= "`host_id` int(11) DEFAULT NULL,`host_name` varchar(255) NOT NULL,"; + $query .= "`hc_id` int(11) DEFAULT NULL,`hc_name` varchar(255) NOT NULL,"; + $query .= "`hg_id` int(11) DEFAULT NULL,`hg_name` varchar(255) NOT NULL"; + if (defined($useMemory) && $useMemory eq "true") { + $query .= ") ENGINE=MEMORY DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + }else { + $query .= ") ENGINE=INNODB DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + } + $db->query($query); +} + +sub createCRC32Table { + my ($self) = @_; + my $db = $self->{"centstorage"}; + + $db->query("DROP TABLE IF EXISTS `".$self->{"CRC32"}."`"); + my $query = "CREATE TABLE `".$self->{"CRC32"}."` CHARSET=utf8 COLLATE=utf8_general_ci"; + $query .= " SELECT `id`, CRC32(CONCAT_WS('-', COALESCE(service_id, '?'),COALESCE(service_description, '?'),"; + $query .= " COALESCE(host_id, '?'),COALESCE(host_name, '?'), COALESCE(sc_id, '?'),COALESCE(sc_name, '?'),"; + $query .= " COALESCE(hc_id, '?'),COALESCE(hc_name, '?'), COALESCE(hg_id, '?'),COALESCE(hg_name, '?'))) as mycrc"; + $query .= " FROM ".$self->{"table"}; + $db->query($query); + $query = "ALTER TABLE `".$self->{"CRC32"}."` ADD INDEX (`mycrc`)"; + $db->query($query); +} + +sub insertNewEntries { + my ($self) = @_; + my $db = $self->{"centstorage"}; + my $fields = "service_id, service_description, host_name, host_id, sc_id, sc_name, hc_id, hc_name, hg_id, hg_name"; + my $tmpTableFields = "tmpTable.service_id, tmpTable.service_description, tmpTable.host_name, tmpTable.host_id, tmpTable.sc_id,"; + $tmpTableFields .= "tmpTable.sc_name, tmpTable.hc_id, tmpTable.hc_name, tmpTable.hg_id, tmpTable.hg_name"; + my $query = " INSERT INTO `".$self->{"table"}."` (".$fields.") "; + $query .= " SELECT ".$tmpTableFields." FROM ".$self->{"tmpTable"}." as tmpTable"; + $query .= " LEFT JOIN (".$self->{"CRC32"}. " INNER JOIN ".$self->{"table"}." as finalTable using (id))"; + $query .= " ON CRC32(CONCAT_WS('-', COALESCE(tmpTable.service_id, '?'),COALESCE(tmpTable.service_description, '?'),"; + $query .= " COALESCE(tmpTable.host_id, '?'),COALESCE(tmpTable.host_name, '?'), COALESCE(tmpTable.sc_id, '?'),COALESCE(tmpTable.sc_name, '?'),"; + $query .= " COALESCE(tmpTable.hc_id, '?'),COALESCE(tmpTable.hc_name, '?'), COALESCE(tmpTable.hg_id, '?'),COALESCE(tmpTable.hg_name, '?'))) = mycrc"; + $query .= " AND tmpTable.service_id=finalTable.service_id AND tmpTable.service_description=finalTable.service_description"; + $query .= " AND tmpTable.host_id=finalTable.host_id AND tmpTable.host_name=finalTable.host_name"; + $query .= " AND tmpTable.sc_id=finalTable.sc_id AND tmpTable.sc_name=finalTable.sc_name"; + $query .= " AND tmpTable.hc_id=finalTable.hc_id AND tmpTable.hc_name=finalTable.hc_name"; + $query .= " AND tmpTable.hg_id=finalTable.hg_id AND tmpTable.hg_name=finalTable.hg_name"; + $query .= " WHERE finalTable.id is null"; + $db->query($query); +} + +sub createTodayTable { + my ($self,$useMemory) = @_; + my $db = $self->{"centstorage"}; + + $db->query("DROP TABLE IF EXISTS `".$self->{"today_table"}."`"); + my $query = "CREATE TABLE `".$self->{"today_table"}."` ("; + $query .= "`id` INT NOT NULL,"; + $query .= "`service_id` int(11) NOT NULL,`service_description` varchar(255) NOT NULL,"; + $query .= "`sc_id` int(11) NOT NULL,`sc_name` varchar(255) NOT NULL,"; + $query .= "`host_id` int(11) DEFAULT NULL,`host_name` varchar(255) NOT NULL,"; + $query .= "`hc_id` int(11) DEFAULT NULL,`hc_name` varchar(255) NOT NULL,"; + $query .= "`hg_id` int(11) DEFAULT NULL,`hg_name` varchar(255) NOT NULL,"; + $query .= " KEY `host_service` (`host_id`, `service_id`)"; + if (defined($useMemory) && $useMemory eq "true") { + $query .= ") ENGINE=MEMORY DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + }else { + $query .= ") ENGINE=INNODB DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;"; + } + $db->query($query); +} + +sub insertTodayEntries { + my ($self) = @_; + my $db = $self->{"centstorage"}; + my $query = "INSERT INTO ".$self->{"today_table"}. " (id, service_id, service_description, host_name, host_id, sc_id, sc_name, hc_id, hc_name, hg_id, hg_name)"; + $query .= " SELECT s.id, t.service_id, t.service_description, t.host_name, t.host_id, t.sc_id, t.sc_name, t.hc_id, t.hc_name, t.hg_id, t.hg_name FROM ".$self->{"tmpTable"}." t"; + $query .= " LEFT JOIN (".$self->{"CRC32"}." INNER JOIN ".$self->{"table"}." s USING (id))"; + $query .= " ON CRC32(CONCAT_WS('-', COALESCE(t.service_id, '?'),COALESCE(t.service_description, '?'),"; + $query .= " COALESCE(t.host_id, '?'),COALESCE(t.host_name, '?'), COALESCE(t.sc_id, '?'),COALESCE(t.sc_name, '?'),"; + $query .= " COALESCE(t.hc_id, '?'),COALESCE(t.hc_name, '?'), COALESCE(t.hg_id, '?'),COALESCE(t.hg_name, '?'))) = mycrc"; + $query .= " AND s.service_id=t.service_id AND s.service_description=t.service_description "; + $query .= " AND s.host_id=t.host_id AND s.host_name=t.host_name "; + $query .= " AND s.sc_id=t.sc_id AND s.sc_name=t.sc_name "; + $query .= " AND s.hc_id=t.hc_id AND s.hc_name=t.hc_name "; + $query .= " AND s.hg_id=t.hg_id AND s.hg_name=t.hg_name "; + $db->query($query); +} + +sub truncateTable { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "TRUNCATE TABLE `".$self->{"table"}."`"; + $db->query($query); + $db->query("ALTER TABLE `".$self->{"table"}."` AUTO_INCREMENT=1"); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/BIServiceCategory.pm b/gorgone/modules/centreon/mbi/libs/bi/BIServiceCategory.pm new file mode 100644 index 0000000..a95e785 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/BIServiceCategory.pm @@ -0,0 +1,130 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::BIServiceCategory; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + bless $self, $class; + return $self; +} + + +sub getAllEntries { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "SELECT `sc_id`, `sc_name`"; + $query .= " FROM `mod_bi_servicecategories`"; + my $sth = $db->query($query); + my @entries = (); + while (my $row = $sth->fetchrow_hashref()) { + push @entries, $row->{"sc_id"}.";".$row->{"sc_name"}; + } + $sth->finish(); + return (\@entries); +} + +sub getEntryIds { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "SELECT `id`, `sc_id`, `sc_name`"; + $query .= " FROM `mod_bi_servicecategories`"; + my $sth = $db->query($query); + my %entries = (); + while (my $row = $sth->fetchrow_hashref()) { + $entries{$row->{"sc_id"}.";".$row->{"sc_name"}} = $row->{"id"}; + } + $sth->finish(); + return (\%entries); +} + +sub entryExists { + my $self = shift; + my ($value, $entries) = (shift, shift); + foreach(@$entries) { + if ($value eq $_) { + return 1; + } + } + return 0; +} +sub insert { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my $data = shift; + my $query = "INSERT INTO `mod_bi_servicecategories`". + " (`sc_id`, `sc_name`)". + " VALUES (?,?)"; + my $sth = $db->prepare($query); + my $inst = $db->getInstance; + $inst->begin_work; + my $counter = 0; + + my $existingEntries = $self->getAllEntries; + foreach (@$data) { + if (!$self->entryExists($_, $existingEntries)) { + my ($sc_id, $sc_name) = split(";", $_); + $sth->bind_param(1, $sc_id); + $sth->bind_param(2, $sc_name); + $sth->execute; + if (defined($inst->errstr)) { + $logger->writeLog("FATAL", "servicecategories insertion execute error : ".$inst->errstr); + } + if ($counter >= 1000) { + $counter = 0; + $inst->commit; + if (defined($inst->errstr)) { + $logger->writeLog("FATAL", "servicecategories insertion commit error : ".$inst->errstr); + } + $inst->begin_work; + } + $counter++; + } + } + $inst->commit; +} + +sub truncateTable { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "TRUNCATE TABLE `mod_bi_servicecategories`"; + $db->query($query); + $db->query("ALTER TABLE `mod_bi_servicecategories` AUTO_INCREMENT=1"); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/BIServiceStateEvents.pm b/gorgone/modules/centreon/mbi/libs/bi/BIServiceStateEvents.pm new file mode 100644 index 0000000..3840e1c --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/BIServiceStateEvents.pm @@ -0,0 +1,251 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::BIServiceStateEvents; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + $self->{'timeperiod'} = shift; + $self->{'bind_counter'} = 0; + $self->{'name'} = "mod_bi_servicestateevents"; + $self->{'tmp_name'} = "mod_bi_servicestateevents_tmp"; + $self->{'timeColumn'} = "end_time"; + bless $self, $class; + return $self; +} + +sub getName() { + my $self = shift; + return $self->{'name'}; +} + +sub getTimeColumn() { + my $self = shift; + return $self->{'timeColumn'}; +} + +sub prepareQuery { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "INSERT INTO `".$self->{'name'}."`". + " (`modbiservice_id`, `modbiliveservice_id`,". + " `state`, `start_time`, `sla_duration`,". + " `end_time`, `ack_time`, `last_update`, `duration`) ". + " VALUES (?,?,?,?,?,?,?,?, TIMESTAMPDIFF(SECOND, FROM_UNIXTIME(?), FROM_UNIXTIME(?)))"; + $self->{'statement'} = $db->prepare($query); + $self->{'dbinstance'} = $db->getInstance; + ($self->{'dbinstance'})->begin_work; +} + +sub createTempBIEventsTable { + my ($self) = @_; + my $db = $self->{"centstorage"}; + $db->query("DROP TABLE IF EXISTS `mod_bi_servicestateevents_tmp`"); + my $createTable = " CREATE TABLE `mod_bi_servicestateevents_tmp` ("; + $createTable .= " `host_id` int(11) NOT NULL,"; + $createTable .= " `service_id` int(11) NOT NULL,"; + $createTable .= " `modbiliveservice_id` tinyint(4) NOT NULL,"; + $createTable .= " `state` tinyint(4) NOT NULL,"; + $createTable .= " `start_time` int(11) NOT NULL,"; + $createTable .= " `end_time` int(11) DEFAULT NULL,"; + $createTable .= " `duration` int(11) NOT NULL,"; + $createTable .= " `sla_duration` int(11) NOT NULL,"; + $createTable .= " `ack_time` int(11) DEFAULT NULL,"; + $createTable .= " `last_update` tinyint(4) DEFAULT '0',"; + $createTable .= " KEY `modbiservice_id` (`host_id`,`service_id`)"; + $createTable .= " ) ENGINE=InnoDB DEFAULT CHARSET=utf8"; + $db->query($createTable); +} + +sub prepareTempQuery { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "INSERT INTO `".$self->{'tmp_name'}."`". + " (`host_id`,`service_id`,`modbiliveservice_id`,". + " `state`, `start_time`, `sla_duration`,". + " `end_time`, `ack_time`, `last_update`, `duration`) ". + " VALUES (?,?,?,?,?,?,?,?,?, TIMESTAMPDIFF(SECOND, FROM_UNIXTIME(?), FROM_UNIXTIME(?)))"; + $self->{'statement'} = $db->prepare($query); + $self->{'dbinstance'} = $db->getInstance; + ($self->{'dbinstance'})->begin_work; +} + +sub bindParam { + my ($self, $row) = @_; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my $size = scalar(@$row); + my $sth = $self->{'statement'}; + for (my $i = 0; $i < $size; $i++) { + $sth->bind_param($i + 1, $row->[$i]); + } + $sth->bind_param($size + 1, $row->[4]); + $sth->bind_param($size + 2, $row->[6]); + + ($self->{'statement'})->execute; + if (defined(($self->{'dbinstance'})->errstr)) { + $logger->writeLog("FATAL", $self->{'name'}." insertion execute error : ".($self->{'dbinstance'})->errstr); + } + if ($self->{'bind_counter'} >= 1000) { + $self->{'bind_counter'} = 0; + ($self->{'dbinstance'})->commit; + if (defined(($self->{'dbinstance'})->errstr)) { + $logger->writeLog("FATAL", $self->{'name'}." insertion commit error : ".($self->{'dbinstance'})->errstr); + } + ($self->{'dbinstance'})->begin_work; + } + $self->{'bind_counter'} += 1; + +} + +sub getDayEvents { + my $self = shift; + my $db = $self->{"centstorage"}; + my $timeperiod = $self->{'timeperiod'}; + my ($start, $end, $liveserviceId, $ranges) = @_; + my $liveServiceList = shift; + my %results = (); + + my $query = "SELECT start_time, end_time, state, modbiservice_id"; + $query .= " FROM `" . $self->{'name'} . "`"; + $query .= " WHERE `start_time` < " . $end; + $query .= " AND `end_time` > " . $start; + $query .= " AND `state` IN (0,1,2,3)"; + $query .= " AND modbiliveservice_id=" . $liveserviceId; + my $sth = $db->query($query); + + if (!scalar(@$ranges)) { + return \%results; + } + + my $rows = []; + while (my $row = ( + shift(@$rows) || + shift(@{$rows = $sth->fetchall_arrayref(undef,10_000) || []}) ) + ) { + my $entryID = $row->[3]; + + my ($started, $ended) = (0,0); + my $rangeSize = scalar(@$ranges); + my $eventDuration = 0; + for (my $count = 0; $count < $rangeSize; $count++) { + my $currentStart = $row->[0]; + my $currentEnd = $row->[1]; + + my $range = $ranges->[$count]; + my ($rangeStart, $rangeEnd) = ($range->[0], $range->[1]); + if ($currentStart < $rangeEnd && $currentEnd > $rangeStart) { + if ($currentStart < $rangeStart) { + $currentStart = $rangeStart; + } elsif ($count == 0) { + $started = 1; + } + if ($currentEnd > $rangeEnd) { + $currentEnd = $rangeEnd; + } elsif ($count == $rangeSize - 1) { + $ended = 1; + } + $eventDuration += $currentEnd - $currentStart; + } + } + if (!defined($results{$entryID})) { + my @tab = (0, 0, 0, 0, 0, 0, 0, 0, 0); + + #New table - sync with the real table in centreon_storage database + # 0: OK time , 1: CRITICAL time, 2 : DEGRADED time 3 : alert_unavailable_opened + # 4: alert unavailable_closed 5 : alert_degraded_opened 6 : alertes_degraded_closed + # 7 : alert_unknown_opened 8 : alert_unknown_closed + $results{$entryID} = \@tab; + } + my $stats = $results{$entryID}; + my $state = $row->[2]; + if ($state == 0) { + $stats->[0] += $eventDuration; + } elsif ($state == 1) { + $stats->[2] += $eventDuration; + $stats->[5] += $started; + $stats->[6] += $ended; + } elsif ($state == 2) { + $stats->[1] += $eventDuration; + $stats->[3] += $started; + $stats->[4] += $ended; + } else { + $stats->[7] += $started; + $stats->[8] += $ended; + } + $results{$entryID} = $stats; + } + + return (\%results); +} + +#Deprecated +sub getNbEvents { + my ($self, $start, $end, $groupId, $hcatId, $scatId, $liveServiceID) = @_; + my $db = $self->{"centstorage"}; + + my $query = "SELECT count(state) as nbEvents, state"; + $query .= " FROM mod_bi_services s, ".$self->{'name'}." e"; + $query .= " WHERE s.hg_id = ".$groupId." AND s.hc_id=".$hcatId." AND s.sc_id=".$scatId; + $query .= " AND s.id = e.modbiservice_id"; + $query .= " AND start_time < UNIX_TIMESTAMP('".$end."')"; + $query .= " AND end_time > UNIX_TIMESTAMP('".$start."')"; + $query .= " AND e.modbiliveservice_id=".$liveServiceID; + $query .= " AND e.state in (1,2,3)"; + $query .= " GROUP BY e.state"; + my $sth = $db->query($query); + + my ($warnEvents, $criticalEvents, $otherEvents) = (undef, undef, undef); + while (my $row = $sth->fetchrow_hashref()) { + if ($row->{'state'} == 1) { + $warnEvents = $row->{'nbEvents'}; + }elsif ($row->{'state'} == 2) { + $criticalEvents = $row->{'nbEvents'}; + }else { + $otherEvents = $row->{'nbEvents'}; + } + } + return ($warnEvents, $criticalEvents, $otherEvents); +} + +sub deleteUnfinishedEvents { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "DELETE FROM `".$self->{'name'}."`"; + $query .= " WHERE last_update = 1 OR end_time is null"; + $db->query($query); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/DBConfigParser.pm b/gorgone/modules/centreon/mbi/libs/bi/DBConfigParser.pm new file mode 100644 index 0000000..6c5571b --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/DBConfigParser.pm @@ -0,0 +1,85 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; +use POSIX; +use XML::LibXML; +use Data::Dumper; + +package gorgone::modules::centreon::mbi::libs::bi::DBConfigParser; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database + +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + bless $self, $class; + return $self; +} + +sub parseFile { + my $self = shift; + my $logger = $self->{"logger"}; + my $file = shift; + + my %connProfiles = (); + if (! -r $file) { + $logger->writeLog("ERROR", "Cannot read file ".$file); + } + my $parser = XML::LibXML->new(); + my $root = $parser->parse_file($file); + foreach my $profile ($root->findnodes('/DataTools.ServerProfiles/profile')) { + my $base = $profile->findnodes('@name'); + + foreach my $property ($profile->findnodes('./baseproperties/property')) { + my $name = $property->findnodes('@name')->to_literal; + my $value = $property->findnodes('@value')->to_literal; + if ($name eq 'odaURL') { + if ($value =~ /jdbc\:[a-z]+\:\/\/([^:]*)(\:\d+)?\/(.*)/) { + $connProfiles{$base."_host"} = $1; + if(defined($2) && $2 ne ''){ + $connProfiles{$base."_port"} = $2; + $connProfiles{$base."_port"} =~ s/\://; + }else{ + $connProfiles{$base."_port"} = '3306'; + } + $connProfiles{$base."_db"} = $3; + $connProfiles{$base."_db"} =~ s/\?autoReconnect\=true//; + } + } + if ($name eq 'odaUser') { + $connProfiles{$base."_user"} = sprintf('%s',$value); + } + if ($name eq 'odaPassword') { + $connProfiles{$base."_pass"} = sprintf('%s', $value); + } + } + } + + return (\%connProfiles); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/DataQuality.pm b/gorgone/modules/centreon/mbi/libs/bi/DataQuality.pm new file mode 100644 index 0000000..746ec74 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/DataQuality.pm @@ -0,0 +1,99 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::DataQuality; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database + +sub new { + my $class = shift; + my $self = {}; + + $self->{logger} = shift; + $self->{centreon} = shift; + bless $self, $class; + return $self; +} + +sub searchAndDeleteDuplicateEntries { + my $self = shift; + + $self->{logger}->writeLog("INFO", "Searching for duplicate host/service entries"); + my $relationIDS = $self->getDuplicateRelations(); + if (@$relationIDS) { + $self->deleteDuplicateEntries($relationIDS); + } +} + +# return table of IDs to delete +sub getDuplicateRelations { + my $self = shift; + + my @relationIDS; + #Get duplicated relations and exclude BAM or Metaservices data + my $duplicateEntriesQuery = "SELECT host_host_id, service_service_id, count(*) as nbRelations ". + "FROM host_service_relation t1, host t2 WHERE t1.host_host_id = t2.host_id ". + "AND t2.host_name not like '_Module%' group by host_host_id, service_service_id HAVING COUNT(*) > 1"; + + my $sth = $self->{centreon}->query($duplicateEntriesQuery); + while (my $row = $sth->fetchrow_hashref()) { + if (defined($row->{host_host_id})) { + $self->{logger}->writeLog( + "WARNING", + "Found the following duplicate data (host-service) : " . $row->{host_host_id}." - ".$row->{service_service_id}." - Cleaning data" + ); + #Get all relation IDs related to duplicated data + my $relationIdQuery = "SELECT hsr_id from host_service_relation ". + "WHERE host_host_id = ".$row->{host_host_id}." AND service_service_id = ".$row->{service_service_id}; + my $sth2 = $self->{centreon}->query($relationIdQuery); + while (my $hsr = $sth2->fetchrow_hashref()) { + if (defined($hsr->{hsr_id})) { + push(@relationIDS,$hsr->{hsr_id}); + } + } + $self->deleteDuplicateEntries(\@relationIDS); + @relationIDS = (); + } + } + return (\@relationIDS); +} + +# Delete N-1 duplicate entry +sub deleteDuplicateEntries { + my $self = shift; + + my @relationIDS = @{$_[0]}; + #WARNING : very important so at least 1 relation is kept + pop @relationIDS; + foreach (@relationIDS) { + my $idToDelete = $_; + my $deleteQuery = "DELETE FROM host_service_relation WHERE hsr_id = ".$idToDelete; + $self->{centreon}->query($deleteQuery) + } +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/Dumper.pm b/gorgone/modules/centreon/mbi/libs/bi/Dumper.pm new file mode 100644 index 0000000..6f4d114 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/Dumper.pm @@ -0,0 +1,132 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::Dumper; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + $self->{'tempFolder'} = "/tmp/"; + bless $self, $class; + return $self; +} + +sub setStorageDir { + my $self = shift; + my $logger = $self->{'logger'}; + my $tempFolder = shift; + + if (!defined($tempFolder)) { + $logger->writeLog("ERROR", "Temporary storage folder is not defined"); + } + if (! -d $tempFolder && ! -w $tempFolder) { + $logger->writeLog("ERROR", "Cannot write into directory ".$tempFolder); + } + if ($tempFolder !~ /\/$/) { + $tempFolder .= "/"; + } + $self->{'tempFolder'} = $tempFolder; +} + +# Dump data in a MySQL table. (db connection,table name, [not mandatory] start column, end column,start date,end date,exclude end time?) +# and return the file name created +# Ex $file = $dumper->dumpData($hostCentreon, 'toto', 'data_start', 'date_end', '2015-01-02', '2015-02-01', 0); +sub dumpData { + my $self = shift; + my $db = $self->{"centstorage"}; + my ($hostCentreon, $tableName) = (shift, shift); + my ($day,$month,$year,$hour,$min) = (localtime(time))[3,4,5,2,1]; + my $fileName = $self->{'tempFolder'}.$tableName; + my $query = "SELECT * FROM ".$tableName." "; + my $logger = $self->{'logger'}; + if (@_) { + my ($startColumn, $endColumn, $startTime, $endTime, $excludeEndTime) = @_; + $query .= " WHERE ".$startColumn." >= UNIX_TIMESTAMP('".$startTime."') "; + if ($excludeEndTime == 0) { + $query .= "AND ".$endColumn." <= UNIX_TIMESTAMP('".$endTime."')"; + }else { + $query .= "AND ".$endColumn." < UNIX_TIMESTAMP('".$endTime."')"; + } + } + my @loadCmdArgs = ('mysql', "-q", "-u", $hostCentreon->{'Censtorage_user'}, "-p".$hostCentreon->{'Censtorage_pass'}, + "-h", $hostCentreon->{'Censtorage_host'}, $hostCentreon->{'Censtorage_db'}, + "-e", $query.">".$fileName); + system("mysql -q -u".$hostCentreon->{'Censtorage_user'}." -p".$hostCentreon->{'Censtorage_pass'}." -P".$hostCentreon->{'Censtorage_port'}." -h".$hostCentreon->{'Censtorage_host'}. + " ".$hostCentreon->{'Censtorage_db'}." -e \"".$query."\" > ".$fileName); + $logger->writeLog("DEBUG","mysql -q -u".$hostCentreon->{'Censtorage_user'}." -p".$hostCentreon->{'Censtorage_pass'}." -P".$hostCentreon->{'Censtorage_port'}." -h".$hostCentreon->{'Censtorage_host'}. + " ".$hostCentreon->{'Censtorage_db'}." -e \"".$query."\" > ".$fileName); + return ($fileName); +} + +sub dumpRequest{ + my $self = shift; + my $db = $self->{"centstorage"}; + my ($hostCentreon, $requestName,$query) = (shift, shift,shift); + my $fileName = $self->{'tempFolder'}.$requestName; + my $logger = $self->{'logger'}; + system("mysql -q -u".$hostCentreon->{'Censtorage_user'}." -p".$hostCentreon->{'Censtorage_pass'}." -h".$hostCentreon->{'Censtorage_host'}. " -P".$hostCentreon->{'Censtorage_port'}. + " ".$hostCentreon->{'Censtorage_db'}." -e \"".$query."\" > ".$fileName); + return ($fileName); +} + +sub dumpTableStructure { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{'logger'}; + my ($tableName) = (shift); + + my $sql = ""; + my $sth = $db->query("SHOW CREATE TABLE ".$tableName); + if (my $row = $sth->fetchrow_hashref()) { + $sql = $row->{'Create Table'}; + $sql =~ s/(CONSTRAINT.*\n)//g; + $sql =~ s/(\,\n\s+\))/\)/g; + }else { + $logger->writeLog("WARNING", "Cannot get structure for table : ".$tableName); + return (undef); + } + $sth->finish; + return ($sql); +} + +sub insertData { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my ($tableName, $inFile) = (shift, shift); + my $query = "LOAD DATA INFILE '".$inFile."' INTO TABLE `".$tableName."`"; + my $sth = $db->query($query); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/HGMonthAvailability.pm b/gorgone/modules/centreon/mbi/libs/bi/HGMonthAvailability.pm new file mode 100644 index 0000000..69157de --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/HGMonthAvailability.pm @@ -0,0 +1,92 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::HGMonthAvailability; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + $self->{'name'} = "mod_bi_hgmonthavailability"; + $self->{'timeColumn'} = "time_id"; + bless $self, $class; + return $self; +} + +sub getName() { + my $self = shift; + return $self->{'name'}; +} + +sub getTimeColumn() { + my $self = shift; + return $self->{'timeColumn'}; +} + + +sub insertStats { + my $self = shift; + my ($time_id, $data) = @_; + my $insertParam = 1000; + + my $query_start = "INSERT INTO `".$self->{'name'}."`". + " (`time_id`, `modbihg_id`, `modbihc_id`, `liveservice_id`, `available`, `unavailable_time`,". + " `alert_unavailable_opened`, `alert_unavailable_closed`, ". + " `alert_unreachable_opened`, `alert_unreachable_closed`,". + " `alert_unavailable_total`, `alert_unreachable_total`,". + " `mtrs`, `mtbf`, `mtbsi`)". + " VALUES "; + my $counter = 0; + my $query = $query_start; + my $append = ''; + + foreach my $entry (@$data) { + my $size = scalar(@$entry); + $query .= $append . "($time_id"; + for (my $i = 0; $i < $size; $i++) { + $query .= ', ' . (defined($entry->[$i]) ? $entry->[$i] : 'NULL'); + } + $query .= ')'; + + $append = ','; + $counter++; + if ($counter >= $insertParam) { + $self->{centstorage}->query($query); + $query = $query_start; + $counter = 0; + $append = ''; + } + } + $self->{centstorage}->query($query) if ($counter > 0); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/HGServiceMonthAvailability.pm b/gorgone/modules/centreon/mbi/libs/bi/HGServiceMonthAvailability.pm new file mode 100644 index 0000000..b8d23c3 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/HGServiceMonthAvailability.pm @@ -0,0 +1,93 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::HGServiceMonthAvailability; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + $self->{'name'} = "mod_bi_hgservicemonthavailability"; + $self->{'timeColumn'} = "time_id"; + bless $self, $class; + return $self; +} + +sub getName() { + my $self = shift; + return $self->{'name'}; +} + +sub getTimeColumn() { + my $self = shift; + return $self->{'timeColumn'}; +} + +sub insertStats { + my $self = shift; + my ($time_id, $data) = @_; + my $insertParam = 1000; + + my $query_start = "INSERT INTO `".$self->{'name'}."`". + " (`time_id`, `modbihg_id`, `modbihc_id`, `modbisc_id`, `liveservice_id`, `available`,". + " `unavailable_time`, `degraded_time`, `alert_unavailable_opened`, `alert_unavailable_closed`, ". + " `alert_degraded_opened`, `alert_degraded_closed`, ". + " `alert_other_opened`, `alert_other_closed`, ". + " `alert_degraded_total`, `alert_unavailable_total`,". + " `alert_other_total`, `mtrs`, `mtbf`, `mtbsi`)". + " VALUES "; + my $counter = 0; + my $query = $query_start; + my $append = ''; + + foreach my $entry (@$data) { + my $size = scalar(@$entry); + + $query .= $append . "($time_id"; + for (my $i = 0; $i < $size; $i++) { + $query .= ', ' . (defined($entry->[$i]) ? $entry->[$i] : 'NULL'); + } + $query .= ')'; + + $append = ','; + $counter++; + if ($counter >= $insertParam) { + $self->{centstorage}->query($query); + $query = $query_start; + $counter = 0; + $append = ''; + } + } + $self->{centstorage}->query($query) if ($counter > 0); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/HostAvailability.pm b/gorgone/modules/centreon/mbi/libs/bi/HostAvailability.pm new file mode 100644 index 0000000..e6a2d7b --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/HostAvailability.pm @@ -0,0 +1,175 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; +use POSIX; +use Time::Local; + +package gorgone::modules::centreon::mbi::libs::bi::HostAvailability; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + $self->{"name"} = "mod_bi_hostavailability"; + $self->{"timeColumn"} = "time_id"; + $self->{"nbLinesInFile"} = 0; + $self->{"commitParam"} = 500000; + bless $self, $class; + return $self; +} + +sub getName() { + my $self = shift; + return $self->{'name'}; +} + +sub getTimeColumn() { + my $self = shift; + return $self->{'timeColumn'}; +} + +#Only for daily mode +sub insertStats { + my $self = shift; + my ($data, $time_id, $liveserviceId) = @_; + my $insertParam = 10000; + + my $query_start = "INSERT INTO `" . $self->{name} . "`". + " (`modbihost_id`, `time_id`, `liveservice_id`, `available`, ". + " `unavailable`,`unreachable`, `alert_unavailable_opened`, `alert_unavailable_closed`, ". + " `alert_unreachable_opened`, `alert_unreachable_closed`) ". + " VALUES "; + my $counter = 0; + my $query = $query_start; + my $append = ''; + + while (my ($modBiHostId, $stats) = each %$data) { + my @tab = @$stats; + if ($stats->[0] + $stats->[1] + $stats->[2] == 0) { + next; + } + + $query .= $append . "($modBiHostId, $time_id, $liveserviceId"; + for (my $i = 0; $i < scalar(@$stats); $i++) { + $query .= ', ' . $stats->[$i]; + } + $query .= ')'; + + $append = ','; + $counter++; + if ($counter >= $insertParam) { + $self->{centstorage}->query($query); + $query = $query_start; + $counter = 0; + $append = ''; + } + } + $self->{centstorage}->query($query) if ($counter > 0); +} + +sub saveStatsInFile { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($data, $time_id, $liveserviceId,$fh) = @_; + my $query; + my $row; + + while (my ($modBiHostId, $stats) = each %$data) { + my @tab = @$stats; + if ($stats->[0]+$stats->[1]+$stats->[4] == 0) { + next; + } + + #Filling the dump file with data + $row = $modBiHostId."\t".$time_id."\t".$liveserviceId; + for (my $i = 0; $i < scalar(@$stats); $i++) { + $row.= "\t".$stats->[$i] + } + $row .= "\n"; + + #Write row into file + print $fh $row; + $self->{"nbLinesInFile"}+=1; + } +} + +sub getCurrentNbLines{ + my $self = shift; + return $self->{"nbLinesInFile"}; +} + +sub getCommitParam{ + my $self = shift; + return $self->{"commitParam"}; +} +sub setCurrentNbLines{ + my $self = shift; + my $nbLines = shift; + $self->{"nbLinesInFile"} = $nbLines; +} + +sub getHGMonthAvailability { + my ($self, $start, $end, $eventObj) = @_; + my $db = $self->{"centstorage"}; + + $self->{"logger"}->writeLog("DEBUG","[HOST] Calculating availability for hosts"); + my $query = "SELECT h.hg_id, h.hc_id, hc.id as cat_id, hg.id as group_id, ha.liveservice_id, avg(available/(available+unavailable+unreachable)) as av_percent,"; + $query .= " sum(available) as av_time, sum(unavailable) as unav_time, sum(alert_unavailable_opened) as unav_opened, sum(alert_unavailable_closed) as unav_closed,"; + $query .= " sum(alert_unreachable_opened) as unr_opened, sum(alert_unreachable_closed) as unr_closed"; + $query .= " FROM ".$self->{"name"}." ha"; + $query .= " STRAIGHT_JOIN mod_bi_time t ON (t.id = ha.time_id )"; + $query .= " STRAIGHT_JOIN mod_bi_hosts h ON (ha.modbihost_id = h.id)"; + $query .= " STRAIGHT_JOIN mod_bi_hostgroups hg ON (h.hg_name=hg.hg_name AND h.hg_id=hg.hg_id)"; + $query .= " STRAIGHT_JOIN mod_bi_hostcategories hc ON (h.hc_name=hc.hc_name AND h.hc_id=hc.hc_id)"; + $query .= " WHERE t.year = YEAR('".$start."') AND t.month = MONTH('".$start."') and t.hour=0"; + $query .= " GROUP BY h.hg_id, h.hc_id, ha.liveservice_id"; + my $sth = $db->query($query); + + $self->{"logger"}->writeLog("DEBUG","[HOST] Calculating MTBF/MTRS/MTBSI for Host"); + my @data = (); + while (my $row = $sth->fetchrow_hashref()) { + my ($totalDownEvents, $totalUnrEvents) = $eventObj->getNbEvents($start, $end, $row->{'hg_id'}, $row->{'hc_id'}, $row->{'liveservice_id'}); + my ($mtrs, $mtbf, $mtbsi) = (undef, undef, undef); + if (defined($totalDownEvents) && $totalDownEvents != 0) { + $mtrs = $row->{'unav_time'}/$totalDownEvents; + $mtbf = $row->{'av_time'}/$totalDownEvents; + $mtbsi = ($row->{'unav_time'}+$row->{'av_time'})/$totalDownEvents; + } + my @tab = ($row->{'group_id'}, $row->{'cat_id'}, $row->{'liveservice_id'}, $row->{'av_percent'}, $row->{'unav_time'}, + $row->{'unav_opened'}, $row->{'unav_closed'}, $row->{'unr_opened'}, $row->{'unr_closed'}, + $totalDownEvents, $totalUnrEvents, $mtrs, $mtbf, $mtbsi); + push @data, \@tab; + } + + return \@data; +} +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/LiveService.pm b/gorgone/modules/centreon/mbi/libs/bi/LiveService.pm new file mode 100644 index 0000000..18849c1 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/LiveService.pm @@ -0,0 +1,223 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::LiveService; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + + $self->{logger} = shift; + $self->{centstorage} = shift; + if (@_) { + $self->{centreon} = shift; + } + bless $self, $class; + return $self; +} + +sub getLiveServicesByName { + my $self = shift; + my $db = $self->{"centstorage"}; + my $name = shift; + my $interval = shift; + my $query = "SELECT `id`, `name`"; + $query .= " FROM `mod_bi_liveservice`"; + $query .= " WHERE `name` like '".$name."%'"; + my $sth = $db->query($query); + my %result = (); + while (my $row = $sth->fetchrow_hashref()) { + $result{ $row->{name} } = $row->{id}; + } + return (\%result); +} + +sub getLiveServicesByTpId { + my $self = shift; + my $db = $self->{"centstorage"}; + my $name = shift; + my $interval = shift; + my $query = "SELECT `id`, `timeperiod_id`"; + $query .= " FROM `mod_bi_liveservice` "; + my $sth = $db->query($query); + my %result = (); + while (my $row = $sth->fetchrow_hashref()) { + $result{$row->{'timeperiod_id'}} = $row->{"id"}; + } + return (\%result); +} + +sub getLiveServicesByNameForTpId { + my $self = shift; + my $db = $self->{"centstorage"}; + my $tpId = shift; + my $query = "SELECT `id`, `name`"; + $query .= " FROM `mod_bi_liveservice` "; + $query .= "WHERE timeperiod_id = ".$tpId; + my $sth = $db->query($query); + my ($name, $id); + + while (my $row = $sth->fetchrow_hashref()) { + ($name, $id) = ($row->{'name'}, $row->{'id'}); + } + return ($name,$id); +} + +sub getLiveServiceIdsInString { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{'logger'}; + my $ids = shift; + + my $idStr = ""; + + my $query = "SELECT `id`"; + $query .= " FROM mod_bi_liveservice"; + $query .= " WHERE timeperiod_id IN (".$ids.")"; + my $sth = $db->query($query); + my %result = (); + while (my $row = $sth->fetchrow_hashref()) { + $idStr .= $row->{'id'}.","; + } + $idStr =~ s/\,$//; + return $idStr; +} + +sub getLiveServicesByNameForTpIds { + my $self = shift; + my $db = $self->{"centstorage"}; + my $ids = shift; + + my $idStr = ""; + + foreach my $key (keys %$ids) { + if ($idStr eq "") { + $idStr .= $key; + }else { + $idStr .= ",".$key; + } + } + if ($idStr eq "") { + $self->{logger}->writeLog("ERROR", "Select a timeperiod in the ETL configuration menu"); + } + my $query = "SELECT `id`, `name`"; + $query .= " FROM mod_bi_liveservice"; + $query .= " WHERE timeperiod_id IN (".$idStr.")"; + my $sth = $db->query($query); + my %result = (); + while (my $row = $sth->fetchrow_hashref()) { + $result{ $row->{name} } = $row->{id}; + } + return \%result; +} + +sub getTimeperiodName { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $id = shift; + my $query = "SELECT name FROM mod_bi_liveservice WHERE timeperiod_id=".$id; + my $sth = $db->query($query); + my $name = ""; + if (my $row = $sth->fetchrow_hashref()) { + $name = $row->{'name'}; + } + return($name); +} + +sub getTimeperiodId { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $name = shift; + my $query = "SELECT timeperiod_id FROM mod_bi_liveservice WHERE name='".$name."'"; + my $sth = $db->query($query); + my $id = 0; + if (my $row = $sth->fetchrow_hashref()) { + $id = $row->{'timeperiod_id'}; + } + return($id); +} + +sub insert { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $name = shift; + my $id = shift; + my $query = "INSERT INTO `mod_bi_liveservice` (`name`, `timeperiod_id`) VALUES ('".$name."', ".$id.")"; + my $sth = $db->query($query); +} + +sub insertList { + my $self = shift; + my $db = $self->{"centstorage"}; + my $list = shift; + + while (my ($id, $name) = each %$list) { + my $tpName = $self->getTimeperiodName($id); + my $tpId = $self->getTimeperiodId($name); + if ($tpName ne "" && $name ne $tpName) { + $self->updateById($id, $name); + }elsif ($tpId > 0 && $tpId != $id) { + $self->update($name, $id); + }elsif ($tpId == 0 && $tpName eq "") { + $self->insert($name, $id); + } + } +} + +sub update { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $name = shift; + my $id = shift; + my $query = "UPDATE `mod_bi_liveservice` SET `timeperiod_id`=".$id." WHERE name='".$name."'"; + $db->query($query); +} + +sub updateById { + my $self = shift; + my $db = $self->{"centstorage"}; + + my ($id, $name) = (shift, shift); + my $query = "UPDATE `mod_bi_liveservice` SET `name`='".$name."' WHERE timeperiod_id=".$id; + $db->query($query); +} + +sub truncateTable { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "TRUNCATE TABLE `mod_bi_liveservice`"; + $db->query($query); + $db->query("ALTER TABLE `mod_bi_liveservice` AUTO_INCREMENT=1"); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/Loader.pm b/gorgone/modules/centreon/mbi/libs/bi/Loader.pm new file mode 100644 index 0000000..9cfe6d3 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/Loader.pm @@ -0,0 +1,121 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; +use POSIX; + +package gorgone::modules::centreon::mbi::libs::bi::Loader; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + $self->{'tempFolder'} = "/tmp/"; + bless $self, $class; + return $self; +} + +sub setStorageDir { + my $self = shift; + my $logger = $self->{'logger'}; + my $tempFolder = shift; + if (!defined($tempFolder)) { + $logger->writeLog("ERROR", "Temporary storage folder is not defined"); + } + if (! -d $tempFolder && ! -w $tempFolder) { + $logger->writeLog("ERROR", "Cannot write into directory ".$tempFolder); + } + if ($tempFolder !~ /\/$/) { + $tempFolder .= "/"; + } + $self->{'tempFolder'} = $tempFolder; +} +sub getStorageDir { + my $self = shift; + return $self->{'tempFolder'}; +} +sub loadData { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($tableName, $inFile) = (shift, shift); + my $query = "LOAD DATA LOCAL INFILE '".$inFile."' INTO TABLE `".$tableName."` CHARACTER SET UTF8 IGNORE 1 LINES"; + my $sth = $db->query($query); +} +sub disableKeys { + my $self = shift; + my $db = $self->{"centstorage"}; + my $tableName = shift; + my $query = "ALTER TABLE `".$tableName."` DISABLE KEYS"; + my $sth = $db->query($query); +} + +sub enableKeys { + my $self = shift; + my $db = $self->{"centstorage"}; + my $tableName = shift; + my $query = "ALTER TABLE `".$tableName."` ENABLE KEYS"; + my $sth = $db->query($query); +} + +sub dumpTableStructure { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{'logger'}; + my ($tableName) = (shift); + + my $sql = ""; + my $sth = $db->query("SHOW CREATE TABLE ".$tableName); + if (my $row = $sth->fetchrow_hashref()) { + $sql = $row->{'Create Table'}; + }else { + $logger->writeLog("WARNING", "Cannot get structure for table : ".$tableName); + return (undef); + } + $sth->finish; + return ($sql); +} + +sub truncateTable { + my $self = shift; + my $db = $self->{"centstorage"}; + my $tableName = shift; + my $query = "TRUNCATE TABLE `".$tableName."`"; + my $sth = $db->query($query); +} +sub dropTable { + my $self = shift; + my $db = $self->{"centstorage"}; + my $tableName = shift; + my $query = "DROP TABLE IF EXISTS `".$tableName."`"; + my $sth = $db->query($query); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/MetricCentileValue.pm b/gorgone/modules/centreon/mbi/libs/bi/MetricCentileValue.pm new file mode 100644 index 0000000..79db771 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/MetricCentileValue.pm @@ -0,0 +1,182 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::MetricCentileValue; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centstorage: Instance of centreonDB class for connection to Centreon database +# $centreon: Instance of centreonDB class for connection to Centstorage database +sub new { + my ($class, %options) = (shift, @_); + my $self = {}; + $self->{logger} = $options{logger}; + $self->{centstorage} = $options{centstorage}; + $self->{centreon} = $options{centreon}; + $self->{time} = $options{time}; + $self->{centileProperties} = $options{centileProperties}; + $self->{timePeriod} = $options{timePeriod}; + $self->{liveService} = $options{liveService}; + + $self->{today_servicemetrics} = "mod_bi_tmp_today_servicemetrics"; #BIMetric -> createTodayTable + + #Daily values + $self->{name} = "mod_bi_metriccentiledailyvalue"; + + #Week values + $self->{name_week} = "mod_bi_metriccentileweeklyvalue"; + + #Month values + $self->{name_month} = "mod_bi_metriccentilemonthlyvalue"; + + $self->{timeColumn} = "time_id"; + bless $self, $class; + return $self; +} + +#getName($granularity) : "month","week" +sub getName { + my $self = shift; + my $granularity = shift; + my $name = $self->{name}; + + if (defined($granularity) && ($granularity eq "month" || $granularity eq "week")) { + my $key = 'name_' . $granularity; + $name = $self->{$key}; + } + return $name; +} + +sub getTmpName { + my ($self, $granularity) = @_; + my $name = $self->{tmp_name}; + if (defined $granularity && ($granularity eq "month" || $granularity eq "week")) { + my $key = 'tmp_name_' . $granularity; + $name = $self->{$key}; + } + + return $name; +} + +sub getTimeColumn { + my $self = shift; + + return $self->{timeColumn}; +} + +sub getMetricsCentile { + my ($self, %options) = @_; + + my $results = {}; + my $centileServiceCategories = $options{etlProperties}->{'centile.include.servicecategories'}; + my $query = 'SELECT id, metric_id FROM ' . $self->{today_servicemetrics} . ' sm ' . + ' WHERE sm.sc_id IN (' . $centileServiceCategories . ')'; + my $sth = $self->{centstorage}->query($query); + while (my $row = $sth->fetchrow_arrayref()) { + $results->{$$row[1]} = [] if (!defined($results->{$$row[1]})); + push @{$results->{$$row[1]}}, $$row[0]; + } + + return $results; +} + +sub getTimePeriodQuery { + my ($self, %options) = @_; + + my $subQuery = ''; + # Get the time period to apply to each days of the period given in parameter + my $totalDays = $self->{time}->getTotalDaysInPeriod($options{start}, $options{end}) + 1; # +1 because geTotalDaysInPeriod return the number of day between start 00:00 and end 00:00 + my $counter = 1; + my $currentStart = $options{start}; + my $append = ''; + while ($counter <= $totalDays) { + my $rangeDay = $self->{timePeriod}->getTimeRangesForDayByDateTime($options{liveServiceName}, $currentStart, $self->{time}->getDayOfWeek($currentStart)); + if (scalar($rangeDay)) { + my @tabPeriod = @$rangeDay; + my ($start_date, $end_date); + my $tabSize = scalar(@tabPeriod); + for (my $count = 0; $count < $tabSize; $count++) { + my $range = $tabPeriod[$count]; + if ($count == 0) { + $start_date = $range->[0]; + } + if ($count == $tabSize - 1) { + $end_date = $range->[1]; + } + $subQuery .= $append . "(ctime >= UNIX_TIMESTAMP(" . ($range->[0]) . ") AND ctime < UNIX_TIMESTAMP(" . ($range->[1]) . "))"; + $append = ' OR '; + } + } + $currentStart = $self->{time}->addDateInterval($currentStart, 1, "DAY"); + $counter++; + } + + return $subQuery; +} + +sub calcMetricsCentileValueMultipleDays { + my ($self, %options) = @_; + + my $centileParam = $self->{centileProperties}->getCentileParams(); + foreach (@$centileParam) { + my ($centile, $timeperiodId) = ($_->{centile_param}, $_->{timeperiod_id}); + my ($liveServiceName, $liveServiceId) = $self->{liveService}->getLiveServicesByNameForTpId($timeperiodId); + + #Get Id for the couple centile / timeperiod + my $centileId; + my $query = "SELECT id FROM mod_bi_centiles WHERE centile_param = " . $centile . " AND liveservice_id = (SELECT id FROM mod_bi_liveservice WHERE timeperiod_id = " . $timeperiodId . ")"; + my $sth = $self->{centstorage}->query($query); + while (my $row = $sth->fetchrow_hashref()) { + if (defined($row->{id})) { + $centileId = $row->{id}; + } + } + + next if (!defined($centileId)); + + my $total = scalar(keys %{$options{metricsId}}); + $self->{logger}->writeLog("INFO", "Processing " . $options{granularity} . " for Centile: [" . $options{start} . "] to [" . $options{end} . "] - " . $liveServiceName . " - " . $centile . ' (' . $total . ' metrics)'); + my $sub_query_timeperiod = $self->getTimePeriodQuery(start => $options{start}, end => $options{end}, liveServiceName => $liveServiceName); + $query = 'SELECT value FROM (SELECT value, @counter := @counter + 1 AS counter FROM (select @counter := 0) AS initvar, data_bin WHERE id_metric = ? AND (' . $sub_query_timeperiod . ') ORDER BY value ASC) AS X where counter = ceil(' . $centile . ' * @counter / 100)'; + my $sth_centile = $self->{centstorage}->prepare($query); + my $current = 1; + foreach my $metricId (keys %{$options{metricsId}}) { + $self->{logger}->writeLog("DEBUG", "Processing metric id for Centile: " . $metricId . " ($current/$total)"); + $sth_centile->execute($metricId); + my $row = $sth_centile->fetchrow_arrayref(); + $current++; + next if (!defined($row)); + + foreach (@{$options{metricsId}->{$metricId}}) { + my $query_insert = 'INSERT INTO ' . $self->getName($options{granularity}) . + '(servicemetric_id, time_id, liveservice_id, centile_value, centile_param, centile_id, total, warning_treshold, critical_treshold)' . + "SELECT '" . $_ . "', '" . $options{timeId} . "', '" . $liveServiceId . "', '" . $$row[0] . "', '" . $centile . "', '" . $centileId . "', " . + 'm.max, m.warn, m.crit FROM metrics m WHERE m.metric_id = ' . $metricId; + $self->{centstorage}->query($query_insert); + } + } + } +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/MetricDailyValue.pm b/gorgone/modules/centreon/mbi/libs/bi/MetricDailyValue.pm new file mode 100644 index 0000000..c667005 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/MetricDailyValue.pm @@ -0,0 +1,146 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::MetricDailyValue; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{logger} = shift; + $self->{centstorage} = shift; + + $self->{name_minmaxavg_tmp} = 'mod_bi_tmp_minmaxavgvalue'; + $self->{name_firstlast_tmp} = 'mod_bi_tmp_firstlastvalues'; + if (@_) { + $self->{name_minmaxavg_tmp} .= $_[0]; + $self->{name_firstlast_tmp} .= $_[0]; + } + + $self->{today_servicemetrics} = "mod_bi_tmp_today_servicemetrics"; + $self->{name} = "mod_bi_metricdailyvalue"; + $self->{timeColumn} = "time_id"; + bless $self, $class; + return $self; +} + +sub getName() { + my $self = shift; + return $self->{'name'}; +} + +sub getTimeColumn() { + my $self = shift; + return $self->{'timeColumn'}; +} + +sub dropTempTables { + my $self = shift; + my $db = $self->{"centstorage"}; + my $query = "DROP TABLE `" . $self->{name_minmaxavg_tmp} . "`"; + $db->query($query); + $query = "DROP TABLE `" . $self->{name_firstlast_tmp} . "`"; + $db->query($query); +} + +sub insertValues { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my $liveServiceId = shift; + my $timeId = shift; + + my $query = "INSERT INTO " . $self->{"name"}; + $query .= " SELECT sm.id as servicemetric_id, '".$timeId."', ".$liveServiceId." as liveservice_id,"; + $query .= " mmavt.avg_value, mmavt.min_value, mmavt.max_value, flvt.`first_value`, flvt.`last_value`, m.max,"; + $query .= " m.warn, m.crit"; + $query .= " FROM " . $self->{name_minmaxavg_tmp} . " mmavt"; + $query .= " JOIN (metrics m, " . $self->{'today_servicemetrics'} . " sm)"; + $query .= " ON (mmavt.id_metric = m.metric_id and mmavt.id_metric = sm.metric_id)"; + $query .= " LEFT JOIN " . $self->{name_firstlast_tmp} . " flvt ON (mmavt.id_metric = flvt.id_metric)"; + $db->query($query); + + $self->dropTempTables(); +} + +sub getMetricCapacityValuesOnPeriod { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($start_time_id, $end_time_id, $etlProperties) = @_; + + my $query = " SELECT servicemetric_id, liveservice_id, "; + $query .= " `first_value`, total"; + $query .= " FROM mod_bi_liveservice l, mod_bi_servicemetrics m, ".$self->{"name"}." v"; + $query .= " WHERE timeperiod_id IN (".$etlProperties->{'capacity.include.liveservices'}.")"; + $query .= " AND l.id = v.liveservice_id"; + $query .= " AND time_id = ".$start_time_id; + if (defined($etlProperties->{'capacity.exclude.metrics'}) && $etlProperties->{'capacity.exclude.metrics'} ne "") { + $query .= " AND metric_name NOT IN (".$etlProperties->{'capacity.exclude.metrics'}.")"; + } + $query .= " AND sc_id IN (".$etlProperties->{'capacity.include.servicecategories'}.")"; + $query .= " AND v.servicemetric_id = m.id"; + $query .= " GROUP BY servicemetric_id, liveservice_id"; + my $sth = $db->query($query); + my %data = (); + while (my $row = $sth->fetchrow_hashref()) { + my @table = ($row->{"servicemetric_id"}, $row->{"liveservice_id"}, $row->{first_value}, $row->{"total"}); + $data{$row->{"servicemetric_id"}.";".$row->{"liveservice_id"}} = \@table; + } + + $query = " SELECT servicemetric_id, liveservice_id, "; + $query .= "`last_value`, total"; + $query .= " FROM mod_bi_liveservice l, mod_bi_servicemetrics m, ".$self->{"name"}." v"; + $query .= " WHERE timeperiod_id IN (".$etlProperties->{'capacity.include.liveservices'}.")"; + $query .= " AND l.id = v.liveservice_id"; + $query .= " AND time_id = ".$end_time_id; + if (defined($etlProperties->{'capacity.exclude.metrics'}) && $etlProperties->{'capacity.exclude.metrics'} ne "") { + $query .= " AND metric_name NOT IN (".$etlProperties->{'capacity.exclude.metrics'}.")"; + } + $query .= " AND sc_id IN (".$etlProperties->{'capacity.include.servicecategories'}.")"; + $query .= " AND v.servicemetric_id = m.id"; + $query .= " GROUP BY servicemetric_id, liveservice_id"; + + $sth = $db->query($query); + while (my $row = $sth->fetchrow_hashref()) { + my $entry = $data{$row->{servicemetric_id} . ';' . $row->{liveservice_id}}; + if (defined($entry)) { + $entry->[4] = $row->{last_value}; + $entry->[5] = $row->{total}; + } else { + my @table; + $table[0] = $row->{servicemetric_id}; + $table[1] = $row->{liveservice_id}; + $table[4] = $row->{last_value}; + $table[5] = $row->{total}; + $data{$row->{servicemetric_id} . ';' . $row->{liveservice_id}} = \@table; + } + } + return \%data; +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/MetricHourlyValue.pm b/gorgone/modules/centreon/mbi/libs/bi/MetricHourlyValue.pm new file mode 100644 index 0000000..037e52c --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/MetricHourlyValue.pm @@ -0,0 +1,72 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::MetricHourlyValue; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{logger} = shift; + $self->{centstorage} = shift; + + $self->{name_minmaxavg_tmp} = 'mod_bi_tmp_minmaxavgvalue'; + if (@_) { + $self->{name_minmaxavg_tmp} .= $_[0]; + } + + $self->{servicemetrics} = "mod_bi_tmp_today_servicemetrics"; + $self->{name} = "mod_bi_metrichourlyvalue"; + $self->{timeColumn} = "time_id"; + bless $self, $class; + return $self; +} + +sub getName() { + my $self = shift; + return $self->{'name'}; +} + +sub getTimeColumn() { + my $self = shift; + return $self->{'timeColumn'}; +} + +sub insertValues { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my $query = "INSERT INTO ".$self->{"name"}; + $query .= " SELECT sm.id as servicemetric_id, t.id as time_id, mmavt.avg_value, mmavt.min_value, mmavt.max_value, m.max , m.warn, m.crit"; + $query .= " FROM " . $self->{name_minmaxavg_tmp} . " mmavt"; + $query .= " JOIN (metrics m, " . $self->{servicemetrics} . " sm, mod_bi_time t)"; + $query .= " ON (mmavt.id_metric = m.metric_id and mmavt.id_metric = sm.metric_id AND mmavt.valueTime = t.dtime)"; + $db->query($query); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/MetricMonthCapacity.pm b/gorgone/modules/centreon/mbi/libs/bi/MetricMonthCapacity.pm new file mode 100644 index 0000000..89b3f20 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/MetricMonthCapacity.pm @@ -0,0 +1,89 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::MetricMonthCapacity; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + $self->{"name"} = "mod_bi_metricmonthcapacity"; + $self->{"timeColumn"} = "time_id"; + bless $self, $class; + return $self; +} + +sub getName() { + my $self = shift; + return $self->{'name'}; +} + +sub getTimeColumn() { + my $self = shift; + return $self->{'timeColumn'}; +} + +sub insertStats { + my $self = shift; + my $db = $self->{centstorage}; + my ($time_id, $data) = @_; + my $insertParam = 5000; + + my $query_start = "INSERT INTO `" . $self->{name} . "`". + "(`time_id`, `servicemetric_id`, `liveservice_id`,". + " `first_value`, `first_total`, `last_value`, `last_total`)". + " VALUES "; + my $counter = 0; + my $query = $query_start; + my $append = ''; + + while (my ($key, $entry) = each %$data) { + $query .= $append . "($time_id"; + + for (my $i = 0; $i <= 5; $i++) { + $query .= ', ' . (defined($entry->[$i]) ? $entry->[$i] : 'NULL'); + } + $query .= ')'; + + $append = ','; + $counter++; + if ($counter >= $insertParam) { + $db->query($query); + $query = $query_start; + $counter = 0; + $append = ''; + } + } + $db->query($query) if ($counter > 0); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/MySQLTables.pm b/gorgone/modules/centreon/mbi/libs/bi/MySQLTables.pm new file mode 100644 index 0000000..a4f94e7 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/MySQLTables.pm @@ -0,0 +1,308 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::libs::bi::MySQLTables; + +use strict; +use warnings; +use POSIX; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + + $self->{logger} = shift; + $self->{centstorage} = shift; + if (@_) { + $self->{centreon} = shift; + } + bless $self, $class; + return $self; +} + +sub tableExists { + my $self = shift; + + my ($name) = (shift); + my $statement = $self->{centstorage}->query("SHOW TABLES LIKE '".$name."'"); + + if (!(my @row = $statement->fetchrow_array())) { + return 0; + } else { + return 1; + } +} + +sub createTable { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($name, $structure, $mode) = @_; + my $statement = $db->query("SHOW TABLES LIKE '".$name."'"); + if (!$self->tableExists($name)) { + if (defined($structure)) { + $logger->writeLog("DEBUG", "[CREATE] table [".$name."]"); + $db->query($structure); + return 0; + }else { + $logger->writeLog("FATAL", "[CREATE] Cannot find table [".$name."] structure"); + } + } + return 1; +} + +sub dumpTableStructure { + my $self = shift; + my ($tableName) = (shift); + + my $sql = ""; + my $sth = $self->{centstorage}->query("SHOW CREATE TABLE " . $tableName); + if (my $row = $sth->fetchrow_hashref()) { + $sql = $row->{'Create Table'}; + $sql =~ s/(CONSTRAINT.*\n)//g; + $sql =~ s/(\,\n\s+\))/\)/g; + }else { + die "Cannot get structure for table : ".$tableName; + } + return ($sql); +} + +# create table data_bin with partitions +sub createParts { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my ($start, $end, $tableStructure, $tableName, $column) = @_; + if (!defined($tableStructure)) { + $logger->writeLog("FATAL", "[CREATE] Cannot find table [".$tableName."] structure"); + } + if ($self->tableExists($tableName)) { + return 1; + } + $tableStructure =~ s/\n.*PARTITION.*//g; + $tableStructure =~ s/\,[\n\s]+\)/\)/; + $tableStructure .= " PARTITION BY RANGE(`".$column."`) ("; + my $timeObj = Time->new($logger,$db); + my $runningStart = $timeObj->addDateInterval($start, 1, "DAY"); + while ($timeObj->compareDates($end, $runningStart) > 0) { + my @partName = split (/\-/, $runningStart); + $tableStructure .= "PARTITION p" . $partName[0] . $partName[1] . $partName[2] . " VALUES LESS THAN (FLOOR(UNIX_TIMESTAMP('".$runningStart."'))),"; + $runningStart= $timeObj->addDateInterval($runningStart, 1, "DAY"); + } + my @partName = split (/\-/, $runningStart); + $tableStructure .= "PARTITION p".$partName[0].$partName[1].$partName[2]." VALUES LESS THAN (FLOOR(UNIX_TIMESTAMP('".$runningStart."'))));"; + $logger->writeLog("DEBUG", "[CREATE] table partitionned [".$tableName."] min value: ".$start.", max value: ".$runningStart.", range: 1 DAY\n"); + $db->query($tableStructure); + return 0; +} + +sub updateParts { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($rangeEnd, $tableName) = @_; + my $timeObj = Time->new($logger,$db); + + my $isPartitioned = $self->isTablePartitioned($tableName); + if (!$isPartitioned) { + $logger->writeLog("WARNING", "[UPDATE PARTS] partitioning is not activated for table [".$tableName."]"); + } else { + my $range = $self->getLastPartRange($tableName); + $range = $timeObj->addDateInterval($range, 1, "DAY"); + while ($timeObj->compareDates($rangeEnd, $range) >= 0) { + $logger->writeLog("DEBUG", "[UPDATE PARTS] Updating partitions for table [".$tableName."] (last range : ".$range.")"); + my @partName = split (/\-/, $range); + my $query = "ALTER TABLE `".$tableName."` ADD PARTITION (PARTITION `p".$partName[0].$partName[1].$partName[2]."` VALUES LESS THAN(FLOOR(UNIX_TIMESTAMP('".$range."'))))"; + $db->query($query); + $range = $timeObj->addDateInterval($range, 1, "DAY"); + } + } +} + +sub isTablePartitioned { + my $self = shift; + my $tableName = shift; + my $db = $self->{"centstorage"}; + + my $sth = $db->query("SHOW TABLE STATUS LIKE '".$tableName."'"); + if (my $row = $sth->fetchrow_hashref()) { + my $createOptions = $row->{"Create_options"}; + if (defined($createOptions) && $createOptions =~ m/partitioned/i) { + return 1; + } elsif (!defined($createOptions) || $createOptions !~ m/partitioned/i) { + return 0; + } + } + die "[TABLE STATUS CHECK] Cannot check if table is partitioned [".$tableName."]"; +} + +sub getLastPartRange { + my $self = shift; + my $tableName = shift; + + my $query = "SHOW CREATE TABLE $tableName"; + + my $partName; + my $sth = $self->{centstorage}->query($query); + if (my $row = $sth->fetchrow_hashref()) { + while ($row->{'Create Table'} =~ /PARTITION.*?p(\d{4})(\d{2})(\d{2}).*?VALUES LESS THAN \([0-9]+?\)/g) { + $partName = "$1-$2-$3"; + } + } + + if (!defined($partName)) { + die "[UPDATE PARTS] Cannot find table [data_bin] in database"; + } + + return $partName; +} + +sub deleteEntriesForRebuild { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($start, $end, $tableName) = @_; + + if (!$self->isTablePartitioned($tableName)) { + $db->query("DELETE FROM ".$tableName." WHERE time_id >= UNIX_TIMESTAMP('".$start."') AND time_id < UNIX_TIMESTAMP('".$end."')"); + }else { + my $query = "SELECT partition_name FROM information_schema.partitions "; + $query .= "WHERE table_name='".$tableName."' AND table_schema='".$db->db."'"; + $query .= " AND CONVERT(PARTITION_DESCRIPTION, SIGNED INTEGER) > UNIX_TIMESTAMP('".$start."')"; + $query .= " AND CONVERT(PARTITION_DESCRIPTION, SIGNED INTEGER) <= UNIX_TIMESTAMP('".$end."')"; + my $sth = $db->query($query); + while(my $row = $sth->fetchrow_hashref()) { + $db->query("ALTER TABLE ".$tableName." TRUNCATE PARTITION ".$row->{'partition_name'}); + } + $self->updateParts($end, $tableName); + } +} + +sub emptyTableForRebuild { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my $tableName = shift; + my $structure = shift; + my $column = shift; + + $structure =~ s/KEY.*\(\`$column\`\)\,//g; + $structure =~ s/KEY.*\(\`$column\`\)//g; + $structure =~ s/\,[\n\s+]+\)/\n\)/g; + if (!defined($_[0]) || !$self->isPartitionEnabled()) { + $db->query("DROP TABLE IF EXISTS ".$tableName); + $db->query($structure); + }else { + my ($start, $end) = @_; + $db->query("DROP TABLE IF EXISTS ".$tableName); + $self->createParts($start, $end, $structure, $tableName, $column); + } + $db->query("ALTER TABLE `".$tableName."` ADD INDEX `idx_".$tableName."_".$column."` (`".$column."`)"); +} + +sub dailyPurge { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my ($retentionDate, $tableName, $column) = @_; + if (!$self->isTablePartitioned($tableName)) { + $db->query("DELETE FROM `".$tableName."` WHERE ".$column." < UNIX_TIMESTAMP('".$retentionDate."')"); + }else { + my $query = "SELECT GROUP_CONCAT(partition_name SEPARATOR ',') as partition_names FROM information_schema.partitions "; + $query .= "WHERE table_name='".$tableName."' AND table_schema='".$db->db."'"; + $query .= " AND CONVERT(PARTITION_DESCRIPTION, SIGNED INTEGER) < UNIX_TIMESTAMP('".$retentionDate."')"; + my $sth = $db->query($query); + if(my $row = $sth->fetchrow_hashref()) { + if (defined($row->{'partition_names'}) && $row->{'partition_names'} ne "") { + $db->query("ALTER TABLE ".$tableName." DROP PARTITION ".$row->{'partition_names'}); + } + } + } +} + +sub checkPartitionContinuity { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($table) = @_; + my $message = ""; + my $query = "select CONVERT(1+datediff(curdate(),(select from_unixtime(PARTITION_DESCRIPTION) from information_schema.partitions"; + $query .= " where table_schema = '".$db->{"db"}."' and table_name = '".$table."' and PARTITION_ORDINAL_POSITION=1)), SIGNED INTEGER) as nbDays,"; + $query .= " CONVERT(PARTITION_ORDINAL_POSITION, SIGNED INTEGER) as ordinalPosition "; + $query .= " from information_schema.partitions where table_schema = '".$db->{"db"}."' and table_name = '".$table."' order by PARTITION_ORDINAL_POSITION desc limit 1 "; + my $sth = $db->query($query); + while (my $row = $sth->fetchrow_hashref()) { + my $nbDays = int($row->{'nbDays'}); + my $ordinalPosition = int($row->{'ordinalPosition'}); + my $dif = int($nbDays - $ordinalPosition); + if($dif > 0){ + $message .= "[".$table.", last partition:".$self->checkLastTablePartition($table)." missing ".$dif." part.]"; + } + } + $sth->finish; + return($message); +} + +sub checkLastTablePartition{ + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($table) = @_; + my $message = ""; + my $query = "select from_unixtime(PARTITION_DESCRIPTION) as last_partition, IF(from_unixtime(PARTITION_DESCRIPTION)=CURDATE() AND HOUR(from_unixtime(PARTITION_DESCRIPTION))=0,1,0) as partition_uptodate "; + $query .="from information_schema.partitions where table_schema = '".$db->{"db"}."'"; + $query .= "and table_name = '".$table."'order by PARTITION_ORDINAL_POSITION desc limit 1"; + my $sth = $db->query($query); + while (my $row = $sth->fetchrow_hashref()) { + if($row->{'partition_uptodate'} == 0){ + $message = $row->{'last_partition'}; + } + } + $sth->finish; + return($message); +} + +sub dropIndexesFromReportingTable { + my $self = shift; + my $table = shift; + my $db = $self->{"centstorage"}; + my $indexes = $db->query("SHOW INDEX FROM ".$table); + my $previous = ""; + while (my $row = $indexes->fetchrow_hashref()) { + + if ($row->{"Key_name"} ne $previous) { + if (lc($row->{"Key_name"}) eq lc("PRIMARY")) { + $db->query("ALTER TABLE `".$table."` DROP PRIMARY KEY"); + }else { + $db->query("ALTER TABLE `".$table."` DROP INDEX ".$row->{"Key_name"}); + } + } + $previous = $row->{"Key_name"}; + } +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/ServiceAvailability.pm b/gorgone/modules/centreon/mbi/libs/bi/ServiceAvailability.pm new file mode 100644 index 0000000..5f4b116 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/ServiceAvailability.pm @@ -0,0 +1,237 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::bi::ServiceAvailability; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + $self->{"name"} = "mod_bi_serviceavailability"; + $self->{"timeColumn"} = "time_id"; + $self->{"nbLinesInFile"} = 0; + $self->{"commitParam"} = 500000; + bless $self, $class; + return $self; +} + +sub getName { + my $self = shift; + return $self->{'name'}; +} + +sub getTimeColumn { + my $self = shift; + return $self->{'timeColumn'}; +} + +sub saveStatsInFile { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($data, $time_id, $liveserviceId,$fh) = @_; + my $query; + my $row; + + while (my ($modBiServiceId, $stats) = each %$data) { + my @tab = @$stats; + if ($stats->[0]+$stats->[1]+$stats->[2] == 0) { + next; + } + + #Filling the dump file with data + $row = $modBiServiceId."\t".$time_id."\t".$liveserviceId; + for (my $i = 0; $i < scalar(@$stats); $i++) { + $row.= "\t".$stats->[$i] + } + $row .= "\n"; + + #Write row into file + print $fh $row; + $self->{"nbLinesInFile"}++; + } +} + +sub insertStats { + my $self = shift; + my ($data, $time_id, $liveserviceId) = @_; + my $insertParam = 10000; + my $query_start = "INSERT INTO `" . $self->{name} . "`". + " (`modbiservice_id`, `time_id`, `liveservice_id`, `available`, ". + " `unavailable`, `degraded`, `alert_unavailable_opened`, `alert_unavailable_closed`, ". + " `alert_degraded_opened`, `alert_degraded_closed`, ". + " `alert_other_opened`, `alert_other_closed`)". + " VALUES "; + + #available+unvailable+alert_unavailable_closed + + my $counter = 0; + my $query = $query_start; + my $append = ''; + while (my ($modBiServiceId, $stats) = each %$data) { + my @tab = @$stats; + if ($stats->[0] + $stats->[1] + $stats->[2] == 0) { + next; + } + + $query .= $append . "($modBiServiceId, $time_id, $liveserviceId"; + for (my $i = 0; $i < scalar(@$stats); $i++) { + $query .= ', ' . $stats->[$i]; + } + $query .= ')'; + $append = ','; + $counter++; + + if ($counter >= $insertParam) { + $self->{centstorage}->query($query); + $query = $query_start; + $counter = 0; + $append = ''; + } + } + + $self->{centstorage}->query($query) if ($counter > 0); +} + +sub getCurrentNbLines { + my $self = shift; + return $self->{"nbLinesInFile"}; +} + +sub getCommitParam { + my $self = shift; + return $self->{"commitParam"}; +} + +sub setCurrentNbLines { + my $self = shift; + my $nbLines = shift; + $self->{"nbLinesInFile"} = $nbLines; +} + +sub getHGMonthAvailability { + my ($self, $start, $end, $eventObj) = @_; + my $db = $self->{"centstorage"}; + + my $query = "SELECT s.hg_id, s.hc_id, s.sc_id, sa.liveservice_id,"; + $query .= " hc.id as hcat_id, hg.id as group_id, sc.id as scat_id,"; + $query .= " avg((available+degraded)/(available+unavailable+degraded)) as av_percent,"; + $query .= " sum(available) as av_time, sum(unavailable) as unav_time, sum(degraded) as degraded_time,"; + $query .= " sum(alert_unavailable_opened) as unav_opened,sum(alert_unavailable_closed) as unav_closed,"; + $query .= " sum(alert_degraded_opened) as deg_opened,sum(alert_degraded_closed) as deg_closed,"; + $query .= " sum(alert_other_opened) as other_opened,sum(alert_other_closed) as other_closed "; + $query .= " FROM ".$self->{'name'}." sa"; + $query .= " STRAIGHT_JOIN mod_bi_time t ON (t.id = sa.time_id )"; + $query .= " STRAIGHT_JOIN mod_bi_services s ON (sa.modbiservice_id = s.id)"; + $query .= " STRAIGHT_JOIN mod_bi_hostgroups hg ON (s.hg_name=hg.hg_name AND s.hg_id=hg.hg_id)"; + $query .= " STRAIGHT_JOIN mod_bi_hostcategories hc ON (s.hc_name=hc.hc_name AND s.hc_id=hc.hc_id)"; + $query .= " STRAIGHT_JOIN mod_bi_servicecategories sc ON (s.sc_id=sc.sc_id AND s.sc_name=sc.sc_name)"; + $query .= " WHERE t.year = YEAR('".$start."') AND t.month = MONTH('".$start."') and t.hour=0"; + $query .= " GROUP BY s.hg_id, s.hc_id, s.sc_id, sa.liveservice_id"; + my $sth = $db->query($query); + + my @data = (); + while (my $row = $sth->fetchrow_hashref()) { + my ($totalwarnEvents, $totalCritEvents, $totalOtherEvents) = $eventObj->getNbEvents($start, $end, $row->{'hg_id'}, $row->{'hc_id'}, $row->{'sc_id'}, $row->{'liveservice_id'}); + + my ($mtrs, $mtbf, $mtbsi) = (undef, undef, undef); + if (defined($totalCritEvents) && $totalCritEvents != 0) { + $mtrs = $row->{'unav_time'}/$totalCritEvents; + $mtbf = $row->{'av_time'}/$totalCritEvents; + $mtbsi = ($row->{'unav_time'}+$row->{'av_time'})/$totalCritEvents; + } + my @tab = ($row->{'group_id'}, $row->{'hcat_id'}, $row->{'scat_id'}, $row->{'liveservice_id'}, + $row->{'av_percent'}, $row->{'unav_time'}, $row->{'degraded_time'}, + $row->{'unav_opened'}, $row->{'unav_closed'}, $row->{'deg_opened'}, $row->{'deg_closed'}, $row->{'other_opened'}, $row->{'other_closed'}, + $totalwarnEvents, $totalCritEvents, $totalOtherEvents, $mtrs, $mtbf, $mtbsi); + push @data, \@tab; + } + return \@data; +} + +sub getHGMonthAvailability_optimised { + my ($self, $start, $end, $eventObj) = @_; + my $db = $self->{"centstorage"}; + + my $query = "SELECT * from ( SELECT s.hg_id, s.hc_id, s.sc_id, sa.liveservice_id, hc.id as hcat_id, hg.id as group_id, sc.id as scat_id,"; + $query .= "avg((available+degraded)/(available+unavailable+degraded)) as av_percent, "; + $query .= "sum(available) as av_time, sum(unavailable) as unav_time, sum(degraded) as degraded_time, "; + $query .= "sum(alert_unavailable_opened) as unav_opened,sum(alert_unavailable_closed) as unav_closed, "; + $query .= "sum(alert_degraded_opened) as deg_opened,sum(alert_degraded_closed) as deg_closed, "; + $query .= "sum(alert_other_opened) as other_opened,sum(alert_other_closed) as other_closed "; + $query .= "FROM mod_bi_serviceavailability sa "; + $query .= "STRAIGHT_JOIN mod_bi_services s ON (sa.modbiservice_id = s.id) "; + $query .= "STRAIGHT_JOIN mod_bi_hostgroups hg ON (s.hg_name=hg.hg_name AND s.hg_id=hg.hg_id) "; + $query .= "STRAIGHT_JOIN mod_bi_hostcategories hc ON (s.hc_name=hc.hc_name AND s.hc_id=hc.hc_id) "; + $query .= "STRAIGHT_JOIN mod_bi_servicecategories sc ON (s.sc_id=sc.sc_id AND s.sc_name=sc.sc_name)"; + $query .= " WHERE YEAR(from_unixtime(time_id)) = YEAR('".$start."') AND MONTH(from_unixtime(time_id)) = MONTH('".$start."') and hour(from_unixtime(time_id)) = 0 "; + $query .= "GROUP BY s.hg_id, s.hc_id, s.sc_id, sa.liveservice_id ) availability "; + $query .= "LEFT JOIN ( SELECT s.hg_id,s.hc_id,s.sc_id,e.modbiliveservice_id, "; + $query .= "SUM(IF(state=1,1,0)) as warningEvents, SUM(IF(state=2,1,0)) as criticalEvents, "; + $query .= "SUM(IF(state=3,1,0)) as unknownEvents FROM mod_bi_servicestateevents e "; + $query .= "STRAIGHT_JOIN mod_bi_services s ON (e.modbiservice_id = s.id) "; + $query .= "STRAIGHT_JOIN mod_bi_hostgroups hg ON (s.hg_name=hg.hg_name AND s.hg_id=hg.hg_id) "; + $query .= "STRAIGHT_JOIN mod_bi_hostcategories hc ON (s.hc_name=hc.hc_name AND s.hc_id=hc.hc_id) "; + $query .= "STRAIGHT_JOIN mod_bi_servicecategories sc ON (s.sc_id=sc.sc_id AND s.sc_name=sc.sc_name) "; + $query .= "AND s.id = e.modbiservice_id AND start_time < UNIX_TIMESTAMP('".$end."') "; + $query .= "AND end_time > UNIX_TIMESTAMP('".$start."') AND e.state in (1,2,3) "; + $query .= "GROUP BY s.hg_id, s.hc_id, s.sc_id, e.modbiliveservice_id ) events "; + $query .= "ON availability.hg_id = events.hg_id AND availability.hc_id = events.hc_id "; + $query .= "AND availability.sc_id = events.sc_id "; + $query .= "AND availability.liveservice_id = events.modbiliveservice_id"; + + #Fields returned : + #hg_id | hc_id | sc_id | liveservice_id | hcat_id | group_id | scat_id | av_percent | av_time | unav_time | degraded_time | + #unav_opened | unav_closed | deg_opened | deg_closed | other_opened | other_closed | hg_id | hc_id | sc_id | + #modbiliveservice_id | warningEvents | criticalEvents | unknownEvents + my $sth = $db->query($query); + + my @data = (); + while (my $row = $sth->fetchrow_hashref()) { + my ($totalwarnEvents, $totalCritEvents, $totalUnknownEvents) = ($row->{'warningEvents'},$row->{'criticalEvents'},$row->{'unknownEvents'}); + + my ($mtrs, $mtbf, $mtbsi) = (undef, undef, undef); + if (defined($totalCritEvents) && $totalCritEvents != 0) { + $mtrs = $row->{'unav_time'}/$totalCritEvents; + $mtbf = $row->{'av_time'}/$totalCritEvents; + $mtbsi = ($row->{'unav_time'}+$row->{'av_time'})/$totalCritEvents; + } + my @tab = ($row->{'group_id'}, $row->{'hcat_id'}, $row->{'scat_id'}, $row->{'liveservice_id'}, + $row->{'av_percent'}, $row->{'unav_time'}, $row->{'degraded_time'}, + $row->{'unav_opened'}, $row->{'unav_closed'}, $row->{'deg_opened'}, $row->{'deg_closed'}, $row->{'other_opened'}, $row->{'other_closed'}, + $totalwarnEvents, $totalCritEvents, $totalUnknownEvents, $mtrs, $mtbf, $mtbsi); + push @data, \@tab; + } + return \@data; +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/bi/Time.pm b/gorgone/modules/centreon/mbi/libs/bi/Time.pm new file mode 100644 index 0000000..c731326 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/bi/Time.pm @@ -0,0 +1,264 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::libs::bi::Time; + +use strict; +use warnings; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{logger} = shift; + $self->{centstorage} = shift; + if (@_) { + $self->{centreon} = shift; + } + $self->{insertQuery} = "INSERT IGNORE INTO `mod_bi_time` (id, hour, day, month_label, month, year, week, dayofweek, utime, dtime) VALUES "; + bless $self, $class; + return $self; +} + +sub getEntriesDtime { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my ($start, $end) = @_; + my $query = "SELECT date_format('%Y-%m-%d', dtime) as dtime"; + $query .= " FROM `mod_bi_time`"; + $query .= " WHERE dtime >= '".$start."' AND dtime <'".$end."'"; + + my $sth = $db->query($query); + my @results = (); + if (my $row = $sth->fetchrow_hashref()) { + push @results, $row->{dtime}; + } + $sth->finish(); + return (@results); +} + +sub getEntryID { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my $dtime = shift; + my ($interval, $type); + if (@_) { + $interval = shift; + $type = shift; + } + my $query = "SELECT `id`, `utime`, date_format(dtime,'%Y-%m-%d') as dtime"; + $query .= " FROM `mod_bi_time`"; + if (!defined($interval)) { + $query .= " WHERE dtime = '".$dtime."'"; + }else { + $query .= " WHERE dtime = DATE_ADD('".$dtime."', INTERVAL ".$interval." ".$type.")"; + } + my $sth = $db->query($query); + my @results = (); + if (my $row = $sth->fetchrow_hashref()) { + $results[0] = $row->{'id'}; + $results[1] = $row->{'utime'}; + } + $sth->finish(); + if (!scalar(@results)) { + $logger->writeLog("ERROR", "Cannot get time ID for date:".$dtime); + } + return (@results); +} + +sub getDayOfWeek { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my $date = shift; + + my $sth = $db->query("SELECT LOWER(DAYNAME('".$date."')) as dayOfWeek"); + my $dayofweek; + if (my $row = $sth->fetchrow_hashref()) { + $dayofweek = $row->{"dayOfWeek"}; + }else { + $logger->writeLog("ERROR", "TIME: Cannot get day of week for date :".$date); + } + if (!defined($dayofweek)) { + $logger->writeLog("ERROR", "TIME: day of week for date ".$date." is null"); + } + return $dayofweek; +} + +sub getYesterdayTodayDate { + my $self = shift; + + # get yesterday date. date format : YYYY-MM-DD + my $sth = $self->{centstorage}->query("SELECT CURRENT_DATE() as today, DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY) as yesterday"); + + my $yesterday; + my $today; + if (my $row = $sth->fetchrow_hashref()) { + $yesterday = $row->{yesterday}; + $today = $row->{today}; + } else { + $self->{logger}->writeLog('ERROR', "TIME: cannot get yesterday date"); + } + if (!defined($yesterday)) { + $self->{logger}->writeLog('ERROR', "TIME: Yesterday start date is null"); + } + if (!defined($today)) { + $self->{logger}->writeLog('ERROR', "TIME: today start date is null"); + } + return ($yesterday, $today); +} + +sub addDateInterval { + my $self = shift; + my ($date, $interval, $intervalType) = @_; + + # get new date. date format : YYYY-MM-DD + my $sth = $self->{centstorage}->query("SELECT DATE_ADD('".$date."', INTERVAL ".$interval." ".$intervalType.") as newDate"); + + my $newDate; + if (my $row = $sth->fetchrow_hashref()) { + $newDate = $row->{newDate}; + } + if (!defined($newDate)) { + $self->{logger}->writeLog('ERROR', "TIME: DATE_ADD('".$date."', INTERVAL ".$interval." ".$intervalType.") returns null value"); + } + return $newDate; +} + +sub compareDates { + my $self = shift; + my ($date1, $date2) = @_; + + my $sth = $self->{centstorage}->query("SELECT DATEDIFF('".$date1."','".$date2."') as nbDays"); + if (my $row = $sth->fetchrow_hashref()) { + return $row->{nbDays}; + } + + $self->{logger}->writeLog('ERROR', "TIME: Cannot compare two dates : ".$date1." and ".$date2); +} + +sub insertTimeEntriesForPeriod { + my $self = shift; + my $db = $self->{"centstorage"}; + my ($start, $end) = @_; + + my $interval = $self->getTotalDaysInPeriod($start, $end) * 24; + my $counter = 0; + my $date = "ADDDATE('".$start."',INTERVAL ".$counter." HOUR)"; + my $query_suffix = ""; + while ($counter <= $interval) { + $query_suffix .= "(UNIX_TIMESTAMP(".$date."),"; + $query_suffix .= "HOUR(".$date."),"; + $query_suffix .= "DAYOFMONTH(".$date."),"; + $query_suffix .= "LOWER(DATE_FORMAT(".$date.",'%M')),"; + $query_suffix .= "MONTH(".$date."),"; + $query_suffix .= "YEAR(".$date."),"; + $query_suffix .= "WEEK(".$date.", 3),"; + $query_suffix .= "LOWER(DAYNAME(".$date.")),"; + $query_suffix .= "UNIX_TIMESTAMP(".$date."),"; + $query_suffix .= "".$date."),"; + $counter++; + $date = "ADDDATE('".$start."',INTERVAL ".$counter." HOUR)"; + if ($counter % 30 == 0) { + chop($query_suffix); + $db->query($self->{insertQuery} . $query_suffix); + $query_suffix = ""; + } + } + chop($query_suffix); + if ($query_suffix ne "") { + $db->query($self->{insertQuery} . $query_suffix); + } +} + +# Delete duplicated entries inserted on winter/summer time change (same timestamp for 02:00 and 03:00) +sub deleteDuplicateEntries { + my $self = shift; + my $db = $self->{"centstorage"}; + my ($start, $end) = @_; + my $query = "SELECT max(id) as id"; + $query .= " FROM mod_bi_time"; + $query .= " WHERE dtime >='".$start."'"; + $query .= " AND dtime <= '".$end."'"; + $query .= " GROUP BY utime"; + $query .= " HAVING COUNT(utime) > 1"; + my $sth = $db->query($query); + my $ids_to_delete = ""; + while (my $row = $sth->fetchrow_hashref()) { + $ids_to_delete .= $row->{'id'}.","; + } + if ($ids_to_delete ne "") { + chop ($ids_to_delete); + $db->query("DELETE FROM mod_bi_time WHERE id IN (".$ids_to_delete.")"); + } +} + +sub getTotalDaysInPeriod { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($start, $end) = @_; + + my $query = "SELECT DATEDIFF('".$end."', '".$start."') diff"; + my $sth = $db->query($query); + my $diff; + if (my $row = $sth->fetchrow_hashref()) { + $diff = $row->{'diff'}; + }else { + $logger->writeLog("ERROR", "TIME : Cannot get difference between period start and end"); + } + if (!defined($diff)){ + $logger->writeLog("ERROR", "TIME : Cannot get difference between period start and end"); + } + if($diff == 0) { + $logger->writeLog("ERROR", "TIME : start date is equal to end date"); + }elsif ($diff < 0) { + $logger->writeLog("ERROR", "TIME : start date is greater than end date"); + } + return $diff; +} + +sub truncateTable { + my $self = shift; + my $db = $self->{"centstorage"}; + + my $query = "TRUNCATE TABLE `mod_bi_time`"; + $db->query($query); + $db->query("ALTER TABLE `mod_bi_time` AUTO_INCREMENT=1"); +} + +sub deleteEntriesForPeriod { + my $self = shift; + my $db = $self->{"centstorage"}; + my ($start, $end) = @_; + + my $query = "DELETE FROM `mod_bi_time` WHERE dtime >= '".$start."' AND dtime < '".$end."'"; + $db->query($query); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/centreon/CentileProperties.pm b/gorgone/modules/centreon/mbi/libs/centreon/CentileProperties.pm new file mode 100644 index 0000000..a719c95 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/centreon/CentileProperties.pm @@ -0,0 +1,60 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::centreon::CentileProperties; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{logger} = shift; + $self->{centreon} = shift; + if (@_) { + $self->{centstorage} = shift; + } + bless $self, $class; + return $self; +} + +sub getCentileParams { + my $self = shift; + my $centreon = $self->{centreon}; + my $logger = $self->{logger}; + + my $centileParams = []; + my $query = "SELECT `centile_param`, `timeperiod_id` FROM `mod_bi_options_centiles`"; + my $sth = $centreon->query($query); + while (my $row = $sth->fetchrow_hashref()) { + if (defined($row->{centile_param}) && $row->{centile_param} ne '0' && defined($row->{timeperiod_id}) && $row->{timeperiod_id} ne '0'){ + push @{$centileParams}, { centile_param => $row->{centile_param}, timeperiod_id => $row->{timeperiod_id} }; + } + } + + return $centileParams; +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/centreon/ETLProperties.pm b/gorgone/modules/centreon/mbi/libs/centreon/ETLProperties.pm new file mode 100644 index 0000000..1181394 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/centreon/ETLProperties.pm @@ -0,0 +1,119 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::centreon::ETLProperties; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + + $self->{logger} = shift; + $self->{centreon} = shift; + if (@_) { + $self->{centstorage} = shift; + } + bless $self, $class; + return $self; +} + +# returns two references to two hash tables => hosts indexed by id and hosts indexed by name +sub getProperties { + my $self = shift; + + my $activated = 1; + if (@_) { + $activated = 0; + } + my (%etlProperties, %dataRetention); + + my $query = "SELECT `opt_key`, `opt_value` FROM `mod_bi_options` WHERE `opt_key` like 'etl.%'"; + my $sth = $self->{centreon}->query($query); + while (my $row = $sth->fetchrow_hashref()) { + if ($row->{opt_key} =~ /etl.retention.(.*)/) { + $dataRetention{$1} = $row->{opt_value}; + } elsif ($row->{opt_key} =~ /etl.list.(.*)/) { + my @tab = split (/,/, $row->{opt_value}); + my %hashtab = (); + foreach(@tab) { + $hashtab{$_} = 1; + } + $etlProperties{$1} = \%hashtab; + } elsif ($row->{opt_key} =~ /etl.(.*)/) { + $etlProperties{$1} = $row->{opt_value}; + } + } + if (defined($etlProperties{'capacity.exclude.metrics'})) { + $etlProperties{'capacity.exclude.metrics'} =~ s/^/\'/; + $etlProperties{'capacity.exclude.metrics'} =~ s/$/\'/; + $etlProperties{'capacity.exclude.metrics'} =~ s/,/\',\'/; + } + + return (\%etlProperties, \%dataRetention); +} + +# returns the max retention period defined by type of statistics, monthly stats are excluded +sub getMaxRetentionPeriodFor { + my $self = shift; + my $logger = $self->{'logger'}; + + my $type = shift; + my $query = "SELECT date_format(NOW(), '%Y-%m-%d') as period_end,"; + $query .= " date_format(DATE_ADD(NOW(), INTERVAL MAX(CAST(`opt_value` as SIGNED INTEGER))*-1 DAY), '%Y-%m-%d') as period_start"; + $query .= " FROM `mod_bi_options` "; + $query .= " WHERE `opt_key` IN ('etl.retention.".$type.".hourly','etl.retention.".$type.".daily', 'etl.retention.".$type.".raw')"; + my $sth = $self->{centreon}->query($query); + + if (my $row = $sth->fetchrow_hashref()) { + return ($row->{period_start}, $row->{period_end}); + } + + die 'Cannot get max perfdata retention period. Verify your data retention options'; +} + +# Returns a start and a end date for each retention period +sub getRetentionPeriods { + my $self = shift; + my $logger = $self->{'logger'}; + + my $query = "SELECT date_format(NOW(), '%Y-%m-%d') as period_end,"; + $query .= " date_format(DATE_ADD(NOW(), INTERVAL (`opt_value`)*-1 DAY), '%Y-%m-%d') as period_start,"; + $query .= " opt_key "; + $query .= " FROM `mod_bi_options` "; + $query .= " WHERE `opt_key` like ('etl.retention.%')"; + my $sth = $self->{centreon}->query($query); + my %periods = (); + while (my $row = $sth->fetchrow_hashref()) { + $row->{'opt_key'} =~ s/etl.retention.//; + $periods{$row->{'opt_key'}} = { start => $row->{period_start}, end => $row->{period_end}} ; + } + if (!scalar(keys %periods)){ + $logger->writeLog("FATAL", "Cannot retention periods information. Verify your data retention options"); + } + return (\%periods); +} +1; diff --git a/gorgone/modules/centreon/mbi/libs/centreon/Host.pm b/gorgone/modules/centreon/mbi/libs/centreon/Host.pm new file mode 100644 index 0000000..2b869ce --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/centreon/Host.pm @@ -0,0 +1,307 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +package gorgone::modules::centreon::mbi::libs::centreon::Host; + +use strict; +use warnings; +use Data::Dumper; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centreon"} = shift; + $self->{'etlProperties'} = undef; + #Hash that will contains all relation between host and hostcategories after calling the function getHostCategoriesWithTemplate + $self->{"hostCategoriesWithTemplates"} = undef; + if (@_) { + $self->{"centstorage"} = shift; + } + bless $self, $class; + return $self; +} + +#Set the etl properties as a variable of the class +sub setEtlProperties{ + my $self = shift; + $self->{'etlProperties'} = shift; +} + +# returns two references to two hash tables => hosts indexed by id and hosts indexed by name +sub getAllHosts { + my $self = shift; + my $centreon = $self->{"centreon"}; + my $activated = 1; + if (@_) { + $activated = 0; + } + my (%host_ids, %host_names); + + my $query = "SELECT `host_id`, `host_name`". + " FROM `host`". + " WHERE `host_register`='1'"; + if ($activated == 1) { + $query .= " AND `host_activate` ='1'"; + } + my $sth = $centreon->query($query); + while (my $row = $sth->fetchrow_hashref()) { + $host_ids{$row->{"host_name"}} = $row->{"host_id"}; + $host_names{$row->{"host_id"}} = $row->{"host_name"}; + } + $sth->finish(); + return (\%host_ids,\%host_names); +} + +# Get all hosts, keys are IDs +sub getAllHostsByID { + my $self = shift; + my ($host_ids, $host_names) = $self->getAllHosts(); + return ($host_ids); +} + +# Get all hosts, keys are names +sub getAllHostsByName { + my $self = shift; + my ($host_ids, $host_names) = $self->getAllHosts(); + return ($host_names); +} + + +# returns host groups linked to hosts +# all hosts will be stored in a hash table +# each key of the hash table is a host id +# each key is linked to a table containing entries like : "hostgroup_id;hostgroup_name" +sub getHostGroups { + my $self = shift; + my $centreon = $self->{"centreon"}; + my $activated = 1; + my $etlProperties = $self->{'etlProperties'}; + if (@_) { + $activated = 0; + } + my %result = (); + + my $query = "SELECT `host_id`, `host_name`, `hg_id`, `hg_name`". + " FROM `host`, `hostgroup_relation`, `hostgroup`". + " WHERE `host_register`='1'". + " AND `hostgroup_hg_id` = `hg_id`". + " AND `host_id`= `host_host_id`"; + if ($activated == 1) { + $query .= " AND `host_activate` ='1'"; + } + if(!defined($etlProperties->{'dimension.all.hostgroups'}) && $etlProperties->{'dimension.hostgroups'} ne ''){ + $query .= " AND `hg_id` IN (".$etlProperties->{'dimension.hostgroups'}.")"; + } + my $sth = $centreon->query($query); + while (my $row = $sth->fetchrow_hashref()) { + my $new_entry = $row->{"hg_id"}.";".$row->{"hg_name"}; + if (defined($result{$row->{"host_id"}})) { + my $tab_ref = $result{$row->{"host_id"}}; + my @tab = @$tab_ref; + my $exists = 0; + foreach(@tab) { + if ($_ eq $new_entry) { + $exists = 1; + last; + } + } + if (!$exists) { + push @tab, $new_entry; + } + $result{$row->{"host_id"}} = \@tab; + }else { + my @tab = ($new_entry); + $result{$row->{"host_id"}} = \@tab; + } + } + $sth->finish(); + return (\%result); +} + +#Get the link between host and categories using templates +sub getRecursiveCategoriesForOneHost{ + my $self = shift; + my $host_id = shift; + my $ref_hostCat = shift; + my $centreon = $self->{"centreon"}; + my $etlProperties = $self->{"etlProperties"}; + + + #Get all categories linked to the templates associated with the host or just template associated with host to be able to call the method recursively + + my $query = "SELECT host_id, host_name, template_id,template_name, categories.hc_id as category_id, categories.hc_activate as hc_activate,". + " categories.hc_name as category_name ". + " FROM ( SELECT t1.host_id,t1.host_name,templates.host_id as template_id,templates.host_name as template_name ". + " FROM host t1, host_template_relation t2, host templates ". + " WHERE t1.host_id = t2.host_host_id AND t2.host_tpl_id = templates.host_id AND t1.host_activate ='1' AND t1.host_id = ".$host_id." ) r1 ". + " LEFT JOIN hostcategories_relation t3 ON t3.host_host_id = r1.template_id LEFT JOIN hostcategories categories ON t3.hostcategories_hc_id = categories.hc_id "; + + + my @hostCategoriesAllowed = split /,/, $etlProperties->{'dimension.hostcategories'}; + + my $sth = $centreon->query($query); + while (my $row = $sth->fetchrow_hashref()) { + my @tab = (); + my $new_entry; + my $categoryId = $row->{"category_id"}; + my $categoryName = $row->{"category_name"}; + my $categoryActivate = $row->{"hc_activate"}; + + #If current category is in allowed categories in ETL configuration + #add it to the categories link to the host, + #Then check for templates categories recursively + if(defined($categoryId) && defined($categoryName) && $categoryActivate=='1'){ + if ((grep {$_ eq $categoryId} @hostCategoriesAllowed) || (defined($etlProperties->{'dimension.all.hostcategories'}) && $etlProperties->{'dimension.all.hostcategories'} ne '')){ + $new_entry = $categoryId.";".$categoryName; + #If no hostcat has been found for the host, create the line + if (!scalar(@$ref_hostCat)){ + @$ref_hostCat = ($new_entry); + }else { #If the tab is not empty, check wether the combination already exists in the tab + @tab = @$ref_hostCat; + my $exists = 0; + foreach(@$ref_hostCat) { + if ($_ eq $new_entry) { + $exists = 1; + last; + } + } + #If the host category did not exist, add it to the table @$ref_hostCat + if (!$exists) { + push @$ref_hostCat, $new_entry; + } + } + } + } + $self->getRecursiveCategoriesForOneHost($row->{"template_id"},$ref_hostCat); + } + $sth->finish(); +} + +#Get the link between host and categories using direct link hc <> host +sub getDirectLinkedCategories{ + my $self = shift; + my $host_id = shift; + my $ref_hostCat = shift; + my $centreon = $self->{"centreon"}; + my $etlProperties = $self->{"etlProperties"}; + my @tab = (); + + my $query = "SELECT `host_id`, `host_name`, `hc_id`, `hc_name`". + " FROM `host`, `hostcategories_relation`, `hostcategories`". + " WHERE `host_register`='1'". + " AND `hostcategories_hc_id` = `hc_id`". + " AND `host_id`= `host_host_id`". + " AND `host_id`= ".$host_id." ". + " AND `host_activate` ='1' AND hostcategories.hc_activate = '1' "; + + if(!defined($etlProperties->{'dimension.all.hostcategories'}) && $etlProperties->{'dimension.hostcategories'} ne ''){ + $query .= " AND `hc_id` IN (".$etlProperties->{'dimension.hostcategories'}.")"; + } + + my $sth = $centreon->query($query); + while (my $row = $sth->fetchrow_hashref()) { + my $new_entry = $row->{"hc_id"}.";".$row->{"hc_name"}; + if (!scalar(@$ref_hostCat)){ + @$ref_hostCat = ($new_entry); + }else { + @tab = @$ref_hostCat; + my $exists = 0; + foreach(@$ref_hostCat) { + if ($_ eq $new_entry) { + $exists = 1; + last; + } + } + if (!$exists) { + push @$ref_hostCat, $new_entry; + } + } + } + $sth->finish(); +} + +#Fill a class Hash table that contains the relation between host_id and table[hc_id,hc_name] +sub getHostCategoriesWithTemplate{ + my $self = shift; + my $centreon = $self->{"centreon"}; + my $activated = 1; + + #Hash : each key of the hash table is a host id + #each key is linked to a table containing entries like : "hc_id,hc_name" + my $hostCategoriesWithTemplate = $self->{'hostCategoriesWithTemplates'}; + if (@_) { + $activated = 0; + } + + my $query = "SELECT `host_id`". + " FROM `host`". + " WHERE `host_activate` ='1'"; + + my $sth = $centreon->query($query); + while (my $row = $sth->fetchrow_hashref()) { + my @tab = (); + my $host_id = $row->{"host_id"}; + $self->getRecursiveCategoriesForOneHost($host_id,\@tab); + $self->getDirectLinkedCategories($host_id,\@tab); + $hostCategoriesWithTemplate->{$row->{"host_id"}} = [@tab]; + undef @tab; + } + $self->{'hostCategoriesWithTemplates'} = $hostCategoriesWithTemplate; + $sth->finish(); +} + +sub getHostGroupAndCategories { + my $self = shift; + + my $hostGroups = $self->getHostGroups(); + $self->getHostCategoriesWithTemplate(); + my $hostCategories = $self->{"hostCategoriesWithTemplates"}; + my $hosts = $self->getAllHostsByName; + my @results; + + while (my ($hostId, $groups) = each (%$hostGroups)) { + my $categories_ref = $hostCategories->{$hostId}; + my @categoriesTab = (); + if (defined($categories_ref) && scalar(@$categories_ref)) { + @categoriesTab = @$categories_ref; + } + my $hostName = $hosts->{$hostId}; + foreach(@$groups) { + my $group = $_; + if (scalar(@categoriesTab)) { + foreach(@categoriesTab) { + push @results, $hostId.";".$hostName.";".$group.";".$_; + } + }else { + #If there is no category + push @results, $hostId.";".$hostName.";".$group.";0;NoCategory"; + } + } + } + return \@results; +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/centreon/HostCategory.pm b/gorgone/modules/centreon/mbi/libs/centreon/HostCategory.pm new file mode 100644 index 0000000..d881daa --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/centreon/HostCategory.pm @@ -0,0 +1,70 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::centreon::HostCategory; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centreon"} = shift; + $self->{'etlProperties'} = undef; + if (@_) { + $self->{"centstorage"} = shift; + } + bless $self, $class; + return $self; +} + +#Set the etl properties as a variable of the class +sub setEtlProperties{ + my $self = shift; + $self->{'etlProperties'} = shift; +} + + +sub getAllEntries { + my $self = shift; + my $db = $self->{"centreon"}; + my $etlProperties = $self->{'etlProperties'}; + + my $query = "SELECT `hc_id`, `hc_name`"; + $query .= " FROM `hostcategories`"; + if(!defined($etlProperties->{'dimension.all.hostcategories'}) && $etlProperties->{'dimension.hostcategories'} ne ''){ + $query .= " WHERE `hc_id` IN (".$etlProperties->{'dimension.hostcategories'}.")"; + } + my $sth = $db->query($query); + my @entries = (); + while (my $row = $sth->fetchrow_hashref()) { + push @entries, $row->{"hc_id"}.";".$row->{"hc_name"}; + } + $sth->finish(); + return (\@entries); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/centreon/HostGroup.pm b/gorgone/modules/centreon/mbi/libs/centreon/HostGroup.pm new file mode 100644 index 0000000..5c91af3 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/centreon/HostGroup.pm @@ -0,0 +1,136 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::centreon::HostGroup; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centreon"} = shift; + $self->{'etlProperties'} = undef; + if (@_) { + $self->{"centstorage"} = shift; + } + bless $self, $class; + return $self; +} + +#Set the etl properties as a variable of the class +sub setEtlProperties{ + my $self = shift; + $self->{'etlProperties'} = shift; +} + +# returns in a table all host/service of a group of host +sub getHostgroupServices { + my $self = shift; + my $db = $self->{"centreon"}; + my $etlProperties = $self->{'etlProperties'}; + my $hgId = 0; + if (@_) { + $hgId = shift; + } + my %result = (); + my $query = "SELECT h.`host_id`, h.`host_name`, s.`service_id`, s.`service_description`"; + $query .= " FROM `hostgroup` hg, `host_service_relation` hsr, `service` s, `hostgroup_relation` hgr, `host` h"; + $query .= " WHERE hg.`hg_id` = ".$hgId; + $query .= " AND hg.`hg_id` = hsr.`hostgroup_hg_id`"; + $query .= " AND hsr.`service_service_id` = s.`service_id`"; + $query .= " AND s.`service_activate` = '1'"; + $query .= " AND s.`service_register` = '1'"; + $query .= " AND hg.hg_id = hgr.`hostgroup_hg_id`"; + $query .= " AND hgr.`host_host_id` = h.`host_id`"; + $query .= " AND h.`host_activate` = '1'"; + $query .= " AND h.`host_register` = '1'"; + if(!defined($etlProperties->{'dimension.all.hostgroups'}) && $etlProperties->{'dimension.hostgroups'} ne ''){ + $query .= " AND hg.`hg_id` IN (".$etlProperties->{'dimension.hostgroups'}.")"; + } + my $sth = $db->query($query); + while (my $row = $sth->fetchrow_hashref()) { + $result{$row->{"host_id"}.";".$row->{"service_id"}} = 1; + } + $sth->finish(); + return (\%result); +} + + +# returns in a table all host/service of a group of host +sub getHostgroupHostServices { + my $self = shift; + my $db = $self->{"centreon"}; + my %etlProperties = $self->{'etlProperties'}; + + my $hgId = 0; + if (@_) { + $hgId = shift; + } + my %result = (); + my $query = "SELECT h.`host_id`, s.`service_id`"; + $query .= " FROM `host` h, `hostgroup` hg, `hostgroup_relation` hgr, `host_service_relation` hsr, `service` s"; + $query .= " WHERE hg.`hg_id` = ".$hgId; + $query .= " AND hgr.`hostgroup_hg_id` = hg.`hg_id`"; + $query .= " AND hgr.`host_host_id` = h.`host_id`"; + $query .= " AND h.`host_activate` = '1'"; + $query .= " AND h.`host_register` = '1'"; + $query .= " AND h.`host_id` = hsr.`host_host_id`"; + $query .= " AND hsr.`service_service_id` = s.`service_id`"; + $query .= " AND s.`service_activate` = '1'"; + $query .= " AND s.`service_register` = '1'"; + if(!defined($etlProperties{'etl.dimension.all.hostgroups'}) && $etlProperties{'etl.dimension.hostgroups'} ne ''){ + $query .= " AND hg.`hg_id` IN (".$etlProperties{'etl.dimension.hostgroups'}.")"; + } + my $sth = $db->query($query); + while (my $row = $sth->fetchrow_hashref()) { + $result{$row->{"host_id"}.";".$row->{"service_id"}} = 1; + } + %result = (%result, $self->getHostgroupServices($hgId)); + return (\%result); +} + +sub getAllEntries { + my $self = shift; + my $db = $self->{"centreon"}; + my $etlProperties = $self->{'etlProperties'}; + + my $query = "SELECT `hg_id`, `hg_name`"; + $query .= " FROM `hostgroup`"; + if(!defined($etlProperties->{'dimension.all.hostgroups'}) && $etlProperties->{'dimension.hostgroups'} ne ''){ + $query .= " WHERE `hg_id` IN (".$etlProperties->{'dimension.hostgroups'}.")"; + } + my $sth = $db->query($query); + my @entries = (); + while (my $row = $sth->fetchrow_hashref()) { + push @entries, $row->{"hg_id"}.";".$row->{"hg_name"}; + } + $sth->finish(); + return (\@entries); +} + + +1; diff --git a/gorgone/modules/centreon/mbi/libs/centreon/Service.pm b/gorgone/modules/centreon/mbi/libs/centreon/Service.pm new file mode 100644 index 0000000..4471517 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/centreon/Service.pm @@ -0,0 +1,214 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::centreon::Service; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centreon"} = shift; + $self->{'etlProperties'} = undef; + + if (@_) { + $self->{"centstorage"} = shift; + } + bless $self, $class; + return $self; +} + +sub setEtlProperties{ + my $self = shift; + $self->{'etlProperties'} = shift; +} + +# returns two references to two hash tables => services indexed by id and services indexed by name +sub getServicesWithHostAndCategory { + my $self = shift; + my $centreon = $self->{"centreon"}; + my $serviceId = ""; + my $hosts = shift; + if (@_) { + $serviceId = shift; + } + my $templateCategories = $self->getServicesTemplatesCategories; + + my (@results); + # getting services linked to hosts + my $query = "SELECT service_description, service_id, host_id, service_template_model_stm_id as tpl". + " FROM host, service, host_service_relation". + " WHERE host_id = host_host_id and service_service_id = service_id". + " AND service_register = '1'". + " AND host_activate = '1'". + " AND service_activate = '1'"; + + + my $sth = $centreon->query($query); + while(my $row = $sth->fetchrow_hashref()) { + # getting all host entries + my $serviceHostTable = $hosts->{$row->{"host_id"}}; + # getting all Categories entries + my @categoriesTable = (); + # getting categories directly linked to service + my $categories = $self->getServiceCategories($row->{"service_id"}); + while(my ($sc_id, $sc_name) = each(%$categories)) { + push @categoriesTable, $sc_id.";".$sc_name; + } + # getting categories linked to template + if (defined($row->{"tpl"}) && defined($templateCategories->{$row->{"tpl"}})) { + my $tplCategories = $templateCategories->{$row->{"tpl"}}; + while(my ($sc_id, $sc_name) = each(%$tplCategories)) { + if(!defined($categories->{$sc_id})) { + push @categoriesTable, $sc_id.";".$sc_name; + } + } + } + if (!scalar(@categoriesTable)) { + #ToDo push @categoriesTable, "0;NULL"; + } + if (defined($serviceHostTable)) { + foreach(@$serviceHostTable) { + my $hostInfos = $_; + foreach(@categoriesTable) { + push @results, $row->{"service_id"}.";".$row->{"service_description"}.";".$_.";".$hostInfos; + } + } + } + } + #getting services linked to hostgroup + $query = "SELECT DISTINCT service_description, service_id, host_id, service_template_model_stm_id as tpl". + " FROM host, service, host_service_relation hr, hostgroup_relation hgr". + " WHERE hr.hostgroup_hg_id is not null". + " AND hr.service_service_id = service_id". + " AND hr.hostgroup_hg_id = hgr.hostgroup_hg_id". + " AND hgr.host_host_id = host_id". + " AND service_register = '1'". + " AND host_activate = '1'". + " AND service_activate = '1'"; + + $sth = $centreon->query($query); + while(my $row = $sth->fetchrow_hashref()) { + # getting all host entries + my $serviceHostTable = $hosts->{$row->{"host_id"}}; + # getting all Categories entries + my @categoriesTable = (); + # getting categories directly linked to service + my $categories = $self->getServiceCategories($row->{"service_id"}); + while(my ($sc_id, $sc_name) = each(%$categories)) { + push @categoriesTable, $sc_id.";".$sc_name; + } + # getting categories linked to template + if (defined($row->{"tpl"}) && defined($templateCategories->{$row->{"tpl"}})) { + my $tplCategories = $templateCategories->{$row->{"tpl"}}; + while(my ($sc_id, $sc_name) = each(%$tplCategories)) { + if(!defined($categories->{$sc_id})) { + push @categoriesTable, $sc_id.";".$sc_name; + } + } + } + if (!scalar(@categoriesTable)) { + push @categoriesTable, "0;NULL"; + } + if (defined($serviceHostTable)) { + foreach(@$serviceHostTable) { + my $hostInfos = $_; + foreach(@categoriesTable) { + push @results, $row->{"service_id"}.";".$row->{"service_description"}.";".$_.";".$hostInfos; + } + } + } + } + $sth->finish(); + return (\@results); +} + +sub getServicesTemplatesCategories { + my $self = shift; + my $db = $self->{"centreon"}; + my %results = (); + + my $query = "SELECT service_id, service_description, service_template_model_stm_id FROM service WHERE service_register = '0'"; + my $sth = $db->query($query); + while(my $row = $sth->fetchrow_hashref()) { + my $currentTemplate = $row->{"service_id"}; + my $categories = $self->getServiceCategories($row->{"service_id"}); + my $parentId = $row->{"service_template_model_stm_id"}; + if (defined($parentId)) { + my $hasParent = 1; + # getting all parent templates category relations + while ($hasParent) { + my $parentQuery = "SELECT service_id, service_template_model_stm_id "; + $parentQuery .= "FROM service "; + $parentQuery .= "WHERE service_register = '0' and service_id=".$parentId; + my $sthparentQuery = $db->query($parentQuery); + if(my $parentQueryRow = $sthparentQuery->fetchrow_hashref()) { + my $newCategories = $self->getServiceCategories($parentQueryRow->{"service_id"}); + while(my ($sc_id, $sc_name) = each(%$newCategories)) { + if (!defined($categories->{$sc_id})) { + $categories->{$sc_id} = $sc_name; + } + } + if (!defined($parentQueryRow->{'service_template_model_stm_id'})) { + $hasParent = 0; + last; + } + $parentId = $parentQueryRow->{'service_template_model_stm_id'}; + $sthparentQuery->finish(); + }else { + $hasParent = 0; + } + } + } + $results{$currentTemplate} = $categories; + } + $sth->finish(); + return \%results; +} + +sub getServiceCategories { + my $self = shift; + my $db = $self->{"centreon"}; + my $id = shift; + my %results = (); + my $etlProperties = $self->{'etlProperties'}; + + my $query = "SELECT sc.sc_id, sc_name "; + $query .= " FROM service_categories sc, service_categories_relation scr"; + $query .= " WHERE service_service_id = ".$id; + $query .= " AND sc.sc_id = scr.sc_id"; + if(!defined($etlProperties->{'dimension.all.servicecategories'}) && $etlProperties->{'dimension.servicecategories'} ne ''){ + $query .= " AND sc.sc_id IN (".$etlProperties->{'dimension.servicecategories'}.")"; + } + my $sth = $db->query($query); + while(my $row = $sth->fetchrow_hashref()) { + $results{$row->{"sc_id"}} = $row->{"sc_name"}; + } + return (\%results); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/centreon/ServiceCategory.pm b/gorgone/modules/centreon/mbi/libs/centreon/ServiceCategory.pm new file mode 100644 index 0000000..47a532d --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/centreon/ServiceCategory.pm @@ -0,0 +1,96 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::centreon::ServiceCategory; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centreon"} = shift; + $self->{'etlProperties'} = undef; + if (@_) { + $self->{"centstorage"} = shift; + } + bless $self, $class; + return $self; +} + +#Set the etl properties as a variable of the class +sub setEtlProperties{ + my $self = shift; + $self->{'etlProperties'} = shift; +} + +# returns two references to two hash tables => services indexed by id and services indexed by name +sub getCategory { + my $self = shift; + my $db = $self->{"centreon"}; + my $etlProperties = $self->{'etlProperties'}; + my $scName = ""; + if (@_) { + $scName = shift; + } + + my $result = ""; + # getting services linked to hosts + my $query = "SELECT sc_id from service_categories WHERE sc_name='".$scName."'"; + if(!defined($etlProperties->{'dimension.all.servicecategories'}) && $etlProperties->{'dimension.servicecategories'} ne ''){ + $query .= " WHERE `sc_id` IN (".$etlProperties->{'dimension.servicecategories'}.")"; + } + my $sth = $db->query($query); + if(my $row = $sth->fetchrow_hashref()) { + $result = $row->{"sc_id"}; + }else { + ($self->{"logger"})->writeLog("error", "Cannot find service category '" . $scName . "' in database"); + } + $sth->finish(); + + return ($result); +} + +sub getAllEntries { + my $self = shift; + my $db = $self->{"centreon"}; + my $etlProperties = $self->{'etlProperties'}; + + my $query = "SELECT `sc_id`, `sc_name`"; + $query .= " FROM `service_categories`"; + if(!defined($etlProperties->{'dimension.all.servicecategories'}) && $etlProperties->{'dimension.servicecategories'} ne ''){ + $query .= " WHERE `sc_id` IN (".$etlProperties->{'dimension.servicecategories'}.")"; + } + my $sth = $db->query($query); + my @entries = (); + while (my $row = $sth->fetchrow_hashref()) { + push @entries, $row->{"sc_id"}.";".$row->{"sc_name"}; + } + $sth->finish(); + return (\@entries); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/centreon/Timeperiod.pm b/gorgone/modules/centreon/mbi/libs/centreon/Timeperiod.pm new file mode 100644 index 0000000..97e0f84 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/centreon/Timeperiod.pm @@ -0,0 +1,247 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; +use Time::Local; +use gorgone::modules::centreon::mbi::libs::Utils; + +package gorgone::modules::centreon::mbi::libs::centreon::Timeperiod; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centreon"} = shift; + if (@_) { + $self->{"centstorage"} = shift; + } + bless $self, $class; + return $self; +} + +sub getTimeRangesForDay { + my $self = shift; + my $db = $self->{"centreon"}; + my ($weekDay, $name, $unixtime) = @_; + my @results = (); + + my @weekDays = ("sunday", "monday", "tuesday", "wednesday", "thursday", "friday", "saturday"); + my $query = "SELECT tp_" . $weekDay; + $query .= " FROM timeperiod"; + $query .= " WHERE tp_name = '" . $name . "'"; + my $sth = $db->query($query); + if (my $row = $sth->fetchrow_hashref()) { + if (defined($row->{'tp_'.$weekDay})) { + my @ranges = split(",", $row->{'tp_' . $weekDay}); + foreach (@ranges) { + my ($start, $end) = split("-", $_); + my ($start_hour, $start_min) = split(':', $start); + my ($end_hour, $end_min) = split(':', $end); + my @range = ($unixtime+ $start_hour * 60 * 60 + $start_min * 60, $unixtime + $end_hour * 60 * 60 + $end_min * 60); + $results[scalar(@results)] = \@range; + } + } + } + + return (\@results); +} + +sub getTimeRangesForDayByDateTime { + my $self = shift; + my $db = $self->{"centreon"}; + my ($name, $dateTime, $weekDay) = @_; + my @results = (); + + my $query = "SELECT tp_".$weekDay; + $query .= " FROM timeperiod"; + $query .= " WHERE tp_name='".$name."'"; + my $sth = $db->query($query); + if(my $row = $sth->fetchrow_hashref()) { + if (defined($row->{'tp_'.$weekDay})) { + my @ranges = split(",", $row->{'tp_'.$weekDay}); + foreach(@ranges) { + my ($start, $end) = split("-", $_); + my $range_end = "'".$dateTime." ".$end.":00'"; + if ($end eq '24:00') { + $range_end = "DATE_ADD('".$dateTime."', INTERVAL 1 DAY)"; + } + my @range = ("'".$dateTime." ".$start.":00'", $range_end); + $results[scalar(@results)] = \@range; + } + } + } + $sth->finish(); + + return (\@results); +} + +sub getRangeTable { + my ($self, $rangeStr) = @_; + if (!defined($rangeStr)) { + $rangeStr = ""; + } + my @ranges = split(",", $rangeStr); + + my @results = (); + foreach(@ranges) { + my ($start, $end) = split("-", $_); + my ($start_hour, $start_min) = split(":", $start); + my ($end_hour, $end_min) = split(":", $end); + push @results, [$start_hour * 60 * 60 + $start_min * 60, $end_hour * 60 * 60 + $end_min * 60]; + } + return [@results]; +} + +sub getAllRangesForTpId { + my ($self, $timeperiod_id) = @_; + my $db = $self->{"centreon"}; + my $logger = $self->{"logger"}; + my $query = "SELECT tp_monday, tp_tuesday, tp_wednesday, tp_thursday, tp_friday, tp_saturday, tp_sunday"; + $query .= " FROM timeperiod"; + $query .= " WHERE tp_id='".$timeperiod_id."'"; + my $sth = $db->query($query); + + my @results = (); + if(my $row = $sth->fetchrow_hashref()) { + $results[0] = $self->getRangeTable($row->{'tp_sunday'}); + $results[1] = $self->getRangeTable($row->{'tp_monday'}); + $results[2] = $self->getRangeTable($row->{'tp_tuesday'}); + $results[3] = $self->getRangeTable($row->{'tp_wednesday'}); + $results[4] = $self->getRangeTable($row->{'tp_thursday'}); + $results[5] = $self->getRangeTable($row->{'tp_friday'}); + $results[6] = $self->getRangeTable($row->{'tp_saturday'}); + }else { + $logger->writeLog("ERROR", "Cannot find time period with id '".$timeperiod_id."' in Centreon Database"); + } + return [@results]; +} + +sub getTimeRangesForPeriod { + my $self = shift; + my ($timeperiodId, $start, $end) = @_; + my @results = (); + my @weekDays = ("sunday", "monday", "tuesday", "wednesday", "thursday", "friday", "saturday"); + my $days = gorgone::modules::centreon::mbi::libs::Utils->getRebuildPeriods($start, $end); + my $weekRanges = $self->getAllRangesForTpId($timeperiodId); + foreach (@$days) { + my $dayStart = $_->{'start'}; + my $dayRanges = $weekRanges->[(localtime($dayStart))[6]]; + foreach(@$dayRanges) { + push @results, [$dayStart+$_->[0], $dayStart+$_->[1]]; + } + } + return [@results]; +} + +sub getTimeRangesForPeriodAndTpList { + my $self = shift; + my ($timeperiodList, $start, $end) = @_; + + my %rangesByTP = (); + while (my ($key, $value) = each %$timeperiodList) { + $rangesByTP{$key} = $self->getTimeRangesForPeriod($key, $start, $end); + } + return \%rangesByTP; +} + +sub getId { + my $self = shift; + my $db = $self->{"centreon"}; + my $name = shift; + + my $query = "SELECT tp_id"; + $query .= " FROM timeperiod"; + $query .= " WHERE tp_name = '".$name."'"; + my $sth = $db->query($query); + my $result = -1; + if(my $row = $sth->fetchrow_hashref()) { + $result = $row->{'tp_id'}; + } + return $result; +} + +sub getPeriodsLike { + my $self = shift; + my $db = $self->{"centreon"}; + my $name = shift; + + my $query = "SELECT tp_id, tp_name"; + $query .= " FROM timeperiod"; + $query .= " WHERE tp_name like '".$name."%'"; + my $sth = $db->query($query); + my %result = (); + while (my $row = $sth->fetchrow_hashref()) { + $result{$row->{'tp_id'}} = $row->{'tp_name'}; + } + return \%result; +} + +sub getPeriods { + my $self = shift; + my $db = $self->{"centreon"}; + my $logger = $self->{'logger'}; + my $ids = shift; + + my $idStr = ""; + + foreach my $key (keys %$ids) { + if ($idStr eq "") { + $idStr .= $key; + }else { + $idStr .= ",".$key; + } + } + if ($idStr eq "") { + $logger->writeLog("ERROR", "Select a timeperiod in the ETL configuration menu"); + } + my $query = "SELECT tp_id, tp_name"; + $query .= " FROM timeperiod"; + $query .= " WHERE tp_id IN (".$idStr.")"; + my $sth = $db->query($query); + my %result = (); + while (my $row = $sth->fetchrow_hashref()) { + $result{$row->{'tp_id'}} = $row->{'tp_name'}; + } + return \%result; +} + +sub getCentilePeriods { + my $self = shift; + my $db = $self->{"centreon"}; + my $logger = $self->{'logger'}; + + my $query = "SELECT tp_id, tp_name"; + $query .= " FROM timeperiod"; + $query .= " WHERE tp_id IN (select timeperiod_id from mod_bi_options_centiles)"; + my $sth = $db->query($query); + my %result = (); + while (my $row = $sth->fetchrow_hashref()) { + $result{$row->{'tp_id'}} = $row->{'tp_name'}; + } + return \%result; +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/centstorage/HostStateEvents.pm b/gorgone/modules/centreon/mbi/libs/centstorage/HostStateEvents.pm new file mode 100644 index 0000000..a43002e --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/centstorage/HostStateEvents.pm @@ -0,0 +1,184 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::centstorage::HostStateEvents; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + $self->{"biHostStateEventsObj"} = shift; + $self->{"timePeriodObj"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + $self->{"name"} = "hoststateevents"; + $self->{"timeColumn"} = "end_time"; + bless $self, $class; + return $self; +} + +sub getName() { + my $self = shift; + return $self->{'name'}; +} + +sub getTimeColumn() { + my $self = shift; + return $self->{'timeColumn'}; +} +sub agreggateEventsByTimePeriod { + my ($self, $timeperiodList, $start, $end, $liveServiceByTpId, $mode) = @_; + my $logger = $self->{"logger"}; + my $nbEvents; + my $db = $self->{"centstorage"}; + + my $rangesByTP = ($self->{"timePeriodObj"})->getTimeRangesForPeriodAndTpList($timeperiodList, $start, $end); + my $query = " SELECT e.host_id, start_time, end_time, ack_time, state, last_update"; + $query .= " FROM `hoststateevents` e"; + $query .= " RIGHT JOIN (select host_id from mod_bi_tmp_today_hosts group by host_id) t2"; + $query .= " ON e.host_id = t2.host_id"; + $query .= " WHERE start_time < ".$end.""; + $query .= " AND end_time > ".$start.""; + $query .= " AND in_downtime = 0 "; + $query .= " ORDER BY start_time "; + + + my $hostEventObjects = $self->{"biHostStateEventsObj"}; + my $sth = $db->query($query); + $hostEventObjects->createTempBIEventsTable(); + $hostEventObjects->prepareTempQuery(); + + while (my $row = $sth->fetchrow_hashref()) { + if (!defined($row->{'end_time'})) { + $row->{'end_time'} = $end; + } + while (my ($timeperiodID, $timeRanges) = each %$rangesByTP) { + my @tab = (); + $tab[0] = $row->{'host_id'}; + $tab[1] = $liveServiceByTpId->{$timeperiodID}; + $tab[2] = $row->{'state'}; + if ($mode eq "daily") { + $timeRanges = ($self->{"timePeriodObj"})->getTimeRangesForPeriod($timeperiodID, $row->{'start_time'}, $row->{'end_time'}); + } + ($tab[3], $tab[4]) = $self->processIncidentForTp($timeRanges,$row->{'start_time'}, $row->{'end_time'}); + $tab[5] = $row->{'end_time'}; + $tab[6] = defined($row->{ack_time}) ? $row->{ack_time} : 0; + $tab[7] = $row->{'last_update'}; + if (defined($tab[3]) && $tab[3] != -1) { + $hostEventObjects->bindParam(\@tab); + } + + } + } + ($db->getInstance)->commit; +} + +sub processIncidentForTp { + my ($self, $timeRanges, $start, $end) = @_; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my $rangeSize = scalar(@$timeRanges); + my $duration = 0; + my $slaDuration = 0; + my $range = 0; + my $i = 0; + my $processed = 0; + my $slaStart = $start; + my $slaStartModified = 0; + + foreach(@$timeRanges) { + my $currentStart = $start; + my $currentEnd = $end; + + $range = $_; + my ($rangeStart, $rangeEnd) = ($range->[0], $range->[1]); + + if ($currentStart < $rangeEnd && $currentEnd > $rangeStart) { + $processed = 1; + if ($currentStart > $rangeStart) { + $slaStartModified = 1; + } elsif ($currentStart < $rangeStart) { + $currentStart = $rangeStart; + if (!$slaStartModified) { + $slaStart = $currentStart; + $slaStartModified = 1; + } + } + if ($currentEnd > $rangeEnd) { + $currentEnd = $rangeEnd; + } + $slaDuration += $currentEnd - $currentStart; + } + } + if (!$processed) { + return (-1, -1, -1); + } + + return ($slaStart, $slaDuration); +} + + +sub dailyPurge { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($end) = @_; + + $logger->writeLog("DEBUG", "[PURGE] [hoststateevents] purging data older than ".$end); + my $query = "DELETE FROM `hoststateevents` where end_time < UNIX_TIMESTAMP('".$end."')"; + $db->query($query); +} + +sub getNbEvents{ + my $self = shift; + my $db = $self->{"centstorage"}; + my ($start, $end) = @_; + my $logger = $self->{"logger"}; + my $nbEvents = 0; + + my $query = "SELECT count(*) as nbEvents"; + $query .= " FROM `hoststateevents` e"; + $query .= " RIGHT JOIN (select host_id from mod_bi_tmp_today_hosts group by host_id) t2"; + $query .= " ON e.host_id = t2.host_id"; + $query .= " WHERE start_time < ".$end.""; + $query .= " AND end_time > ".$start.""; + $query .= " AND in_downtime = 0 "; + + + my $sth = $db->query($query); + + while (my $row = $sth->fetchrow_hashref()) { + $nbEvents = $row->{'nbEvents'}; + } + return $nbEvents; +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/centstorage/Metrics.pm b/gorgone/modules/centreon/mbi/libs/centstorage/Metrics.pm new file mode 100644 index 0000000..d4fc5bf --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/centstorage/Metrics.pm @@ -0,0 +1,230 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::centstorage::Metrics; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{logger} = shift; + $self->{centstorage} = shift; + + $self->{metrics} = (); + $self->{name} = 'data_bin'; + $self->{timeColumn} = 'ctime'; + $self->{name_minmaxavg_tmp} = 'mod_bi_tmp_minmaxavgvalue'; + $self->{name_firstlast_tmp} = 'mod_bi_tmp_firstlastvalues'; + $self->{name_minmaxctime_tmp} = 'mod_bi_tmp_minmaxctime'; + if (@_) { + $self->{name_minmaxavg_tmp} .= $_[0]; + $self->{name_firstlast_tmp} .= $_[0]; + $self->{name_minmaxctime_tmp} .= $_[0]; + } + bless $self, $class; + return $self; +} + +sub getName() { + my $self = shift; + return $self->{'name'}; +} + +sub getTimeColumn() { + my $self = shift; + return $self->{'timeColumn'}; +} + +sub createTempTableMetricMinMaxAvgValues { + my ($self, $useMemory, $granularity) = @_; + my $db = $self->{"centstorage"}; + $db->query("DROP TABLE IF EXISTS `" . $self->{name_minmaxavg_tmp} . "`"); + my $createTable = " CREATE TABLE `" . $self->{name_minmaxavg_tmp} . "` ("; + $createTable .= " id_metric INT NULL,"; + $createTable .= " avg_value FLOAT NULL,"; + $createTable .= " min_value FLOAT NULL,"; + $createTable .= " max_value FLOAT NULL"; + if ($granularity eq "hour") { + $createTable .= ", valueTime DATETIME NULL"; + } + if (defined($useMemory) && $useMemory eq "true") { + $createTable .= ") ENGINE=MEMORY CHARSET=utf8 COLLATE=utf8_general_ci;"; + }else { + $createTable .= ") ENGINE=INNODB CHARSET=utf8 COLLATE=utf8_general_ci;"; + } + $db->query($createTable); +} + +sub getMetricValueByHour { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my ($start, $end, $useMemory) = @_; + my $dateFormat = "%Y-%c-%e %k:00:00"; + + # Getting min, max, average + $self->createTempTableMetricMinMaxAvgValues($useMemory, "hour"); + my $query = "INSERT INTO `" . $self->{name_minmaxavg_tmp} . "` SELECT id_metric, avg(value) as avg_value, min(value) as min_value, max(value) as max_value, "; + $query .= " date_format(FROM_UNIXTIME(ctime), '".$dateFormat."') as valueTime "; + $query .= "FROM data_bin "; + $query .= "WHERE "; + $query .= "ctime >=UNIX_TIMESTAMP('".$start."') AND ctime < UNIX_TIMESTAMP('".$end."') "; + $query .= "GROUP BY id_metric, date_format(FROM_UNIXTIME(ctime), '".$dateFormat."')"; + + $db->query($query); + $self->addIndexTempTableMetricMinMaxAvgValues("hour"); +} + +sub getMetricsValueByDay { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my ($period, $useMemory) = @_; + my $dateFormat = "%Y-%c-%e"; + + # Getting min, max, average + $self->createTempTableMetricMinMaxAvgValues($useMemory, "day"); + my $query = "INSERT INTO `" . $self->{name_minmaxavg_tmp} . "` SELECT id_metric, avg(value) as avg_value, min(value) as min_value, max(value) as max_value "; + #$query .= " date_format(FROM_UNIXTIME(ctime), '".$dateFormat."') as valueTime "; + $query .= "FROM data_bin "; + $query .= "WHERE "; + my @tabPeriod = @$period; + my ($start_date, $end_date); + my $tabSize = scalar(@tabPeriod); + for (my $count = 0; $count < $tabSize; $count++) { + my $range = $tabPeriod[$count]; + if ($count == 0) { + $start_date = $range->[0]; + } + if ($count == $tabSize - 1) { + $end_date = $range->[1]; + } + $query .= "(ctime >= UNIX_TIMESTAMP(".($range->[0]). ") AND ctime < UNIX_TIMESTAMP(".($range->[1]) .")) OR "; + } + + $query =~ s/OR $//; + $query .= "GROUP BY id_metric"; + + $db->query($query); + $self->addIndexTempTableMetricMinMaxAvgValues("day"); + $self->getFirstAndLastValues($start_date, $end_date, $useMemory); +} + +sub createTempTableMetricDayFirstLastValues { + my ($self, $useMemory) = @_; + my $db = $self->{"centstorage"}; + $db->query("DROP TABLE IF EXISTS `" . $self->{name_firstlast_tmp} . "`"); + my $createTable = " CREATE TABLE `" . $self->{name_firstlast_tmp} . "` ("; + $createTable .= " `first_value` FLOAT NULL,"; + $createTable .= " `last_value` FLOAT NULL,"; + $createTable .= " id_metric INT NULL"; + if (defined($useMemory) && $useMemory eq "true") { + $createTable .= ") ENGINE=MEMORY CHARSET=utf8 COLLATE=utf8_general_ci;"; + } else { + $createTable .= ") ENGINE=INNODB CHARSET=utf8 COLLATE=utf8_general_ci;"; + } + $db->query($createTable); +} + +sub addIndexTempTableMetricDayFirstLastValues { + my $self = shift; + my $db = $self->{"centstorage"}; + $db->query("ALTER TABLE " . $self->{name_firstlast_tmp} . " ADD INDEX (`id_metric`)"); +} + +sub addIndexTempTableMetricMinMaxAvgValues { + my $self = shift; + my $granularity = shift; + my $db = $self->{"centstorage"}; + my $index = "id_metric"; + if ($granularity eq "hour") { + $index .= ", valueTime"; + } + my $query = "ALTER TABLE " . $self->{name_minmaxavg_tmp} . " ADD INDEX (" . $index . ")"; + $db->query($query); +} + +sub createTempTableCtimeMinMaxValues { + my ($self, $useMemory) = @_; + my $db = $self->{"centstorage"}; + $db->query("DROP TABLE IF EXISTS `" . $self->{name_minmaxctime_tmp} . "`"); + my $createTable = " CREATE TABLE `" . $self->{name_minmaxctime_tmp} . "` ("; + $createTable .= " min_val INT NULL,"; + $createTable .= " max_val INT NULL,"; + $createTable .= " id_metric INT NULL"; + if (defined($useMemory) && $useMemory eq "true") { + $createTable .= ") ENGINE=MEMORY CHARSET=utf8 COLLATE=utf8_general_ci;"; + } else { + $createTable .= ") ENGINE=INNODB CHARSET=utf8 COLLATE=utf8_general_ci;"; + } + $db->query($createTable); +} + +sub dropTempTableCtimeMinMaxValues { + my $self = shift; + my $db = $self->{"centstorage"}; + $db->query("DROP TABLE `" . $self->{name_minmaxctime_tmp} . "`"); +} + +sub getFirstAndLastValues { + my $self = shift; + my $db = $self->{"centstorage"}; + + my ($start_date, $end_date, $useMemory) = @_; + + $self->createTempTableCtimeMinMaxValues($useMemory); + my $query = "INSERT INTO `" . $self->{name_minmaxctime_tmp} . "` SELECT min(ctime) as min_val, max(ctime) as max_val, id_metric "; + $query .= " FROM `data_bin`"; + $query .= " WHERE ctime >= UNIX_TIMESTAMP(" . $start_date . ") AND ctime < UNIX_TIMESTAMP(" . $end_date . ")"; + $query .= " GROUP BY id_metric"; + $db->query($query); + + $self->createTempTableMetricDayFirstLastValues($useMemory); + $query = "INSERT INTO " . $self->{name_firstlast_tmp} . " SELECT d.value as `first_value`, d2.value as `last_value`, d.id_metric"; + $query .= " FROM data_bin as d, data_bin as d2, " . $self->{name_minmaxctime_tmp} . " as db"; + $query .= " WHERE db.id_metric=d.id_metric AND db.min_val=d.ctime"; + $query .= " AND db.id_metric=d2.id_metric AND db.max_val=d2.ctime"; + $query .= " GROUP BY db.id_metric"; + my $sth = $db->query($query); + $self->addIndexTempTableMetricDayFirstLastValues(); + $self->dropTempTableCtimeMinMaxValues(); +} + +sub dailyPurge { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($end) = @_; + + my $query = "DELETE FROM `data_bin` where ctime < UNIX_TIMESTAMP('" . $end . "')"; + $logger->writeLog("DEBUG", "[PURGE] [data_bin] purging data older than " . $end); + $db->query($query); +} + +1; diff --git a/gorgone/modules/centreon/mbi/libs/centstorage/ServiceStateEvents.pm b/gorgone/modules/centreon/mbi/libs/centstorage/ServiceStateEvents.pm new file mode 100644 index 0000000..af19233 --- /dev/null +++ b/gorgone/modules/centreon/mbi/libs/centstorage/ServiceStateEvents.pm @@ -0,0 +1,179 @@ +# +# Copyright 2019 Centreon (http://www.centreon.com/) +# +# Centreon is a full-fledged industry-strength solution that meets +# the needs in IT infrastructure and application monitoring for +# service performance. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +use strict; +use warnings; + +package gorgone::modules::centreon::mbi::libs::centstorage::ServiceStateEvents; + +# Constructor +# parameters: +# $logger: instance of class CentreonLogger +# $centreon: Instance of centreonDB class for connection to Centreon database +# $centstorage: (optionnal) Instance of centreonDB class for connection to Centstorage database +sub new { + my $class = shift; + my $self = {}; + $self->{"logger"} = shift; + $self->{"centstorage"} = shift; + $self->{"biServiceStateEventsObj"} = shift; + $self->{"timePeriodObj"} = shift; + if (@_) { + $self->{"centreon"} = shift; + } + + $self->{"name"} = "servicestateevents"; + $self->{"timeColumn"} = "end_time"; + bless $self, $class; + return $self; +} + +sub getName() { + my $self = shift; + return $self->{'name'}; +} + +sub getTimeColumn() { + my $self = shift; + return $self->{'timeColumn'}; +} + +sub agreggateEventsByTimePeriod { + my ($self, $timeperiodList, $start, $end, $liveServiceByTpId, $mode) = @_; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my $rangesByTP = ($self->{"timePeriodObj"})->getTimeRangesForPeriodAndTpList($timeperiodList, $start, $end); + my $query = "SELECT e.host_id,e.service_id, start_time, end_time, ack_time, state, last_update"; + $query .= " FROM `servicestateevents` e"; + $query .= " RIGHT JOIN (select host_id,service_id from mod_bi_tmp_today_services group by host_id,service_id) t2"; + $query .= " ON e.host_id = t2.host_id AND e.service_id = t2.service_id"; + $query .= " WHERE start_time < ".$end.""; + $query .= " AND end_time > ".$start.""; + $query .= " AND in_downtime = 0 "; + $query .= " ORDER BY start_time "; + + my $serviceEventObjects = $self->{"biServiceStateEventsObj"}; + my $sth = $db->query($query); + $serviceEventObjects->createTempBIEventsTable(); + $serviceEventObjects->prepareTempQuery(); + + while (my $row = $sth->fetchrow_hashref()) { + if (!defined($row->{'end_time'})) { + $row->{'end_time'} = $end; + } + while (my ($timeperiodID, $timeRanges) = each %$rangesByTP) { + my @tab = (); + $tab[0] = $row->{'host_id'}; + $tab[1] = $row->{'service_id'}; + $tab[2] = $liveServiceByTpId->{$timeperiodID}; + $tab[3] = $row->{'state'}; + if ($mode eq 'daily') { + $timeRanges = ($self->{"timePeriodObj"})->getTimeRangesForPeriod($timeperiodID, $row->{'start_time'}, $row->{'end_time'}); + } + ($tab[4], $tab[5]) = $self->processIncidentForTp($timeRanges,$row->{'start_time'}, $row->{'end_time'}); + $tab[6] = $row->{'end_time'}; + $tab[7] = defined($row->{ack_time}) ? $row->{ack_time} : 0; + $tab[8] = $row->{last_update}; + if (defined($tab[4]) && $tab[4] != -1) { + $serviceEventObjects->bindParam(\@tab); + } + } + } + ($db->getInstance)->commit; +} + +sub processIncidentForTp { + my ($self, $timeRanges, $start, $end) = @_; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + + my $rangeSize = scalar(@$timeRanges); + my $duration = 0; + my $slaDuration = 0; + my $range = 0; + my $i = 0; + my $processed = 0; + my $slaStart = $start; + my $slaStartModified = 0; + + foreach(@$timeRanges) { + my $currentStart = $start; + my $currentEnd = $end; + $range = $_; + my ($rangeStart, $rangeEnd) = ($range->[0], $range->[1]); + if ($currentStart < $rangeEnd && $currentEnd > $rangeStart) { + $processed = 1; + if ($currentStart > $rangeStart) { + $slaStartModified = 1; + } elsif ($currentStart < $rangeStart) { + $currentStart = $rangeStart; + if (!$slaStartModified) { + $slaStart = $currentStart; + $slaStartModified = 1; + } + } + if ($currentEnd > $rangeEnd) { + $currentEnd = $rangeEnd; + } + $slaDuration += $currentEnd - $currentStart; + } + } + if (!$processed) { + return (-1, -1, -1); + } + return ($slaStart, $slaDuration); +} + +sub dailyPurge { + my $self = shift; + my $db = $self->{"centstorage"}; + my $logger = $self->{"logger"}; + my ($end) = @_; + + $logger->writeLog("DEBUG", "[PURGE] [servicestateevents] purging data older than ".$end); + my $query = "DELETE FROM `servicestateevents` where end_time < UNIX_TIMESTAMP('".$end."')"; + $db->query($query); +} + +sub getNbEvents { + my $self = shift; + my $db = $self->{"centstorage"}; + my ($start, $end) = @_; + my $nbEvents = 0; + my $logger = $self->{"logger"}; + + my $query = "SELECT count(*) as nbEvents"; + $query .= " FROM `servicestateevents` e"; + $query .= " RIGHT JOIN (select host_id,service_id from mod_bi_tmp_today_services group by host_id,service_id) t2"; + $query .= " ON e.host_id = t2.host_id AND e.service_id = t2.service_id"; + $query .= " WHERE start_time < ".$end.""; + $query .= " AND end_time > ".$start.""; + $query .= " AND in_downtime = 0 "; + + my $sth = $db->query($query); + + while (my $row = $sth->fetchrow_hashref()) { + $nbEvents = $row->{'nbEvents'}; + } + return $nbEvents; +} + +1; diff --git a/gorgone/modules/core/action/class.pm b/gorgone/modules/core/action/class.pm index 74db97a..8750297 100644 --- a/gorgone/modules/core/action/class.pm +++ b/gorgone/modules/core/action/class.pm @@ -60,6 +60,13 @@ sub new { $connector->{allowed_cmds} = $connector->{config}->{allowed_cmds} if (defined($connector->{config}->{allowed_cmds}) && ref($connector->{config}->{allowed_cmds}) eq 'ARRAY'); + if (defined($connector->{config}->{tar_insecure_extra_mode}) && $connector->{config}->{tar_insecure_extra_mode} =~ /^(?:1|true)$/) { + $Archive::Tar::INSECURE_EXTRACT_MODE = 1; + } + + $connector->{paranoid_plugins} = defined($connector->{config}->{paranoid_plugins}) && $connector->{config}->{paranoid_plugins} =~ /true|1/i ? + 1 : 0; + $connector->{return_childs} = {}; $connector->{engine_childs} = {}; $connector->{max_concurrent_engine} = defined($connector->{config}->{max_concurrent_engine}) ? @@ -154,7 +161,7 @@ sub get_package_manager { $self->{package_manager} = 'unknown'; if ($os =~ /Debian|Ubuntu/i) { $self->{package_manager} = 'deb'; - } elsif ($os =~ /CentOS|Redhat|rhel|almalinux/i) { + } elsif ($os =~ /CentOS|Redhat|rhel|almalinux|rocky/i) { $self->{package_manager} = 'rpm'; } elsif ($os eq 'ol' || $os =~ /Oracle Linux/i) { $self->{package_manager} = 'rpm'; @@ -245,6 +252,7 @@ sub install_plugins { redirect_stderr => 1, logger => $self->{logger} ); + $self->{logger}->writeLogDebug("[action] install plugins. Command output: [\"$stdout\"]"); if ($error != 0) { return (-1, 'install plugins command issue: ' . $stdout); } @@ -256,18 +264,20 @@ sub validate_plugins_rpm { my ($self, %options) = @_; my ($rv, $message, $installed) = $self->check_plugins_rpm(%options); - return ($rv, $message) if ($rv == -1); + return (1, $message) if ($rv == -1); return 0 if ($rv == 0); if ($rv == 1) { ($rv, $message) = $self->install_plugins(type => 'rpm', installed => $installed); - return ($rv, $message) if ($rv == -1); + return (1, $message) if ($rv == -1); } ($rv, $message, $installed) = $self->check_plugins_rpm(%options); - return ($rv, $message) if ($rv == -1); + return (1, $message) if ($rv == -1); if ($rv == 1) { - $self->{logger}->writeLogError("[action] validate plugins - still some to install: " . join(' ', @$installed)); + $message = 'validate plugins - still some to install: ' . join(' ', @$installed); + $self->{logger}->writeLogError("[action] $message"); + return (1, $message); } return 0; @@ -282,18 +292,20 @@ sub validate_plugins_deb { } my ($rv, $message, $installed) = $self->check_plugins_deb(plugins => $plugins); - return ($rv, $message) if ($rv == -1); + return (1, $message) if ($rv == -1); return 0 if ($rv == 0); if ($rv == 1) { ($rv, $message) = $self->install_plugins(type => 'deb', installed => $installed); - return ($rv, $message) if ($rv == -1); + return (1, $message) if ($rv == -1); } ($rv, $message, $installed) = $self->check_plugins_deb(plugins => $plugins); - return ($rv, $message) if ($rv == -1); + return (1, $message) if ($rv == -1); if ($rv == 1) { - $self->{logger}->writeLogError("[action] validate plugins - still some to install: " . join(' ', @$installed)); + $message = 'validate plugins - still some to install: ' . join(' ', @$installed); + $self->{logger}->writeLogError("[action] $message"); + return (1, $message); } return 0; @@ -307,7 +319,7 @@ sub validate_plugins { my $plugins; eval { - $plugins = JSON::XS->new->utf8->decode($content); + $plugins = JSON::XS->new->decode($content); }; if ($@) { return (1, 'cannot decode json'); @@ -657,7 +669,7 @@ sub action_actionengine { if (defined($options{data}->{content}->{plugins}) && $options{data}->{content}->{plugins} ne '') { my ($rv, $message) = $self->validate_plugins(file => $options{data}->{content}->{plugins}); - if ($rv) { + if ($rv && $self->{paranoid_plugins} == 1) { $self->{logger}->writeLogError("[action] $message"); $self->send_log( socket => $options{socket_log}, diff --git a/gorgone/modules/core/httpserver/class.pm b/gorgone/modules/core/httpserver/class.pm index 1ed7a79..4eeb09b 100644 --- a/gorgone/modules/core/httpserver/class.pm +++ b/gorgone/modules/core/httpserver/class.pm @@ -217,7 +217,10 @@ sub run { exit(0); } - next if (!defined($connection)); + if (!defined($connection)) { + gorgone::standard::api::event(httpserver => $self); + next; + } while (my $request = $connection->get_request) { if ($connection->antique_client eq '1') { @@ -325,7 +328,7 @@ sub api_call { my $content; eval { - $content = JSON::XS->new->utf8->decode($request->content) + $content = JSON::XS->new->decode($request->content) if ($request->method =~ /POST|PATCH/ && defined($request->content)); }; if ($@) { diff --git a/gorgone/modules/core/httpserver/hooks.pm b/gorgone/modules/core/httpserver/hooks.pm index c36515a..2258753 100644 --- a/gorgone/modules/core/httpserver/hooks.pm +++ b/gorgone/modules/core/httpserver/hooks.pm @@ -138,7 +138,11 @@ sub check { return $count; } -sub broadcast {} +sub broadcast { + my (%options) = @_; + + routing(%options); +} # Specific functions sub create_child { diff --git a/gorgone/modules/core/httpserverng/class.pm b/gorgone/modules/core/httpserverng/class.pm index 7b17745..b57bbdb 100644 --- a/gorgone/modules/core/httpserverng/class.pm +++ b/gorgone/modules/core/httpserverng/class.pm @@ -69,7 +69,7 @@ websocket '/' => sub { my $content; eval { - $content = JSON::XS->new->utf8->decode($msg); + $content = JSON::XS->new->decode($msg); }; if ($@) { $connector->close_websocket( @@ -316,7 +316,7 @@ sub read_log_event { if (defined($options{data})) { my $content; eval { - $content = JSON::XS->new->utf8->decode($options{data}); + $content = JSON::XS->new->decode($options{data}); }; if ($@) { $response = { error => 'decode_error', message => 'Cannot decode response' }; @@ -345,7 +345,7 @@ sub read_listener { my $content; eval { - $content = JSON::XS->new->utf8->decode($options{data}); + $content = JSON::XS->new->decode($options{data}); }; if ($@) { $self->{token_watch}->{ $options{token} }->{mojo}->render(json => { error => 'decode_error', message => 'Cannot decode response' }); @@ -393,7 +393,10 @@ sub read_zmq_events { } } if ((my $method = $connector->can('action_' . lc($action)))) { - $method->($connector, token => $token, data => $data); + my ($rv, $decoded) = $connector->json_decode(argument => $data, token => $token); + if (!$rv) { + $method->($connector, token => $token, data => $decoded); + } } } } else { diff --git a/gorgone/modules/core/httpserverng/hooks.pm b/gorgone/modules/core/httpserverng/hooks.pm index b43a601..fc2e698 100644 --- a/gorgone/modules/core/httpserverng/hooks.pm +++ b/gorgone/modules/core/httpserverng/hooks.pm @@ -139,7 +139,11 @@ sub check { return $count; } -sub broadcast {} +sub broadcast { + my (%options) = @_; + + routing(%options); +} # Specific functions sub create_child { diff --git a/gorgone/modules/core/proxy/hooks.pm b/gorgone/modules/core/proxy/hooks.pm index eab2332..374fc3c 100644 --- a/gorgone/modules/core/proxy/hooks.pm +++ b/gorgone/modules/core/proxy/hooks.pm @@ -135,7 +135,7 @@ sub routing { my $data; eval { - $data = JSON::XS->new->utf8->decode($options{data}); + $data = JSON::XS->new->decode($options{data}); }; if ($@) { $options{logger}->writeLogError("[proxy] Cannot decode json data: $@"); @@ -192,7 +192,7 @@ sub routing { routing( action => 'PROXYADDNODE', target => $node_id, - data => JSON::XS->new->utf8->encode($register_nodes->{$node_id}), + data => JSON::XS->new->encode($register_nodes->{$node_id}), gorgone => $options{gorgone}, dbh => $options{dbh}, logger => $options{logger} @@ -438,7 +438,7 @@ sub check { routing( target => $_, action => 'PROXYCLOSECONNECTION', - data => JSON::XS->new->utf8->encode({ id => $_ }), + data => JSON::XS->new->encode({ id => $_ }), gorgone => $options{gorgone}, dbh => $options{dbh}, logger => $options{logger} @@ -565,7 +565,7 @@ sub pathway { routing( target => $_, action => 'PROXYCLOSEREADCHANNEL', - data => JSON::XS->new->utf8->encode({ id => $_ }), + data => JSON::XS->new->encode({ id => $_ }), gorgone => $options{gorgone}, dbh => $options{dbh}, logger => $options{logger} @@ -638,8 +638,7 @@ sub setlogs { code => $_->{code}, token => $_->{token}, instant => $_->{instant}, - data => $_->{data}, - encode_utf8 => 1 + data => $_->{data} ); last if ($status == -1); $ctime_recent = $_->{ctime} if ($ctime_recent < $_->{ctime}); @@ -890,7 +889,7 @@ sub unregister_nodes { routing( action => 'PROXYDELNODE', target => $node->{id}, - data => JSON::XS->new->utf8->encode($node), + data => JSON::XS->new->encode($node), gorgone => $options{gorgone}, dbh => $options{dbh}, logger => $options{logger} @@ -1036,7 +1035,7 @@ sub register_nodes { routing( action => 'PROXYADDNODE', target => $node->{id}, - data => JSON::XS->new->utf8->encode($register_nodes->{ $node->{id} }), + data => JSON::XS->new->encode($register_nodes->{ $node->{id} }), gorgone => $options{gorgone}, dbh => $options{dbh}, logger => $options{logger} @@ -1045,7 +1044,7 @@ sub register_nodes { routing( action => 'PROXYADDNODE', target => $node->{id}, - data => JSON::XS->new->utf8->encode($node), + data => JSON::XS->new->encode($node), gorgone => $options{gorgone}, dbh => $options{dbh}, logger => $options{logger} diff --git a/gorgone/modules/core/proxy/httpserver.pm b/gorgone/modules/core/proxy/httpserver.pm index 884598e..90ef57a 100644 --- a/gorgone/modules/core/proxy/httpserver.pm +++ b/gorgone/modules/core/proxy/httpserver.pm @@ -47,7 +47,7 @@ websocket '/' => sub { tx => $mojo->tx, logged => 0, last_update => time(), - authentication => $mojo->tx->req->headers->header('authentication') + authorization => $mojo->tx->req->headers->header('authorization') }; $mojo->on(message => sub { @@ -301,11 +301,11 @@ sub is_logged_websocket { return 1 if ($self->{ws_clients}->{ $options{ws_id} }->{logged} == 1); - if (!defined($self->{ws_clients}->{ $options{ws_id} }->{authentication}) || - $self->{ws_clients}->{ $options{ws_id} }->{authentication} !~ /^\s*Bearer\s+$self->{config}->{httpserver}->{token}/) { + if (!defined($self->{ws_clients}->{ $options{ws_id} }->{authorization}) || + $self->{ws_clients}->{ $options{ws_id} }->{authorization} !~ /^\s*Bearer\s+$self->{config}->{httpserver}->{token}\s*$/) { $self->close_websocket( code => 500, - message => 'token authentication unallowed', + message => 'token authorization unallowed', ws_id => $options{ws_id} ); return 0; @@ -322,7 +322,7 @@ sub is_logged_websocket { my $content; eval { - $content = JSON::XS->new->utf8->decode($1); + $content = JSON::XS->new->decode($1); }; if ($@) { $self->close_websocket( diff --git a/gorgone/modules/core/proxy/sshclient.pm b/gorgone/modules/core/proxy/sshclient.pm index 7c371dd..d0f4030 100644 --- a/gorgone/modules/core/proxy/sshclient.pm +++ b/gorgone/modules/core/proxy/sshclient.pm @@ -30,7 +30,6 @@ use gorgone::standard::misc; use File::Basename; use Time::HiRes; use gorgone::standard::constants qw(:all); -use Encode; use MIME::Base64; sub new { @@ -313,7 +312,6 @@ sub action_enginecommand { if ($options{target_direct} == 0) { foreach my $command (@{$options{data}->{content}->{commands}}) { chomp $command; - $command = Encode::decode('UTF-8', $command); my $msg = "[sshclient] Handling command 'EXTERNALCMD'"; $msg .= ", Target: '" . $options{target} . "'" if (defined($options{target})); $msg .= ", Parameters: '" . $command . "'" if (defined($command)); @@ -362,7 +360,6 @@ sub action_enginecommand { }; foreach my $command (@{$options{data}->{content}->{commands}}) { - $command = Encode::decode('UTF-8', $command); $self->{logger}->writeLogInfo("[sshclient] Processing external command '" . $command . "'"); if ($self->{sftp}->write(handle_file => $file, data => $command . "\n") != Libssh::Session::SSH_OK) { $self->{logger}->writeLogError("[sshclient] Command file '$command_file' must be writeable"); diff --git a/gorgone/modules/core/pull/class.pm b/gorgone/modules/core/pull/class.pm index bf02a13..7baa1ba 100644 --- a/gorgone/modules/core/pull/class.pm +++ b/gorgone/modules/core/pull/class.pm @@ -117,7 +117,7 @@ sub transmit_back { if ($options{message} =~ /^\[ACK\]\s+\[(.*?)\]\s+(.*)/m) { my $data; eval { - $data = JSON::XS->new->utf8->decode($2); + $data = JSON::XS->new->decode($2); }; if ($@) { return $options{message}; diff --git a/gorgone/modules/core/pullwss/class.pm b/gorgone/modules/core/pullwss/class.pm index c957663..793aee0 100644 --- a/gorgone/modules/core/pullwss/class.pm +++ b/gorgone/modules/core/pullwss/class.pm @@ -67,6 +67,27 @@ sub handle_TERM { my $self = shift; $self->{logger}->writeLogDebug("[pullwss] $$ Receiving order to stop..."); $self->{stop} = 1; + + my $message = gorgone::standard::library::build_protocol( + action => 'UNREGISTERNODES', + data => { + nodes => [ + { + id => $self->get_core_config(name => 'id'), + type => 'wss', + identity => $self->get_core_config(name => 'id') + } + ] + }, + json_encode => 1 + ); + + if ($self->{connected} == 1) { + $self->{tx}->send({text => $message }); + $self->{tx}->on(drain => sub { Mojo::IOLoop->stop_gracefully(); }); + } else { + Mojo::IOLoop->stop_gracefully(); + } } sub class_handle_TERM { @@ -130,7 +151,7 @@ sub wss_connect { } $self->{ua}->websocket( - $proto . '://' . $self->{config}->{address} . ':' . $self->{config}->{port} . '/' => { Authentication => 'Bearer ' . $self->{config}->{token} } => sub { + $proto . '://' . $self->{config}->{address} . ':' . $self->{config}->{port} . '/' => { Authorization => 'Bearer ' . $self->{config}->{token} } => sub { my ($ua, $tx) = @_; $connector->{tx} = $tx; @@ -217,7 +238,7 @@ sub transmit_back { if ($options{message} =~ /^\[ACK\]\s+\[(.*?)\]\s+(.*)/m) { my $data; eval { - $data = JSON::XS->new->utf8->decode($2); + $data = JSON::XS->new->decode($2); }; if ($@) { return $options{message}; diff --git a/gorgone/modules/plugins/newtest/class.pm b/gorgone/modules/plugins/newtest/class.pm index 8095161..1fcef6e 100644 --- a/gorgone/modules/plugins/newtest/class.pm +++ b/gorgone/modules/plugins/newtest/class.pm @@ -596,7 +596,7 @@ sub event { if ((my $method = $connector->can('action_' . lc($1)))) { $message =~ /^\[(.*?)\]\s+\[(.*?)\]\s+\[.*?\]\s+(.*)$/m; my ($action, $token) = ($1, $2); - my $data = JSON::XS->new->utf8->decode($3); + my $data = JSON::XS->new->decode($3); $method->($connector, token => $token, data => $data); } } diff --git a/gorgone/modules/plugins/newtest/hooks.pm b/gorgone/modules/plugins/newtest/hooks.pm index 33b7007..b6ca486 100644 --- a/gorgone/modules/plugins/newtest/hooks.pm +++ b/gorgone/modules/plugins/newtest/hooks.pm @@ -68,7 +68,7 @@ sub routing { my $data; eval { - $data = JSON::XS->new->utf8->decode($options{data}); + $data = JSON::XS->new->decode($options{data}); }; if ($@) { $options{logger}->writeLogError("[newtest] Cannot decode json data: $@"); @@ -209,7 +209,7 @@ sub get_containers { my $list_scenario; eval { - $list_scenario = JSON::XS->new->utf8->decode($_->{list_scenario_status}); + $list_scenario = JSON::XS->new->decode($_->{list_scenario_status}); }; if ($@) { $options{logger}->writeLogError("[newtest] cannot load container '" . $_->{name} . "' - cannot decode list scenario option"); diff --git a/gorgone/modules/plugins/scom/class.pm b/gorgone/modules/plugins/scom/class.pm index 2585f9c..c59fe3e 100644 --- a/gorgone/modules/plugins/scom/class.pm +++ b/gorgone/modules/plugins/scom/class.pm @@ -484,7 +484,7 @@ sub event { if ((my $method = $connector->can('action_' . lc($1)))) { $message =~ /^\[(.*?)\]\s+\[(.*?)\]\s+\[.*?\]\s+(.*)$/m; my ($action, $token) = ($1, $2); - my $data = JSON::XS->new->utf8->decode($3); + my $data = JSON::XS->new->decode($3); $method->($connector, token => $token, data => $data); } } diff --git a/gorgone/modules/plugins/scom/hooks.pm b/gorgone/modules/plugins/scom/hooks.pm index 3af2127..a8b8306 100644 --- a/gorgone/modules/plugins/scom/hooks.pm +++ b/gorgone/modules/plugins/scom/hooks.pm @@ -67,7 +67,7 @@ sub routing { my $data; eval { - $data = JSON::XS->new->utf8->decode($options{data}); + $data = JSON::XS->new->decode($options{data}); }; if ($@) { $options{logger}->writeLogError("[scom] Cannot decode json data: $@"); diff --git a/gorgone/standard/api.pm b/gorgone/standard/api.pm index bd4f941..651aaa5 100644 --- a/gorgone/standard/api.pm +++ b/gorgone/standard/api.pm @@ -30,7 +30,7 @@ use JSON::XS; my $module; my $socket; -my $results; +my $results = {}; my $action_token; sub root { @@ -163,14 +163,14 @@ sub call_internal { if (defined($results->{$action_token}->{data})) { my $content; eval { - $content = JSON::XS->new->utf8->decode($results->{$action_token}->{data}); + $content = JSON::XS->new->decode($results->{$action_token}->{data}); }; if ($@) { $response = '{"error":"decode_error","message":"Cannot decode response"}'; } else { if (defined($content->{data})) { eval { - $response = JSON::XS->new->utf8->encode($content->{data}); + $response = JSON::XS->new->encode($content->{data}); }; if ($@) { $response = '{"error":"encode_error","message":"Cannot encode response"}'; @@ -231,13 +231,13 @@ sub get_log { if (defined($results->{ $token_log }) && defined($results->{ $token_log }->{data})) { my $content; eval { - $content = JSON::XS->new->utf8->decode($results->{ $token_log }->{data}); + $content = JSON::XS->new->decode($results->{ $token_log }->{data}); }; if ($@) { $response = '{"error":"decode_error","message":"Cannot decode response"}'; } elsif (defined($content->{data}->{result}) && scalar(@{$content->{data}->{result}}) > 0) { eval { - $response = JSON::XS->new->utf8->encode( + $response = JSON::XS->new->encode( { message => "Logs found", token => $options{token}, @@ -255,8 +255,11 @@ sub get_log { } sub event { + my (%options) = @_; + + my $httpserver = defined($options{httpserver}) ? $options{httpserver} : $module; while (1) { - my $message = $module->read_message(); + my $message = $httpserver->read_message(); last if (!defined($message)); if ($message =~ /^\[(.*?)\]\s+\[([a-zA-Z0-9:\-_]*?)\]\s+\[.*?\]\s+(.*)$/m || @@ -267,8 +270,10 @@ sub event { token => $token, data => $data }; - if ((my $method = $module->can('action_' . lc($action)))) { - $method->($module, token => $token, data => $data); + if ((my $method = $httpserver->can('action_' . lc($action)))) { + my ($rv, $decoded) = $httpserver->json_decode(argument => $data, token => $token); + next if ($rv); + $method->($httpserver, token => $token, data => $decoded); } } } diff --git a/gorgone/standard/constants.pm b/gorgone/standard/constants.pm index c730343..789863f 100644 --- a/gorgone/standard/constants.pm +++ b/gorgone/standard/constants.pm @@ -31,6 +31,7 @@ BEGIN { GORGONE_ACTION_FINISH_KO => 1, GORGONE_ACTION_FINISH_OK => 2, GORGONE_STARTED => 3, + GORGONE_ACTION_CONTINUE => 4, GORGONE_MODULE_ACTION_COMMAND_RESULT => 100, GORGONE_MODULE_ACTION_PROCESSCOPY_INPROGRESS => 101, @@ -43,7 +44,9 @@ BEGIN { GORGONE_MODULE_CENTREON_AUTODISCO_SVC_PROGRESS => 400, - GORGONE_MODULE_CENTREON_AUDIT_PROGRESS => 500 + GORGONE_MODULE_CENTREON_AUDIT_PROGRESS => 500, + + GORGONE_MODULE_CENTREON_MBIETL_PROGRESS => 600 ); } diff --git a/gorgone/standard/library.pm b/gorgone/standard/library.pm index 7a464f5..622e489 100644 --- a/gorgone/standard/library.pm +++ b/gorgone/standard/library.pm @@ -295,7 +295,7 @@ sub addlistener { my $data; eval { - $data = JSON::XS->new->utf8->decode($options{data}); + $data = JSON::XS->new->decode($options{data}); }; if ($@) { return (GORGONE_ACTION_FINISH_KO, { message => 'request not well formatted' }); @@ -340,7 +340,7 @@ sub unloadmodule { my $data; eval { - $data = JSON::XS->new->utf8->decode($options{data}); + $data = JSON::XS->new->decode($options{data}); }; if ($@) { return (GORGONE_ACTION_FINISH_KO, { message => 'request not well formatted' }); @@ -365,7 +365,7 @@ sub loadmodule { my $data; eval { - $data = JSON::XS->new->utf8->decode($options{data}); + $data = JSON::XS->new->decode($options{data}); }; if ($@) { return (GORGONE_ACTION_FINISH_KO, { message => 'request not well formatted' }); @@ -392,7 +392,7 @@ sub synclogs { my $data; eval { - $data = JSON::XS->new->utf8->decode($options{data}); + $data = JSON::XS->new->decode($options{data}); }; if ($@) { return (GORGONE_ACTION_FINISH_KO, { message => 'request not well formatted' }); @@ -437,7 +437,7 @@ sub setmodulekey { my $data; eval { - $data = JSON::XS->new->utf8->decode($options{data}); + $data = JSON::XS->new->decode($options{data}); }; if ($@) { return (GORGONE_ACTION_FINISH_KO, { message => 'request not well formatted' }); @@ -467,7 +467,7 @@ sub setcoreid { my $data; eval { - $data = JSON::XS->new->utf8->decode($options{data}); + $data = JSON::XS->new->decode($options{data}); }; if ($@) { return (GORGONE_ACTION_FINISH_KO, { message => 'request not well formatted' }); @@ -513,7 +513,7 @@ sub putlog { my $data; eval { - $data = JSON::XS->new->utf8->decode($options{data}); + $data = JSON::XS->new->decode($options{data}); }; if ($@) { return (GORGONE_ACTION_FINISH_KO, { message => 'request not well formatted' }); @@ -538,7 +538,7 @@ sub getlog { my $data; eval { - $data = JSON::XS->new->utf8->decode($options{data}); + $data = JSON::XS->new->decode($options{data}); }; if ($@) { return (GORGONE_ACTION_FINISH_KO, { message => 'request not well formatted' }); @@ -577,6 +577,26 @@ sub getlog { sub kill { my (%options) = @_; + my $data; + eval { + $data = JSON::XS->new->decode($options{data}); + }; + if ($@) { + return (GORGONE_ACTION_FINISH_KO, { message => 'request not well formatted' }); + } + + if (defined($data->{content}->{package}) && defined($options{gorgone}->{modules_register}->{ $data->{content}->{package} })) { + $options{gorgone}->{modules_register}->{ $data->{content}->{package} }->{kill}->(logger => $options{gorgone}->{logger}); + return (GORGONE_ACTION_FINISH_OK, { action => 'kill', message => "module '$data->{content}->{package}' kill in progress" }); + } + if (defined($data->{content}->{name}) && + defined($options{gorgone}->{modules_id}->{ $data->{content}->{name} }) && + defined($options{gorgone}->{modules_register}->{ $options{gorgone}->{modules_id}->{ $data->{content}->{name} } })) { + $options{gorgone}->{modules_register}->{ $options{gorgone}->{modules_id}->{ $data->{content}->{name} } }->{kill}->(logger => $options{gorgone}->{logger}); + return (GORGONE_ACTION_FINISH_OK, { action => 'kill', message => "module '$data->{content}->{name}' kill in progress" }); + } + + return (GORGONE_ACTION_FINISH_KO, { action => 'kill', message => 'cannot find module' }); } ####################### @@ -662,8 +682,7 @@ sub add_history { $listener->event_log( token => $options{token}, code => $options{code}, - data => $options{data}, - encode_utf8 => $options{encode_utf8} + data => $options{data} ); } return $status; @@ -678,7 +697,7 @@ sub json_encode { my $data; eval { - $data = JSON::XS->new->utf8->encode($options{data}); + $data = JSON::XS->new->encode($options{data}); }; if ($@) { if (defined($options{logger})) { @@ -695,7 +714,7 @@ sub json_decode { my $data; eval { - $data = JSON::XS->new->utf8->decode($options{data}); + $data = JSON::XS->new->decode($options{data}); }; if ($@) { if (defined($options{logger})) { diff --git a/gorgone/standard/misc.pm b/gorgone/standard/misc.pm index db75393..689a907 100644 --- a/gorgone/standard/misc.pm +++ b/gorgone/standard/misc.pm @@ -164,7 +164,6 @@ sub backtick { } if ($pid) { - binmode(KID, ":encoding(UTF-8)"); eval { local $SIG{ALRM} = sub { die "Timeout by signal ALARM\n"; }; alarm( $arg{timeout} ); diff --git a/install/src/centreon-api.yaml b/install/src/centreon-api.yaml index 4d335ee..39b7eb1 100644 --- a/install/src/centreon-api.yaml +++ b/install/src/centreon-api.yaml @@ -3,7 +3,7 @@ gorgone: - name: centreonv2 base_url: "http://127.0.0.1/centreon/api/latest/" username: admin - password: centreon + password: Centreon!2021 - name: clapi username: admin - password: centreon + password: Centreon!2021 diff --git a/packaging/centreon-gorgone.spectemplate b/packaging/centreon-gorgone.spectemplate index bfb9d20..5c1783c 100644 --- a/packaging/centreon-gorgone.spectemplate +++ b/packaging/centreon-gorgone.spectemplate @@ -1,5 +1,5 @@ Name: centreon-gorgone -Version: 22.04.0 +Version: 22.04.1 Release: @RELEASE@%{?dist} Summary: Perl daemon task handlers Group: Applications/System @@ -19,6 +19,7 @@ Requires: perl(ZMQ::Constants) Requires: perl(JSON::XS) Requires: perl(JSON::PP) Requires: perl(XML::Simple) +Requires: perl(XML::LibXML::Simple) Requires: perl(Net::SMTP) Requires: perl(YAML::XS) Requires: perl(DBD::SQLite)