topic Re: generate logs in grafana loki using nso observability-exporter in Observability Discussions

generate logs in grafana loki using nso observability-exporter

sahooanup — Thu, 30 May 2024 20:10:34 GMT

hi ,

I am trying to monitor all the nso logs in my Grafana Loki dashboard using nso Observability-exporter package. I have loaded the observability-exporter package successfully in my nso. When I run the setup.sh which is inside bservability-exporter package, I get 7 different container created. Those containers are 1. observability-exporter-jaeger, 2. observability-exporter-jaeger-collector, 3. observability-exporter-elasticsearch, 4. observability-exporter-influxdb, 5. observability-exporter-grafana, 6. observability-exporter-prometheus, 7. observability-exporter-otelcol.

Now my goal is to create a loki container and integrate that with Grafana dashboard so that I should be able to see all the logs in my Grafana Dashboard. How to achieve this please suggest.

Thanks

Re: generate logs in grafana loki using nso observability-exporter

bigevilbeard — Fri, 31 May 2024 13:00:10 GMT

You can use the official Loki Docker image to create a Loki container. I found an example so here is how to command to create a Loki container named observability-exporter-loki you will need to do this to create a Docker volume loki-volume to store the Loki data, and a network observability-exporter-net to allow communication between the containers fyi.

docker run -d --name observability-exporter-loki --volume=loki-volume:/mnt/loki --network=observability-exporter-net grafana/loki:2.4.0 --config.file=/mnt/loki/local-config.yaml

You will also need to create a local-config.yaml file to configure Loki. Here's an example configuration, i found on their website

auth_enabled: false server: http_listen_port: 3100 grpc_listen_port: 9095 log_level: info chunk_target_size: 1048576 table_manager: retention_deletes_enabled: false retention_period: 168h retention_target_size: 10485760 chunk_idle_period: 5m retention_recheck_interval: 1h retention_soft_deletes_enabled: false retention_soft_delete_period: 24h retention_compaction_window: 168h retention_compaction_retries: 3 retention_compaction_retry_delay: 1m retention_compaction_max_series_per_chunk: 1000 retention_compaction_max_age: 168h retention_compaction_min_size: 1048576 retention_compaction_schedule: "0 0 * * *" retention_compaction_working_set: 1000000000 retention_compaction_index_cache_size: 100000000 retention_compaction_index_cache_ttl: 1h retention_compaction_index_cache_max_age: 1h retention_compaction_index_cache_max_entries: 1000000 retention_compaction_index_cache_max_entry_size: 10000 retention_compaction_index_cache_max_entry_ttl: 1h retention_compaction_index_cache_max_entry_ttl_age: 1h retention_compaction_index_cache_max_entry_ttl_size: 10000 retention_compaction_index_cache_max_entry_ttl_size_age: 1h retention_compaction_index_cache_max_entry_ttl_size_age_ttl: 1h retention_compaction_index_cache_max_entry_ttl_size_age_ttl_ttl: 1h retention_compaction_index_cache_max_entry_ttl_size_age_ttl_ttl_ttl: 1h retention_compaction_index_cache_max_entry_ttl_size_age_ttl_ttl_ttl_ttl: 1h retention_compaction_index_cache_max_entry_ttl_size_age_ttl_ttl_ttl_ttl: 1h retention_compaction_index_cache_max_entry_ttl_size_age_ttl_ttl_ttl_ttl: 1h retention_compaction_index_cache_max_entry_ttl_size_age_ttl_ttl_ttl_ttl: 1h retention_compaction_index_cache_max

Re: generate logs in grafana loki using nso observability-exporter

sahooanup — Mon, 03 Jun 2024 11:42:14 GMT

I used the above local-config.yaml bit that throws error and container is exiting. We must use a proper local-config.yaml file.

Re: generate logs in grafana loki using nso observability-exporter

bigevilbeard — Mon, 03 Jun 2024 13:27:43 GMT

Ok try rthis version, also create a volume for the loki data and update the store section accordingly, then also, create an overrides.yaml file to configure runtime overrides if you need.

auth_enabled: false server: http_listen_port: 3100 grpc_listen_port: 9095 log_level: info ingester: lifecycler: ring: kvstore: store: inmemory chunk_idle_period: 5m chunk_target_size: 1048576 max_chunk_age: 2h store: boltdb: path: /mnt/loki/index schema_config: configs: - from: 2020-05-15 store: boltdb object_store: file schema: v11 index: period: 24h prefix: index_ limits_config: ingestion_rate_mb: 10 ingestion_burst_size_mb: 20 runtime_config: file: overrides.yaml

Re: generate logs in grafana loki using nso observability-exporter

sahooanup — Mon, 03 Jun 2024 20:17:42 GMT

Now I am able to create loki container with the above local-config.yaml . I am not using overrides.yaml. But when I go to my Grafana Dashboard I am not seeing any logs. I am not able to pass any queries because the Label filters section is show blank.
PFA screen shot.

Re: generate logs in grafana loki using nso observability-exporter

bigevilbeard — Tue, 04 Jun 2024 07:07:27 GMT

To add label filters, click on the Label filters button and select the labels you want to filter by. For example, if you want to filter the logs based on the level and method labels, you can select them from the list and enter the filter values.

Hope this helps.

Re: generate logs in grafana loki using nso observability-exporter

sahooanup — Tue, 04 Jun 2024 07:58:48 GMT

I am not finding any options when i click on Label filters button. It shows 'no options found'. Please refer to the attached screenshot. I think loki not able to fetch the nso logs.

Re: generate logs in grafana loki using nso observability-exporter

bigevilbeard — Tue, 04 Jun 2024 11:38:35 GMT

Try to manually inspect the logs in the Loki UI or use a command-line tool like loki-cli to see if labels are present.

Re: generate logs in grafana loki using nso observability-exporter

sahooanup — Wed, 05 Jun 2024 07:15:14 GMT

I am not able to access loki UI. I exposed 3100 as 50109 but not able to access the UI. please suggest whats wrong i am doing. The goal is I should be able to see Label filters option in the grafana dashboard and should be able to pass queries.
Attaching Screenshots for better clarity.

Re: generate logs in grafana loki using nso observability-exporter

bigevilbeard — Wed, 05 Jun 2024 11:04:31 GMT

To troubleshoot the inaccessible Loki UI, begin by examining the container logs with docker logs -f observability-exporter-loki for error messages. Next, confirm that the port mapping in your docker-compose file or docker run command correctly exposes port 3100 and maps it to port 50109 on the host. Then, inspect the Loki configuration file (local-config.yaml) to ensure it points to the appropriate storage backend (like Elasticsearch). Finally, verify that the Loki container shares the same network as other containers (such as Grafana, Elasticsearch) and that the network configuration is accurate.

Hope this helps.